Linux Needs Resource Management For Complex Workloads 161
storagedude writes: Resource management and allocation for complex workloads has been a need for some time in open systems, but no one has ever followed through on making open systems look and behave like an IBM mainframe, writes Henry Newman at Enterprise Storage Forum. Throwing more hardware at the problem is a costly solution that won't work forever, he notes.
Newman writes: "With next-generation technology like non-volatile memories and PCIe SSDs, there are going to be more resources in addition to the CPU that need to be scheduled to make sure everything fits in memory and does not overflow. I think the time has come for Linux – and likely other operating systems – to develop a more robust framework that can address the needs of future hardware and meet the requirements for scheduling resources. This framework is not going to be easy to develop, but it is needed by everything from databases and MapReduce to simple web queries."
Newman writes: "With next-generation technology like non-volatile memories and PCIe SSDs, there are going to be more resources in addition to the CPU that need to be scheduled to make sure everything fits in memory and does not overflow. I think the time has come for Linux – and likely other operating systems – to develop a more robust framework that can address the needs of future hardware and meet the requirements for scheduling resources. This framework is not going to be easy to develop, but it is needed by everything from databases and MapReduce to simple web queries."
Is this real or fantasy? (Score:4, Interesting)
I read the article and I can't tell if this is a real problem that is really affecting thousands of users and companies, or a fantasy that the author wrote up in 30 minutes after having a discussion with an old IBM engineer.
Sure, IBM has all these resource prioritization in mainframes because mainframes cost a lot of money. Nowadays, hardware is so cheap you don't have to do all that stuff.
If some young programmer undertook the challenge and created the framework, would anyone use it and test it? Will there be an actual need for something like this?
My point is that an insider information to what is really going on in the cutting edge usage of linux or just some smoke being blown around to an obligated write up.
Re:This belongs in the cluster manager (Score:5, Interesting)
Yes and no.
No, large (Linux using) companies like Google, Facebook, Twitter have always used some kind of Linux container solution, not virtualization.
Yes, policy is controlled by the cluster manager.
But for example Google uses nested CGroups for implemeting those policies for controlling resources/priorities on their hosts.
Virtualization is very ineffcient and Docker/Linux containers are a perfect example of how peole are starting to see that again:
https://www.youtube.com/watch?... [youtube.com] / https://www.youtube.com/watch?... [youtube.com]
Suppposedly, CPU utilization on AWS is very low, maybe even only 7%:
http://huanliu.wordpress.com/2... [wordpress.com]
The reason for that is, is that VMs get allocated resources they never end up using. Because the host kernel/hypervisor doesn't know what the VM (kernel) is going to do/need.
For their own services Google doesn't use VMs, but Google does offer VMs to customers and to control the resources used by VM they run the VM inside a container.
Here are some talks Google did at DockerCon that mentions some of the details of how they work:
https://www.youtube.com/watch?... [youtube.com]
https://www.youtube.com/watch?... [youtube.com]