Shuttleworth on Open Source Development

An anonymous reader writes "Mark Shuttleworth (retired cosmonaut and Ubuntu daddy) has written an informative blog entry about the problems associated with open source development. He found that paying geeks to code without assigning them managers lead to "shiny geek toys", rather than the product he was actually paying for. Shuttleworth says that left-field thinking is required when it comes to managing open source teams. See also Andrew Orlowski's analysis of why AOL eventually killed the Netscape project from a few years ago, where he describes Mozilla developers as "wandering off into Lotus-eating land"."

Exactly...

You can have all the creativity you want - but without proper leadership, all that effort and talent goes wasted. I have a few creative friends that have all these wonderful ideas - but they have no idea on the concepts of project planning or management of resources. Needless to say, their killer applications are still brain children - and not actually out here where the rest of us can use them.

Not some huge revelation...

I'm not sure exactly how it went down, but it sounds like he hired a bunch of developers for a project and just sent them off to go do it without leadership. This isn't a problem with open source, it's just a boneheaded decision. When you hire someone you have to train them and tell them what you expect. It's no wonder that they gravitated towards whatever they wanted to do since they had no direction.

You can have all the creativity you want - but without proper leadership, all that effort and talent goes wasted. I have a few creative friends that have all these wonderful ideas - but they have no idea on the concepts of project planning or management of resources. Needless to say, their killer applications are still brain children - and not actually out here where the rest of us can use them.

In that case self-management is the key. I've been there. Working for years in an educational environment where the actual workload was less than 20 hours, I had a lot of freedom to take things in new directions. I ended up coming up with some of my best ideas and was able to develop the discipline to implement them. But it was really hard not to get distracted. You have to develop a manager mentality--be results oriented. As a programmer / designer / creative, sometimes spending 8 hours just researching or learning something is well worth it, but at some point you have to jump in and focus hard on the final product until its done. Then you can go back into creative mode and dream up version 2.0.

Old article

From the article:

This entry was posted on Friday, November 21st, 2003 at 6:48 pm...

A little out of touch maybe?

Re:Old article

From the article:
This entry was posted on Friday, November 21st, 2003 at 6:48 pm...
A little out of touch maybe?

No, no, it just took that long to be signed off by all the department heads and then approved by upper management.

I'm not so sure....

that shiny geek toys are all that bad. I can't think of one thing that my grandmother (who is as far from a geek as one can get) uses every day that wasn't once a shiny geek toy to someone.

Mozilla - ouch.

I can relate to that comment, I've been waiting for Mozilla to implement Internet Explorer compatibility (XSLT extensions) and ACID2 compliance for a while. Even with the 10% market share Firefox enjoys, they still don't facilitate the programmers to replace existing IE applications.

I also agree with this:

Creating a neat C++ framework when what the world really needs a non-Microsoft browser is nothing but a deriliction of duty: a piece of vanity code. What we Brits call pointless "willy waving".

I really hated Internet Explorer. When I heard about Mozilla, I tried Milestone 8 (around 1999), and it was slow as a snail on my poor machine. WTF were they thinking? The Netscape code might have been difficult to maintain, but what really needed a revamp was the html renderer.

The reason Firefox did get a huge market share is not because of the XUL framework, but because it was finished. I'm sure all that delay could've been avoided.

His project needs an architect

I think his project ultimately would have been successful if he'd started with a strong architectural design. Get the documentation out there first and get the developers coding to it, rather than to some nebulous desire for a GUI tool. An architect would also have made a GUI decision, either picking XUL or some other framework, or he certainly could have designed his own. But to let the programmers run without focus was simply asking for what he got.

Who woulda thunk it?

He found that paying geeks to code without assigning them managers lead to "shiny geek toys", rather than the product he was actually paying for.

Do ya think? How long did it take him to reach that conclusion?

Seriously folks, this is a given and one of the main reasons I don't buy into all the hype about the electronic toy du jour. Everytime I see an article somewhere which says that 'X' is the latest electronic whiz toy that everyone must have I just roll my eyes and move along. (As a side note to marketers, I don't watch your commercials or read your flyers in the paper. You may now explode with unmitigated rage because I'm stealing from you for not watching what you produce.)

I don't want to be forced to buy a DVD player which plays DVDs, mpegs, connects to the net, calls my vet or offers me advice on what wine goes well with acadian rigatoni. I want the machine to play DVDs. Period.

By their very nature geeks (true geeks) will shovel every bell and whistle into a device they can get away with because that is what they do. They want to see how much cruft they can tack onto the hardware simply to see if it can be done. Top that off with manuals (the paper ones if you're lucky enough to get one) which are so poorly written and obtuse that the average user has to take lessons to learn how to program their device, and the market becomes filled with devices whose half-life is as long as the life of a fruit fly.

To all who produce this crap, here's a hint: Stop making a swiss army knife out of every product. If you absolutely must put tinsel on the tree, make three trees. The first is bare bones (i.e. just a cell phone. no music, games, etc). The second has a few more items (include games and music). The third has everything (bleeding edge). If you check your sales figures you'll be surprised to learn which one sells the best (hint: it's not number three).

Re:Who woulda thunk it?

If you check your sales figures you'll be surprised to learn which one sells the best (hint: it's not number three).>

Actually, I think, you will be quite surprised to find out that it actually IS number three.

Oh noes !

"It seemed as if, given free reign, the developers pursued their own personal interests rather than the goals of the project." [...]
"So I canned the project and shutdown the development office, letting the developers go."

For Pete's sake, don't anyone let my boss see that !! O.O

Not only open source projects...

I agree with most of what he said, except I don't think its limited to open source projects. I have seen that on purely commercial context as well. The problem is that you *need* some kind of "geek toys" occasionnally, because they sometimes give birth to a very valuable technology (I've seen that many times). That's a complex task to find the fair balance between what is reasonable/valuable and what is not in term of focus diversion, and that's a hell of a management task to deal with people who can't see that balance (either way).

Why blame OSS?

It sounds like he hired a team of talented but flakey developers and is generalizing this to all OSS development. I don't think the problem has anything to do with OSS - it has to do with a team of guys thinking they have free reign to do what they want with no expectations, deadlines or oversight.

Olde news?

Ok. For those that didn't realize it, that blog entry was from 2003. Today, three years later on, where is the SchoolTool project? Did Mark really learn a lesson and develop a solution or did he just relive a trend he noticed so long ago?

Seems to be a long development cycle for a specialized calendar. [schooltool.org] I'm glad I'm not paying for it.

And what else did you expect?

As far as I could see, very little in the way of specification, design / architecting.
Without a reasonable framework it was inevitable the project collapsed.

The actual coding should be a minor part of a project, the real blood, sweat and tears is the spec and the architecting / design (and usability / test side of things): If that is done well enough then the coding should be a simple join the dots task.

Without architecture / design constraints then you will get toys for the boys (and girls) as there is no pressure / direction on them to do otherwise.

Happy Shiney Faces

But it's often these oddball programmer projects that end up being the next big thing. Manage your coders but don't stiffle them. I think that's the real secret to Google's success and it can be yours too. Steer them towards finishing your projects and finishing their own projects.

XUL

I always kept wondering what exactly XUL was developed for when a browser was needed. I don't know the timeline, but wasn't Gtk ready about the time they started Mozilla? I know that Qt was for a long time worthless for cross platform free stuff, because Trolltech charged money for the win32 version (which they had every right to do so).

Schooltool link

In case people are too lazy to spend 3 seconds on google... (Which from some comments above seems to be the case)

http://www.schooltool.org/ [schooltool.org]

Summary of current status as I read it: SchoolTool still isn't really there, but they did manage to get the spinoff 'SchoolBell' out there, and the SchoolTool work is ongoing and being included in the 'Edubuntu' distro.

A Generic Failure

This has nothing to do with Open Source. It is about trying to develop a product without a spec., without an architect, without management and without a timeline. Kind of like pointing a group of carpenters at an empty lot and telling them to build a school.

It wouldn't be any more or less successful at Microsoft, IBM or SAS.

Orlowski is sooo wrong (and today we know it)

Actually, Orlowski reasons for deriding the Mozilla team in "wander[ing] off into Lotus-eating land" are:

"creating esoteric frameworks". Later we learn that means "Creating a neat C++ framework when what the world really needs a non-Microsoft browser is nothing but a deriliction of duty: a piece of vanity code". Except http://weblogs.mozillazine.org/ben/archives/009698 .html/ [mozillazine.org] shows XUL creation was a direct effect of AOL pressure on advertising and netscape portal integration

"note-perfect bug tracking systems that only a nerd could appreciate". Anyone who ever looked at bugzilla's internal knows it's a quick and ugly hack who could never mobilise a whole team for years.

So when Orlowski writes jokingly "corrupt suits at AOL" "betrayed the Great Noble Project" he's right, and when he rants about nerds he's dead wrong (you'll notice once the inspired suits where taken of the picture the nerds did produce a successful browser).

If anything, returning on Orlowski's pontifications today only emonstrates the depths of his prejudices and cluelessness.

30 year old philosphy...

...is still valid? Do one thing, do it well [wikipedia.org].

Imagine that - simple, solid advice survives time. Reminds me of the Twelve Networks Truths of RFC 1925 Section 2-11 [faqs.org]

Speaking of managers...

Speaking of managers, we need one at the Ubuntu art team. It's insane to see that place's activity painted by uncreative icon set wars. All while there's minimal organization and whenever you DO finish something interesting, it's too difficult to figure out how to get the right people to notice it. There's zero management, and therefore there's nothing useful happening with the time that people seem to put in it. Like stated, this leads to "geek toys".

I'm not even being a troll here. Ubuntu artwork development would be very well-off having SOME sort of centralized body to coordinate efforts.

Not quite

"retired cosmonaut"

He paid a bunch of money for the Russians to take him up. "Retired space tourist" maybe.

Developing for Others

I think it can be summed up that Open Source faces challenges when the developers are working on code "for other people" not themselves. Two of the most successful Open Source projects are GNU (excuse me, Free Software) and the Linux Kernel. I think you can categorize both of these as situations where the developer is classified as a user of the end product. It is in their interest to make the best product possible because it actually helps their own cause.

The case of the SchoolTool was that it was being developed by developers who didn't have a vested interest in making it work or at least faced no consequence if the tool didn't work.

Not to open another can of worms but I think this can extend to other situations such as Desktop Environments. A lot of talk about making stuff "Grandma Friendly" or similar mantra removes the developer even further from an interest in the user experience rather than add the focus people are hoping for.

-Eyst

much more so for proprietary projects

This sort of thing is a general problems with developers, and how it manifests itself depends on the environment.

In fact, it tends to be more a problem with closed source projects in large companies (as well as with lavishly funded open source projects). Why? Because the developers in large companies are well funded, they can go on forever doing their pet things, and upper management is often easily fooled about what's going on. The only reason Shuttleworth caught this is because he has a clue. Arguably, most of Vista is like that, although there the problem isn't that Microsoft management doesn't understand what their engineers are doing, it's that Microsoft management has the same set of warped gearhead goals as their engineers. It's also the reason why the Java enterprise platform is getting ever more bloated and unwieldy, while most real people write stuff in PHP.

In contrast, many open source projects, as Shuttleworth observes, are projects that have "an itch to scratch"; the code may not be pretty, but it helps get the primary jobs of the people who are developing it done--and their primary job is not to create complex software systems or re-invent XUL.

It's not reasonable to blame developers for it--after all, they aren't generally paid to make customers happy, they don't benefit from happy customers, so why should they care? Developers have to worry about making the resume look good, and a bigger, more complex project using the latest technology looks better than pushing a small VB or PHP app out the door.

So, if you're funding open source projects, make sure that you're getting your money's worth and keep in touch with the engineers you're paying. And if you're a manager in a big corporation with a large software development staff and you actually care about delivering good software, you have your choice of jumping off a roof or changing jobs.

You can't have it both ways

Developers of OSS often forget that they have two choices in most cases:

1) Meet the needs of their users and especially those who want to use their products
2) Meet their own needs

OSS developers need to stop using the argument that "feature X is missing because we're hobbyists." If you want to compete with the big guys, you need to give your users the features they want. It's certainly your right to prioritize based on your wants, but don't kid yourselves. If you don't give the users what they want... they'll leave.

You need somebody who cares

He was really going about it the wrong way the first time, and he was still going about it the wrong way the second time. What he should have done was start by hiring a retired school administrator who is willing to play with computers (but doesn't necessarily know anything about them). Then somebody who to keep the computer working. Then a couple of developers, chosen mostly by the school administrator based on whether they find the manager's excitement infectious.

You always get shiny geek toys. Knowing this, what you have to do is make the result you're after the most shiny thing around. It's probably too hard to find a group of developers who are mostly interested in school administration, but you can get the programming skill and interest from different people, so long as the people mix well. Of course, ideally, you want to enlist users as soon as possible, too, because even someone who used to need the software but doesn't now is going to have less of a focus on getting it. Pick some school system that's really in bad shape organizationally (so what you can do quickly is an improvement), get the people who would use the system to spend the summer working with the project, and actually use it in the fall.

This gives the team pressure to have something working and useable and real soon, so they don't get sidetracked into all the millions of things they could work on, which would advance the state of the art but not actually lead to the actual goal.

Misleading Blurb

Take note: In the Orlowski article, the line following the bit about the lotus-eating is: "Both these points of view are caricatures, of course." Thanks to the poster for the sensational, and ultimately false, summary.

Take the funding out of the equation...

...and he probably would've had a better product in the end. I think alot of the best open source software out there became great, simply because the developers *weren't* getting paid. It was a labor of love -- they saw something great, grabbed a CSV account and started contributing. You can't just throw money at people and expect them to follow your vision, especially if you're not there to lead and manage.

Blasphemy

Blasphemy : Mark Shuttleworth and Andrew Orlowski in the same Paragraph Mark is a God, I am in homage to him, (also as a Ubuntu user) Fellow "/.ers" Andrew Orlowski is a pathetic writer, that does not share any of our views.. wikipedia sums it up better than I do . http://en.wikipedia.org/wiki/Andrew_Orlowski [wikipedia.org]

different proprities

Andrew Orlowski says:

Creating a neat C++ framework when what the world really needs a non-Microsoft browser is nothing but a deriliction of duty: a piece of vanity code. What we Brits call pointless "willy waving".

This is correct for a commercial project but for an OSS project a modular well-designed code is an important quality because often programmers work on it for fun. Now that Firefox has an easy to extend structure (XUL, plugins, ecc..) the OSS development process can show its strengths because many developers can enhance it in a pleasant way.

I spoke with the head of SchoolTool

I spoke with the project manager for SchoolTool last year when he was at an educational conference in my area. He said that what Mark basicaly learned with the first (Java based) SchoolTool is not to start a project and then go into space. Goo advice for anyone committing to large product development.

The current SchoolTool is being written in Zope3 and is under tighter development control.

This is very old news and does not reflect the current understandings of either SchoolTool or Marc Shuttleworth. This article could also be called "My first babysteps in the universe of Open Source development", file under ancient history.

Kind Regards

Also Why Microsoft Is Infected With "Featuritis"

Of course, it may be their product managers so infected, and not so much the programmers, as I can't imagine programmers being that enthused over Group Policy Management Consoles...

Certainly Microsoft is the home of "Lotus eating" when it comes to security and reliability. I mean, their antispyware product disables Norton Anti-Virus? Who thought that one up?

SchoolTool Update

Funny to see this piece dredged up again. I'm the blogger Mark references in the story, Tom Hoffman, and for the past year and a half I've been managing the renewed SchoolTool development effort, after Steve Alexander created a new Zope 3 based architecture.

It is definitely tricky to manage a project with such broad and lofty goals, and we've still had our share of mis-steps and mis-directions. I have a background as a teacher and self-taught Zope hacker, so I've learned a lot of lessons about software development.

Nonetheless, a useful application is in sight. We'll have a beta this spring and serious testing in real schools in the fall of 2006. One key this time around was keeping the burn rate down and not creating specific expectations in schools and with governments that we subsequently failed to meet.

If you're interested in open source software for schools, check out http://schooltool.org./ [schooltool.org.]

Peer Review

what we need is not management BUT

-payment to coder only when the product meets requirements
(why did anyone get paid if all you got were shiny toys!)
-select coders who can self manage
-peer review

Peer Review is very important! You could have college students doing it, as long as someone goes in there and checks that the code does what it says it should.

Was in process of moderating but removed my moderation to make this comment

Faulty crystal ball

Creating a neat C++ framework when what the world really needs a non-Microsoft browser is nothing but a deriliction of duty: a piece of vanity code. What we Brits call pointless "willy waving".

...

So it's more than a mere accident that Opera, with its relentless focus on the Human Interface - and looking after the needs of users like your grandmother - stands set to reap the rewards. Instead of investigating the nerd-options, Opera invested in smartphones and embedded appliances.

Now I wonder which company has the largest browser market share? Maybe that willy waving paid off.

That's it! KDE and GNOME-- shiny geek toys.

This guy hit the nail on the head. Though it might be simply because these guys bought the party line that you have to use X.

Paying geeks to code without managers

Seems lots of places do quite well with egalitarian structures. No-one complains about India, Japan, China putting out shiny geek toys or wandering into lotus land even though they don't have American "org charts".

The issue is more to do with programmers who can't stay on track rather than programmers who ignore the "org chart".

Zope is a good pick for this sort of thing

Zope is a good technology choice for this project. And SchoolTool is a very neat project indeed. I always thought the world lacked such a product.

But no matter what technology and what sort of software you're building - be it OSS or not - you need a plan how to do it and should stick to that plan as far as possible. That's the lesson he learned.

I wonder if this has anything to do with The GIMP?

People have been wanting 16-bit color and CMYK support in the gimp since the previous century. FilmGimp aka Cinepaint was the gimp with 16-bit years ago. Why does the gimp still not have 16-bit color when the code has been around for years?

The answer is GEGL, a non-existant "shiny geek toy". GEGL is supposed to be some amazing framework that will handle image operations the Right Way. It will make 16-bit color, CMYK, and adjustment layers appear by magic. It will be fast and generalized and light-years beyond anything Adobe has and wash your windows for you. Who knows what it is supposed to do now? Unlike the codebase of GEGL, the legend of GEGL grows by leaps and bounds.

It you read the gimp devel list archives, you'll see many cases of people saying, "I want to code CMYK", or, "I have 16 bit support". The developers always send them away, "You are doing things the Wrong Way, you must work on GEGL instead!" The result is, development is killed.

What of GEGL? Years go by and it's nothing more a "design document" aka Musings of a Lotus-Eater, that hasn't been updated since the Clinton administration. A CVS repository that goes eight months at a time between commits. No code that actually compiles and does anything. It's still just a pipe-dream shiny geek toy.

Mark Shutteworth tried to fund someone to work on GEGL. I imagine nothing ever came of it.

Shouldn't he have changed his name...

...to "Soyuzworth?"

Hire the itch scratchers?

I'm not clear on why he's hiring programmers generically and not hiring people whose itch *is* the software he wants. They may not be super hotshot programmers but they've got the right motivations. In other words, enable people that would produce the software already if they could just afford to do it -- so many people have dreams but can't afford to follow them because their day job keeps them too busy. Particularly if you're working in Python the barriers to entry for less skilled programmers are much lower.

Re:Obviously!

Actually, Firefox came after that period of lotus-eating Orlowski describes.

I'm not sure there's much to disagree with in his analysis.

Re:Obviously!

Orlowski is miles off.

See how Firefox developed once it came out from under a corporate yoke. All those shiny geek toys (XUL, plugins, etc) started getting the attention they needed instead of making it work on an infinite number of badly written web pages.

Orlowski is just a hack who slags things off on the cusp of thier sucess. Hes turned The Register into a personal rant blogg, dont be suprised when it goes bankrupt.

Mark Shuttleworth on the other hand clearly states the problem, gives a lucid account of its causes, and proposes a solution. Maybe you should read his article and actually learn something.

Re:A bit old?

Geez, the guy posts a minute after the previous post (so he probably didn't see it) and he gets modded -1 (redundant). Won't anyone think of the latency?