Bitten By the Red Hat Perl Bug 234
snydeq writes "Smart coders always optimize the slowest thing. But what if 'the slowest thing' is the code supplied by your vendor? That was exactly the situation Vipul Ved Prakash discovered when he tinkered with a company Linux box on which Perl code was running at least 100 times slower than expected. The code, he found, was running on CentOS Linux, using Perl packages built by Red Hat. So Prakash got rid of the Perl executable that came with CentOS, compiled a new one from stock, and the bug disappeared. 'What's more disturbing,' McAllister writes, 'is that this Red Hat Perl performance issue is a known bug,' first documented in 2006 on Red Hat's own Bugzilla database. Folks affected by the current bug have two options: sit tight, or compile the Perl interpreter from source — effectively waiving your support contract. If a Linux vendor can't provide comprehensive maintenance and support for the open source software projects you depend on, McAllister asks, who ever will?"
If they aren't fixing simple bugs... (Score:4, Interesting)
...then what good is the support contract anyway? Either stop paying for it or make enough of a stink that they fix it. Of course, RedHat being an enormous company won't pay attention to anyone making a stink unless they are an enormous customer, which is a problem in both open-source and proprietary worlds. At least there's a workaround in the open-source world, one better than object patching the binary...
Re:That's what you get. (Score:3, Interesting)
However, I agree with your analysis that with something that's mission critical, you need to have the flexibility to compile and maintain the program yourself instead of relying on the vendor. In my view, you get the support contract for the things that aren't central to the business and you don't have 15+ people hired for.
Modules (Score:3, Interesting)
We used "modules" in a similar situation. http://modules.sourceforge.net/ [sourceforge.net]
There was a lot of development where I worked before. Things were written with dependancies to specific versions of programs. They would "probably" work with a later version but the developer didn't want to risk that and neither did the business side, so we implemented modules and let the devoper load the version s/he needed.
"module load perl" would always load the latest version but if you depended on a specific version you could always do a "module load perl/5.5.8" which would set environment variables to only get things from that version.
Worked like a charm.
My guess is it's the UTF bug (Score:2, Interesting)
Well the reason to want to use vendor perl is that perl does some very fancy file management stuff that makes it extremely fast for opening and closing files. So one presume a vendor could optimize this to their kernel.
My guess here is that this is just the recurrence of the UTF-8/16 bug.
Grep and Sort on unix set to UTF-8/16 as opposed to LANG = C (or Posix) are about three orders of magnitude slower on large files. (No I'm not exagerating). Recently this Bug showed up once again on the Leopard on mac.
About ten years ago this showed up in Perl on Redhat. Then it went away. I bet it is back.
I think the reason it comes and goes is it depends on if things are compiled to default to UTF-8/16 or to ascii.
Re:waiving your support contract? (Score:2, Interesting)
It does on any issues that crop up in the applications using the locally compiled Perl. What's so stupid about a vendor not supporting something you compiled yourself?
Re:Don't throw out the baby with the bath water (Score:5, Interesting)
Perhaps not, but I know that they will never get another dime of my money.
For years, I always purchased Red Hat even though I never had occasion to use the support that came with it. I was (and still am) bought into the FOSS concept and wanted to make it work for me and my business. But I stopped sending RH my money sometime about 8.0, when I called their support to try and get some help with a printer issue. I would have been satisfied if they had been able to get either one of my printers (HP LaserJet 1100 or LaserJet 4L) to work with RH. A surly woman with almost unrecognizable English--obviously reading off of a cue card--tried for a few minutes and then dismissed my support case with the comment that "RedHat doesn't work with all printers." When I mentioned that I had paid for the RedHat just so that I could have support, she hung up on me. I called back to get another support person with an equally incompetent and rude tech.
Eventually, someone at experts-exchange.com gave me the answer to my problem. Now I just download Centos and if I need support, I pay someone on a case-by-case basis.
Re:That's what you get. (Score:3, Interesting)
I always use the vendor perl on Fedora. Why should I pretend that I am cleverer than the people who did the work to package perl for my distribution? And this particular bug does not change my view of the our relative intelligence.
There's also the issue of using rpm to maintain your system, which like many good habits is a bit of a pain at first but soon becomes indispensible. Even when I wanted to upgrade perl to 5.10 (with Fedora 8, since 9 wasn't out yet), I preferred to build 5.10 as an RPM based on the Fedora source package for 5.8 and a new upstream tarball. That way I got a smooth upgrade when the official perl 5.10 package was included with Fedora 9 - the same smooth upgrade as all the other packages in the distro.
Multiple versions of perl on the system? No thanks. This isn't Java where you have a dozen crappy third-party apps all insisting on their own exact version of the interpreter or they won't be 'certified'. Just take one good version - preferably the package that comes with your distro, which you know thousands of others are using and testing - and stick with it.
Why is he using RH anyway? (Score:3, Interesting)
Re:That's what you get. (Score:5, Interesting)
The vendor version is always inferior.
The vendor version in this case has a bug fixed. The bug caused incorrect behaviour. In this case the vendor version is only inferior if you prefer fast but incorrect results. There isn't anything wrong with preferring fast incorrect results over slow correct results, but most people probably want slow and correct to be the default if given the choice.
Fast and correct always wins, and the real Perl hackers are working on that. In the meantime we take what we can get.
Re:That's what you get. (Score:2, Interesting)
It's not a Fedora vs Red Hat thing; it's a 5.8 vs 5.10 thing. Apparently, there are different package maintainers for the different versions, and the 5.10 guys are much better about working with the upstream.
Re:That's what you get. (Score:4, Interesting)
I disagree. You pay Red Hat to provide a baseline server with all of the provided applications and languages. You pay Red Hat to provide timely updates for their distribution.
You pay your 15+ staff to work on a custom application that may depend on something Red Hat provides.
Now you may need to custom compile something as a short-term solution, but if Red Hat can't do something as simple as keeping their Perl interpreter working correctly I would seriously reconsider paying them any more money.
Think about it. If your paid staff has to make a custom compilation of a vendor supplied application (and consequently keep it updated) then what are you paying Red Hat for? The days of paying Red Hat out of the goodness of our hearts are over. It's time for Red Hat to act like one of the big boys and earn their money.
Re:waiving your support contract? (Score:5, Interesting)
This is nearly opposite my experience. I'm working at a very large Wall Street firm.
Red Hat does *not* tell us: "Oh, I'm sorry you're not running the latest support pack, no support for you."
We've had to run a modified GCC for a while and Red Hat, *again* didn't say "You've changed it, so support for you." What they *did* say was, "Can you reproduce this on *our* gcc?" Which again is better than We've gotten from some other vendors.
We're still running AS4U4 in some places and RH has worked with us to track down bugs. Sometimes it ends with: "This was fixed in 4.5 please update." Sometimes it ends with "This is a bug, and here is the HF, please update to the released version when it becomes available."
In fact I have a hard time sometimes of getting our Admins to open tickes with the *right* vendor, because they'd rather open a ticket with RH, because it gets solved sooner. (Course that is more a dig on HP, Veritas, EMC and some other "Enterprise Software" companies.)
Unfortunately for Both us and RH, we don't like to update either, and even when RH has proven an update solves the problem, it's hard to get the Admins to actually update the boxen.
Re:waiving your support contract? (Score:5, Interesting)
My experiences are the opposite to that listed above. RedHat have been more forgiving and sensible supporting live production systems than my experiences from SUN or Veritas. Both experiences were for mega billion-dollar companies.
Re:waiving your support contract? (Score:3, Interesting)
Totaally lame article (Score:2, Interesting)
1) CentOS is *not* a RedHat product. So, Vipul, you don't lose any support... because you're entitled to none!
2) For the sake of clearing the FUD with RedHat's bugzilla's policies, here are some details on how it works:
- There's no rule that patches applied on bugzilla are to be reported upstream! Some (large) packages have over 100 patches applied. How would *you* feel if RedHat spammed you every time they applied a patch to whatever software you wrote?
- Upstream devs can register to receive any and all buzilla tickets for their products, if they chose to do so. Some do, some don't. Those that don't sometimes end up complaining that "RedHat/Fedora is keeping patches secret!". Duh.
- When a bug is reported on bugzilla, especially a crashing bug, only a _minimal_ fix is applied that fixes the crash. The patched package goes on QA for a while.
In the Perl story above an upstream patch was applied, so it made *no* sense for RedHat to report it upstream. Turns out that the patch (alone) caused another problem (slowdown). The normal thing to do was to file another bug report on it. Which did happen! And, lo and behold the bug will be fixed:
-- Comment #14 From Marcela Maslanova 2008-08-08 02:42:52 EDT --
This bug will be solved in RHEL5U3.
So Vipul, which didn't contribute a cent to RedHat by using CentOS, is biatchin' that RedHat is simply following the cautious QA policy that RHEL is known for. Perhaps he needs to use another distro that provides more bleeding edge updates, with the obvious risk entailed by less QA. Not that RedHat would give a hoot what Vipul is using (for free)...
Re:waiving your support contract? (Score:3, Interesting)
Red Hat does *not* tell us: "Oh, I'm sorry you're not running the latest support pack, no support for you."
I've found it's more likely that _I'm_ telling customers that we should patch to fix problems they have with Redhat systems than Redhat telling me.
We're still running AS4U4
Heh, we have some AS4U2 systems, with only selected patches applied. I suspect some customers have used software from 'enterprise' vendors who don't have quite as strict don't-break-stuff policies with their updates. I can sympathize; I've had vendors change actual major versions of software in 'patches', rather than backport bugfixes.
So I have to agree with you, Redhat is a rare pleasure to deal with in the enterprise software support space.
Re:waiving your support contract? (Score:2, Interesting)
they won't support a system that doesn't have the latest updates
I am a Tech Support Engineer at Red Hat. Your statement is incorrect.
We do provide support for older packages (assuming you are running a RHEL version that isn't over 7 years old [redhat.com] and even then we usually provide best effort support). However if you hit a known bug when running package version X and we have released a fix in version Y we'll ask you to upgrade to version Y and won't provide a version X.y just for you. Well, you can still apply patch from Bugzilla and compile your own X.y but we can't provide real support for that. Of course, if you get into troubles when upgrading to version Y we'll support you (and provide version Z if you hit another bug).
Now, when we need to quickly find the cause of a problem (and a workaround and a proper fix) it is not unusual to ask if you can reproduce it with latest version, especially with complex packages that change quickly (like the kernel) but we understand this is not always practical and we surely don't systematically ask you to upgrade to latest version.
Re:Don't throw out the baby with the bath water (Score:2, Interesting)
You had an individual license over half a decade ago as the .com bubble was still bursting. The company has completely changed since then. Hold a grudge much?
Re:waiving your support contract? (Score:3, Interesting)
Ummm... other than shiny pictures, what is the difference between redhat and centos?
Oh, support. Wait, you mean, a bug in centos isn't a bug in redhat until someone magically buys a redhat license? Now I've learnt something new.