Bitten By the Red Hat Perl Bug 234
snydeq writes "Smart coders always optimize the slowest thing. But what if 'the slowest thing' is the code supplied by your vendor? That was exactly the situation Vipul Ved Prakash discovered when he tinkered with a company Linux box on which Perl code was running at least 100 times slower than expected. The code, he found, was running on CentOS Linux, using Perl packages built by Red Hat. So Prakash got rid of the Perl executable that came with CentOS, compiled a new one from stock, and the bug disappeared. 'What's more disturbing,' McAllister writes, 'is that this Red Hat Perl performance issue is a known bug,' first documented in 2006 on Red Hat's own Bugzilla database. Folks affected by the current bug have two options: sit tight, or compile the Perl interpreter from source — effectively waiving your support contract. If a Linux vendor can't provide comprehensive maintenance and support for the open source software projects you depend on, McAllister asks, who ever will?"
Re:Red Hat on CentOS? (Score:3, Informative)
Um...because CentOS is basically Redhat Enterprise Linux with the proprietary bits stripped out. So if the bug appears in RHEL, it will also appear in CentOS.
Personally, I always stash a copy of the latest version in /usr/local as another poster already mentioned. I've tripped on this bug before as well.
Cheers,
CentOS it NOT Red Hat (Score:5, Informative)
If it is "supplied by CentOS" then it was compiled by "CentOS" not Red Hat. Red Hat Enterprise Linux enterprise had a hotfix for this weeks ago. So if Vipul had been using a Red Hat product, he would not have had this problem.
Re:It's not just perl (Score:3, Informative)
I noticed this too, but then found the cause. By default, your language setting is en_US.UTF-8 (echo "$LANG" variable). This includes unicode support, so any text utilities such as grep have a bit more work to do when searching through files (UTF-8 uses 8 bits for normal ascii charcters 0-127, but uses 16 or 24 bits for extended unicode characters). To fix this, change LANG environment variable to "en_US" (http://linux.slashdot.org/article.pl?sid=08/08/29/1423201# /etc/sysconfig/i18n.
Previewleave off the "UTF-8") prior to running these tools (if you don't need unicode support). To change the language permanently on bootup, change it in
Heck, this might even fix this perl "bug" (I haven't looked at it yet though).
Re:It's not just perl (Score:3, Informative)
Maybe, but I'm not completely convinced. I have my locale to en_US.UTF8 on my Gentoo machines and they're still markedly faster.
Re:Article is a troll (Score:5, Informative)
This is what a perl core hacker [perl.org] has to say about the issue:
Example of the UTF-8 default "bug" (Score:4, Informative)
here's an example of this on a mac OS 10.5 /tmp/foo
[CODE]
perl -we 'for (0..1000000) { print rand(1),"\n"}' >
time sort /tmp/foo
real 0m36.169s
user 0m35.731s
sys 0m0.264s
echo $LANG
en_US.UTF-8
setenv LANG C
time sort /tmp/foo > /tmp/foo2
0.966u 0.201s 0:01.27 91.3%
SO for a small file it's 40 times faster when not defaulting to UTF-8
Perl on leopard however appears to assume LANG=C bey default and is quite fast. As is grep.
Using your support contract? (Score:1, Informative)
If this benchmark was a real issue for your production systems and you had a support contract with Red Hat you would obviously have requested and gotten a hotfix from them already. Done.
That is what support contracts are for to get support and fixes timely even if the generally available package is still going through the full support cycles. Sure CentOS is great and very cheap, but you would go to Red Hat to get timely support and updates if something like this is a real business concern.
Re:yum (Score:3, Informative)
Re:Red Hat on CentOS? (Score:2, Informative)
Re:CentOS it NOT Red Hat (Score:2, Informative)
Don't link to bugzilla (Score:5, Informative)
Here's the text currently in the bugzilla entry (edited to meet slashdot's filter requirements):
Bug 379791 - perl bless/overload performance problem
Summary: perl bless/overload performance problem
Status: VERIFIED
Product: Red Hat Enterprise Linux 5
Component: perl (Show Red Hat Enterprise Linux 5/perl bugs)
Version: 5.2
Platform: All Linux
Priority: urgent Severity: high
Target Milestone: rc
Assigned To: Marcela Maslanova
QA Contact: desktop-bugs@redhat.com
URL:
Whiteboard: GSSApproved
Keywords: ZStream
Depends on:
Blocks:
Reported: 2007-11-13 07:14 EDT by Nigel Metheringham
Modified: 2008-08-29 10:30 EDT (History)
Fixed In Version:
Release Notes:
Description From Nigel Metheringham 2007-11-13 07:14:04 EDT
RHEL5 perl shows the same performance issues as the Fedora 7 perl did - see
Bug #196836 and Bug #253728
This has been demonstrated in the recent perl update perl-5.8.8-10.el5_0.2
Same fix needs taking across to RHEL, ideally as a update release rather than
waiting for next major release cycle.
I do not have RHEL5.1 to test against right now, but the timing of the Fedora
fixes leads me to believe these would be much too late for the 5.1 release
cycle.
-- Comment #2 From Martin Kutter 2007-11-30 05:24:01 EDT --
The issue can be observed running the benchmark script from the recent
SOAP::WSDL package.
To do so, download SOAP-WSDL-2.00_24 (and its dependencies) from CPAN, run perl
Build.PL && perl Build, cd into benchmark and run perl -I../blib/lib 01_expat.t
This is the Output from RHEL4:
perl -I../lib 01_expat.t
Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 4 wallclock secs ( 3.48 usr + 0.01 sys = 3.49 CPU) @1432.66/s (n=5000)
XML::Simple (Hash): 7 wallclock secs ( 7.19 usr + 0.03 sys = 7.22 CPU) @692.52/s (n=5000)
XSD (SOAP::WSDL): 6 wallclock secs ( 6.06 usr + 0.01 sys = 6.07 CPU) @823.72/s (n=5000)
And this (with reduced n) is from RHEL5 (different machine, perl-5.8.8-10):
Benchmark: timing 500 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 1 wallclock secs ( 0.59 usr + 0.00 sys = 0.59 CPU) @847.46/s (n=500)
XML::Simple (Hash): 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @471.70/s (n=500)
XSD (SOAP::WSDL): 11 wallclock secs (11.34 usr + 0.01 sys = 11.35 CPU) @44.05/s (n=500)
Increasing the number of runs shows the O(n^2) nature of the performance problem
- increasing the number of runs by a factor of 10 increases the runtime for the
XSD bench by a factor of nearly 100:
Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 6 wallclock secs ( 6.19 usr + 0.03 sys = 6.22 CPU) @ 803.86/s (n=5000)
XML::Simple (Hash): 11 wallclock secs (11.20 usr + 0.02 sys = 11.22 CPU) @ 445.63/s (n=5000)
XSD (SOAP::WSDL): 851 wallclock secs (847.36 usr + 2.28 sys = 849.64 CPU) @ 5.88/s (n=5000)
-- Comment #3 From RHEL Product and Program Management 2007-12-03 15:47:35 EDT --
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the cur
Re:That's what you get. (Score:1, Informative)
The reason to run vendor code is support. Suppose your CTO gave you a few million dollars to spend on purchasing a computer cluster and writing some software to parse logs or convert files or whatever these guys were doing. This is a large investment for the company, needed to get some critical analysis done fast every day. Now you've set it all up and you go download some recent version of an interpreter from a group of developers ranging from semi-paid to hobbyists that was released a couple of months ago to run it. If this interpreter breaks, runs slowly, leaks memory, or otherwise fails you, your expensive cluster is sitting idle, and your deadlines aren't being met, who is to blame? Only you. Even worse, for you the only problem is that you might get fired, but for the company, there will be serious trouble explaining to investors and customers why some core part of the infrastructure isn't working and the company can't meet the consumer demands it said it would. This is why businesses buy software with support. If you're a hobbyist running a personal project or even some admin at a real business doing something non-critical, and you want to use random recent software you downloaded off the net, go ahead. But if your business depends on it, it's worth having a team of experts ready to help with any issues as soon as you call, as well as the knowledge that your software has been built and tested by a proven profitable company and the potential for getting compensation if things go wrong.
Re:waiving your support contract? (Score:5, Informative)
Very true. And this has been an ongoing issue with Linux adoption... I have a friend who runs mega-million-dollar, mission-critical systems and they've had to move off of Linux in favor of (Sun? Don't remember right now). It isn't about functionality. It isn't about open source. It's about support. Red Hat, et. al. want to be enterprise systems, and claim to offer enterprise support. But they don't perform enterprise support. As indicated here, change something to fix a bug, and you don't get support for that piece anymore. More, they won't support a system that doesn't have the latest updates, which is a problem on mission-critical systems. We don't update needlessly, and we certainly don't update to 'today's' patch. We have to wait and be sure the patch is stable and provides an improvement without risking our mission.
Until the players selling support realize all of this, Linux will be a difficult sell for such key systems (and the PHBs all think ALL their systems are mission-critical).
Keep in mind, I say this lovingly.... I want Linux to succeed and prefer it over the popular alternative.
Re:That's what you get. (Score:5, Informative)
No released version of Perl ever had this bug. Red Hat pulled a patch from a development version of Perl and maintained it over released versions of Perl which did not need it. That's the source of this bug. The Perl developers fixed this bug before releasing the next stable version of Perl.
Re:Good thing it was open source (Score:3, Informative)
Why are those benifits mutually exclusive?
Over 99% of the time you can install common software with a single apt/yum command and have it work perfectly. Occasionally, when a packaging mistake is made, it's reasonably straightforward for an expert to install the software manually.
This is the general usability case for FOSS software: In the normal case, it Just Works. In the exceptional case, you can make it work. With proprietary software, the supplier's business model frequently makes you sacrifice *both* of those properties. You have to deal with license compliance stuff to make it work at all, and if it breaks you don't have what you need to fix it.
Re:That's what you get. (Score:2, Informative)
p5p uses a tool called perlbench [cpan.org] to check performance periodically. It doesn't cover every combination of language features, but it's a good baseline.
Re:Wrong tool for the job? (Score:3, Informative)
It's not all-or-nothing. The code doesn't have to be super fast, just fast enough. Perl can (when running at it's normal speed) accomplish the needed task at a sufficient speed. If Perl couldn't get it done fast enough when running properly, then yes it would be the wrong tool for the job.