Unreliable Linux Dumped from Crest Electronics 960
nri writes "The Age writes, Linux misses Windows of opportunity. Crest Electronics chose a Linux operating system, then seven months on, the company chose to abandon it for Windows.
Mr Horton says. ".. the machine would basically, putting it in Windows terms, core dump or blue screen at random. It would run for weeks or so and then just bang, it would stop....I fully support Linux but if I had to make the decision again I'd pick Windows. A big reason is the fact Windows was up and running in two hours at all the right patch levels. The installation of SAP took two days on Windows, the installation on Linux Red Hat took two weeks. The total cost of ownership is actually lower in this case than with Linux because of the hidden costs of the support.""
Wndows BSOD (Score:5, Interesting)
Odd that the Windows terminology for the blue screen of death now seems to be the standard term for a computer crashing. Or maybe that's not so odd.
(please don't mod this as funny, I am very serious here.)
What is SAP? (Score:3, Interesting)
blue screens? (Score:2, Interesting)
either way none of this reflects on linux's stablity at all, just on the skill of the admin running it.
want another hint this is a case of a total retard running the system? "2 weeks to install sap on redhat"
even the most stable system will go bad with an idiot in charge.
Re:Sometimes this doesn't suprise me (Score:3, Interesting)
Yeah, this is a general problem with common modern programming languages. Dealing with dependencies is just hard since we've had a reuse model that is largely based on saving disk space by having one copy of a function.
Today, I'm convinced we need a system where every version of every library is stored and programs are able to use whatever version they have been tested against.
Re:blue screens? (Score:3, Interesting)
A) You are admitting you truly know nothing about the NT architecture.
B) And it is normally called Kernel Panic, or a Random Reboot in your world.
C) If you never saw any OS fail in ALL YOUR YEARS ADMINING, are you sure they are really years?
Comment removed (Score:3, Interesting)
Comment removed (Score:2, Interesting)
Re:Smells fishy. (Score:2, Interesting)
I doubt that. From the article:
"We asked the customer to do a diagnostic test and the customer never responded, so it was impossible for us to address the issue," [Redhat's] Mr McLaren says.
Re:Sometimes this doesn't suprise me (Score:1, Interesting)
With Gentoo this is never an issue because programs are built against the current version of the library that you have installed on your system.
Re:I wish he would have given us more info. (Score:5, Interesting)
* 2 weeks to install to SAP standards? Hmm. How about 1 day to install Linux, and the rest is setting up SAP and testing? 2 days to install on Windows? How much testing was included there, eh?
* "Software updates had to be manually installed to ensure SAP certification." So that's like, rpm -Uvh the_update.rpm. The HORROR!
* "We asked the customer to do a diagnostic test and the customer never responded, so it was impossible for us to address the issue," Mr McLaren says. Most folks who are serious about making it work would probably get back to them when having these problems. Almost sounds like some geek personalities were the problem, not Linux.
RedHat, IBM and SAP are all cool about running this setup - but the IT department of this consumer electronics distribution company can't handle it effectively? I think I can see where the problem is...
The cynic in me suspects they got a VERY good deal from MS for publicising this move.
Re:Qualified is the operative term. (Score:3, Interesting)
I find 00 buckshot works quite well.
Seriously though, IIS runs really poorly on Linux too. Insist on such combinations and you get what you deserve. Did anyone investigate SAP before spending two weeks trying to get it running? I also find it quite hard to believe that they were getting crashes every week or so. Although I avoid RedHat...
Re:Windows vs Linux (Score:5, Interesting)
Besides the reference they were running IBM hardware, I wonder what their configuration was. That's the tough part of these kind of articles. Very little information and a conclusion. Sure it was IBM certified hardware and it was ruled out as the problem. Perhaps the RedHat engineers simply screwed up. Not like that couldn't happen
"We asked the customer to do a diagnostic test and the customer never responded, so it was impossible for us to address the issue," Mr McLaren says.
I wonder why they never bothered to respond to RedHat. If it was important then they would have worked with the Vendor. I'd like to see someone work with ANY Operating System and ignore their vendors help. With these tidbits of information, it's difficult to take such a conclusion seriously.
Re:Real Story - SAP implementation fails miserably (Score:2, Interesting)
I'm sure our future will be to run Linux on Red Hat and use DB2 as our database. But I'm also sure we'll do fine...I myself am an RHCE and we have several highly skilled Unix admins and DBAs.
Probably the article has an unwritten story of wanting to pay outside consultants to do everything and lacking the necessary talent in house. Also I suspect there was an impatience and underestimation of the complexity of an SAP implementation. Add in an unresolved bug and you have a disaster. But not necessarily the fault of Red Hat, SAP, or IBM.
BTW, Sun support isn't what it once was, but for SAP systems, there's nothing like the platinum Sun enterprise support. The only reason we'll probably switch to Linux is the questionable future of Sun itself.
Re:Two weeks -- hahahah some ppl are useless (Score:1, Interesting)
here are some quotes from the article.....
"Last November it began migrating to an SAP enterprise resource planning system running on Red Hat Enterprise Linux 3.0. Despite using a version of Linux certified by SAP and SAP-certified IBM servers, stability issues and the complexities of keeping Linux up to date and secure forced Crest Electronics to abandon Linux for Windows Server 2003, Enterprise Edition in July this year."
"Mr Horton called in Red Hat-recommended contractors to install Red Hat Enterprise Linux and ensure it was configured according to SAP standards, a process which took two weeks."
This company appears to have done everything right in terms of getting an application running and using only certified components and staff to get it done.
If it only takes a couple of days to get Windows to get this done and work right - think business sense (Does the business run or wait for their "Linux/SAP certified system" to work) than there needs to be some serious thought given by busnesses who want to have Linux run their business software.
Re:your admins are not qualified (Score:5, Interesting)
Re:windows code dumps (Score:3, Interesting)
I wonder what the hell they are doing that causes the system to crash? Or did they check for hardware problems before deciding to pull the plug? The only times I've had Linux crash in more than 10 years of using it were related to serious hardware failures.
Non sequitur (Score:3, Interesting)
Case in point. The most unreliable system I ever installed was a Linux server for a small retail management installation (2 registers, one server that was supposedly fault tolerant, etc). Well, we ran into two problems. The first was that the application was sitting on top of PostgreSQL and was trying to do an outer join between a very large table and two empty tables. Since PostgreSQL assumes that zero-length tables are really say 10 pages in size, it was doing a nested loop join against two empty tables every time the new invoice window would be opened, meaning that the registers were full, the 20 seconds or so was not only slowing the business down but also causing the server to be under very high load most of the time. No problems from that aside from poor performance. We did, however, hack the application to make it stop such braindead behavior.
However, a few weeks after we were able to fix this problem, we started getting database server errors (most commonly corrupted hash table messages). These were intermittant, but usually occurred at about the same time every day if they occurred at all. Turned out to be hardware failure of course.
In neither of these cases were the problems the fault of Linux. One was a correctable hardware failure and the other was a correctable softrware error. OTOH, I am not brave enough to try to run SAP on Linux.
Re:Windows vs Linux (Score:3, Interesting)
True, of course, but in this case it looks like someone seriously screwed up the configuration. By default neither Linux nor Windows will crash every two weeks, so somebody came along and made it worse. I don't know much about SAP, but if it took two weeks to install and configure on Linux and only two days on Windows, then the people who did on Linux it are either incompetent or the software is not very good on Linux, which is an SAP problem.
Maybe it's fair to say that if you're running SAP you shouldn't do it on Linux. I think that's where most of the pros and cons come into play when choosing an OS. I wouldn't run Apache or a DNS server on Windows for the same reason, but I also wouldn't blame the problem on Windows, I'd blame it on, for instance, the Apache Group not properly supporting Windows. (I hear Apache is pretty good on Windows now, but that wasn't always the case).
Re:Sometimes this doesn't suprise me (Score:3, Interesting)
Yeah, this is another difficult problem. How do you ensure that security updates are applied.
If you have something better than a linear versioning system, then you can distiguish between security updates and other sorts of updates. I still think you need a system where you have multiple versions of libraries that are functionally different. I think it's just a fact of life of software development. Libraries aren't going to just stagnat.
Re:"A" Linux Operating System? (Score:5, Interesting)
I've run busy mail servers hosting about 6,000 email addresses. I've seen a server run with a load average between 2.0 and 20.0, 24x7 for WEEKS ON END without any complaints. A full megabit of traffic, 24x7, just for EMAIL...
I've seen millions of website hits per month, month after month, year after year. No complaints, reliability simply excellent. And, I've seen this using Linux kernel 2.2, 2.4, and 2.6.
Sorry, pal. Maybe it's true for some other slashweenies, but in my experience, the reliability of Linux IS truly legendary, and is why I've standardized on Linux anywhere I can possibly use it.
Heck, when I'm putting together a new, high-capacity system, one of the first things I do is load a series of "torture tests" and run them. I put the server through its paces, running with a load average between 5.0 and 10.0, compiling the kernel or PHP in a loop, copying files, reading large files into memory and clearing memory out, while stressing whatever service the server will be using. (EG: if it's a mail server, while all the above is running, I have a script sending 10,000-20,000 emails per hour to 25 pseudo-accounts, while another script POPs them all to the bit bucket. If it's a web server, I have 10-20 wget shell scripts beating the webserver continuously)
Hour after hour, for a week or so.
A few disclaimers:
1) I make sure all the components for a high-capacity server (esp. the chipset & NIC) are on the RedHat compatability list.
2) When I'm buying hardware for a cheapie embedded server, I try to buy hardware that's been on the market for at least 6 months or so.
With this formula, I've had nothing but stellar results!
Re:Different results (Score:3, Interesting)
The only reason to reboot a 'nix system is if you're patching the kernel. We've got systems (bastion systems that do email filtering, very stripped down) that have uptimes coming up on two years. Yes, the applications and services have been upgraded and/or patched a few times, but the (linux) kernel hasn't needed it, and it's still going strong. (Heck, we even discovered one crazy process that leaked about a meg of memory a day -- a cron job to restart it every month took care of that until we got an update.)
Re:Windows vs Linux (Score:3, Interesting)
What I cannot understand here is why did they not go for a cluster. After-all SAP is not cheap and if you want reliability and availability then a cluster is the only way to go. If you try to do things on cheap (the MS way) then you are going to get bitten.
If the two weeks included the physical hardware installation and network setup then that is most likely acceptable, MS Windows would not make it any faster. If this was the case then what we have here is pure FUD.
SAP certification of hardware - has to be done for all hardware platforms and all OS's so that part is pure FUD. Evidently he must have got a bad hardware platform and the vendor should have come to the party quickly. The Intel platform requires alot more certification than the other major vendors hardware. It would be interesting to know if they used the same hardware platform.
Patch updates - can be done automatically by MS Windows and also Linux. If you do this for MS windows you will most likely have to reboot, with Linux (unless you change the kernel) you don't need to do this.
All I can really say to this that for every failure (or perceived failure) of Linux there are many successful ones.
Re:Non sequitur (Score:3, Interesting)
Keep in mind that the table was something to the effect of:
CREATE TABLE stuff (
recno INTEGER PRIMARY KEY DEFAULT NEXTVAL('recno_seq'),
empno INTEGER NOT NULL,
account VARCHAR(32) NOT NULL,
);
Only after turning on query logging did I discover that every time a new record was entered, Access would send a query of:
SELECT * FROM stuff WHERE empno IS NULL;
Uh, hello, empno is defined as NOT NULL -- it can't be null! Ever! I verified that the ODBC driver was providing the correct table definition, so it was just Access being stupid. Indeed, defining a partial index for the condition (empno IS NULL) made things lightning quick again...
Gah, I hate closed source apps.
Re:Days VS Months (Score:3, Interesting)
I have had much more patch downtime on Windows because it has to restart (usually minutes) while on Linux only the services usually need restarting (usually less than a second), but even then we install patches during off-hours, and most patches are really optional.
We run mostly Linux servers, but some server software (our ERP for instance) is only supported on Windows. In this case I suspect it'll run on Linux, but I'd rather run it on a configuration that's supported, tested, and recommended by the developer. So we bought a Windows server just to run this one piece of software, the most expensive server we've ever bought, but less than 1/10th as expensive as the software itself. In this case our decision of which operating system to use had nothing to do with which operating system was better.
Re:Different results (Score:2, Interesting)
Patching in general and especially kernel patches are more important in "high-availability" systems.
This patching does not mean you are taking the entire enviroment down though. Doesn't everyone run there "high-availability" systems in clusters? You simply take one node down at a time and patch.
How often do you patch your systems?
Quite true... (Score:3, Interesting)
Everyone knows the Apple Store, one of the largest online stores, runs on.. oh, wait.
We do know that Macs are useless for clustering [apple.com] and could never be used to build a supercomputer [macnewsworld.com].
I know, old ideas die hard.
AHA - I know whats was wrong (Score:5, Interesting)
Comment removed (Score:3, Interesting)
blue screen at random - costs more for most (Score:4, Interesting)
when your servers on windows will blue screen at the middle of the production day that wiwill most likely cost a lot more on the long term in productivity loss and people sitting in their offices not being able to access resources
yes i can install windows box in 30 minutes with webserver, however i have bsd boxes running 365days+ with dns/apache restart and having a good sleep while my non windows machines run is just cheaper me than having a blue screened server for 8 hours and loose customers or receive pages to "fix that crap" in the middle of the night
of course your mileage might vary
just a note: how can an installation of a software last 2 weeks vs 2days ? Same software ? I know sometimes clicking a defult config together takes less time than building a config file (text) from a bad template/example but 2 weeks ?
God created all that in 1 week! (including basics for SAP and Linux in a way) -OK He knows more than us I guess
Re:I wish he would have given us more info. (Score:3, Interesting)
I have a good hunch you're right. I think I'll post anonymously and let you know that at one point I've worked for EV1Servers. When they first offered Windows servers, I saw the press release that we were releasing. It was full of quotes about how we love our Windows servers, and how easy it is to install and set up Windows. It included graphs comparing Windows and Linux setup times, and how we can push out Windows servers quicker and how it costs less.
The article was complete bullshit. From what I heard, Microsoft wrote the article and sent it to EV1 to sign off on it so they could publish it. How much do you want to bet that EV1 got a discount for that?
Look at these quotes from the article:
"Mr Horton also found the total cost of ownership included soft costs such as the hard work required to keep Linux up and running"
"The total cost of ownership is actually lower in this case than with Linux because of the hidden costs of the support."
Does anyone have any doubt that this isnt straight out of the mouth of Microsoft's anti-Linux "Get-The-Facts" press department? This is Microsoft's current anti-Linux slogan that they're hammering into everyone's heads.
Check it out:
http://www.microsoft.com/windowsserversystem/fact
Subject (Score:5, Interesting)
"I fully support Linux but if I had to make the decision again I'd pick Windows. A big reason is the fact Windows was up and running in two hours at all the right patch levels. The installation of SAP took two days on Windows, the installation on Linux Red Hat took two weeks. The total cost of ownership is actually lower in this case than with Linux because of the hidden costs of the support."
I feel like I'm reading a Microsoft brochure. And keep in mind that I *like* Windows as a desktop OS, for the most part.
Cups is my troublespot (Score:3, Interesting)
my core troublespot appears to be cups, which will spin at 100% of a CPU within a few hours of starting. So I have to restart cups every morning. This is so, irritating. I suppose i could just get cron to reset it for me, but still.
Whereas on windows, I havent had to reboot my laptop since, what, yesterday, when the clipboard stopped working. I didnt even know the clipboard could stop working, but no, you can suddenly stop being able to cut and paste. Trust me to be the one to find out.
Re:Windows vs Linux (Score:4, Interesting)
I, we - the team I work in, have recently finished a migration of SAP4.6 on windows to SAP4.7 on solaris. The windows installation was formerly the largest single installation of SAP on windows in western europe, and it had severe scaling and support problems.
I also know from some years back, this having been discussed in an interview I attended, that Unilever, who before they switched held the crown for the largest installation of SAP on windows, switched to Digital UNIX (before it was Tru64) for exactly the same reason.
SAP on any OS has a hefty hardware requirement list. In addition, to my mind, they make some stupid recommendations about memory. Viz - for our setup we have multiple V440 and V490 suns with 16GB memory, and SAP want (and, on some servers, having hit this problem, *need*) THREE or FOUR TIMES that in swap. I would have suggested either less databases on each machine, or more memory (not sure if 16GB is the physical limit for V440/490) or just bigger machines, but then it's not my job to spec these things.
Anyway, having vast quantities of swap actively used as working memory may have contributed to the instability of a SAP system on linux, if as someone suggested that VM on linux is currently not as good as it could be.
Re:Lets see in seven months (Score:2, Interesting)
This is exactly the sort of stupid statement that gives Linux and OSS a bad name. My properly maintained Windows notebooks do not crash. They are completely stable and go for weeks without reboots, only going to standby and hibernate. Sounds stable to me.
You cannot just eliminate "driver issues" from the equation. If Linux has an issue with drivers (which is does) than that has to be included in the equation, unless you can run your system without drivers. Blaim vendors all you want, but as an IT manager I don't really give a damn whose fault it is that the system is unstable.
Re:Cups is my troublespot (Score:3, Interesting)
Yes, that can happen. Actually, it's probably a miracle it works at all - I looked at the Windows clipboard API [microsoft.com] a while back, and let's just say it's... interesting. (I'm not sure if they really still use a chain and rely on programs to pass on messages [microsoft.com], and I probably don't want to know.)
Wouldn't be suprised (Score:3, Interesting)
I've had ongoing issues like that before of random crashes spaced weeks apart (userland software problem, not OS problem). I worked with the vendor very hard and we got the issue resolved over the period of a few months. Some suggested we switch to windows. Myself and my contact at the software vendor didn't think it was a good idea. In fact, it wouldn't have been a good idea, because there was a corruption in the data itself that was crashing it. An OS switch would have been loads of time and effort, just to have the problem still be there.
The fact that he never even returned Red Hat's calls leads me to believe he really didn't want the problem fixed. He wanted to make Linux look as bad as possible to his superiors so he could switch to what he really wanted. I doubt the whole operating system crashed. A misconfigured SAP was probably crashing and he was too incompetant to be able to tell the difference.
Also, what lameass autopatches on a mission critical server? That's such an incredibly bad idea. I'm sure all Red Hat's patches are of the highest quality, but if downtime could be a problem at all, take 20 minutes out of your day to look over the patches and make sure none will conflict with your particular setup. There's no replacement for human intervention if it's that important.
Ultimately I highly doubt the problems are rooted in Red Hat or SAP. They are rooted in a stubborn admin who didn't know what he's doing on Linux and found it easier to blame everyone else.
Re:Lets see in seven months (Score:2, Interesting)
When dealing with SAP, on ANY operating system, you NEVER EVER NEVER turn on autopatching of the OS. When I was doing it, we were primarily running on Windows servers and moving some to Sun. OS Patches almost always break some part of SAP's software. As such, admins are explicitly instructed to WAIT on patching the OS until SAP has had time to evaluate the OS patch and create any patches of their own to fix issues that may have been introduced.
I agree with the parent, this admin is an idiot and its only a matter of time before he hoses up his Windows SAP install as well.
Re:Lets see in seven months (Score:3, Interesting)
apt-get remove gaim
Yep, that's a hell of a lot of time. And if you want to install something with a different prefix, you probably have a good reason (i.e. it's unstable), and people shouldn't try that if they don't know much about their Linux box.