Unreliable Linux Dumped from Crest Electronics 960
nri writes "The Age writes, Linux misses Windows of opportunity. Crest Electronics chose a Linux operating system, then seven months on, the company chose to abandon it for Windows.
Mr Horton says. ".. the machine would basically, putting it in Windows terms, core dump or blue screen at random. It would run for weeks or so and then just bang, it would stop....I fully support Linux but if I had to make the decision again I'd pick Windows. A big reason is the fact Windows was up and running in two hours at all the right patch levels. The installation of SAP took two days on Windows, the installation on Linux Red Hat took two weeks. The total cost of ownership is actually lower in this case than with Linux because of the hidden costs of the support.""
Bigger Problems (Score:1, Informative)
Well, if this Crest Electronics [crestelectronics.com] is their website they have more problems than just Linux. From their homepage:
Currently some people are having problems accessing portions of our website. We apologize for any inconvenience this may cause.
Real Story - SAP implementation fails miserably (Score:5, Informative)
I just left a company that was $10M and 2 years behind on their "$2M" SAP implementation. It's a joke. Once SAP gets their foot in the door, they flood your company with incompetent consultants and rebuild your business around SAP-approved procedures and architecture. At the end of this clusterfuck you end up WAY over budget and desperately looking for a scape goat. Clearly Crest Electronics chose Linux.
SAP products require patch after patch, and take MONTHS to really install. We had a team of engineers working around the clock (literally) for 5 months to get our base systems set up to SAP specs. Even then we would receive "mystery" patches, frequently resulting in system crashes as they weren't designed to work with other patches. Bottom line - SAP is the problem. They churn out highly unstable software and have armies of consultants who will sweep problems like this under the carpet or find something else to blame.
I Have My Doubts About the Guy (Score:5, Informative)
Blue screen is a Windows thing but core dump is not [wikipedia.org].
Crest Electronics is trialling Microsoft's Windows Server Update Service, which allows automatic patching for the operating system and other Microsoft software on servers and desktop machines across a corporate network. Its benefits are one of the key reasons why Mr Horton stands by his decision to switch from Linux to Windows.
"We run Linux on our web server and for an accounting package with great success and we do use the auto-patching in those environments,"
I work in a Windows shop but we don't do automatic patching. We don't patch until we've done extensive testing on our own to make sure it works in our environment first. SUS/WUSS/whatever is great in the sense that it allows you to control how patches to your Windows workstations are distributed. You can change the workstations' auto-update behavior so they only update from your SUS servers, etc. But the automatic update thing, from what I've heard, is rarely used in a production environment. In fact, Microsoft gives you a considerable amount of control over its behavior, probably because in recognition of the dangers of auto updating in a production environment.
Mr Horton disagrees: "It might be fine for things like security patches, which don't impact SAP certification rules but with some patches you still actually have to check the release levels and then check against the SAP site. Otherwise SAP might ask you to roll back to the previous version before they will support it."
Give me a break! The same thing happens in the Windows environment. It took Bloomberg and our other vendors a while before they supported Windows XP SP2. When SP2 first came out, a lot of vendors blamed SP2 for problems that may or may not have been SP2's fault. It took Windows vendors a while to adpot SP2 as well.
In any case, the whole patching issue he takes with Linux seems absurd. Just a few days ago, I think our server guys patched their cluster with a Microsoft service pack. Now the cluster refuses to fail over properly. Patching in a production environment is ALWAYS a big headache if you want to do it right. Unfortunately for our server guys, we don't have a spare cluster sitting around for them to test patches on like they normally do with other servers.
Re:your admins are not qualified (Score:5, Informative)
Re:I wish he would have given us more info. (Score:3, Informative)
Red Hat Enterprise Linux 3.0 was the distro. More info would've been nice, but they DID give this one (which a lot of people seem to be asking about).
Re:What is SAP? (Score:5, Informative)
Re:I wish he would have given us more info. (Score:3, Informative)
Well ITFA it said they were running RHEL 3 and for the server it was an IBM server
The server was also setup by a contractor that Redhat had recommended per specs that SAP had provided.
Re:I wish he would have given us more info. (Score:3, Informative)
Re:Windows vs Linux (Score:1, Informative)
ERP stuff is (generally) much more mature and easier to support on Windows. I was at a PeopleSoft conference right before they were eaten and went to the Linux sessions they had and it was a mess. They weren't going to officially support Linux until their next major version which in ERP is a death sentance (no support from the vendor, so if something goes wrong and you cant fix it, its off to consultants...). The install and config process was also detailed and even the speaker mentioned it wasnt for the faint of heart comparatively.
Short of Merck/IBM/other large company doing an implimentation on *nix (which is the only place I've ever seen anyone actually do it on *nix), everyone else does it on Windows cause its just easier to get up and running and maintain.
right tool for right job, this is just one of those jobs that doesnt fit at the moment.
Re:Bigger Problems (Score:1, Informative)
The article is in an Aussie paper. So I'd suspect that the company is actually at: http://www.crest.com.au/ [crest.com.au]
Re:Real Story - SAP implementation fails miserably (Score:3, Informative)
Out in the world, there's a software configuration management (SCM) tool called ClearCase. It's developed by a company called Rational, which is now known as a subsidiary of IBM.
As best as I can tell, the only reason anyone uses this thing is because it integrates cleanly into another product, Rational Rose (a UML modeling tool). Or in my case, because the company says it's a standard (no matter how many stories I hear that every project that uses it tends to do so miserably)
They, do, helpfully, have a linux version of both the client and the server. Unfortunately, this thing requires a binary kernel module to support their own proprietary filesystem (mvfs)
Of course, as binary-only modules usually go, if it wasn't compiled against the same version you're running (and in some cases, the same branch, stinking distribution vendors and their nonstandard patches), bad things happen.
I've spent a month fighting with one such machine that's running one of their blessed distribution/version combinations, and it's still causing problems. Seems the thing is trashing some part of kernel memory and causing both the ext3 and ext2 filesystems to go wonky.
But then, isn't this the exact reason why there's a separation between kernel and user space?
I've seen so many warts on this thing, I can't help but believe that Linux support for them was at best an afterthought, at worst a deceptive bulletpoint on marketing's spec sheets.
And this is what's considered "Enterprise Class Software" (like that term ever meant anything)
Re:What is SAP? (Score:5, Informative)
One doesn't usually run anything from SAP without a small army from SAP (or one of their annointed consulting firms) completely stroking the install. They don't usually tolerate failed installs. And there's usually a LOT of money involved in these installations, and a lot at stake. SAP products are rarely used with modifications and customization to both the infrastructure and the apps themselves.
Re:your admins are not qualified (Score:5, Informative)
from TFA: "We asked the customer to do a diagnostic test and the customer never responded, so it was impossible for us to address the issue," Mr McLaren says.
Re:your admins are not qualified (Score:1, Informative)
here are some quotes from the article....
"Last November it began migrating to an SAP enterprise resource planning system running on Red Hat Enterprise Linux 3.0. Despite using a version of Linux certified by SAP and SAP-certified IBM servers, stability issues and the complexities of keeping Linux up to date and secure forced Crest Electronics to abandon Linux for Windows Server 2003, Enterprise Edition in July this year."
"Mr Horton called in Red Hat-recommended contractors to install Red Hat Enterprise Linux and ensure it was configured according to SAP standards, a process which took two weeks."
This company appears to have done everything right in terms of getting an application running and using only certified components and staff to get it done.
If it only takes a couple of days to get Windows to get this done and work right - think business sense (Does the business run or wait for their "Linux/SAP certified system" to work) than there needs to be some serious thought given by busnesses who want to have Linux run their business software.
RTFA (Score:3, Informative)
RTFA or shut up.
Re:The key point to note in TFA is..... (Score:3, Informative)
"Crest's IT manager, Anthony Horton, oversaw the deployment of SAP on Linux in November 2004, after inheriting the decision when he took the job. Having previously run SAP on AIX - IBM's version of Unix - Horton was comfortable with deploying such a mission-critical application on Linux."
This isn't a Windows guy.
Re:blue screens? (Score:2, Informative)
Two Weeks! (Score:5, Informative)
Only an absolute moron would admit to that. You have idiots working for you fire them immediately! With absolutely no experience with any unix/linux system and very little windows experience, I setup a mail server, webserver and started creating a website for a company. I did that back in 1996 with RedHat 5 & a Linux for Dummies Book. Linux has come a long way since then. If they can't figure out how to install a modern linux distro in less than 4 hours, you should not be let near any computer ever! I could build a PC clone system from parts and install Fedora Core 4 configure it with apache, mysql, ftp and secure it before lunch. I've done it several times at work.
Re:RTFA (Score:2, Informative)
Regards,
Steve
Re:blue screens? (Score:2, Informative)
The majority of the time that bluescreens happen on Windows, it's because some 3rd party vendor's drivers are written poorly. Given the plethora of hardware that's supported under Windows, how likely do you suspect that it is that the average home user has at least *1* driver with at least *1* bug in it somewhere? That's your BSOD.
That's also why companies such as Dell make so much money -- they provide a fully-supported system with verified drivers. Computers you get from, say, www.cyberpowersystem.com or pieced together from Newegg parts are less likely to be consistent. Writing drivers is not easy.
How often do you see bug free programs? Never. I've had Firefox crash on me; I've had IE crash on me; I've had Visual Studio crash; I've had Eclipse crash. Everything crashes, it just so happens that when drivers do it it brings down the system. And there happens to be a *lot* more hardware support on Windows, which means a lot more people writing software, and a lot more bugs.
I'd challenge you to find me a set of API calls that can actually crash an out-of-the-box Win2k machine from "userland".
Re:Windows vs Linux (Score:5, Informative)
Re:Windows vs Linux (Score:4, Informative)
SAP??? No such product! (Score:2, Informative)
I am currently involved in an SAP Netweaver '04 implementation at one of the largest SAP customers in the SF Bay area. I have to admit that I have no experience with SAP software on a Linux platform -- my experience is with ERP 2004/Netweaver 2004 on Wintel and Solaris. Even so, I think I am accurate in stating that any significant part of the suite that you install on either of these platforms would not be useful in just a few hours. You probably won't have finished installing the base components, the patches, the service packs and the relevant business packages until towards the end of the first day. And then you still would not have even begun the lengthy task of configuring all the backend architecture to play together. And keep in mind, this is NOT a single server business solution, even for the smallest SMB customer!
So, what exactly does it mean if someone claims to have "SAP running" on a box in a couple of hours? It sounds kind of like a mail server with no network interface -- runs like a champ for months on end, no problems!
Maybe I'm missing some deeper insight, but this so-called "news" tells me nothing about SAP, Linux, Windows or Crest Electronics. Nada. Zilch. Click the back button and keep scrolling folks, there's nothing to see here.
Re:Windows vs Linux (Score:5, Informative)
Do the same applications run on each of the *BSDs without recompliation? I tend to doubt it, but I haven't tried it ...
No, it's not. Is it Apache 1 or Apache 2? The two are very different. Which modules are configured? Default configurations vary wildly. Yes, if you know what you're doing you can easily bring them under control, but for an amateur who's just using the Apache that came with his installation, things can be VERY different from distribution to distribution. (Personally, I find myself installing my own Apache and similar daemons, even if one is provided for me, on *BSD, whatever Linux, Solaris, etc. -- it just makes things easier, starting from a known quantity. And more secure.) It's not up to Stallman. Call it whatever you want. Your *BSD box has a lot of GNU stuff on it tooIn any event, certain hardware devices have buggy drivers, even in the latest versions of whatever Linux kernels and distributions you prefer. The vendors generally make Windows drivers, where the Linux drivers are often reverse engineered, and it often shows in the quality.
For the *BSDs, the drivers you get are generally more reliable than those in Linux, but if you've got some new device, where Linux would support it (and the driver might have some issues), *BSD is likely to not support it at all.
But I do agree with you too -- FreeBSD does make a better server than any of the Linux distributions. However, the commercial application support is very spotty. However, I've heard that the Linux emulation is quite good, and it can run most Linux applications with little trouble. Though that just sounds so ... wrong ... to use it for a production server. But if it works ...
Re:Windows vs Linux (Score:2, Informative)
Not necessarily. Windows is very tolerant of poor quality hardware. Linux is not. Case in point: I had a P3-800 with 3 256MB DIMMs. One was bad. I had Windows98 on it for years and it ran as well as expected. I added Gentoo as a dualboot and it was unstable as balls. Windows still ran fine. I replaced the RAM and BAM Gentoo was stable again, in fact, way more stable than Windows. Windows was unchanged. Moral of the story: Windows is crap designed to run on crap; Linux is good stuff (when done right) that requires good hardware to run. Not top-end, but good, as in functional.
FreeBSD is much the same as Linux, although I like better for stock uses like Apache and Samba. Linux, I think, is better for 3rd party apps, like Steam-based Dedicated Servers (Counter-Strike, Day of Defeat and related) and other game dedicated servers. Windows is not a very good server OS at all. While administration may "easier" because it's GUI, that GUI just serves to slow it down and causes instability. Case in point: I have a Cybercafe whose administration software runs only on Windows. It uses an MS Access backend. I also have to run a license server for Steam-based games on the same machine. Those two servers alone cause the sound to lag to the point where sound is useless even when they idle. Web Browsing is a bitch and it's not like we can afford to have this box not be used as an interface to the cybercafe software. And we're not talking a crap machine either. It's got a 2.6GHz Celeron D and 1GB PC3200-DDR2 SDRAM. Contrast this with a FreeBSD machine running a Samba printer and 5 shared drives used by 22 other system, on a P3-800 with 768MB PC100. No lag in Firefox in X even under a heavy load, like 20 machines grabbing 2GB of files each and one machine printing out report after report.
Linux is great as a server OS for many tasks but it's definitely not all things. Windows, on the other hand, sucks as a server OS but it's usuability makes it very worthwhile for apps that can handle a slight amount of instability, like web browsing, games and typing documents.
</RANT>
while tomcat on linux can be a pain (Score:3, Informative)
sum.zero
Re:Windows vs Linux (Score:3, Informative)
Re:Windows vs Linux (Score:3, Informative)
We asked them to do a diagnostic test... (Score:2, Informative)
So in other words, hardware fault that they never bothered to trace. What's the bet the Windows system was on fresh hardware?
I'm an SAP BASIS Admin (Score:2, Informative)
That means watching memory usage, extents, indices, disk controller utilization, network interface utilization, swap space, processor load, and on and on and on.... And trend it all out... then monitor some more - when you see the problem, you fix it. Sometimes that leads to other problems that get exposed as you move up the line - but when it's all done the system rocks...
I know this because I've built HUGE systems used by thousands of agents every day... they run on Sun E10000 boxes, E4000's, HP K-class boxes and so on... If you're installing SAP and you can't handle the diagnostics - get the fuck out of the computer room and go back to asking if you'd like fries with that...
I read the article and thought that the guy had a bunch of dingos for admins...
Ask Sanjay to help (Score:1, Informative)
This is a couple of blog postings that one person's experience with installing SAP. They're long writeups, but pretty darn amusing (unless it's happened to you of course!).
Part 1 [w1c.ca]
Part 2 [w1c.ca]
Part 3 [w1c.ca]
Part 4 [w1c.ca]
You really can't make this sort of stuff up.
Re:Windows vs Linux (Score:1, Informative)
Linux is not an operating system. Linux is a kernel, and this utilizes whatever userland you choose.
I doubt i have any GNU software on my FreeBSD 5.4-P6 boxen.
The reason you call it GNU/Linux is because of the GNU userland alongside the Linux kernel, it makes no sense calling it GNU/BSD when the *BSDs are complete OS's (both kernel and userland).
Re:Lets see in seven months (Score:5, Informative)
Right tool for the right job. Most of the time I think Unix is the better tool, but sometimes you don't need a swiss army knife. Sometimes you need a hammer. Windows is a very effective hammer
Re:Lets see in seven months (Score:4, Informative)
Re:Windows vs Linux (Score:3, Informative)
The point is that FreeBSD (for example) is developed as a whole with it's own kernel developed for nothing else but FreeBSD. The FreeBSD kernel, userland tools, bootloader, fs layout, installer, etc are all developed together, specifically for each other. A Linux distribution can be to an extent also, but they would tend to want to track the official Linux kernel and continue to make changes as they see fit.
FreeBSD is developed seperately as a whole. The same can be said for OpenBSD and NetBSD. There are no external forces that they need to deal with, as there is with any Linux distro. If Debian (I'm wearing a swirl t-shirt right now) does not like a Linux kernel change, they need to work it into what they do want, assuming they notice ALL the changes and ALL the consequences of seemingly innocent changes that they did not make. That is the fragmentation which the BSD's don't have.
Maybe not a HUGE deal, but any Linux distro is certainly developed in a more fragmented manner than the three main BSD's.
Theo de Raadt does not have to constantly change someone elses kernel or back-port desired changes between two large trees that continue to grow more and more different. The OpenBSD kernel is his and fits OpenBSD as a whole as it needs to, when it needs to.
BTW, I like Debian the most of all the Linux distro's and feel that it comes closest to being the least fragmented in development. I am aware that Debian developers work on the Linux kernel, including the official Linux kernel. The fragmentation, is between the fact that there are a bunch of kernels, various versions of official, plus Debian kernels and that the rest of the system is built to perform around them.
I realise that the BSD's also have various versions of their kernels, but their kernels are theirs, first and foremost.
Re:Windows vs Linux (Score:3, Informative)
Re:Hang on - auto patching Windows? (Score:3, Informative)
SAP "Supported configuration" is such and such version of Redhat Linux at a very specific patch level. I.E. This specific sub-release of glibc and the kernel that Redhat has published. If you let up2date or rhn automatically update the server, we won't support you because we've only tested at this certain level.
Now why that's ANY different than Windows updating itself is beyond me BUT I have a feeling that since he mentioned installing a Windows SUS, that SAP will support automatic Windows updates, as long as you control the update server. FYI, you can also do that with a Redhat Satellite Server but it costs money whereas Windows SUS does not (other than the cost of another server license).
One of the problems that we've seen with RHN updates is that older versions of packages are not available if a new one comes out. This has caused us problems in the past with our TSM server.
Case in point:
IBM in thier infinite wisdom decided that the Tape Library driver would not only check for the specific version of the kernel RPM but also check for the specific subrelease! I.E. kernel-smp-2.4.9-48.i386.rpm was supported but kernel-smp-2.4.9-65.i386.rpm was NOT. This drove us batshit because we went to install a new TSM server and we had two kernel choices, keep the one on the install media which had known bugs or upgrade via RHN to the newest kernel that had the performance bug squashed. Meanwhile the driver was only available for a release somewhere in between those two and RHN wasn't making it available for download anymore. The RPM installer for the driver (which was only available in RPM) refused to install if you didn't have the specific subrelease! The only way around it was a little rpm2tgz magic and remain "unsupported" until IBM saw fit to release a version of the driver for that kernel.
What ended up happening was that a NEWER kernel was released shortly after that. IBM built to THAT kernel instead of the one we had so we were forced to do a kernel upgrade just to install the driver.
Now having said all that, I can understand why this guy might have had to deal with that if SAP were a hardware vendor but they aren't. As MANY people have mentioned before in this thread, the problem really lies with SAP and the fact that the product will only be "supported" at very specific levels of certain Redhat packages.
The only real choice is to make sure you download each and every package that get mentioned in the errata from RHN when it's announced and then wait for SAP to announce which combination is newly supported and upgrade at that time. If you don't then you may be SOL because a package could be superceded in RHN and you won't even be able to get that package again.