NZX Moves To Oracle On Linux 213
sn00ker writes "In this story in The New Zealand Herald, we learn that the NZX stock exchange has moved their database systems to Oracle running on RedHat Linux, running on commodity Intel-based hardware. What's really impressive are the performance numbers they're claiming. Quoth the article, "One key query - searching the data on historical trades to identify maximum trade values - has been cut from 36 seconds to 0.03 seconds." An improvement of over 1000 times is spectacular in anybody's books, and is one hell of a boost for the proponents of Linux at the back-end of the financial world."
Linux? (Score:5, Insightful)
Oh come on! They consolidated 21 databases and moved to Oracle. That's why it is 1000 times faster. The move to Linux is a footnote as far as the performance issue is concerned -- as stated in the article, the move to Linux was for cost. I'm sure Solaris or god help me, Windows Server 2003 would have given similar performance results. Now if they had moved to MySQL...
Re:Linux? (Score:5, Insightful)
Either way though the fact that a major exchange is running linux is big news. Their database is their life and they are trusting it to linux. That says a lot.
Some kind of cluster (Score:5, Informative)
Re:Some kind of cluster (Score:5, Insightful)
Re:Some kind of cluster (Score:5, Funny)
Re:Some kind of cluster (Score:2)
Re:Some kind of cluster (Score:2)
Also you have to factor in the (supposedly) more expensive Sun training and sysadmins vs. cheap-to-train Linux sysadmins
That's not true. If you compare Red Hat and Sun courses and certification prices they are on par.
And there are just Unix sysadmins - companies rarely distinguish between Linux, Solaris. FreeBSD, HP/UX, etc admins when hire.
and the cost of 3
Re:Linux? (Score:5, Insightful)
Re:Linux? (Score:1, Insightful)
Re:Linux? (Score:3, Informative)
Re:Linux? (Score:2)
Obligatory Simpson's quote, sorry...
"BLAMM" and/or "PUFF" (Score:2)
PUFF == Partisan Usability Fact Filtration
Re:Linux? (Score:5, Insightful)
Re:Linux? (Score:5, Funny)
It's great news that Larry Ellison has Open Sourced Oracle!
And to think people criticize me for getting all my news from Slashdot.
Re:Linux? (Score:5, Insightful)
And while Windows Server 2k3 can run on the same cheap hardware, can you get a production quality release for AMD64 if you need gobs of RAM? What about the cost of multi-CPU licenses, and any client licenses needed? All money that buys you more power in the Linux world.
It may well have been the case "well, with Linux, we can buy enough CPU, I/O, and DBA tuning time to make this thing sing. With Windows we blow money of software licenses. With Solaris we blow it on licenses and proprietary hardware."
Re:Linux? (Score:5, Informative)
Using an Oracle RAC cluster of Sun V440s would have actually been cheaper than clustering 4 way Dells - Sparc kit's a lot cheaper than it was. You'd also have had some decent 64bit capable boxes. Check out the TPC/E benchmarks - Sun boxes blow everyone else away in terms of price/performance on a real world database app.
Re:Linux? (Score:2)
Seriously: Sun is great; I love the kit, and for a high-end server system I'd highly reccomend it. But you are living in cloud cuckoo land if you think it's cheaper (even using a Microsoft-esque TCO calculation) at the low-end.
Truth is, Moore's law is working relentlessly against Sun. Lintel systems become more powerful by the day, and being commodity, they become cheaper too. Sun's market is slowly, but surely being eaten away. A move (again) to the (even higher) high-end only
Re:Linux? (Score:3, Insightful)
Lintel systems have fast CPUs, but no decent memory bandwidth. On the famous HPC Linux clusters everyone loves on Slashdot you'll see the first CPU running happily at 90% plus, the second sitting way below that - you're simply waiting very quickly for nothing to happen.
Opteron systems are another thing entirely though - that's why Sun are focussing a great deal on that platform. There are even a couple of reference architectures for Oracle on th
Re:Linux? (Score:2)
This makes me think the RHAS scheme is stupid, and is trading marketshare for short-term profit. Which is ok for a short-term plan, but they have to somehow get marketshare up to a critical level (20%?) long-term... they may be eating seedcorn here.
Re:Linux? (Score:2)
Unless, of course, you need to upgrade, where Sun will now charge you for licenses.
Tell it to the UltraSPARC/Linux developers, when Sun refused to release errata information for them to deal with data corruption and halting bugs their processors. Real open.
Re:Linux? (Score:2)
You mean the OpenBSD developers who refused to sign an NDA which the Linux guys were happy to sign?
Sparc is an open standard - http://www.sparc.org
Re:Linux? (Score:2)
That may be true, but not to the degree of a 1000X performance increase, which is what I think OP was protesting.
Re:Linux? (Score:2, Funny)
wait 35 seconds
Re:Linux? (Score:3, Funny)
Don't bring facts into any argument for open source. The zealots will lynch you!
Re:Linux? (Score:5, Insightful)
An inprovement of 1000 times might not be that sectacular, it depends what the system was replacing.
Most likely it would be some unix hardware circa 1997. (say 4 x 200 Mhz Solaris, 512MB, with SCSII II disks, or, perhaps even a VAX complete with snails pace IO would be typical for that period in that environment). So a 2 x 3 gHz, 2 GB, with fibre channel ought to be faster. Plus it looks like they rewrote the whole system to take advantage of Oracle 10 features.
What is perhaps more interesting for slashdot readers is that for most people working at the trading end of finacial services this is very much a non news story. The last two sites I worked at had implemented or were implementing Linux cluster server based systems, and, these were both for volume performance critical systems.
Ya, pretty much (Score:5, Informative)
Where an OS can shine is if you are running lots of stuff (eg webserver, scripts, database server, media server all on one box) and espically when you are screwing around and hence likely to cause problems. However when you do a DB install and run nothing but that, the OS is just a helper. It talks to the hardware and provides some simple APIs. Which OS it is isn't of much consequence to performance.
The cost thing makes me curious too. We tried Solaris on Linux. The DBA couldn't get it to work, and neither could I. Then I looked at the requirements. We are trying SUSE, since that was listed... Well, sorta. It didn't run on normal SUSE, just SUSE Enterprise Server. Likewise not RedHat, but RHEL, and also UnitedLinux. In otherwords, high dollar server Linuxes. Oracle tech support wouldn't even talk to us unless we used a supported OS. We ended up option for Windows XP Pro, since it was supported. As I said, OS didn't much matter, just that it ran Oracle.
Now while I'm sure (or at least pretty sure) Oracle could be made to run on a non-enterprise Linux, what would be the point? They wouldn't support you and support is one of the big reasons to buy Oracle (not cheap in case you were wondering).
Macho, macho ,men (Score:4, Funny)
Man, let me stand up and take my hat in genuine appreciation.
This ladies and gents, is a real hero.
Re:Ya, pretty much (Score:2)
You definitely want a supported enterprise Linux for your production machines, but it's also worthwhile to give developers on workstations copies of Oracle to use locally. Development can be a lot easier when you have a database that's all your ow
Depends how you read it. (Score:3, Informative)
A significant section of the IT world think of linux as nothing more than a hobby Linux - that you can't rely on it for anything mission critical.
What this is saying is that Redhat Linux CAN foot it with the big (commerical) boys like Sun and Microsoft.
Re:Linux? (Score:3, Funny)
That *is* a joke, right?
Re:Linux? (Score:2)
Re:Linux? (Score:2, Funny)
Geek -> Pointy Haired Boss -> PR Bullshit Man -> Journalist
And the final result doesn't seem to make a lot of sense. Why does this come as a surprise?
Re:Linux? (Score:2)
1) Proper indexing
2) Proper allocation of DB files across disk controlers
3) Proper use of in memory cache
4) use of the proper raid configuration (related to point #2)
5) cleaning up queries in the application (one cross product and you can bog down the biggest servers).
My first suspicion would be that the staff that set up the original system were goobers....
Re:Linux? (Score:2)
Always check your indexes (Score:4, Funny)
Re:Always check your indexes (Score:2)
Obviously.
The migration is from where the obvious isn't to where the obvious is.
The obvious is obvious once you see it. It is not equivalent to easy.
"With enough eyes all bugs are shallow"
If the right set of eyes looks at it just right, the bug is obvious.
You will be able to immediately spot obvious bugs I make that I cannot see.
Re:Always check your indexes (Score:2)
A single point of information contains date, symbol(probably and id), open, high, low, close, volume, and sometimes open interest, bid, and ask. All those fields combined are probably going to run you about 40 bytes. There ar
1000 times faster? (Score:3, Insightful)
Of course, if you want to yell from the treetops "Linux runs 1000 times faster..." I'm sure people will back you up.
Re:1000 times faster? (Score:5, Insightful)
Yeah. My call would be that they were operating an RAM-starved server. I've seen similar numbers doing basic PC upgrades!
I remember on case (this was a few years ago) where somebody with a customer information database of about 400,000 records came to me because generating a list from a query would often take several minutes.
They were using a Pentium-90 with 32 MB of RAM. I set them up with a (then) top-of-the-line PIII 600 with 256 MB RAM. Query time dropped to 1 second.
No matter what O/S you run, you're going to get JACK for performance if your running your app in swap.
Re:1000 times faster? (Score:1)
They probably migrated off a 10-20 year old clunker.
Re:1000 times faster? (Score:2, Funny)
Lemme guess. You're a DBA and you've had a hard time with your boss lately.
Re:1000 times faster? (Score:3, Insightful)
Re:1000 times faster? (Score:2)
Re:1000 times faster? (Score:2)
Slashdot: fair and balanced (Score:5, Funny)
Comon guys. What kind of idiots do you take us for?
Re:Slashdot: fair and balanced (Score:3, Funny)
That has *nothing* to do with Linux (Score:4, Insightful)
Sure, a more efficient process scheduler, a more efficient IO scheduler, but really. It would make a lot more sense for the difference to be in the DBM, or even more likely, in the design of the database itself.
Just because someone works for a big company doesn't mean they know what their doing. The most likely reason for the speedup would have been an optimization in their own software, or their database schema. Followed by an improvement in the RDBM, and finally the OS.
A thousand fold increase in speed simply from changing the OS is just impossible to believe -- unless there was something very wrong to begin with.
Re:That has *nothing* to do with Linux (Score:4, Insightful)
I agree totally. I can't see there is any way changing a DB or an OS will change execution time by an order of magnitude such as that. My guess is that they rewrote the code since the system they were moving to is so different, and had smarter programmers that also learned from the mistakes of the previous creators. I would say they recreated the schema, eliminated useless joins, replaced loops with queries in with one single query using an IN, and the rest of the usual optimisations. I think rather than showing how fast the new system is, it showed how poor the old one had become.
Phillip.
Re:That has *nothing* to do with Linux (Score:4, Insightful)
Vindication is sweet (Score:4, Insightful)
I'd say successfullly pulling off a massive consolidation project on a financially critical system probably puts an upper limit on their incompetence.
Just because what they say doesn't really make all that much technical sense doesn't mean they are incapable of making technical sense when it serves their purpose. Getting things done in any organization involves using your successes to push you agenda. It also means sensitivity to the other messages your audience may be receiving and what message you need to counteract it.
For example, the PHBs may have been hearing that Linux was an unsophisticated system cobbled together by a bunch of amateurs from 1980s technology. If you don't think that message is out there, or that it can't possibly be effective , you are extremely naive. If you think you can counter this argument with technical arguments about file systems, virtual memory schemes and schedulers you are even more naive. So, here's a countermessage: "Look, this Linux based system works great. It's a thousand times faster in some important tasks than the systems we spent millions on before. How 'unsophisticated' can that be?"
You might not think this mode of reasoning is entirely valid, and you'd be right. But it's not without its virtues. Successful decision makers put a higher premium on things being demonstrably "good enough" than on their being "best". And this argument meets the admittedly relaxed corporate standards of truth: it is not literally false and its advanced with the best interest of the company in mind.
1000x performance increases... (Score:5, Insightful)
I'm no windows sympathizer, but in the world of enterprise software, only optimizations at the database layer (or reworking badly written networking layer) can yield those kind of results.
Sounds like they data warehoused and redesigned the schema/indexes to better match usage.
Re:1000x performance increases... (Score:5, Funny)
I beg to differ. I have many Slashdot posts from exceptionally informed sources stating that simply by using teh gentoo with -O6 and optimizing for j00r CPU such performance increases are easily obtainable.
Re: (Score:1)
I like the quote (Score:4, Funny)
"We went for Linux, not just because we hated Microsoft, but because the cost was compelling," Phillips said.
(Insert funny remark here because I'm unfunny)
Re:I like the quote (Score:3, Funny)
Anonymous Coward post
+ Admission that the poster had nothing to say
+ Quote verbatim from article
+ Admission that the poster is not funny
= Comment moderated up as Funny
Yep, that's Slashdot.
Re:I like the quote (Score:2, Interesting)
making moderation non-anonymous would be a nice feature
Best quote of the story (Score:1, Redundant)
Re:Best quote of the story (Score:2)
Quite an improvement, but from what? (Score:3, Insightful)
Re:Quite an improvement, but from what? (Score:2)
SELECT MAX(price) FROM trades WHERE ticker = ?;
Very easy to believe culling that data from several different unindexed databases could take 36 seconds (especially on older hardware), and the latter only 30 millis.
as I said (Score:3, Insightful)
One key query - searching the data on historical trades to identify maximum trade values - has been cut from 36 seconds to 0.03 seconds.
Well yeah. They consolidated 21 databases. It sounds like they had an 'overgrown' design, with lots of hacks. That's why it was slow, the consolidated the whole thing into one. Probably with help from Oracle themselves on optimization. Anyone would get a huge speedup out of that.
Also (Score:4, Insightful)
Now, I'm sure people are about to jump on me, given that Linux is free... But WAIT! We are talking Linux for running Oracle here. Well, if one checks Oracle requirements you find that in additon to Windows, HP-UX, xOS and such, it does run on Linux, but it's pickey. They require and only support enterprise Linuxes such as RHEL and SUSE Enterprise.
Ok, fair enough, but these AREN'T free. RHEL is to the effect of $800. Hmmmmm... Given that ORacle will also run on XP Pro, doesn't seem like such a deal any more.
We've dealt with Oracle in this regard and found out that:
1) It won't work on stock SUSE or RedHat systems. Dunno why, but there must be something different in the enterprise versions because it won't install properly on the normal ones.
2) More importantly Oracle REFUSES to support you if you aren't on a supported OS. They just say "run a supported OS" and that's it.
Well, given that, for the kind of apps one would want an Oracle database, support is important,I'm not seeing them running on a normal Linux distro hacked to make Oracle happy. So given that they are probably on an enterprise Linux, I'm not seeing the cost savings.
The whole thing sets off my zealotry bells. It sounds like that had a horrible hacked-ass, old database system. They needed to modernize it. So they elected to use Oracle, Makes sense, when it comes to unlimited scalibility and rock solid reliability, Oracle just has it. However then someone sold them on doing it on Linux. No problem, except it sounds like cost was the selling point, which isn't really valid for Oracle.
So now we have the justification scramble. Make sure everyone, espically the bosses, buy the cost argument. Pointing out the speed increase is also a good idea, never mind what caused it, obviously it was your brilliant decisions.
I've seen this happen plenty, and it's not limited to people advocating Linux, any platform that they like will work. You get a zealot for platform X, that uses BS arguments to sell it. They then produce lots of hype, to make sure people think it was the right choice.
Re:Also (Score:2)
Re:Also (Score:2, Informative)
the "strace" program helps here , it will tell you what function calls the program was making when it stopped.
I was at an oracle + RH installfest during oracle openworld in melbourne a few days ago , and was assisting people with laptops getting redhat and oracle 10i installed. There was a cheat sheet of sorts going around
Re:Also (Score:2)
I run 9i and and 10g on my optimized Gentoo boxes. But don't open a TAR with us on FC2 or Gentoo as the only answer you will get is: Move to a supported platform and reproduce the problem. TAR closed.
Another issue is that when you run SuSE Enterprise Server or RH ES, Oracle will also take all your Linux support in addition to your Oracle support. One stop support, no blame game, nothing.
Re:Also (Score:2)
In our case, the University has a site wide license for Oracle so we used it for that reason. However some anti-Windows zealotry
You answered your own question. (Score:2)
If you are unhappy with the solution provided you can alook around for a better deal because the solution, at least at the OS level is open, so any scripts that you use to juice the DB can be transported with no changes at all.
If you get really fed up with Red HAt you can go to SuSe, and if you are a big company, you can armtwist Oracle to support other Linux distros.
If you get fedup with M
The real costs (Score:3, Insightful)
- $40,000 / CPU for base product
- $10,000 / CPU for partitioning
- $10,000 / CPU for RAC
So, even a trivial Oracle cluster is just not going to come in under a quarter million dollars. Saving a few thousand dollars by going from windows to linux isn't going to make any difference at all.
Unless you have a large unix support staff you can leverage, want to diminish security-related patching & vulnerabilities, etc, etc. B
Re:The real costs (Score:3, Interesting)
Plus, if you are going to go for an expensive OS, why not go for Solaris? Oracle likes Solaris and plays quite well on it. For that matter, Oracle on Solaris on Sparc hardware would probably be a good idea. Much more reliable than normal x86 hardware
Re:The real costs (Score:2)
> servers ought to be on a private, firewalled,
network and only the DB should be accessable and
> only from trusted systems. This is just better for so many reasons.
I understand the concern about patching database servers - it's a *huge* challenge, and can easily knock your databases out. On the other hand, patching gives you 'defense in depth' - and a much more secure environment than just relying
Re:The real costs (Score:2)
I can see plenty of situations where you'd want the DB on comoddity, or at least low end enterprise, hardware and the OS that implies. In that situat
Ja, ja (Score:5, Informative)
Financial organizations are very conservative but even Deutsche Bank [silent-penguin.com] are migrating to Linux some of their less important processes.
In all the cases the future of the financial industry is in cheap linux clusters.
Impressive, but.. (Score:4, Insightful)
I mean, every part of the architecture has its role.
Some other contributing factors not mentioned, I suspect, would includes - focused performance requirements, specific purpose optimised query framework.
Can someone point to some public material on the architecture? It would be a interesting read.
the NZX runs it's trading system under Linux (Score:3, Interesting)
The NZX (ex NZSE) runs the Computershare ASTS trading system for their equities and bond trading. They have done so for 4 years.
This system runs under Linux (Redhat) on Compaq machines.
That they aggregated some of their databases and achieved better performance is non news but the increase in performance stated is worth a conspiracy post!
Linux at the back-end of the financial world." (Score:5, Funny)
Re:Linux at the back-end of the financial world." (Score:2)
C'mon, it ain't nice to call NZ that!
Well, its better than a Sheep at the back-end of the NZ world....
Re:Linux at the back-end of the financial world." (Score:2)
Re:Linux at the back-end of the financial world." (Score:2)
Now if God wanted to give the world an enema, Rotorua is where he'd stick the tube...
Re:Linux at the back-end of the financial world." (Score:2)
Not really (Score:4, Insightful)
Unless specifics about the query and the physical database model are comparable in both systems this isn't really impressive.
Comparable - not equal - since each database engines optimizer has it's individual quirks and strength.
Assuming that you have large joins on huge tables a couple of good indexes, which make the optimizer happy can reduce execution time from hours to seconds.
Table scans are expensive in database speak.
Unbelievable (Score:4, Insightful)
Re:Unbelievable (Score:2)
Re:Unbelievable (Score:2)
10^3 performance increase.. No big deal.... (Score:5, Interesting)
Most of the time the problem is stupid code or operational ignorance. Rarely is hardware, O/S or data base software changes the sole or main solution in performance problems. Hardware is only a factor when the system is underspecified to save money.
Given that they consolidated 21 databases into a single database the problem could simply have been network latency between separate physical servers.
The simplest way to get performance problems is to test on developers personal machines with tiny test databases and implement without full scale testing.
For those of you who wish to ensure that Microsoft SQL server is slow, invoke a user defined function as part of the where clause that the optimizer cannot recognize as a determinate function when joining two tables. This will ensure a nested loop join that will take an eternity.
Controlled environment (Score:4, Interesting)
For some software applications it makes sense to refuse to ship the software on its own and insist on giving away free hardware with the deal, with the operating system of your choice (it isn't really going to matter which operating system) fully configured and installed. That way you know what the client is running your software on, you've tested it, and you've got an identical setup back in the lab to research problems on, and you know it isn't going to crash because the client's box is running some crap driver you've never heard of.
Bullshit it's downtime that counts (Score:3, Insightful)
Linux uptimes are legendary. (Score:2)
An improvement of over 1000 times is spectacular ? (Score:3, Interesting)
"CREATE INDEX"
It's amazing what you can do to optimise a query or two...
Advice to the sarcasm-impared: do not take this posting literally.
Favourite Quote... (Score:3, Funny)
Have To Smile :)
Linux and assumed Performance increase (Score:4, Insightful)
I don't see the article make that claim... they just said that they changed a bunch of stuff, and they now have a different system in which one sample query is 1000x faster.
This could be (and probably is) due to a number of reasons:
-- consolidated many separate databases into 1
-- probable new data model
-- probable new application design
-- upgraded system resources (more RAM, better CPUs, faster SAN, etc.)
-- different OS
-- Oracle tuning / kernel tweaking
It doesn't make sense that they'd just re-implement the exact same system and application design... they probably spent a lot of time redoing the apps to make them smarter and faster.
To assume that Linux is singly responsible for the performance increase is kind of silly.
Re:Linux and assumed Performance increase (Score:2, Insightful)
"Everyone seems to be thinking that the story is all about a thousand times performance increase because they switched to Linux.
I don't see the article make that claim..."
Ah the subtle things that we miss... The whole reason this article was posted here on Slashdot is because Linux is in the mix. Get it? If this was an article with the word "Windows" or "Solaris" substituted for "Linux" then it would never be on Slashdot. If this way about a 1000% increase in performance with our new Solaris, Oracle RAC
36 seconds to 0.03 seconds (Score:4, Funny)
DELETE FROM clients
It took 36 seconds to return. The second time it pretty much came straight back.
Indexing (Score:2)
Orcle on Unix vs Windows (Score:3, Interesting)
After serveral months, it became obvious there wes no comparison in performance. The Solaris-based server out-performed the NT-based box easily by a factor of 4-to-1.
More importantly, the NT system has to be routinely rebooted in order to remain stable. I actually had to schedule reboots just to keep the system from running out of resources!
That was more than four years ago. Since then, the NT server was repurposed into a workstation; the Solaris Oracle server is still running with an uptime of more than two years.
Re:A pretty telling statement in there... (Score:5, Insightful)
They are a serious enterprise, and there must be a reason something as provocative as " not just because we hated Microsoft" would come out in an interview.
IOW - It's likley that Microsoft's products and/or policies have left a very, very bad impression with these people, and they're glad that they have a compeditor with which to smack Microsoft in the head with.
Soko
And what is your problem with that? (Score:2)
Hate is not gratuitous, it should have a reason, not that a could think of any myself of course, since MS is the greatest, more ethical company, I can think of....
MOD PARENT DOWN (Score:3, Funny)
On a more serious note, the statement made above just about applies to any operating system i.e.
If the world goes*insert OS name here*, *insert company name here* won't need to spend so much making sure its technology can run securely and reliably on every weird combination of hardware and operating system its customers adopt.
Re:Oracle (Score:2)
Re:But does it Run Linux? (Score:2)
<AHREF="Remember to always put a space after the A">
Re:But does it Run Linux? (Score:2)