London Stock Exchange Tackles System Problem 237
DMandPenfold writes "The London Stock Exchange has taken steps to resolve a system problem that occurred at 4.30pm Tuesday, which saw a delay to the start of the closing auction and knocked out automatic trades during a 42 second period. The problem occurred a day after the high profile launch of its new matching engine on the main equities market, based on the SUSE Linux system from Novell."
Well, well, well... (Score:2, Interesting)
This just shows that it's hard to build these highly available, low latency, massive usergroup systems. Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)
The problem is more likely to be internal organisation than specific platforms.
Re:Well, well, well... (Score:5, Insightful)
This just shows that it's hard to build these highly available, low latency, massive usergroup systems. Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)
Yes. And let us not forget that a lot of that chatter came from Microsoft's PR department.
Re:Well, well, well... (Score:4, Interesting)
Given that this was a showcase client for MS, it does not make them looks good.
Given that the MS was involved in developing the TradeElect system (the Windows based one), so even is the fault lies in TradeElect rather than the MS platform, then its still at least partly MS's fault.
Microsoft and Accenture developed a system that turned out not to be as good as the one developed by a small Sri Lankan company no-one had ever heard of.
Teething problems (Score:5, Insightful)
Previously there was a lot of chatter about the platforms (.NET, MSSQL 2003, etc...)
It's one thing to have a 42 seconds glitch in the first day a totally new system is powered up. That's perfectly normal, and had been predicted [computerworlduk.com]:
"Observers watching today's Linux-based launch will likely note that such a large change could bring about some teething problems, as with any technology overhaul."
It's a totally different thing to have it stop for a whole day after having been in operation for three months.
So, in conclusion, yes, it's about the platform. .NET, MSSQL 2003, etc aren't robust enough for this kind of job.
Re: (Score:2)
To be fair we don't know the exact reasons TradElec went down. It was mentioned that an upgrade went wrong. It could have been the platform.
There was talk that the platform was slow and wasn't performing as expected. The outage was just the last straw. Like a poor performing employee, you'd like to replace him/her but until they screw up royally you don't have enough justification for HR.
Or it could have been working fine but a public embarasment like that needed a scape goat. Someone had to be fired;
Re: (Score:2)
You do realize that it's not .NET, MSSQL and Windows 2003 alone that made up the trading system, right? It was also a boat load of custom developed software.
There is no evidence to suggest the problems that TradeElect had were directly the problem with the technology it was based on, but more likely due to problems in the custom software developed for it. I'd venture a guess that Microsoft's own web sites, like Hotmail, MSN, etc.. generate more load and have more complexity than TradeElect, and they handl
Re: (Score:2)
Re: (Score:2)
That may be, but there is no evidence to suggest it was the basic stack that was causing the problem.
It may simply be that the TradeElect software itself was not written to withstand the loads that were placed upon it.
Without a thorough analysis, there is no way for us to know. Therefore, claiming it's the stack is just as valid or invalid as claiming it's the custom software.
In any event, this is not an either/or situation. Microsoft did not write the TradeElect software, a third party did... Accenture.
Re: (Score:2)
Re: (Score:2)
It's interesting that you use power tools as an analogy. I once made a presentation to managers in my company on "Why Linux?" where I had the following analogy: Windows XP on personal computers is like Black & Decker drills, cheap and easy to use, found everywhere. Windows server is like Makita drills, sturdier, more professional, and more expensive, but very much like an amateur tool on steroids.
Linux is like an air drill [google.com]. Most people aren't even aware that such thing exists. But it's the only solution
Re: (Score:2)
Re: (Score:2)
Remember folks, you heard it here first:
APK says Windows is better than Linux.
Re: (Score:2)
Cool story, Bro.
Re: (Score:2)
Y U MAD BRAH?
Re: (Score:2)
Oh APK, you are amazing.
Re: (Score:2)
Damn. You have uncovered my secret: I'm part of Linus Torvalds' secret Linux Astroturf Brigade.
You win twelve internets for your insight.
Re: (Score:2)
Don't worry, I'm on the case. Keep your hosts files updated, and your Windows recovery disk warm.
Re: (Score:2)
One trick pony only has one trick. :(
Re: (Score:2)
tl;dr. Please try formulating a coherent point before posting.
Re: (Score:2)
It's not an ad hominem attack, you fool. An ad hominem attack would take this form:
"APK is an unstable psychotic. We may therefore conclude that all of his points are invalid."
I am asking you to formulate a coherent point, rather than your rambling mess of capitals, emphatic text, and "PS" addenda, which is both physically painful to read, and logically incoherent.
Re: (Score:2)
An ad hominem attack is the logical fallacy of using negative statements about your opponent to discredit their arguments.
As I said, "APK is a psychotic. Therefore his points are wrong." Is an ad hominem attack. And it is a fallacy, because whether or not you are a psychotic isn't relevant to whether or not your argument holds water.
Calling you a 1-trick pony is an observation of your behavior: You flog the same snippets of text repeatedly with a rambling mish-mash of capital letters, bold typefaces, an
Re: (Score:2)
From your own link:
"An ad hominem (Latin: "to the man"), also known as argumentum ad hominem, is an attempt to link the validity of a premise to a characteristic or belief of the person advocating the premise."
In other words: If I were trying to dismiss your concerns by calling you a one trick pony, that would be an ad hominem argument.
Since I have simply observed that all you do is flog the same points with poor grammar and rambling and incoherent walls of text, that is not an ad hominem argument. It is
Re: (Score:2)
APK, why do you speak of yourself in the third person in these thinly veiled attempts at sock puppetry?
Noting that you're a self-aggrandizing bore is simply an observation. It is not an ad hominem attack, because frankly, I'm not arguing with any of your points, I'm simply pointing out that you love to repeat yourself!
Y U MAD, BRAH?
Re: (Score:2)
APK, I'll explain this to you one last time in small words with few syllables: for me to be conducting an "ad hominem" attack, I'd have to be saying something negative about you, *and then* suggesting that that negative trait is why your points are wrong and invalid. Name-calling and getting you all lathered with frustration like this is what we call "just a bit of fun," and has no bearing on whether or not your points are true, false, or completely disconnected from all reality.
You're the one who went of
Re: (Score:2)
Yawn.
You're boring me, APK.
1) Learn what constitutes an ad hominem argument. The definition is very clear, and you haven't been subjected to one.
2) Attacking you? Hardly. I've simply called you some names and poked fun. If I were attacking you, you'd know it.
Re: (Score:2)
I see you're able to quote the definition I gave you - good for you! Next, perhaps you can work on understanding what all those big words mean.
If I were attacking you, you'd know it. And not even your precious hosts file would help you!
Re: (Score:2)
You provided a link to the definition of ad hominem, which surprisingly, you appear incapable of comprehending.
I quoted the relevant definition of exactly what it was (not just a link). I win, you lose.
And why do you insist on asking if I'm making threats? I told you I wasn't, because if I was, you'd know it. Since you claim to not know whether or not I am, the answer, by process of deduction, is that I have not made any threats.
Really, APK. Elementary logic. You should read a book sometime.
Re: (Score:2)
APK, stop with the sock puppets. It does nothing to bolster your arguments, just makes you look like a foolish child, desperate for attention.
(And as far as hiding... who, exactly, is hiding behind an "Anonymous Coward" label here? Yeah... I thought so.)
Re: (Score:2)
APK, you seem intent on viewing every response of mine as a threat. Is it because you're a paranoid schizophrenic? Seek help.
We all know that you've got a hosts file and you're not afraid to use it!
But I'll do you a favor - my ip is 127.0.0.1 - please block me with your hosts file. Also, you probably want to block out a system named "localhost," that's sort of my thing, I use that identifier.
Re: (Score:2)
Again with the paranoid delusions.
You must live in such a scary world.
Re: (Score:2)
Don't need a PhD in psychiatry to diagnose you. Just good old-fashioned common sense, girlfriend.
APK == OFF TOPIC TROLL!
Re: (Score:2)
Prognosis is a term to describe a prediction of the "likely outcome of an illness." Diagnosis is a determination of what your illness is.
What I offered was a speculative diagnosis, based on my observations of your behavior here. My prognosis is that you will die alone, unloved, probably of liver failure because you'll turn to alcohol to fill that deep void in your life that Slashdot and your Hosts file can't fill.
See the difference?
APK == OFF TOPIC TROLL!
Re: (Score:2)
If a college degree is what gives you your mangled interpretations of "ad hominem," "prognosis," and other standard terms, I'm glad I'm not a graduate of whatever backwater troll factory you call your alma mater.
I do, in fact, have a college degree. You're wrong on all counts, but you are providing a remarkable demonstration of ad hominem attacks on me, and also, an effective case study in pissing into the wind. I'll let you know when I write you up in JAMA, the specifics of your case are fascinating.
APK
Re: (Score:2)
APK == OFF TOPIC TROLL.
Re: (Score:2)
I have no need to prove myself to you, you troll.
Re: (Score:2)
"We"? What, you got a mouse in your pocket now?
You don't know what an ad hominem argument is, you don't know what the difference between "diagnosis" and "prognosis" are, and you clearly have issues.
Cry more troll. Cry more.
Re: (Score:2)
Now you're just flapping your gums to hear yourself talk. Hope you enjoy it, you off-topic troll!
Re: (Score:2)
Wow, 2 posts that say the same thing, but which obviously aren't copies of one another. Are you having a psychotic break?
Re: (Score:2)
BOOOOOOORING.
Re: (Score:2)
Or it could be that I have nothing to prove to a raving lunatic whose sole source of excitement appears to be talking about how he edited a hosts file once.
Fail more, APK.
Re: (Score:2)
YAWN.
You finished?
Re: (Score:2)
I have nothing to prove to you. And why are you so insistent on me being a "fag" - hoping to score? Sorry to disappoint you, APK, but, psychotic and male isn't exactly my type.
Re: (Score:2)
No, I will not date you.
Cherry picking much, Mr. Ballmer? (Score:2)
That's NOT what London Stock Exchange reps said:
"LSE denied that the collapse was TradElect's fault" - http://blogs.computerworld.com/london_stock_exchange_to_abandon_failed_windows_platform [computerworld.com] by Stephen J. Vaughn-Nichols
Let's quote from your own link. First of all, the line you put between quotes was entirely out of context, because you omitted the initial "while" and the following lines:
"they also refused to explain what the problem really was. Sources at the LSE tell me to this day that the problem was with TradElect.
Since then, the CEO that brought TradElect to the LSE, Clara Furse, has left without saying why she was leaving. Sources in the City-London's equivalent of New York City's Wall Street--tell me that TradElect
Re: (Score:2)
Interesting that you are actually trying to argue with apk. Having some experience here, let me tell you how this will go. You might say something completely innocuous like:
"I like vanilla."
Pretty soon you will get a 3000 word ramble from an AC about how chocolate is good for you and how Thalian chocolate is the best since they age the beans 400 years. Then you'll get a 4000 word diatribe from an AC about how chocolate is bad for you because he's an expert in Thalian chocolate. If you're lucky, he'll accidentally sign h
42 seconds, that's eternity! (Score:5, Insightful)
Re: (Score:2)
Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.
At least it would be The Answer to.. !
What I was wondering (Score:2)
Was the consequences of the loss of the automatic trade?
Everybody won? :D
Re: (Score:2)
Imagine telling a trader in the 1970s that we had a 42 second outage in the stock market, it was all over the news, and a few companies probably lost hundreds of millions in revenue.
"Hi, 1970s trader! I am just going to skip over the momentous fact that I have accomplished time travel - the ultimate cool technology - to let you know that one day people will be even more obsessive about the stock market than they are in the 1970s.
Thought you'd like to know. Oh, and in case you are interested, I can travel through time. Now going back to talk to HG Wells about a great book idea I have."
Re: (Score:2)
Man, I want you to come talk to my boss and tell him that 42 seconds isn't an "outage of any length."
When you work with market data and trading systems, a "significant outage" is anything with time > 0 that your functionality is unavailable. Business critical applications are, susprisingly, considered critical to the operation of the business.
This is why you have hot-spare High Availability / Disaster Recovery configurations, robust monitoring and load balancing, and a team of operations staff standing
42 seconds (Score:2)
Liffe[sic] the universe and everything.
Re: (Score:3)
LIFFE != LSE
Re: (Score:2)
Restate my assumptions:
One, Mathematics is the language of nature.
Two, Everything around us can be represented and understood through numbers.
Three: If you graph the numbers of any system, patterns emerge. Therefore, there are patterns everywhere in nature.
Evidence: The cycling of disease epidemics;the wax and wane of caribou populations; sun spot cycles; the rise and fall of the Nile. So, what about the stock market? The universe of numbers that represents the global economy. Millions of hands at work, billions of minds. A vast network, screaming with life. An organism. A natural organism. My hypothesis: Within the stock market, there is a pattern as well... Right in front of me... hiding behind the numbers. Always has been.
— Maximillian Cohen, Pi [imdb.com]
Someone skimped on testing (Score:5, Insightful)
I've seen this too often in my 25+ professional years in IT. The system test manager produces an excellent plan, that fully simulates the anticipated workload. But it requires X testers, Y test case developers and Z machines. The program manager rejects the plan, "because he is under pressure to reduce costs." The program manager says, "The testing that the developers do should be enough." He then moves on, before the system goes into production.
The result? It always ends in tears.
Re:Someone skimped on testing (Score:5, Interesting)
Testing is just hard. Even with a perfect plan, and a perfect execution you're still going to miss things.
I just spend 2 weeks finding a 6 year old bug, which only showed itself when the temperature of our product is somewhere at -5C, and then only on 75% of the total devices. Devices where tested at low as -40C, but those where lucky and kept working correctly, so nobody noticed before. Even after I found it I spend a week looking at the code to find what's really wrong.
Re: (Score:2)
Proposed patch (Score:2)
What hour ? (Score:2)
4:30. What were the sys admins doing 10 minutes before this ? ;-)
42? (Score:2)
well there we have an answer to a popular question
UPDATE: Anger mounts over LSE price data errors (Score:2)
Re: (Score:2)
Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.
I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?
Re: (Score:3)
Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.
One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.
Ah, I see... well, maybe it would be news-worthy, but... too pity there's too much noise coming from the open outcry trading in the pit... nobody hears about an outage and even during an outage the business goes as usual in that pit... everybody furiously shouting and gesticulating.
(just kidding)
Maybe it will help to compare the LSE trade-volume per day with the same for your workplace?
Re: (Score:2)
Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.
I'm afraid I can't remember what you said about the name of company you work for, can you please repeat it?
One of the major exchanges in Chicago, as well as one one of the bigger global banks. Not a small firm.
Besides, if Reuters or Currenex had this same issue for two hours, we STILL wouldn't hear about this.
I can't answer why it is "news", but I can answer why it is on Slashdot. Over the past couple of years, Slashdot has increasingly posted stories that have flamebait as their primary feature and where it isn't intrinsic to the story, it is often editorialised to add a little prompting to get us all arguing. This has an ostensible Linux vs. MS angle hence qualifies to be featured prominently for the sake of the post-count boosting flamewars it could induce. I say "ostensibly" because in reality, any change ov
Re: (Score:2)
I can't answer why it is "news", but I can answer why it is on Slashdot. Over the past couple of years, Slashdot has increasingly posted stories that have flamebait as their primary feature and where it isn't intrinsic to the story, it is often editorialised to add a little prompting to get us all arguing.
It's all starting to make sense, now. The entity calling itself "CmdrTaco" is one of those energy beings from the Start Trek "Day of the Dove" episode, feeding on the strife it creates.
Re: (Score:2)
It's all starting to make sense, now. The entity calling itself "CmdrTaco" is one of those energy beings from the Start Trek "Day of the Dove" episode, feeding on the strife it creates.
Or he gets ad revenues based on page hits, but yours is good too. ;)
Re:Newsworthy? (Score:5, Funny)
Why is this news? None of the several hour long outage calls I've been involved with were ever on the news.
Simple, when a Linux system has a 42 seconds outage that's news. When a windows systems has several hours outage, there is nothing new about.
Odd that you didn't get that..
Re: (Score:2)
Cause it's about Linux, it's News for nerds, stuff that matters (to us, at least).
Re: (Score:2)
Actually, cue someone claiming that X millions of dollars or pounds were lost during that 42 seconds of downtime...
It's all about trading speed these days, with milliseconds being critical. Not that I agree, because usually this means someone is gaming the system through automated computer-based trading systems. I would like to see a mandatory delay added to all trades to slow things a bit and let the market become more manageable, by you know, humans. After all, it's we, as humans, that are ultimately doin
Re: (Score:2)
God, why won't this stupid meme die? "Inserting a mandatory delay" does NOTHING to solve the problem you're asserting it will. All it does is mean that it'll take $delay seconds after market open for high-speed / automated trading to begin hitting the exchange, and it will continue ALL DAY LONG until market close.
This assertion that a delay would somehow make the market "manageable" for individual humans is like saying that making everybody wait in their driveway for 2 minutes before they begin their comm
Re: (Score:2)
$delay won't solve squat as you point out, but rounding everything down to the nearest second might alleviate some issues. "All orders between 08:13:35.000 and 08:13:35.999 are considered simultaneous and will all be recorded as 08:13:35".
With 1 second "tick" even a human player might have a chance vs computers! Cue Watson jokes.
Re: (Score:2)
I guess I don't understand what problem you think this will solve, at all. "Treating them as simultaneous" doesn't change the fact that they're received in an order, and processed in a particular order. All you're going to do is invent a new mouse trap for the mice to engineer around - either they'll stuff the queue with more orders to try and increase the chances that some of their trades will be randomly selected to go first, or they'll figure out the delay and algorithm you're using to process orders,
Re: (Score:2)
At the same time, they're providing additional liquidity to the market, and acting as a damping mechanism on that price volatility.
that liquidity is fake and artificial. HFT algorithms trade the handful of shares back and forth to set trends, confuse competing algorithms and discover optimal prices triggering trades. Flash crash showed how great they are at providing liquidity and damping the price volatility.
Introducing ticks would mean that there is less profit in cutting into a trade that would happen anyway. There would be no time to probe buyers and sellers with bogus transactions to discover their prices and use that knowledge ag
Re: (Score:2)
Well, since it's fake AND artificial, I'm sure your point is completely valid.
If all HFT algos were doing was passing the same set of shares back and forth, nobody would be making any money at it. But I'm sure you already knew that. They're buying shares that investors want to sell, and selling shares that investors want to buy. They make money off it... and if they weren't there offering to buy & sell those shares, the price would probably be different, or at least certainly more volatile, when the
Re: (Score:2)
Cause it's about Linux
It's also about 42 [wikipedia.org]
Simple (Score:2)
Re: (Score:2)
Re:First posters are lame (Score:5, Insightful)
No test can ever be a 100% accurate representation or real use.
It's probably somewhere in the 0.1% mismatch where this problem occurred.
Re: (Score:2)
Platform changes don't actually address the underlying issues of incompetent programmers to begin with. Platform changes just wash the bad taste of a now fired programmer out of your mouth when you replace them with someone else who works in a different environment.
I would argue that in that particular case platform matters.
Most modern platforms (C#/.Net, Java) encourage sloppy programming. (*) That works fine on desktops and even in corporate data centers, where minor problems can be brushed off, where excessive resource consumption can be mitigated by throwing in more hardware. But that isn't acceptable for customers like LSE, where even adding piece of dumb hardware might require exhausting test: to make sure that software works with more hardware and the extra
Re: (Score:3, Insightful)
Re:First posters are lame (Score:5, Insightful)
This works for a component with one set of inputs and one set of outputs.
A trading system is essentially chaotic in the way it processes data because it gets so many inputs and their relative arrival times determine the system's behavior. You'd have to be replacing the old system with an identical new one, and then add heavy and slow synchronization to all the inputs going to both systems (so e.g. a trade A hitting the old system one microsecond before trade B also hits the new system the same way).
So yes, it comes down to running a whole lot of offline tests using real data and then bringing it online.
Re:First posters are lame (Score:5, Informative)
The system has been online since the middle of last year. Nice for you to drop by and dish out out your golden nuggets of wisdom tough.
Re: (Score:2)
Re: (Score:2)
Re:First posters are lame (Score:4, Informative)
The summary is even lamer. It didn't knock out any trading. What happened is that the closing auction announcement was delayed by 42 seconds. Normally any trades submitted prior to the announcement get ignored. However, in this case I'll quote from the article:
LSE traders are required to wait for the message before trading. Normally auction trade instructions sent before the message would not complete, however because of the surprise delay, on this occasion order book trades were allowed to complete later.
So, no outage, no downtime, no lost trades.
Re: (Score:2)
So, no outage, no downtime, no lost trades.
It amazes me how quickly the press tries to make a big deal about a Linux outage that isn't even an outage. The Windows monopoly must be nervous.
Re: (Score:2)
Perhaps you'd like to compare it to the old system which shutdown trading for a whole day if not longer? I don't remember the specifics.
A 42 second delay at the end of trading is nothing. I really don't understand why people hold this Linux platform to a much higher standard and overlook the poor performance of the previous system which was terrible.
Re: (Score:2)
Had this been a Windows based system and it stopped for only 42 seconds it would have been hailed as high reliability. But you've got the Windows fanbois who can't wait to find an issue, any issue, please, with a Linux system.
Remember the article a couple of weeks ago where there was an outage on the LSE while still running on the C# system? The article's title was something like LSE Outage during Linux test. Of course, the Linux test had nothing to do with it, but they had to find something.
Re:"Oh well I guess Linux sucks then (Score:5, Insightful)
To be fair, if this was Windows you *know* the Linux fans (myself included) would be berating them for choosing such a crappy platform. And it was Windows, and we did berate them for it...
Of course, the truth is somewhere in the middle - isn't it always? The most important part of a trading computer system set up is, well, the trading computer system software. The system's biggest problems aren't going to be due to Coolwebsearch or a bluescreen, they'll be with the trading software itself. The OS doesn't really matter. Buggy software is buggy software.
That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.
Re: (Score:2)
That's a sad but true commentary. If I heard of a high profile Linux->Windows migration that failed in some way, I'm sure I would've jumped on it. The truth is that any new systems roll-out is hard, and the pressures on anything involving large amounts of money is even worse. That's why it takes forever to develop and deploy these systems. I hope their issue was something perceivable and easily addressable.
Re: (Score:2)
The move to Windows happened three and a half years ago. That is hardly a new system: its mature and its still crap.
Re: (Score:2)
Sorry, I misunderstood the GP.
Re:"Oh well I guess Linux sucks then (Score:4, Interesting)
Licensing costs are trivial in the context of these sorts of systems. TradeElect cost £40m, the new system was £50m - but that was to buy the company, not just the software.
Re: (Score:2)
The Unix Philosophy (Score:2)
That being said, Linux is just a better platform to build something like this on. Sure, you can do it with Windows and make it work, but it's just more and unnecessarily difficult.
I keep hearing this, but never see any technical details. Why is this so?
I had extensive Windows programming experience before I switched to Linux about ten years ago and, basically, it boils down to the Unix philosophy [faqs.org].
Linux is a system made by programmers for programmers. Windows is a system made following directions of the marketing department.
Re: (Score:3)
Re: (Score:2)
I keep hearing this, but never see any technical details. Why is this so?
I wonder if that is because there are some hard real-time requirements for those systems (or at least parts of them)? Stock Linux isn't that, either, but there are hard RT forks which still count as "Linux". There's no such option with Windows.
Re:"Oh well I guess Linux sucks then (Score:5, Funny)
Re: (Score:3, Funny)
But wont Apple keep 30% of the money that flows through it?
Re: (Score:2)
I am not camp!
Also, I just think Linux is a better operating system, with reduced time spent managing the system, fixing errors and waiting for it to open applications.
why you don't need to worry about HST (Score:2)
Granted, it increases costs overall for everybody in the market, but those costs should be spread evenly. This shouldn't greatly adversely impact long term investors.
But it does still amount to an privileged class siphoning value out of the system. That's bad.