London Stock Exchange Price Errors 'Emerged At Linux Launch' 168
DMandPenfold writes "Within the first 20 seconds of the London Stock Exchange's new matching engine going live on Monday, price data vendors began displaying incorrect prices, blank prices and wrong trading volumes, according to Computerworld UK sources. Thomson Reuters, Interactive Data and Netbuilder are among the largest data vendors, providing share prices to traders, that have been displaying pricing problems on some stocks throughout the week. Even the LSE's own data vendor, ProQuote, experienced problems. Concerns are being raised that there could be mistakenly setup connections or incorrect software interfaces at some of the large data vendors. Alternatively, there may be a data caching issue at the LSE that means data going out is not properly synchronised between different systems."
Re:Linux fails... AGAIN (Score:1, Interesting)
Btw, they are _upgrading_ to Linux, because the previous system (Windows and .net) failed several times, required MW of energy and was slow.
This new _upgraded_ system is many orders of magnitude faster.
Unfortunately, it appears, it wasn't properly tested... WTH? I'd expect for the system to run only 100x faster, instead of 1000x faster, during the first few weeks. Or maybe that some API from 1990 stopped working. Wrong data is just too much. Maybe the story isn't telling all the facts? ...Hmmm... Maybe the affected clients are running Windows?!!!!!!
Re:either sympathy or accusation (Score:5, Interesting)
Or the big vendors were doing excessive cost cutting to maintain profits during the recession.
As long as this is just a one weekend nightmare, the execs that ordered the QA not done are looking like great business monkeys. they took a risk that cut costs and managed to defect the blame to the exchange when it blew up, and it looks like nothing bad will happen to the big vendors, unless one of the small vendors gets a really unethical sales monkey that steals a large chunk of business from a couple of the big vendors.
Re:Linux fails... AGAIN (Score:5, Interesting)
I work on large scale air traffic control systems which run Linux and I don't envy the LSE in their task. Most of our interfaces are relatively simple and go out to organisations with a good history of validating interfaces. This trading system seems to have to interface to a lot of little offices around the place running various implementations. Its no surprise some of the interfaces weren't tested to the point where they are known to work 100%, though they may be 100% correct.
Re:either sympathy or accusation (Score:4, Interesting)
These are teething issues. And it apparently worked for the smaller traders/vendors so it looks more like a problem at the trader's end in that they didn't get their crap straight. But when changing out a large system like that it isn't that incredible there were problems not caught in testing.
> I don't blame Linux at all, but nor was I partisan fool enough to blame Windows or .NET for a poor implementation of a trading tool.
Of course Linux isn't to blame, unless they turn up kernel or other low level library problems, that system is a mass of spanking new code running atop linux. But we already know Linux can handle the transactional load reliability requirements and real time needs of a stock exchange since several are already doing it. The question is whether the guys working for the London Exchange have good enough code-fu.
But apparently they blamed the performance of .NET/Windows or the cost benefit of .NET/Windows otherwise they wouldn't have embarked on an expensive rip and replace operation in the first place. Personally I'm shocked .NET ever managed such a task in the first place, even poorly.
Re:Quote Services (Score:5, Interesting)
Hi,
The NYSE Arca FIX docs are not quite relevant to this - as what we're talking about here is the market data API.
The FIX protocol (although it does support market data transactions), is typically not the protocol of choice for market data. Typically it'll be non standard provider specific protocols which are optimised for delivering partial updates of information in a very bandwidth efficient manner.
In the parent post someone mentions that Reuters, Netbuilder etc fetch quotes - which is also not entirely correct either. They accept push feeds from the exchange, which contain tick updates to the information about the instruments traded on that exchange. The generation of these ticks is triggered when a match occurs on the order book - which may (or may not) involve a change in price data. They then typically augment this information with other data - such as other identifiers, or derived information such as previous days close value / percentage etc.
It is with these derived fields that the ongoing issues seem to lie, although for a certain period of time, the exchange was actually pushing out zero values for the bid/offer fields. As you can imagine, these plays merry hell with systems that attempt to drive automatic execution based on current price.
I am informed by Netbuilder that they will have pushed a final fix to live within a subset of their servers - but it will probably be Tuesday before this is available across their whole system. This is over a week since the original identification of the issue, not good!
LOL (Score:0, Interesting)
I am curious which part of the Linux (kernel) they'll be going to blame and how Slashdot will turn it into FUD with adverts around it... xD
Re:Effect on stockbrokers (postmortem) (Score:3, Interesting)
Re:either sympathy or accusation (Score:5, Interesting)
As I work for one of the vendors cited in the article, I can assure you that the reason is freaking simple: the systems that consume the feeds to move them into the trader's desktop are absolute mess of epic proportions riddled with mind boggling wtfs. Missing data, blank values, wrong numbers are common on those systems (root cause are mainly race conditions everywhere, and assumptions about missing data), assembled from random left and right acquisitions. There is no way to really know how they will behave under stress, and corporate culture will not allow managers to actually fix the issues (extremely costly, and corporate bonuses are not aligned to quality).
In fact, it is much cheaper for everyone to just pretend this is a one time occurrence, to fire a couple of middle managers and continue frantically fixing bugs in the mess until it sorta works.
Re:either sympathy or accusation (Score:5, Interesting)
We don't like High Frequency Trading because it fucks up the economy for everyone else.