OSDL Releases TPC Benchmark Tests For Linux

trialsboy writes "A news.com article announces that ODSL has released a set of workloads and test frameworks which conform to the TPC benchmark specifications." (Slashdot parent company VA Software is one of the OSDL sponsors.)

ODSL?

[BLAG-blast]

Is that fast than ADSL?

Re:But Around Here...

[MisterFancypants]

That might have been funny except they are TPS reports. You're FIRED!

Re:But Around Here...

[NotAnotherReboot]

yeeaaahhhh...

Don't you love the VA Software disclaimers?

[Drakonian]

It really proves to me that Slashdot is all about professional, responsible journalism. ;)

Re:Don't you love the VA Software disclaimers?

[jdkincad]

Slashdot is all about professional, responsible journalism.

You're new here aren't you.

benchmark handbook

[suhit]

For those that didnt understand what TPC is and did not get enough information from http://www.tpc.org [tpc.org], check out further information that is very well explained at The Benchmark Handbook [benchmarkresources.com]. The direct link is at http://www.benchmarkresources.com/handbook/content s.asp [benchmarkresources.com].

Suhit

Question

[Anonymous Coward]

Why should we trust this any more than the Microsoft-sponsored tests?

Re:Question

[Old Uncle Bill]

Damn, I was wondering the same thing. Did Mindcraft put out this benchmark? Maybe they shoulda done a Pet Shop comparison while they were at it...

Re:Question

[sloanster]

Trust what?

I think you are confused, this is a test suite, period.

Re:Question

[GooberToo]

Because you can run and validate it your self. Don't trust your self? Don't run it. Know someone you do trust? Have them run it.

Now, there is no excuse to say you don't trust the results because the suite of tools are readily available for you to run your own tests. It's as simple as that.

Does this mean you should trust any result quoting OSDL TCP results? Of course not. Does that mean you might be able to validate them for a change? Absolutely!

Ignorant question:

[Geaty]

Why is "Transaction Processing Performance Council" abbreviated "TPC"?

Re:Ignorant question:

[mbadolato]

Strictly so we can make Office Space jokes

Re:Ignorant question:

[mbadolato]

...and before the rest of you jump on me... yes i realize office space is TPS not TPC. I'm an ass, i know...I fucked up, get over it.

Re:Ignorant question:

[EvilTwinSkippy]

But didn't you get the memo?

Re:Ignorant question:

[dark-br]

Memo? No! Im really about to do something about that damm printer!

Re:Ignorant question:

[DAldredge]

THEY MEMO! I just want my damn stapler...

Yeah, Linux is great...

[EvilTwinSkippy]

...but I always thought it excelled as the bright kid who frankly didn't care about grades. Now it's going for MilSpec certifications and benchmarks.

I don't know what I'm going to do now. When I tell poeple my server run Linux they don't look at me like some granola head. I'm actually starting to see Linux showing up on Intern's resume along side Microsoft Word and Excel as computer experience.

It getting to be cool, and I've never been cool before. I backed apple back in '94. I had a PC when everyone had a C64. Could it be, that for a shining moment in time, I could actually be doing something that the rest of the world considers cool?

Nah, back in your box Skippy.

Re:Yeah, Linux is great...

[tupps]

Bright kid now wants to go to college, so he will do a bit of work for his grades.

Re:Yeah, Linux is great...

[EvilTwinSkippy]

...Must fight urge to post gratiuitous Beowulf Cluster of those line...

Benchmark implementations in question use SAP DB

[kcbrown]

...which I think is quite interesting, considering that it's one of the most recent arrivals on the Linux scene (MySQL and PostgreSQL are much more widely used on the Linux platform).

They're looking for contributors to help port the tests over to other databases, and I for one will be very interested in seeing how the various database engines for Linux stack up against each other when tested using these tools.

Re:Benchmark implementations in question use SAP D

[Anonymous Coward]

Except that MySQL can not be used in TPC benchmarking because it lacks the most critical part, ANSI SQL92 conformant transactions.

Re:Benchmark implementations in question use SAP D

[GGardner]

Except that MySQL can not be used in TPC benchmarking because it lacks the most critical part, ANSI SQL92 conformant transactions.

Technically speaking, yes. But, if you read the section where they state the differences between this benchmark and the "real" TPC-C, they admit that they don't actually check that the database is ACID. So you could theoretically run MySQL against this benchmark. I haven't looked at the code, but if this benchmark has only been run on the SAPDB database before, it probably is more work porting to a new database than the authors care to admit.

Come to think of it, it would be interesting to see if you could run the benchmark with a bogus null database, and see what the speed-of-light performance is.

Re:Benchmark implementations in question use SAP D

[fimath]

Well, the code uses an ODBC interface to call stored procedures in SAP DB. So only the code the interfaces the database needs to be tweaked for other databases. Unfortunitely, that's the bulk of the code...

Actually, the DBT-1 kit has a mode where it fakes the database transaction part. The other two kits lack that at the moment.

Re:Benchmark implementations in question use SAP D

[fimath]

And DBT-3 is mostly script based, so that should be really easy to run against other databases. It will just be a matter of SQL compatability.

Re:Benchmark implementations in question use SAP D

[vandan]

I was under the impression that MySQL supported transactions with the innodb table type.
I use them anyway, and have done for 2 years now.

Re:

[account_deleted]

Comment removed based on user account deletion

Re:Benchmark implementations in question use SAP D

[vandan]

I see.
Maybe those funding the benchmarking should throw some money at the MySQL people to make their transactions pass the test. I think it would increase the performance over SAP DB incredibly.

Re:

[account_deleted]

Comment removed based on user account deletion

Re:Benchmark implementations in question use SAP D

[wookie_osdl]

The intent wasn't to be comparable to the TPC benchmarks. If you want to know what your system does against other systems on a TPC benchmark you are going to have to go and run one of those. We chose to use a database that had ACID properties as to us it didn't make sense to do this type of operations without being ACID compliant. Tim

Re:Benchmark implementations in question use SAP D

[fimath]

I think it would still be worthwhile to give MySQL a shot, to see how it does. Anyone can get help from the OSDL folks from a couple of mailing lists:

[e-mail]
osdl_perf [e-mail]

Re:Benchmark implementations in question use SAP D

[zak_greant]

We have supported ANSI SQL 92 conformant transaction for about 22 months now.

Granted, we did use BEGIN as the keyword to indicate the start of a transaction. This syntax is not ANSI SQL 92 conformant.

However, in version 4.0.10 we introduced the comformant START TRANSACTION keyword as a synonym for BEGIN.

Re:Benchmark implementations in question use SAP D

[sql*kitten]

...which I think is quite interesting, considering that it's one of the most recent arrivals on the Linux scene (MySQL and PostgreSQL are much more widely used on the Linux platform).

The reason is that MySQL simply couldn't run the TPC benchmarks - its SQL parser isn't sophisticated enough. SAP DB is feature-comparable to Oracle 7 - it's a real, commercial-grade database engine. Of course, I have to wonder why they don't simply use Oracle 9 on Linux to run the benchmark.

Re:Benchmark implementations in question use SAP D

[wookie_osdl]

Of course the reason that we don't use a "commercial" database is that most of their licenses have something that is commonly refered to as a "Dewitt clause". That states something like, "You can't publish performance information without our permission." This would of course invalidate the goal of being an engineering tool if we had to wait until somebodies legal and marketing department approved of the test run. :-)

Re:Benchmark implementations in question use SAP D

[afidel]

Did you just say Oracle and benchmark in the same paragraph? I hope you don't have an Oracle liscense or you might be due for an audit =) Basically Oracle doesn't allow ANY benchmarks to be released, their spin is that it's fast enough and it's really not about the speed but the scalability and reliability.

Re:Benchmark implementations in question use SAP D

[sql*kitten]

Basically Oracle doesn't allow ANY benchmarks to be released, their spin is that it's fast enough and it's really not about the speed but the scalability and reliability.

The real issue is that one wrong setting in init.ora can make orders of magnitude difference in performance. Set sort_area_size too small, and you'll be hitting the temp tablespace on disk for every sort, set your shared pool to large and latch contention will wipe out performance. Oracle are simply scared that a misconfigured database will benchmark poorly (and to be fair, it will). A database like MySQL is so simple that it's almost impossible to mistune it.

Oracle do release benchmarks when it suits them, tho', but at the moment, they're really obsessing on the performance of 9i Application Server versus WebLogic and WebSphere.

Re:Or

[tupps]

Not really, Apple creates a benchmark in Photoshop, which highlights Apple Hardware. Eg all of the tests are tests that fully utilise AltiVec.

Microsoft has TPC benchmarks of their own...

[ryanvm]

For what it's worth, I noticed this article on Microsoft's web site the other day:

Windows 2003 & SQL Server 2000 is the fastest 32-way OLTP server in the world [microsoft.com]

Unfortunately, I'm too stupid to spot any holes in the testing, but that's what you guys are for.

Re:Microsoft has TPC benchmarks of their own...

[xixax]

"Spot the dog ate M$ brand dog food after we poured sump oil on our competitor's dog food"

Who knows, may they don't bother with bothersome checksums that would only inhibit performance, or maybe it really is faster.

Xix.

TPC? TPS...

[ppc970]

I kept telling you guys to get your TPS reports in. About time.

Enterprise Linux

[Anonymous Coward]

TPC-C is one of the most respected benchmarks in the industry. Only one major server vendor has dropped out of "publishing" results...Sun. And this is only because their hardware sucks ass compared to the competition.

The more benchmarks showing Linux seriously competing against other commercial OSes the better.

TPC-C benchmarks typically cost millions of dollars to run because of the expensive auditing that is required.

This tool lets any business or end user quickly gauge the performance of their servers for free, who could ask for anything more.

Clarifications

[wookie_osdl]

Actually the tests do not confirm to the TPC benchmarks but are fair use subset implimentations.

The TPC tests are designed, implemented and verified to enable comparisons between different systems (System A to System B comparisons) the DBT tests are engineered for working in an environment where you are trying to find out if your changes to a single system have made it faster or slower.