20 Years Later, Real-Time Linux Makes It To the Kernel (zdnet.com) 93
ZDNet's Steven Vaughan-Nichols reports: After 20 years, Real-Time Linux (PREEMPT_RT) is finally -- finally -- in the mainline kernel. Linus Torvalds blessed the code while he was at Open Source Summit Europe. [...] The real-time Linux code is now baked into all Linux distros as of the forthcoming Linux 6.12 kernel. This means Linux will soon start appearing in more mission-critical devices and industrial hardware. But it took its sweet time getting here. An RTOS is a specialized operating system designed to handle time-critical tasks with precision and reliability. Unlike general-purpose operating systems like Windows or macOS, an RTOS is built to respond to events and process data within strict time constraints, often measured in milliseconds or microseconds. As Steven Rostedt, a prominent real-time Linux developer and Google engineer, put it, "Real-time is the fastest worst-case scenario." He means that the essential characteristic of an RTOS is its deterministic behavior. An RTOS guarantees that critical tasks will be completed within specified deadlines. [...]
So, why is Real-Time Linux only now completely blessed in the kernel? "We actually would not push something up unless we thought it was ready," Rostedt explained. "Almost everything was usually rewritten at least three times before it went into mainline because we had such a high bar for what would go in." In addition, the path to the mainline wasn't just about technical challenges. Politics and perception also played a role. "In the beginning, we couldn't even mention real-time," Rostedt recalled. "Everyone said, 'Oh, we don't care about real-time.'" Another problem was money. For many years funding for real-time Linux was erratic. In 2015, the Linux Foundation established the Real-Time Linux (RTL) collaborative project to coordinate efforts around mainlining PREEMPT_RT.
The final hurdle for full integration was reworking the kernel's print_k function, a critical debugging tool dating back to 1991. Torvalds was particularly protective of print_k --He wrote the original code and still uses it for debugging. However, print_k also puts a hard delay in a Linux program whenever it's called. That kind of slowdown is unacceptable in real-time systems. Rostedt explained: "Print_k has a thousand hacks to handle a thousand different situations. Whenever we modified print_k to do something, it would break one of these cases. The thing about print_k that's great about debugging is you can know exactly where you were when a process crashed. When I would be hammering the system really, really hard, and the latency was mostly around maybe 30 microseconds, and then suddenly it would jump to five milliseconds." That delay was the print_k message. After much work, many heated discussions, and several rejected proposals, a compromise was reached earlier this year. Torvalds is happy, the real-time Linux developers are happy, print_K users are happy, and, at long last, real-time Linux is real.
So, why is Real-Time Linux only now completely blessed in the kernel? "We actually would not push something up unless we thought it was ready," Rostedt explained. "Almost everything was usually rewritten at least three times before it went into mainline because we had such a high bar for what would go in." In addition, the path to the mainline wasn't just about technical challenges. Politics and perception also played a role. "In the beginning, we couldn't even mention real-time," Rostedt recalled. "Everyone said, 'Oh, we don't care about real-time.'" Another problem was money. For many years funding for real-time Linux was erratic. In 2015, the Linux Foundation established the Real-Time Linux (RTL) collaborative project to coordinate efforts around mainlining PREEMPT_RT.
The final hurdle for full integration was reworking the kernel's print_k function, a critical debugging tool dating back to 1991. Torvalds was particularly protective of print_k --He wrote the original code and still uses it for debugging. However, print_k also puts a hard delay in a Linux program whenever it's called. That kind of slowdown is unacceptable in real-time systems. Rostedt explained: "Print_k has a thousand hacks to handle a thousand different situations. Whenever we modified print_k to do something, it would break one of these cases. The thing about print_k that's great about debugging is you can know exactly where you were when a process crashed. When I would be hammering the system really, really hard, and the latency was mostly around maybe 30 microseconds, and then suddenly it would jump to five milliseconds." That delay was the print_k message. After much work, many heated discussions, and several rejected proposals, a compromise was reached earlier this year. Torvalds is happy, the real-time Linux developers are happy, print_K users are happy, and, at long last, real-time Linux is real.
Could there be desktop linux benefits? (Score:2)
Re:Could there be desktop linux benefits? (Score:5, Interesting)
Audio production comes to mind.
Re: (Score:1)
Not really. RTOS is important when you're dealing with potentially expensive interactions in the physical world - you can't pause physics, and you can't magically replace destroyed raw materials.
For audio *production*, computers are already fast enough such that imperceptibly small latency builds in all the protection you need against starvation, even in high end production.
RTOS may be attractive for certain applications that *have* audio playback, say an immersive ride with motors that should act in sync w
Re: (Score:2, Insightful)
Put in speakers loud enough and you want to have it ensured that there won't be gaps in the audio.
I suppose that's still really mostly relevant to live performances, though, and just having the lowest layer perform won't be enough. It's nice to have, but people have done without.
The lowest level could perhaps protect against loss of input stream by having a dynamically generated backup track to ensure sound continuity.
Re: Yet even more bloated. (Score:2)
Fat porkers make the sweetest bacon. Yum!
Re: (Score:2)
Re: (Score:1)
Re:Could there be desktop linux benefits? (Score:5, Interesting)
You don't need a real-time kernel for real-time audio. And you don't need real-time kernel to ensure that there aren't gaps in audio playback. This is almost always handled via special hardware that does not require RTOS kernel.
Yes, the old "write 500 milliseconds of audio into a big FIFO and let the sound card DMA it out to the audio port when it needs to" approach works great, as long as you can tolerate many milliseconds of latency. It's fine for canned music playback.
For live sound, however, you need both reliable playback AND minimal latency -- for example, monitor mix engineers want less than 5 milliseconds of round-trip latency from microphone to speaker; anything more than that can throw off the musicians. Unless you have custom hardware that can be programmed to handle all of your audio-processing algorithms internally, that means you need a real-time OS to execute your audio filters reliably. (and if you do have custom hardware that powerful... then that custom hardware is probably running a real-time OS, or else no OS at all)
My company has been using Linux with Xenomai patches to implement real-time/low-latency audio processing on Intel motherboards, and it works quite well. I will be interested in seeing if the new vanilla Linux kernel can perform well enough to make Xenomai unnecessary.
Re: (Score:2)
I'm sure there are applications where using the RTOS is better, but I wouldn't have thought that live mixing was one of them. Can you elabora
Re: (Score:2)
You can use DAWs in live environments (and some do), e.g. Bitwig has features towards that kind of use. In such context you would have MIDI equipment and audio inputs that are then sent to synthesizers or effect pipelines, and finally to the audio output device. Some such uses could could be live playing keys, MIDI drum pads, or running amplifier simulators for electric guitars. Here's Richie Hawtin using it live, though basically for controlling filters and selecting clips: https://www.youtube.com/watch?.. [youtube.com]
Re: (Score:2)
Everything I have in terms of mixers is old Yamaha MG equipment which does process digitally but doesn't introduce any noticeable delay. Even the class D amplifiers in my subwoofers don
Re: (Score:2)
I'm sure there are applications where using the RTOS is better, but I wouldn't have thought that live mixing was one of them. Can you elaborate on the use case?
Our use-case is acoustic enhancement [meyersound.com] and automated and semi-automated front-of-house mixing [lsionline.com] for live shows (Broadway/LasVegas/Cirque-style productions).
Re: (Score:2)
Re: (Score:2)
For audio *production*, computers are already fast enough such that imperceptibly small latency builds in all the protection you need against starvation, even in high end production.
Music production is literally the current primary use of realtime on the desktop because the latency is not imperceptible, especially when it adds up.
RTOS may be attractive for certain applications that *have* audio playback, say an immersive ride with motors that should act in sync with an audio and video track, but even then, the "costs" of stutter or momentary desync are a not that high - compared to say a vehicle carrying humans that smashes into a wall or a robotic tool that fucks up valuable stock material.
Oh look, technical whataboutism.
Re: (Score:2)
Re: (Score:2)
Not really. RTOS is important when you're dealing with potentially expensive interactions in the physical world - you can't pause physics, and you can't magically replace destroyed raw materials.
For audio *production*, computers are already fast enough such that imperceptibly small latency builds in all the protection you need against starvation, even in high end production.
Up to now, I've not found that to be the case at all with Linux. Sometimes it works. But if enough background processes demand attention at the wrong moment? Bzzzt. And digital distortion. The only solution up to now has been to shut down all background processes when recording or producing. Even with the RTOS setups that, up to now, haven't been official. I wonder if full integration into the kernel proper will fix that or not?
I will say, I'm a massive resource hog with audio production. I tend to hear sco
Re: (Score:2)
I beg to differ. In an RTOS your process is guaranteed to be given the time it needs to finish a certain task, regardless of what else is going on in the system. It doesn't matter that computers are already fast enough to process audio. An example would be when the system is recording audio and writing it to disk A, and something else tries to write to disk B, but disk B throws an error. In an event driven (scheduled) O/S, the disk B issues could cause other processes, including your recording process.
Re: (Score:2)
Audio requires low latency not real time. There's currently no RTOS in audio gear that isn't running finely on a microcontroller layer in a process where a the Linux kernel would be orders of magnitude pointlessly complex.
The more complex part of the audio world where you benefit from a proper OS sitting on your hardware runs systems such as Dante which happily plod away on Windows, or AES67 for which there has been a linux kernel driver for decades.
Re:Could there be desktop linux benefits? (Score:5, Insightful)
Audio requires low latency not real time.
Non-real-time low latency means you are gambling that audio underruns won't happen often enough (or be noticeable enough) to be a showstopper for you. In many cases, that risk is worth taking; but in other cases (large or high-profile venues) it's considered unacceptable to gamble with the audience's ears, so in those cases a real-time OS can be used to guarantee that underruns will not occur.
There's currently no RTOS in audio gear that isn't running finely on a microcontroller layer in a process where a the Linux kernel would be orders of magnitude pointlessly complex.
I know for a fact that isn't true, because my job is writing software for high-end audio gear that runs on Linux via the Xenomai real-time patches.
Re: (Score:2)
Are you implying that Linux, without these realtime hooks, is not suitable for audio production? But Windows and MacOS clearly are, despite not being RTOSs.
Re: (Score:1)
People may get that impression from using a shitty middleware daemon like pulseaudio.
In the earlier days of linux, the need for low-latency scheduling for audio was in part what drove a number of linux scheduling optimizations. (Wiki article about JACK goes a bit into that history, with references [wikipedia.org])
Re: (Score:2)
It might avoid an embarrassingly delayed boss key?
Re: Could there be desktop linux benefits? (Score:2)
Exactly! Mod this up!
Re: (Score:2)
In theory yes for low latency multimedia and industrial uses, although don't quote me on that, but in practice modern desktop machines can achieve low latencies for audio simply by being fast as hell.
Re: Could there be desktop linux benefits? (Score:2)
Re:Could there be desktop linux benefits? (Score:5, Interesting)
I've been using Debian/Devuan with parameter preempt=full in grub, and the usual desktop use feels just that bit more snappier.
Re: (Score:2)
I don't think that has anything to do with PREEMPT_RT, though. I don't think PREEMPT_RT would help a desktop GUI at all. But if you need accurate timing to control a signal through a GPIO pin, that's what PREEMPT_RT is for.
Re: (Score:2)
Preempt is all about interrupting processing in favour of other processes. Unfortunately I was not able to find a good page to explain this after a quick google.
In practice, using RT for a desktop would probably mean performance would be slower, not faster. The overriding purpose of an RTOS is reliable and consistent processing time.
Re: (Score:2)
It may well be snappier but it is not RT.
Preempt is all about interrupting processing in favour of other processes.
What do you think realtime does? Hint, it doesn't make your computer faster.
Re: (Score:2)
Show us benchmarks. Feelings quite possibly driven by your expectation bias are worthless. This kind of thing is testable.
Re: (Score:2)
I don't think so. Since the purpose of RT is to provide precise and accurate timing guarantees to processes (for example toggling a GPIO pin to generate a signal), that might force GUI processes to wait, making it feel sluggish. I read once that a hard real-time system often can't utilize the entire capability of the CPU because of the real-time guarantees (this will run in X cycles, guaranteed). Whereas under normal linux, the scheduler can pre-empt other processes and even change their priority, which
Re: Could there be desktop linux benefits? (Score:1)
I'm wondering if this would be beneficial for emulators and VM environments too. TFA mentioned how some prior changes already made improvements in the data center.
Benefits will be on factory floor, not desktops. (Score:2)
A real time kernel is designed to ensure that an industrial process runs and is absolutely current. If you're, say, a cereal manufacturer with huge ovens automatically baking tons of cereal you don't want to hear that the oven was uncontrolled for several minutes because of a hung thread or a blocking I/O or excess CPU load. A realtime kernel guarantees on-time execution, and it's hard to do - we're modifying the way that preemptive multitasking works and that's pretty fundamental to the way the whole sys
Re: (Score:2)
There are general benefits to this outside of specialty reasons. It means that almost the whole kernel is preemptable. Task switching can happen at anytime, anywhere, in a predictable and timely manner
This is a big deal (Score:3, Funny)
This shouldn't be a big deal, but it's a big deal because it shouldn't have been a big deal.
Important technical achievement (Score:2)
This is an amazing technical achievement.
Re: Your mom (Score:2)
If you run in milliseconds, then you're embarrassingly useless to moms.
Wonder if this really could have been sooner... (Score:2)
The article suggests that print_k was this major blocker to getting all the changes in. I don't know if that was the case, I'd hate to think that all this work was held up because somebody wanted to printf debug the kernel like it was the 90s.
Then again, it could have been just a reasonable milestone on a long list of stuff. Either way, well done to the team.
Could be a big boost to audio applications on the kernel. It is very useful to have features like "hey, make sure this grabs a buffer every so and so.
Re: (Score:3)
"The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." — Brian Kernighan, "Unix for Beginners" (1979)
And he was reacting to the "fancy" command line debuggers like adb and crash; and describing the old techniques which were called Trace Debugging in the 1960's (and output to paper sheets)...
History aside, print debugging is often the only method available for real time code once it is loaded on the embedded device (we do try to find every bug w
This interrupt was deferred for 20 Years... (Score:2)
>> Wonder if this really could have been sooner...
Nope. Real time was not available, this interrupt was deferred for 20 Years...
Will it get real-world commercial use? (Score:5, Insightful)
I think that if a feature has no users then it shouldn't be in the kernel. Maybe in 10 years they'll go, "Yeah OK no one ended up using this feature. It's a pain to maintain. It's now marked as depreciated and we will remove it sometime in the future". I think that is the elephant in the room here. That is - is this something people will actually use, or is it a pointless academic exercise?
Re: (Score:2)
A more responsive GUI with little overhead sounds like one real world use case.
Re: Will it get real-world commercial use? (Score:2)
The goal of an RTOS is not to make things faster -> "more responsive". It's to make things predictable with a deterministic max response latency.
I could benefits in being able to run and debug RT workloads within your desktop... but treating your UX as an RT program and erroring on microsecond sounds like a lot more work for a much worse experience
Re:Will it get real-world commercial use? (Score:5, Informative)
It's been used for years in CNC machine control. Glad to see it in the mainline tree now. Makes life a lot easier for those running LinuxCNC or MachineKit. MachineKit seems to have evolved far beyond CNC and can be used in any situation where you need real-time control over machinery like stepper motors, industrial machines, etc.
It's not something normal users will ever use, and I don't think it is helpful for a desktop GUI. In fact it might slow down a GUI due to the harder scheduling requirements.
So anyway yes this is something people actually use. It's not academic at all.
Re: (Score:2)
Re:Will it get real-world commercial use? (Score:4)
It's not something normal users will ever use, and I don't think it is helpful for a desktop GUI. In fact it might slow down a GUI due to the harder scheduling requirements.
It will probably slow down a GUI with no changes made to it. But when taken into account and utilized for the GUI, it should make a GUI more responsive. This is certainly true of QNX.
Realtime offers direct benefits for jobs like recording music, so there are good reasons at least for people who are less computer nerds and more music nerds to care. It could also affect things like gaming, although we're a ways off from that since most games are still Windows programs — even the ones people run on Linux. Since realtime's whole job is reduction of latency, and latency is what we notice as slowness in most cases, it can potentially improve the UX by a whole lot.
Re:Will it get real-world commercial use? (Score:5, Interesting)
Yes, a European car consortium is already building vehicles around it.
IIRC there was an article here a few months ago.
Brakes gotta brake.
Re: (Score:3)
I think you're getting your announcements confused. As far as I am aware there are no automotive applications which currently use or are even allowed to use Linux RT. What we discussed a few months ago was Linux getting certified to ISO 26262 ASIL-B paving the way to allow it to be used beyond the infotainment system (where it is actively being used) and into some soft time/safety critical applications such as displaying the instrument cluster, handling safety override inputs such as disabling ESP / changin
Re:Will it get real-world commercial use? (Score:5, Interesting)
Talked with a dude who worked on automotive firmware. Said realtime linux was better than QNX and that essentially QNX was more fragile and crash-prone.
(Not in the sense that QNX just crashes randomly, but that it was fragile and difficult to develop for.)
Re:Will it get real-world commercial use? (Score:4, Informative)
Indeed, but QNX due to its simplicity is usually used for systems that also require an ASIL-A rating - engine management, ABS, etc. You can't simply migrate this to Linux RT. If anything is used Linux wise it'll be a hybrid of an OS handling user facing system communicating with something running QNX on its metal to control things like spark plug firing.
They are no doubt playing with it (its important for all companies to play with emerging technology and put it through its paces), but I don't know of any cases of Linux RT being used in a production vehicle for a real time application.
Re: (Score:2)
It is already in very wide use in many Embedded Linux systems. Yes it's very practical. But probably not on a PC, a PC is a boring platform because they solve all problems by just waiting for faster hardware.
And the Linux kernel has a lot of tasks now, it's not at all the same as early Linux monolithic kernels. But the kernel vs userspace architecture often means some real time activities end up being modules hooked into the kernel. At the same time, with threaded processes the real time can apply even
Re: (Score:2)
You create a perfect catch-22 that way. Nobody uses the feature because it's not there. It's not there because nobody uses it.
In fact, RT mods to Linux have been in use, just with external patches.
Re: (Score:2)
These days, I think it's going to be more important than ever.
Using an RTOS is one thing - there are plenty of them out there between QNX and many F/OSS versions (FreeRTOS is a particularly popular one).
However, the issue you'll run into is that none of them have the software support Linux does. It actually makes running an RTOS alone hard. Modern infotainment systems are using QNX and another OS in a VM.
RT-LInux was more of a dual kernel solution - it had a RTOS scheduler to do handle the real-time tasks,
BeOS (Score:4, Interesting)
Around 25 years ago I was writing for a company developing a multitrack recording system using BeOS. Man, BeOS was snappy; it was designed to be a media (audio and video) OS. In many ways it was better than Linux as an alternate OS to Macs and Windows, but Linux won.
But in 1998, when you clicked on a background BeOS window, it immediately jumped to the front, and if you clicked a button you got immediate visual feedback, even if the action might take longer to carry out. How come this was not implemented elsewhere, for 25 years?
Re: (Score:1)
How come this was not implemented elsewhere, for 25 years?
Kind of a shame, isn't it? The simple truth of the matter is that not enough software developers cared about that stuff. And it does require more than OS support by interrupting a process because its window got user input, prioritizing the focused application, and ensuring that the relevant object code isn't swapped out of memory. It requires that app developers design their software with a similar focus on UI responsiveness.
To this day I still encounter apps where the UI blocks because the app is busy doin
Re: (Score:2)
Nowadays most apps are web apps, most actions in web apps happen via js code making http requests to retrieve or push data, and most of that code gives _zero_ feedback:
- no visual feedback such as graying out or change of colour of the clicked control
- double-clicking is not prevented
- cursor does not change to a spinner
- worst offender of it all: quite often there is no feedback at all if the http request fails for any reason, or if it brings back unexpected data
Re: (Score:2)
Low-latency and RealTime are not the same thing. In fact an RTOS running a UI is likely to feel less snappy to the end user since the scheduler needs to prioritise the actual real time I/O tasks at a strict regular cadance regardless of whether there are any inputs to handle at the expense of any user interaction. Your UI and interaction become a secondary task which can be done when the kernel has nothing better to do.
Re: (Score:2)
Low-latency and RealTime are not the same thing.
That's true, but only because realtime is a means of achieving low latency. They are not the same thing, they are directly related things. What do you imaging that realtime means or does?
Your UI and interaction become a secondary task which can be done when the kernel has nothing better to do.
Your UI and interaction can become a primary task which can be done first and the heavy lifting can be delayed so that the user gets a better experience. This will slow down processing by a tiny percentage, but improve user interface latency by a massive one. QNX proved this with their public desktop release.
Re: (Score:2)
Realtime isn't about reducing latency; it's about guaranteeing latency. It's about saying that we'll do this thing every 5 ms and we'll do this other thing every 20 ms.
Re: (Score:2)
OK, let's split this hair.
It's about reducing latency to known values for specific tasks.
Re: (Score:2)
Because it wasn't deemed a critical feature by the vast majority of customers. Same as with any other economic decision.
Re: (Score:2)
But in 1998, when you clicked on a background BeOS window, it immediately jumped to the front, and if you clicked a button you got immediate visual feedback, even if the action might take longer to carry out. How come this was not implemented elsewhere, for 25 years?
BeOS was a great OS, because it was designed from the ground up to take advantage of multithreading and multiprocessing to improve responsiveness. It was able to take that approach because it wasn't required to be backwards-compatible with any existing applications or APIs that assumed a single-threaded/single-processor environment... that enabled BeOS to be fast and simple, but also largely doomed it to fail because people need to run their applications of choice more than they need good performance. Th
RTOS guarantees (Score:1)
>An RTOS guarantees that critical tasks will be completed within specified deadlines. [...]
It's a lot simpler than that, even.
All an RTOS guarantees is that your interrupt handler will be called within some worst-case latency, and that the called code can't be preempted. Any further guarantees are up to the developer to make.
I find the world of realtime computing offers interesting, satisfying challenges. Various microcontrollers give you deterministic, sub-microsecond control over timing.
Interestingly,
Re: (Score:2)
Actually, your typical RT design avoids interrupts whenever possible.
Horseshit. RT designs are fundamentally interrupt driven, just that those interrupts are triggered by timers and physical changes rather than software requests. In fact interrupts are the way that an RTOS guarantees that it is operating in real time.
Re: (Score:2)
Jesus christ (Score:1)
Linux RTOS patches existed way back in 2.2-2.4 releases, if memory serves. Torvalds wouldn't let it merge for 24 years because of his precious function? What a dick
Re: (Score:1)
If you weren't a knee-jerking moron you'd know that "printk" was just the *final* obstacle, not the only obstacle, so no, it didn't block merging for 24 years. It was only a couple of years ago that other critical kernel issues were resolved. Just a few years ago, adding the out of tree RT patches gave you a kernel with stability issues, prone to occasional hard lockups. But you, in your infinite wisdom, would have just shoved those patches in rather than being "stupid and stubborn" like Linus and ensuring
Re: (Score:3)
When you're responsible for billions of devices and their interactions, you can get to decide when an out-of-tree patch that hits literally EVERY part of the OS gets included in YOUR version of a huge, open-source, forkable piece of code where things can happily live entirely out of tree for decades without affecting anyone and where anyone with real-time requirements was going while the rest of us normal people just used whatever the distro had chosen (which often wasn't Linus's version of the kernel... th
Baked into all distros? (Score:2)
Isn't it a bit early to claim that?
Most users should NOT be using a RTOS (Score:1)
Virtually all end users - and that includes nearly all enterprise users - don't have any use for a real time kernel. Here's your first test - do you have dedicated A/D equipment, like say, a Symatec F5? If you don't know what that means, we can just consider the answer to that question "no".
If you're still thinking about using real time facilities, I hope you're comfortable with kernel
The article may overstate the case (Score:2)
IIUC PREMPT_RT makes Linux "soft" real-time, not "hard" real-time, so it might not be appearing in "mission critical" applications - at least if I've understood the comments in Phoronix correctly:
https://www.phoronix.com/forum... [phoronix.com]
It's still very good news.
Is there a list of known good drivers? (Score:3)
The kernel might be well behaved, but the drivers can mask interrupt without care of any time bounds. Are all the mainline drivers known good?
Re: (Score:2)
"known good" means bad
kernel and debugging story (Score:2)
In the mid '80s I worked at a research lab running DEC Ultrix on a VAX. There was some sort of problem with the OS, I think related to a device on the system, and the system admin added a mod to the kernel to print out the error message to an old LA-120 printing terminal. That provided an immutable record of error messages. But it also made the kernel block while the message was typed out at 120 characters per second... Needless to say, the rest of us were not happy when the system would hang up while t
FYI, SpaceX avionics don't need hard real-time (Score:2)
...but they use Linux all over the place. See:
https://lwn.net/Articles/54036... [lwn.net]
Sounds a lot like the Titan submersible. (Score:1)