Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Operating Systems Linux

20 Years Later, Real-Time Linux Makes It To the Kernel (zdnet.com) 100

ZDNet's Steven Vaughan-Nichols reports: After 20 years, Real-Time Linux (PREEMPT_RT) is finally -- finally -- in the mainline kernel. Linus Torvalds blessed the code while he was at Open Source Summit Europe. [...] The real-time Linux code is now baked into all Linux distros as of the forthcoming Linux 6.12 kernel. This means Linux will soon start appearing in more mission-critical devices and industrial hardware. But it took its sweet time getting here. An RTOS is a specialized operating system designed to handle time-critical tasks with precision and reliability. Unlike general-purpose operating systems like Windows or macOS, an RTOS is built to respond to events and process data within strict time constraints, often measured in milliseconds or microseconds. As Steven Rostedt, a prominent real-time Linux developer and Google engineer, put it, "Real-time is the fastest worst-case scenario." He means that the essential characteristic of an RTOS is its deterministic behavior. An RTOS guarantees that critical tasks will be completed within specified deadlines. [...]

So, why is Real-Time Linux only now completely blessed in the kernel? "We actually would not push something up unless we thought it was ready," Rostedt explained. "Almost everything was usually rewritten at least three times before it went into mainline because we had such a high bar for what would go in." In addition, the path to the mainline wasn't just about technical challenges. Politics and perception also played a role. "In the beginning, we couldn't even mention real-time," Rostedt recalled. "Everyone said, 'Oh, we don't care about real-time.'" Another problem was money. For many years funding for real-time Linux was erratic. In 2015, the Linux Foundation established the Real-Time Linux (RTL) collaborative project to coordinate efforts around mainlining PREEMPT_RT.

The final hurdle for full integration was reworking the kernel's print_k function, a critical debugging tool dating back to 1991. Torvalds was particularly protective of print_k --He wrote the original code and still uses it for debugging. However, print_k also puts a hard delay in a Linux program whenever it's called. That kind of slowdown is unacceptable in real-time systems. Rostedt explained: "Print_k has a thousand hacks to handle a thousand different situations. Whenever we modified print_k to do something, it would break one of these cases. The thing about print_k that's great about debugging is you can know exactly where you were when a process crashed. When I would be hammering the system really, really hard, and the latency was mostly around maybe 30 microseconds, and then suddenly it would jump to five milliseconds." That delay was the print_k message. After much work, many heated discussions, and several rejected proposals, a compromise was reached earlier this year. Torvalds is happy, the real-time Linux developers are happy, print_K users are happy, and, at long last, real-time Linux is real.

This discussion has been archived. No new comments can be posted.

20 Years Later, Real-Time Linux Makes It To the Kernel

Comments Filter:
  • "could" because I know this mostly benefits embedded stuff
    • by Narcocide ( 102829 ) on Wednesday September 18, 2024 @08:09PM (#64798189) Homepage

      Audio production comes to mind.

      • by SirSlud ( 67381 )

        Not really. RTOS is important when you're dealing with potentially expensive interactions in the physical world - you can't pause physics, and you can't magically replace destroyed raw materials.

        For audio *production*, computers are already fast enough such that imperceptibly small latency builds in all the protection you need against starvation, even in high end production.

        RTOS may be attractive for certain applications that *have* audio playback, say an immersive ride with motors that should act in sync w

        • Re: (Score:2, Insightful)

          by flux ( 5274 )

          Put in speakers loud enough and you want to have it ensured that there won't be gaps in the audio.

          I suppose that's still really mostly relevant to live performances, though, and just having the lowest layer perform won't be enough. It's nice to have, but people have done without.

          The lowest level could perhaps protect against loss of input stream by having a dynamically generated backup track to ensure sound continuity.

          • You don't need a real-time kernel for real-time audio. And you don't need real-time kernel to ensure that there aren't gaps in audio playback. This is almost always handled via special hardware that does not require RTOS kernel.
            • by Jeremi ( 14640 ) on Thursday September 19, 2024 @09:57AM (#64799679) Homepage

              You don't need a real-time kernel for real-time audio. And you don't need real-time kernel to ensure that there aren't gaps in audio playback. This is almost always handled via special hardware that does not require RTOS kernel.

              Yes, the old "write 500 milliseconds of audio into a big FIFO and let the sound card DMA it out to the audio port when it needs to" approach works great, as long as you can tolerate many milliseconds of latency. It's fine for canned music playback.

              For live sound, however, you need both reliable playback AND minimal latency -- for example, monitor mix engineers want less than 5 milliseconds of round-trip latency from microphone to speaker; anything more than that can throw off the musicians. Unless you have custom hardware that can be programmed to handle all of your audio-processing algorithms internally, that means you need a real-time OS to execute your audio filters reliably. (and if you do have custom hardware that powerful... then that custom hardware is probably running a real-time OS, or else no OS at all)

              My company has been using Linux with Xenomai patches to implement real-time/low-latency audio processing on Intel motherboards, and it works quite well. I will be interested in seeing if the new vanilla Linux kernel can perform well enough to make Xenomai unnecessary.

              • Yes. I assumed that live mixing audio equipment runs no OS at all. And then a DAW buffers and streams if you want to do digital recording. I am not up-to-date but historically, mixing boards were a ADC a non-OS microchip to do processing, and a DAC on the output. The alternative would be to replace the bare-metal chip with a much more powerful one and an RTOS.

                I'm sure there are applications where using the RTOS is better, but I wouldn't have thought that live mixing was one of them. Can you elabora

                • by flux ( 5274 )

                  You can use DAWs in live environments (and some do), e.g. Bitwig has features towards that kind of use. In such context you would have MIDI equipment and audio inputs that are then sent to synthesizers or effect pipelines, and finally to the audio output device. Some such uses could could be live playing keys, MIDI drum pads, or running amplifier simulators for electric guitars. Here's Richie Hawtin using it live, though basically for controlling filters and selecting clips: https://www.youtube.com/watch?.. [youtube.com]

                  • The last time I was at a Broadway show (pre-pandemic), the delay from the digital microphones was so bad that I could hear it in the audience. I was close enough that I could hear the unamplified performer and the amplified audio noticeably later. After that, I was completely turned off to most digital audio equipment.

                    Everything I have in terms of mixers is old Yamaha MG equipment which does process digitally but doesn't introduce any noticeable delay. Even the class D amplifiers in my subwoofers don

                • by Jeremi ( 14640 )

                  I'm sure there are applications where using the RTOS is better, but I wouldn't have thought that live mixing was one of them. Can you elaborate on the use case?

                  Our use-case is acoustic enhancement [meyersound.com] and automated and semi-automated front-of-house mixing [lsionline.com] for live shows (Broadway/LasVegas/Cirque-style productions).

                  • That's really impressive. If the company lists Broadway theaters that use the system, I'll give one of them a try. If there's excessive audio delay, I'll assume that the microphones are buffering and not your system!
        • For audio *production*, computers are already fast enough such that imperceptibly small latency builds in all the protection you need against starvation, even in high end production.

          Music production is literally the current primary use of realtime on the desktop because the latency is not imperceptible, especially when it adds up.

          RTOS may be attractive for certain applications that *have* audio playback, say an immersive ride with motors that should act in sync with an audio and video track, but even then, the "costs" of stutter or momentary desync are a not that high - compared to say a vehicle carrying humans that smashes into a wall or a robotic tool that fucks up valuable stock material.

          Oh look, technical whataboutism.

          • by SirSlud ( 67381 )

            I really don't think you're a particularly smart person drinkypoo, so in the future, you can probably just not bother replying to my posts. Your efforts may be better spent downvoting them.

        • by w1z7ard ( 227376 )
          You still need a low latency kernel if you are doing real time recording with midi / soundfonts (from my personal experience at least).
        • Not really. RTOS is important when you're dealing with potentially expensive interactions in the physical world - you can't pause physics, and you can't magically replace destroyed raw materials.

          For audio *production*, computers are already fast enough such that imperceptibly small latency builds in all the protection you need against starvation, even in high end production.

          Up to now, I've not found that to be the case at all with Linux. Sometimes it works. But if enough background processes demand attention at the wrong moment? Bzzzt. And digital distortion. The only solution up to now has been to shut down all background processes when recording or producing. Even with the RTOS setups that, up to now, haven't been official. I wonder if full integration into the kernel proper will fix that or not?

          I will say, I'm a massive resource hog with audio production. I tend to hear sco

        • by lsllll ( 830002 )

          I beg to differ. In an RTOS your process is guaranteed to be given the time it needs to finish a certain task, regardless of what else is going on in the system. It doesn't matter that computers are already fast enough to process audio. An example would be when the system is recording audio and writing it to disk A, and something else tries to write to disk B, but disk B throws an error. In an event driven (scheduled) O/S, the disk B issues could cause other processes, including your recording process.

      • Audio requires low latency not real time. There's currently no RTOS in audio gear that isn't running finely on a microcontroller layer in a process where a the Linux kernel would be orders of magnitude pointlessly complex.

        The more complex part of the audio world where you benefit from a proper OS sitting on your hardware runs systems such as Dante which happily plod away on Windows, or AES67 for which there has been a linux kernel driver for decades.

        • by Jeremi ( 14640 ) on Thursday September 19, 2024 @10:04AM (#64799699) Homepage

          Audio requires low latency not real time.

          Non-real-time low latency means you are gambling that audio underruns won't happen often enough (or be noticeable enough) to be a showstopper for you. In many cases, that risk is worth taking; but in other cases (large or high-profile venues) it's considered unacceptable to gamble with the audience's ears, so in those cases a real-time OS can be used to guarantee that underruns will not occur.

          There's currently no RTOS in audio gear that isn't running finely on a microcontroller layer in a process where a the Linux kernel would be orders of magnitude pointlessly complex.

          I know for a fact that isn't true, because my job is writing software for high-end audio gear that runs on Linux via the Xenomai real-time patches.

      • by necro81 ( 917438 )

        Audio production comes to mind.

        Are you implying that Linux, without these realtime hooks, is not suitable for audio production? But Windows and MacOS clearly are, despite not being RTOSs.

    • by sjames ( 1099 )

      It might avoid an embarrassingly delayed boss key?

    • In theory yes for low latency multimedia and industrial uses, although don't quote me on that, but in practice modern desktop machines can achieve low latencies for audio simply by being fast as hell.

    • If you do real-time right, it makes the system more stable. It sounds like they did a good job carefully looking at each area of the code and analyzing it, so it sounds like this will a make.the code more stable. Now we just need a real-time systemd written in Rust.
    • by JamesTRexx ( 675890 ) on Wednesday September 18, 2024 @09:02PM (#64798305) Journal

      I've been using Debian/Devuan with parameter preempt=full in grub, and the usual desktop use feels just that bit more snappier.

      • by caseih ( 160668 )

        I don't think that has anything to do with PREEMPT_RT, though. I don't think PREEMPT_RT would help a desktop GUI at all. But if you need accurate timing to control a signal through a GPIO pin, that's what PREEMPT_RT is for.

      • by BigZee ( 769371 )
        It may well be snappier but it is not RT.

        Preempt is all about interrupting processing in favour of other processes. Unfortunately I was not able to find a good page to explain this after a quick google.

        In practice, using RT for a desktop would probably mean performance would be slower, not faster. The overriding purpose of an RTOS is reliable and consistent processing time.

        • It may well be snappier but it is not RT.
          Preempt is all about interrupting processing in favour of other processes.

          What do you think realtime does? Hint, it doesn't make your computer faster.

      • Show us benchmarks. Feelings quite possibly driven by your expectation bias are worthless. This kind of thing is testable.

    • by caseih ( 160668 )

      I don't think so. Since the purpose of RT is to provide precise and accurate timing guarantees to processes (for example toggling a GPIO pin to generate a signal), that might force GUI processes to wait, making it feel sluggish. I read once that a hard real-time system often can't utilize the entire capability of the CPU because of the real-time guarantees (this will run in X cycles, guaranteed). Whereas under normal linux, the scheduler can pre-empt other processes and even change their priority, which

    • Comment removed based on user account deletion
    • There are general benefits to this outside of specialty reasons. It means that almost the whole kernel is preemptable. Task switching can happen at anytime, anywhere, in a predictable and timely manner

  • by backslashdot ( 95548 ) on Wednesday September 18, 2024 @07:54PM (#64798169)

    This shouldn't be a big deal, but it's a big deal because it shouldn't have been a big deal.

  • This is an amazing technical achievement.

  • The article suggests that print_k was this major blocker to getting all the changes in. I don't know if that was the case, I'd hate to think that all this work was held up because somebody wanted to printf debug the kernel like it was the 90s.

    Then again, it could have been just a reasonable milestone on a long list of stuff. Either way, well done to the team.

    Could be a big boost to audio applications on the kernel. It is very useful to have features like "hey, make sure this grabs a buffer every so and so.

    • by mtm10 ( 1530769 )

      "The most effective debugging tool is still careful thought, coupled with judiciously placed print statements." — Brian Kernighan, "Unix for Beginners" (1979)

      And he was reacting to the "fancy" command line debuggers like adb and crash; and describing the old techniques which were called Trace Debugging in the 1960's (and output to paper sheets)...

      History aside, print debugging is often the only method available for real time code once it is loaded on the embedded device (we do try to find every bug w

    • >> Wonder if this really could have been sooner...
      Nope. Real time was not available, this interrupt was deferred for 20 Years...

  • by Gavino ( 560149 ) on Wednesday September 18, 2024 @09:01PM (#64798303)
    If I was manufacturing something mission critical that needed a RTOS I'd probably go with QNX due to it's proven track record and micro kernel architecture. I wouldn't go for something that is tacked onto a monolithic kernel, and one that not all kernel developers seem to care about.

    I think that if a feature has no users then it shouldn't be in the kernel. Maybe in 10 years they'll go, "Yeah OK no one ended up using this feature. It's a pain to maintain. It's now marked as depreciated and we will remove it sometime in the future". I think that is the elephant in the room here. That is - is this something people will actually use, or is it a pointless academic exercise?
    • > is this something people will actually use, or is it a pointless academic exercise?

      A more responsive GUI with little overhead sounds like one real world use case.
      • The goal of an RTOS is not to make things faster -> "more responsive". It's to make things predictable with a deterministic max response latency.

        I could benefits in being able to run and debug RT workloads within your desktop... but treating your UX as an RT program and erroring on microsecond sounds like a lot more work for a much worse experience

    • by caseih ( 160668 ) on Wednesday September 18, 2024 @09:19PM (#64798329)

      It's been used for years in CNC machine control. Glad to see it in the mainline tree now. Makes life a lot easier for those running LinuxCNC or MachineKit. MachineKit seems to have evolved far beyond CNC and can be used in any situation where you need real-time control over machinery like stepper motors, industrial machines, etc.

      It's not something normal users will ever use, and I don't think it is helpful for a desktop GUI. In fact it might slow down a GUI due to the harder scheduling requirements.

      So anyway yes this is something people actually use. It's not academic at all.

      • by dargaud ( 518470 )
        I work in data acquisition. We always do the acquisition itself (real-time) in custom components (microcontrollers, FPGA, etc...) and then transfer the data to the Linux system (PC, embedded, etc...) doing the control/command, processing, storage and display. Transfer is via serial, USB, ethernet... It's easy, reliable and easier when something breaks. Trying to tack everything into one single system is a nonsense.
      • It's not something normal users will ever use, and I don't think it is helpful for a desktop GUI. In fact it might slow down a GUI due to the harder scheduling requirements.

        It will probably slow down a GUI with no changes made to it. But when taken into account and utilized for the GUI, it should make a GUI more responsive. This is certainly true of QNX.

        Realtime offers direct benefits for jobs like recording music, so there are good reasons at least for people who are less computer nerds and more music nerds to care. It could also affect things like gaming, although we're a ways off from that since most games are still Windows programs — even the ones people run on Linux. Since realtime's whole job is reduction of latency, and latency is what we notice as slowness in most cases, it can potentially improve the UX by a whole lot.

    • by bill_mcgonigle ( 4333 ) * on Wednesday September 18, 2024 @09:28PM (#64798337) Homepage Journal

      Yes, a European car consortium is already building vehicles around it.

      IIRC there was an article here a few months ago.

      Brakes gotta brake.

      • I think you're getting your announcements confused. As far as I am aware there are no automotive applications which currently use or are even allowed to use Linux RT. What we discussed a few months ago was Linux getting certified to ISO 26262 ASIL-B paving the way to allow it to be used beyond the infotainment system (where it is actively being used) and into some soft time/safety critical applications such as displaying the instrument cluster, handling safety override inputs such as disabling ESP / changin

    • by Anonymous Coward on Wednesday September 18, 2024 @10:17PM (#64798417)

      Talked with a dude who worked on automotive firmware. Said realtime linux was better than QNX and that essentially QNX was more fragile and crash-prone.
      (Not in the sense that QNX just crashes randomly, but that it was fragile and difficult to develop for.)

      • by thegarbz ( 1787294 ) on Thursday September 19, 2024 @05:43AM (#64799039)

        Indeed, but QNX due to its simplicity is usually used for systems that also require an ASIL-A rating - engine management, ABS, etc. You can't simply migrate this to Linux RT. If anything is used Linux wise it'll be a hybrid of an OS handling user facing system communicating with something running QNX on its metal to control things like spark plug firing.

        They are no doubt playing with it (its important for all companies to play with emerging technology and put it through its paces), but I don't know of any cases of Linux RT being used in a production vehicle for a real time application.

    • It is already in very wide use in many Embedded Linux systems. Yes it's very practical. But probably not on a PC, a PC is a boring platform because they solve all problems by just waiting for faster hardware.

      And the Linux kernel has a lot of tasks now, it's not at all the same as early Linux monolithic kernels. But the kernel vs userspace architecture often means some real time activities end up being modules hooked into the kernel. At the same time, with threaded processes the real time can apply even

    • by sjames ( 1099 )

      You create a perfect catch-22 that way. Nobody uses the feature because it's not there. It's not there because nobody uses it.

      In fact, RT mods to Linux have been in use, just with external patches.

    • by tlhIngan ( 30335 )

      These days, I think it's going to be more important than ever.

      Using an RTOS is one thing - there are plenty of them out there between QNX and many F/OSS versions (FreeRTOS is a particularly popular one).

      However, the issue you'll run into is that none of them have the software support Linux does. It actually makes running an RTOS alone hard. Modern infotainment systems are using QNX and another OS in a VM.

      RT-LInux was more of a dual kernel solution - it had a RTOS scheduler to do handle the real-time tasks,

    • by zmooc ( 33175 )

      Just about all music production software depends on this. RT ready kernels have been shipped with many Linux distributions for years if not decades because without it a lot of stuff simply won't work. RT has been mainstream for ages. It's suitability for production use is totally undisputed.

  • BeOS (Score:4, Interesting)

    by yo303 ( 558777 ) on Wednesday September 18, 2024 @09:56PM (#64798385)

    Around 25 years ago I was writing for a company developing a multitrack recording system using BeOS. Man, BeOS was snappy; it was designed to be a media (audio and video) OS. In many ways it was better than Linux as an alternate OS to Macs and Windows, but Linux won.

    But in 1998, when you clicked on a background BeOS window, it immediately jumped to the front, and if you clicked a button you got immediate visual feedback, even if the action might take longer to carry out. How come this was not implemented elsewhere, for 25 years?

    • by Anonymous Coward

      How come this was not implemented elsewhere, for 25 years?

      Kind of a shame, isn't it? The simple truth of the matter is that not enough software developers cared about that stuff. And it does require more than OS support by interrupting a process because its window got user input, prioritizing the focused application, and ensuring that the relevant object code isn't swapped out of memory. It requires that app developers design their software with a similar focus on UI responsiveness.
      To this day I still encounter apps where the UI blocks because the app is busy doin

      • by giuntag ( 833437 )
        Talk about battles lost ages ago...

        Nowadays most apps are web apps, most actions in web apps happen via js code making http requests to retrieve or push data, and most of that code gives _zero_ feedback:
        - no visual feedback such as graying out or change of colour of the clicked control
        - double-clicking is not prevented
        - cursor does not change to a spinner
        - worst offender of it all: quite often there is no feedback at all if the http request fails for any reason, or if it brings back unexpected data
    • Low-latency and RealTime are not the same thing. In fact an RTOS running a UI is likely to feel less snappy to the end user since the scheduler needs to prioritise the actual real time I/O tasks at a strict regular cadance regardless of whether there are any inputs to handle at the expense of any user interaction. Your UI and interaction become a secondary task which can be done when the kernel has nothing better to do.

      • Low-latency and RealTime are not the same thing.

        That's true, but only because realtime is a means of achieving low latency. They are not the same thing, they are directly related things. What do you imaging that realtime means or does?

        Your UI and interaction become a secondary task which can be done when the kernel has nothing better to do.

        Your UI and interaction can become a primary task which can be done first and the heavy lifting can be delayed so that the user gets a better experience. This will slow down processing by a tiny percentage, but improve user interface latency by a massive one. QNX proved this with their public desktop release.

        • Realtime isn't about reducing latency; it's about guaranteeing latency. It's about saying that we'll do this thing every 5 ms and we'll do this other thing every 20 ms.

        • by tlhIngan ( 30335 )

          What do you imaging that realtime means or does?

          Real Time means defined latency. Not necessarily low latency, but guaranteed latency.

          That's the most common definition. Another definition is that an RTOS is able to accomplish everything it needs to do within deadlines. Say you have a task that must complete its work every second. An RTOS can guarantee that. Though, most RTOSes merely guarantee it will run every second - finishing in the second is up to the programmer to ensure there are adequate resources f

    • How come this was not implemented elsewhere, for 25 years?

      Because it wasn't deemed a critical feature by the vast majority of customers. Same as with any other economic decision.

    • by Jeremi ( 14640 )

      But in 1998, when you clicked on a background BeOS window, it immediately jumped to the front, and if you clicked a button you got immediate visual feedback, even if the action might take longer to carry out. How come this was not implemented elsewhere, for 25 years?

      BeOS was a great OS, because it was designed from the ground up to take advantage of multithreading and multiprocessing to improve responsiveness. It was able to take that approach because it wasn't required to be backwards-compatible with any existing applications or APIs that assumed a single-threaded/single-processor environment... that enabled BeOS to be fast and simple, but also largely doomed it to fail because people need to run their applications of choice more than they need good performance. Th

  • by Anonymous Coward

    >An RTOS guarantees that critical tasks will be completed within specified deadlines. [...]
    It's a lot simpler than that, even.
    All an RTOS guarantees is that your interrupt handler will be called within some worst-case latency, and that the called code can't be preempted. Any further guarantees are up to the developer to make.

    I find the world of realtime computing offers interesting, satisfying challenges. Various microcontrollers give you deterministic, sub-microsecond control over timing.
    Interestingly,

  • Linux RTOS patches existed way back in 2.2-2.4 releases, if memory serves. Torvalds wouldn't let it merge for 24 years because of his precious function? What a dick

    • by Anonymous Coward

      If you weren't a knee-jerking moron you'd know that "printk" was just the *final* obstacle, not the only obstacle, so no, it didn't block merging for 24 years. It was only a couple of years ago that other critical kernel issues were resolved. Just a few years ago, adding the out of tree RT patches gave you a kernel with stability issues, prone to occasional hard lockups. But you, in your infinite wisdom, would have just shoved those patches in rather than being "stupid and stubborn" like Linus and ensuring

    • by dgatwood ( 11270 )

      Linux RTOS patches existed way back in 2.2-2.4 releases, if memory serves.

      The Montavista folks were working on introducing realtime support in the Linux kernel back when I worked there (1999 to 2001), so those are probably what you're thinking of. Apparently this specific set of changes started somewhat after that. There have been a *lot* of patches that added various realtime capabilities to the Linux kernel and fixed various latency issues, locking contention issues, etc. that made any sort of RT in the Linux kernel problematic, and those efforts started back as far back as 1

  • Isn't it a bit early to claim that?

  • IIUC PREMPT_RT makes Linux "soft" real-time, not "hard" real-time, so it might not be appearing in "mission critical" applications - at least if I've understood the comments in Phoronix correctly:
    https://www.phoronix.com/forum... [phoronix.com]
    It's still very good news.

  • by Pinky's Brain ( 1158667 ) on Thursday September 19, 2024 @05:47AM (#64799053)

    The kernel might be well behaved, but the drivers can mask interrupt without care of any time bounds. Are all the mainline drivers known good?

  • In the mid '80s I worked at a research lab running DEC Ultrix on a VAX. There was some sort of problem with the OS, I think related to a device on the system, and the system admin added a mod to the kernel to print out the error message to an old LA-120 printing terminal. That provided an immutable record of error messages. But it also made the kernel block while the message was typed out at 120 characters per second... Needless to say, the rest of us were not happy when the system would hang up while t

  • ...but they use Linux all over the place. See:

    https://lwn.net/Articles/54036... [lwn.net]

  • But what is the compromise? Blargh, the most important part of the whole article is just hand waved away.

    • by msew ( 2056 )

      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c903327d3295b135eb8c81ebe0b68c1837718eb8

Experiments must be reproducible; they should all fail in the same way.

Working...