Forgot your password?
typodupeerror
Open Source Linux Hardware

The Ugly State of ARM Support On Linux 94

Posted by Soulskill
from the penguins-have-weak-arms dept.
jfruhlinger writes "Power-efficient ARM processors are moving up the food chain, to the extent that even Windows will soon see an ARM port. Linux, which has long been cross-platform, should have a long head start in this niche, right? Well, blogger Brian Proffitt explains just how messy the state of Linux support for ARM is right now, partially as a result of mutually conflicting kernel hacks from ARM manufacturers who just wanted to get their products out the door and weren't necessarily abiding by the GPL obligations to release code. Things are improving now, not least because Linus is taking a personal hand in things, but sorting the mess out will take time."
This discussion has been archived. No new comments can be posted.

The Ugly State of ARM Support On Linux

Comments Filter:
  • Problem is simple (Score:5, Insightful)

    by JamesP (688957) on Monday June 20, 2011 @05:54PM (#36506314)

    ARM manufacturers are idiots

    Intel gets open source, most ARM manufacturers don't.

    Hence, most BSP rely on proprietary drivers, they don't have up-to-date support for devices in the mainline kernel, etc

    Also, there's a lack of a 'standart platform', even though ARM is pretty much homogeneous

    Things are beginning to change, still. And ARM is still miles ahead from SH, embedded MIPS, etc

    • Re:Problem is simple (Score:5, Interesting)

      by Microlith (54737) on Monday June 20, 2011 @06:12PM (#36506478)

      most BSP rely on proprietary drivers

      Not true. Almost every device released today has full driver support in the kernel sources that are dropped. Userspace components notwithstanding, the kernels released are fully capable of supporting other OSes when recompiled (assuming the device will boot them.)

      What does happen, however, and I stated this elsewhere, is the drivers are released ONLY into those tarballs with no revision history, full of android-specific code and are never merged upstream into the kernel. This makes porting newer kernels to the device even harder, which you can see in the 2.6.36 and 2.6.37 changeup in how some sound drivers are structured. As a result, you've got tons of drivers for hardware sitting, and rotting, in obscure folders on corporate websites.

      And all this mess is before the schism created in the userspace by Android.

      • Re:Problem is simple (Score:4, Informative)

        by JamesP (688957) on Monday June 20, 2011 @06:18PM (#36506540)

        I wasn't talking about Android, but the point stands.

        If wireless controllers on Android devices don't depend on proprietary drivers, great! That's a start

        But try Hw accelerated video playback, 3D drivers, etc

        And some products absolutely depend on those. Think set-top-boxes, multimedia players, etc.

        • Re:Problem is simple (Score:5, Informative)

          by Microlith (54737) on Monday June 20, 2011 @06:25PM (#36506616)

          But try Hw accelerated video playback, 3D drivers, etc

          Working on MeeGo makes me all too keenly aware of that mess. None of it really applies to the kernel though, since all interesting bits are in userspace. And the graphics core IP vendors (Qualcomm most notably) have already been refused entry into the kernel because of this.

      • I totally agree with you, but would reword one point: this has been happening since before Android, but Android has easily made the situation much worse.

    • Re:Problem is simple (Score:4, Interesting)

      by serviscope_minor (664417) on Monday June 20, 2011 @06:22PM (#36506588) Journal

      Also, there's a lack of a 'standart platform', even though ARM is pretty much homogeneous.

      Kind of. Actually things are not that bad. There are a lot of SoCs out there which bundle an arm core with a few other cores (ethernet, usb, etc). There are actually staggeringly few vendors for the peripheral cores. The SoC vendors don't generally mention who the core vendor is, but they provide a datasheet and stick the core at some random place in the address space.

      As a result, there are a lot of reimplementations of the same drivers. This has been recognised and peopls are now trying to spot duplicate drivers and replace them with a asingle unified one.

      • Re:Problem is simple (Score:4, Interesting)

        by Sun (104778) <shachar@shemesh.biz> on Tuesday June 21, 2011 @01:25AM (#36509384) Homepage

        Here's my experience. I did a project for a company that were producing a SoC themselves. We were using the designware SPI peripheral. We wrote the driver ourselves (don't remember right now why - the dw_spi module was not for the right chip or something along these lines. I didn't do the original development).

        Turns out this chip doesn't have proper peripherals support. No NAND controller and no integrated MAC, so we use SPI for both persistent storage and for networking. Except the chip isn't fast enough to service the "SPI queue is almost empty" interrupt, despite the designware having a huge queue (256 bytes), and no matter how high we place the watermark, so we do some serious trickery in order to get things working (in essence - directing SPI chips select to a GPIO and manually controlling activation and deactivation). Poor SPI throughput. Worse, the driver is now unsubmittable, as it contains hacks which really only make sense to this particular chip.

        So I come along, and suggest to hook the SPI driver to the existing on board DMA controller. Get the whole buffers through without the CPU needing to do anything. A bit of hard work, and the DMA is working (not improving performance, but that's another story). Except neither the DMA infrastructure nor the actual hardware are generic enough to do such a thing so that I don't care which DMA controller is hooked into the SPI controller. So, more hacks. In theory, I could rework the infrastructure so that it is more generic, but that's a project that will cost (man hours) about as much as the original SPI driver rewrite.

        The project wound up being canceled, so things never progressed any further, but you can understand that none of that code was ever released. This is not due to the client's desire not to release. Search for Baruch Siach's contributions in the enc286 code for example of vanilla integrated code that were done on that client's dime and with their consent. It's just that there is a limit to how much time a company can authorize merely so that the code is generic enough to go into main.

        Shachar

        • by samjam (256347)

          Often the business case is reached long before the project maintainers standards are met.

          The result is that the code cannot be accepted into the project and often isn't even submitted.

        • by JamesP (688957)

          Yes

          If your customer doesn't know how to do proper hardware, it's difficult to do proper software

          Been there, done that (but not at a such low level of hw changes)

          Still, for example, SPI wouldn't work on PowerQUICC processors (good thing it wasn't essential to the project)

    • by bgat (123664)

      Anyone who thinks Intel "gets" open source has never worked with them on a technical level. I'm looking at you, Poulsbo.

      • by JamesP (688957)

        Poulsbo was not done by Intel, IIRC, it's 3rd part (PowerVR) IP (yes, I got bitten by that) - http://en.wikipedia.org/wiki/System_Controller_Hub [wikipedia.org]

        Apart from that, they are one of the least worse vendors.

        • by bgat (123664)

          Apart from that, they are one of the least worse vendors.

          Point is, any differences between the top-tier vendors where open source is concerned is mostly splitting hairs. :)

      • by Microlith (54737)

        Poulsbo is simply the same problem that every ARM device with a PowerVR graphics core has. The company refuses to release sources for the userspace driver, the kernel space stubs are not in the mainline, and they absolutely don't work with end-users.

    • Yeah and you Linux clowns are genius' - rrrrright....

  • by MatanZ (4571) on Monday June 20, 2011 @05:56PM (#36506336) Homepage

    The ARM vendors (TI, Samsung, etc.) do release their kernel changes. What they do not do is work with Linus and RMK on getting their code merged upstream. The GPL does not require that they do that.

    • The ARM vendors (TI, Samsung, etc.) do release their kernel changes. What they do not do is work with Linus and RMK on getting their code merged upstream. The GPL does not require that they do that.

      I think you're missing two points. One, except for the claim that some smaller ARM vendors might not be so diligently releasing their kernel source changes, the articles points out bigger vendors (presumably including TI, Samsung, etc) are complying properly with the GPL. Two, the article/blog was going out of

      • by Andy Dodd (701)

        Great clarification there.

        As far as really nasty offenders:

        Huawei is one of those "smaller" (at least in terms of the US market for mobile devices) vendors that isn't cooperative. They don't release kernel sources until legally threatened to do so.

        HTC seems to think it's OK to wait 30-90 days to release kernel sources.

    • Some employees try to upsteam some patches. They are free to do so since the vendors' trees are public anyway (you can go look at what the big boys are doing to get 3.0 running right now) Most would probably LOVE to see their stuff merged upstream (either to Linux mainline or at least to Google's) so that they don't have port patches to new kernel versions.

      A few reasons you don't see more upstreaming:
      - Some of the code is chip specific (errata, custom IP)
      - The market demands fa

  • AMEN (Score:5, Informative)

    by synthesizerpatel (1210598) on Monday June 20, 2011 @05:57PM (#36506338)

    Having worked on bring-up on three custom ARM projects, I can personally attest to how gnarly it can be. But it's not necessarily something that Linus will be able to fix, or the Linux kernel community at large.

    The main problem is the custom board support - even though the source code is GPL, they give you full source code and even submit it to back into the eco-system, it's just haphazard code that was pushed out the door too quickly. Linus can't stop people from writing bad kernel code, he can stop them from submitting it back into the mainline, but thats kind of what we have right now. If your code isn't up to snuff it doesn't make it into mainline. That doesn't stop them from shipping a product and giving that code to customers.

    In one case, the documentation for the ARM chip I recieved was a password protected PDF that you can't even cut text out of, describing how to use the features by writing your own device driver. In that case, they had minimal Linux support but for all the bells and whistles you had to do it yourself.

    The problem is as dense and layered as the chips themselves - what really needs to happen is a standardized method for publishing SoC features in a structured format (XML?) where common features (FIFO registers with a bytes_remaining field? Write only configuration registers, Read only configuration register.. etc) could be defined and the code could in many cases just be automatically generated.

    Need to set reg A to all f's, reg B to all zeros, flip bit 12 of reg C and then your PHY is configured - done.

    For more complex interlocking mechanisms that would be difficult or impossible to communicate in a cure-all DSL, but even if you could eliminate 80% of the problems that'd be great.

    Which brings me to the other problem - a lot of what you do to get ARM systems up and running happens way before you run Linux - in U-Boot/RedBoot or whatever else is out there.. And thats a whole other kettle of fish.

    • Re:AMEN (Score:5, Interesting)

      by bgat (123664) on Monday June 20, 2011 @07:08PM (#36507030) Homepage

      You think it's gnarly now? You should have seen it a couple of years ago! Things have improved by light-years since then.

      It's true that ARM isn't as cleanly supported as, say, x86. But the simple explanation is that there is significantly more diversity in the ARM world than in the x86 world, so comparisons between the two are a bit like comparing apples to orangutans.

      There are limits to what can be done to address the problem. I prefer having a diversity of ARM chips to having a BIOS--- and that would be the only way to tame this beast long-term. I think most platform developers (those who do both hardware and software) would agree with me: it's easier to port Linux to a good chip for your end application, than it is to use a less-than-ideal chip in the platform just because it has a mature Linux port. So while we should continue refactoring Linux on ARM, we should also accept that things will never be as clean as they are on x86. It isn't in anyone's best interest to even strive for that goal.

      In parallel with all of this, we must be careful not to kill the goose that lays the golden eggs. ARM is the singular reason why Linux owns the embedded space for 32-bit CPUs that run OSes. Nobody else is even close. So despite all of Linux's warts for ARM, it still works really, really, REALLY well. Vendors of ARM SoC's should recognize this, and pony up some funding to clean up the mess as an investment in their futures.

    • Re:AMEN (Score:4, Interesting)

      by JackDW (904211) on Monday June 20, 2011 @08:34PM (#36507840) Homepage

      I don't know if such a standard SoC description format could ever exist in a useful form. Anything even moderately complex would require driver code, not just descriptive data. Descriptions produced by vendors would inevitably be buggy, like ACPI data. This solution would probably just make the problem worse.

      It would be much better to simply standardise the SoC, so that every ARM system has the same basic elements. Just like a PC, where the interrupt controller and memory are always in the same place, and the timer always has the same register map.

      I assume that SoC vendors do not do this because (1) they don't need to, (2) they want to have "value-added" features like their own custom power management subsystem, and (3) the diversity makes it harder to use a different SoC as a drop-in replacement.

      But they should standardise. There's no advantage to the user, the OEM, or the OS developers in having so many different SoCs.

    • by eknagy (1056622)

      ... was a password protected PDF that you can't even cut text out of ...

      Not even after pdf2ps and ps2pdf? ;)

  • I don't know why people are still acting surprised by Microsoft's support for ARM processor. I had an old Garmin handheld that used an Intel PXA272 Xscale processor that ran WinMo 2003. Maybe full blown Windows isn't ARM friendly but Microsoft has supported ARM in the past.
     
    I guess some people refuse to look into this and see that Microsoft on ARM is nothing new.

    • The Windows Mobile and Windows NT kernel are not the same thing.
      • by nomadic (141991)
        They're not both made by Microsoft?
        • Yes. Just like the F-150, the Escape and the GT-40 are all made by Ford. Same thing, right?
          • by nomadic (141991)
            Huh? What does that have to do with anything? The point raised was, Microsoft has previously supported ARM architecture. Someone else then argues a nonsequitur that one Microsoft platform is not the same as another. If the point is, the COMPANY obviously has no problem with supporting the ARM architecture, merely pointing out that not everything the company builds supports it does not refute the central point. A more appropriate analogy would be, "why should you be surprised that Ford is using Part X
            • Perhaps because the summery and the world at large was talking about Windows, not Windows Mobile. Windows and the NT KErnel ahve never supported ARM.
              • Never publicly supported....

                Remember that - there were rumours of a maintained x86 version of OSX around for years before the PPC to Intel switch was ever seriously considered by anyone following Apple. And low and behold, Apple releases a fully workable x86 build very quickly.

                Its the same here - no public support, but for a major OS developer it makes sense to maintain a low key, low resource port just incase.

    • by CompMD (522020)

      Hey, cool, you're the guy who bought the iQue m5! You really made my day when I heard someone bought one.

      I have an iQue m5 and iQue 3600 in the "museum" on my desk. :)

  • by Anonymous Coward on Monday June 20, 2011 @06:06PM (#36506426)

    weak ARM support is very much related to the constantly moving target of ARM hardware. there are several series of ARM cpus in use today and as soon as one becomes commonplace, it is phased out in favor of a "cheaper and better" cpu, sometimes in the same series, sometimes not.

    this phenomenon is related to wireless providers having an economy of scale that doesn't make sense in an end-user context. for them, having a team of skilled programmers that cost > USD 10 mln / yr is nothing and they leverage the hell out of this fact. expect this sort of stuff to continue despite ARM cpus comprising the majority of cpus on the planet.

    • by bgat (123664)

      I wouldn't call it "weak" support, at all. Rather, it's a challenge to keep Linux abreast of the rapid pace of ARM development. Both on the CPU side, and on the platform side.

      Linux is an incredibly strong OS for ARM. If you want support for the bleeding edge CPUs and SoCs, it can be pretty painful--- but if you step back even a little from that edge, Linux is solid.

  • by Anonymous Coward on Monday June 20, 2011 @06:30PM (#36506656)

    >> a threat that could effect dozens of companies' livelihoods
    A lot of semiconductor companies were releasing linux-based SoCs way before the mainline kernel started consolidating code from vendors. If Linus stopped pulling ARM code, no business would shut down. I personally don't know any companies that rely on Linus' tree to ship their customers.

    >> To make matters worse, even though the GPL v2 license on the Linux kernel requires these changes to be released back upstream to the main Linux kernel, often they were not.
    This doesn't make any sense to me. GPL requires the changes to be released to the person who purchases your device/code. The vendors have zero responsibility to the mainline.

    >> ...this is entirely the reason why the non-profit Linaro consortium (...) was put together...
    One thing I wonder about Linaro is how they are going to be the leader and not play catch up. There are a lot of board-specific drivers they can consolidate, but as they consolidate, the vendors are coming out with even more.

    >> [a]s an indication of the scale of this problem, each new kernel release sees about 70,000 new lines of ARM code, whereas there's roughly 5,000 lines of new x86 code added."
    I find this comparison very unfair. Yes, that 70K number could be more like 20-25K but the devices with ARM processors have very different structures, designs, and end goals. One code can't fit them all. On the flip side, most x86 implementations are on either desktop or server side.

    I'm surprised Likely didn't talk about the device-tree support for the ARM tree. I've implemented a few (ppc-based) boards with device trees. The initial learning curve was a bit painful, but once you understand it, it enables a lot of common code and cuts down development time too. synthesizerpatel above mentioned "a standardized method for publishing SoC features in a structured format" above and the device trees are exactly it (except they're not XML! so, even better!)

    My preference as a lowly bring-up guy would be if the desktop/server kernel split up from the embedded kernel completely. Embedded kernel devs then can emphasize what's important to them (cut down development time, wide variety of device support, aggressive power mgmt) while the desktop/server devs can focus on their stuff.

    • by gcl (1070302)
      We did talk about device tree, but I didn't dwell on it because it is just another piece of the puzzle in cleaning up arm. Device tree support for ARM is now in mainline, and it is full steam ahead to adapt many of the current ARM subarchitectures.
  • by Suiggy (1544213) on Monday June 20, 2011 @07:32PM (#36507312)

    The Kernel isn't the only thing suffering from shoddy support. The ARM backend and code generator for GCC is suboptimal. The GCC __sync_* builtin functions for atomic memory access are unoptimized and call into kernel functions, which isn't always necessary, hopefully this will be fixed with the new C1x/C++0x atomics and memory model. And then the ARM NEON neon intrinsics/builtins implementation is in an absolutely horrendous state, I'm surprised NEON register allocator is even functional.

    I'd fix it myself, but then I'd have to spend 2 months learning how to make changes to GCC, and wait another 6 months for my patches to be accepted.

    • Oh, you forgot to mention about how sometimes char is unsigned on ARM as well as other bizarre anomalies. As someone who was tasked with making some very complex code very efficient on an ARM board I can personally tell you nobody is getting anywhere until the problems Suiggy pointed out and quite a few more are actually fixed in GCC/G++/Binutils.

    • But these days, we also have Clang. Is it any better as far as ARM code generation goes? Can it be realistically used instead of gcc?

      • by KiloByte (825081)

        At least as C++ goes, clang is a bad joke. It mostly works on C and ObjC, but then, it takes a week for a high school student with no former knowledge of grammar parsers to write a C compiler (ok, with hardly any optimizations...).

        That's the front end, but with clang back end being behind gcc as well on x86, I wouldn't hold my breath.

        • by KiloByte (825081)

          To provide some specific numbers on efficiency of ARM code:

          clang trunk 28.493s
          gcc 4.6 13.613s
          (clang 2.9 fails to compile the code in question at all)

          Pretty crushing, I'd say.

          • Wow, that's seriously bad. Why is Apple pushing for Clang as the default compiler on their systems (isn't it already the default in the most recent version of Xcode?) with numbers like that?

            Would also be interesting to see what VC++ can do on ARM.

            • by Suiggy (1544213)

              Because of GCC having adopted GPLv3 with GCC 4.3 and later. Apple doesn't want to use it. Clang/LLVM trunk is actually starting to get pretty decent at x86/x86-64, but ARM is still pretty crappy.

              Think is, Clang has come a very long way in just a couple of years, if they keep up the same pace of development, it will most likely surpass GCC either next year or the year after.

  • by gatkinso (15975) on Monday June 20, 2011 @07:52PM (#36507494)

    it is OSS in general with respect to ARM support.

    My God.... the state of Angstrom, BitBake, OpenEmbedded... while the maintainers are doing great work, they are not nearly as stable and mature as established architectures.

    Seriously, check out OpenEmbedded and try to roll that latest Gumstix omap-console-image. Count the number of things that are broken. It is a travesty!

    This kind of shite would NEVER be accepted in a mature x86 based project.

    • by MtHuurne (602934)

      This kind of shite would NEVER be accepted in a mature x86 based project.

      Many of the problems that embedded Linux projects have come from packages that are mature on x86 but not on embedded architectures. One reason is that x86 is not strict about memory alignment, but on for example PPC or MIPS (I don't know about ARM) you get hit by a SIGBUS if you break alignment rules. But the main issue is that cross compilation is broken on many packages.

      Some try to run the compiled binaries as part of the build process. Some detect cross compilation and then use different implementations

      • by daid303 (843777)

        You sir, hit the nail on the head.

        The kernel is fine. Yes, patches are lacking behind a bit, but in general they are stable and working great.

        On userland it's a whole different level. Automake just needs to die, it's like the 7th circle of hell for cross compilation. I need to build for x86, ARM and PPC (ppc860 in my case) and getting configure to cooperate is just a disaster every time. Anyone who says otherwise should try to cross compile "tcpdump" with configure. Which I never got to build with configure

        • by KiloByte (825081)

          Have you tried compiling cross-compiling projects that use autotools compared to those with hand-written makefiles? There's a world of difference in favour of the former. Unless there are some utterly broken m4 macros mispasted by retarded monkeys (which, I admit, happen way too often), it just works.

          Hand-written makefiles on the other hand support only systems they were explicitely ported to, with inevitable bit rot setting in, and if cross compilation happens to work, it's only for a certain pair of hos

  • by mug funky (910186) on Monday June 20, 2011 @07:55PM (#36507522)

    i know this article is slightly trollish, but it did make me wonder.

    what's going to happen when Linus finally retires? will there be strong enough leadership in the ensuing vacuum? in spite of open-source philosophy, will linux remain Linus' brainchild?

    • by Kjella (173770)

      There's only one benevolent dictator for life, at that is Linus. One of the subsystem maintainers would probably take over the role as project leader after a vote, but he'd not have nearly the same authority. I don't see the project overall going anywhere though, the entire structure, team and commercial backing would still be there - would Windows or OS X collapse if the lead guy disappeared? By the GPL there's not many other ways to run it than as open source collaboration.

      The only thing I see is an incre

    • by NullProg (70833)
      what's going to happen when Linus finally retires?

      I hear he will live on as a kernel module.

      insmod linus

      insmod: can't read 'linus':

  • My suggestion from well before kernel modules era. One thing that is needed for the Linux, is a seperation of the drivers from the base of Linux. To do this, create a standard interface for the kernel and drivers to communicate. Probably through the use of Stub drivers in the kernel. This would allow some level of standardization of the kernels, would allow for 3rd party drivers (I know I will get killed for this one), and more importantly is that urgent fixes in drivers could be pushed before the kern
    • by Alex Belits (437) *

      Oh, for fuck sake!

      Driver interface does exist. It just isn't, and is not supposed to cater to proprietary drivers that exist only in the form of binary blobs. Not that it prevent such things from being developed whenever a hardware vendor wants to do so (by adding a driver-specific wrapper), but it's a stupid idea and this is why it is not done.

      This has also absolutely nothing to do with ARM -- with ARM most problems are related to configuration that Linux reads or detects at initialization or somehow store

  • There are basically three x86 processor manufacturers. The two smaller players work hard to stay compatible because their livelihood depends on it. Most of the interface functionality is off-chip.

    There are many well-known ARM processor licensees. They all strive to differentiate their product offerings. In the majority of cases all of the major peripherals (which are one of the primary opportunities for differentiation) are on-chip.

    As such, where minimizing differences by processor was clean and relativ

    • by gcl (1070302)

      As such, where minimizing differences by processor was clean and relatively straightforward for x86, expecting it to continue to work well for ARM is nonsensical. I really think Linus is missing the forest on this one.

      Not true. Pretty much all of us maintainers agree that the duplication of code and infrastructure in arch/arm is ridiculous. It has to be fixed, and we're actively working on it. Linus was perfectly correct in his statements.

  • I worked with Marvell extensively in Linux and while it was good all driver support was in their own patch which seemed to not integrate well with the kernel. Parts of it conflicted so I had to ensure certain standard features were turned off. Even userland tools were changed. I had to use a custom version of U-Boot tools. It was a mess. What they don't understand is that they end up spending more time maintaining their patches then just getting them right and submitting them upstream.
  • Power-efficient ARM processors are moving up the food chain

    They've changed from herbivores into carnivores?

    It was always a retarded expression, but I hoped it had become extinct. I am disappointed to a literally exponential degree.

  • Windows 8 is supposed to be moving to support ARM and now there's an article suggesting that Linux support for ARM is a buggy collection of hacks and tricks. I'd bet money we'll see more of the same from wide ranging sources. When the Muellertroll writes about it we'll know for certain where it's coming from.

    Makes you wonder if you should be paranoid or not ?

  • RISC is old news. Linux was ported many RISC platforms. Anything names advanced makes me wounder if it is advanced. When ARM is ready for real computing it will be ported to real computers.

    It's is only news for Windows not Linux. Since ARM will break all backwards compatibility. I wounder if there will be a VM compatibility option? Probably not since Steve Ballmer wants you to buy all new software. We can only hope this will be the final nail in the coffin for Windows.
  • I have a couple of GuruPlugs and OpenRD's. One of the openRD's is my NAS/dhcp/etc server. I just spent a week trying to get a newer u-boot to boot from USB with some consistency. Probably 3/4 of my time was spent fighting to get a u-boot and linux build that worked properly together. The remaining 1/4 was actually fixing the problem.

    The list of sins on these devices is _VERY_ long. A partial list includes:

    • First they are sold as "open" devices but the PDF from Marvell just looks like it contains the tech

Systems programmers are the high priests of a low cult. -- R.S. Barton

Working...