SpaceX: Lessons Learned Developing Software For Space Vehicles 160
Posted
by
timothy
from the no-one-can-hear-you-bleep-in-space dept.
from the no-one-can-hear-you-bleep-in-space dept.
jrepin writes "On day two of the 2013 Embedded Linux Conference, Robert Rose of SpaceX spoke about the 'Lessons Learned Developing Software for Space Vehicles.' In his talk, he discussed how SpaceX develops its Linux-based software for a wide variety of tasks needed to put spacecraft into orbit—and eventually beyond. Linux runs everywhere at SpaceX, he said, on everything from desktops to spacecraft."
VxWorks? (Score:5, Insightful)
I thought that for the Falcon rocket and Dragon capsule, SpaceX use the VxWorks realtime OS made by Wind River.
Is the market for spacecraft programming expected to grow significantly in the coming decade(s)?
Re: (Score:1)
Is the market for spacecraft programming expected to grow significantly in the coming decade(s)?
Yes.
Re: (Score:3)
Even if it went up an order of magnitude... you're still only talking a few thousand jobs. Not something I'd want to build a career plan on.
Re: (Score:2)
Even if it went up an order of magnitude... you're still only talking a few thousand jobs. Not something I'd want to build a career plan on.
Derek, you, myself and others have been closely following space for sometime. You know that if Bigelow gets going, and SpaceX really gets launch cheap, then we are going to the moon and mars SOON. And yes, there will be LOADS of coding needed. Nobody will want something like Windows creating the blue screams of death. They will want an OS that works. Period.
Re: (Score:2)
Maybe, probably not. When launch gets cheap, [non professional] people will finally realize what professionals knew all along - vehicles and equipment designed to survive and operate in extreme and unusual environments are inherently expensive. (There's a reason why MIL-SPEC gear is more expensive than consumer g
Re: (Score:2)
BTW, I am not certain how old you are, but, I was using the internet back in t
Re: (Score:2)
The Shuttle recycled the most expensive part - and that was the *wrong* part? The mind boggles at the amount of doublethink required to reach that conclusion.
Re: (Score:3)
Due to miniaturization of sensors, solar panels, CPUs and communication devices (lasers and/or RF), you will be able to do massively useful stuff at something like 50k dollar of cost. For example, you can shoot up microsats just for a single sailing sports event and do some nifty communications with that. Iridium-style messaging will soon be possible for the "little" guy and his 100 million dollars, if you just want texting to be done with it.
Re: (Score:3)
Re: (Score:2)
vxworks does sell a hard realtime linux - maybe they're using that. Technically linux runs on the vx microkernel/hypervisor thingy.
You get to deal with linux but then have a few more knobs to tell it how to behave.
Re: (Score:3)
VxWorks is an operating system, not a company. Wind River, who make VxWorks, sell Wind River Linux.
Re: (Score:2)
hard real time [lmgtfy.com]
Re: (Score:2)
Re: (Score:1)
I thought that for the Falcon rocket and Dragon capsule, SpaceX use the VxWorks realtime OS made by Wind River.
No, they don't use it at all. RFTA.
Re: (Score:3)
Or don't RTFA, and instead actually do some research first.
http://www.spacex.com/downloads/dragonlab-datasheet.pdf [spacex.com]
Search for "Flight Software" (the most mission critical RT bit, of course).
Re: (Score:2)
So, which is more accurate, do you think? A 3+ year old datasheet for a spacecraft that isn't even scheduled to fly for the first time until 2014 (first DragonLab mission), or recent statements by the person IN CHARGE of the flight software?
Your research is flawed.
Re: (Score:2)
No, it's not. Ok, here's more evidence.
http://www.spacex.com/careers.php?jvi=oe2HWfwD,Job [spacex.com]
"Our Flight Software Group currently has opportunities developing software for embedded flight hardware using Linux and VxWorks as well as ground simulation software using Linux. "
They are HIRING people with VxWorks experience and say they use it in their current job descriptions.
They use Linux, obviously. But they also use VxWorks for some mission critical RT applications. You said "they don't use it at all". You ar
Re: (Score:2)
Nope, try again.
Re: (Score:2)
when your cpu isn't able to help out, the fastest way to catch landing in garbage is to place a few nops (to get realigned) and then jmp to the Big Bomb before each block of code. then if something branches wildly or otherwise escapes its block or into a buffer somewhere, it'll eventually plow into the next of those traps and stop itself.
Re: (Score:1)
we don't actually want it to "stop itse;f" as much as we want it to either "gracefully recover" or "gracefully die". For our UAV, when it jumps to somewhere it shouldn't be, it goes to a piece of code that determines whether or not it was in a flight safety critical loop. If it wasn't, it gives up control for the rest of that cycle and resyncs with the other flight computers on the next 5Hz cycle. If it was in a flight safety critical loop, it sends the "I'm brainfucked and dying" message to the other proce
Re:Holy Shit! (Score:5, Funny)
malloc() and new() are non-deterministic in many ways and therefore to be banned in anything truely real-time.
Don't worry. We now have garbage collected languages where we don't need malloc/free any longer. :-)
Garbage (Score:2, Interesting)
Now THAT is hilarious. Complaining that malloc is non-deterministic and then alluding to dependence upon garbage collection. I get the distinct impression you've never written anything requiring high performance memory allocation/deallocation.
Re: (Score:1)
Now THAT is hilarious. Complaining that malloc is non-deterministic and then alluding to dependence upon garbage collection. I get the distinct impression you've never written anything requiring high performance memory allocation/deallocation.
I am pretty sure, what you have encountered is a real (as in not like in Trollface comic [knowyourmeme.com]) instance of trolling. Therefore:
YHBT [catb.org].
YHL.
HAND.
Re: (Score:2)
Since when do troll posts end in smilies?
Re: (Score:1)
Since the very beginning of the tradition of trolling?
Re: (Score:2)
Since when do troll posts end in smilies?
Er...? It could be interpreted that the emiley meant you were happy about having GC languages...
Anyway, trolling has a bad rap now (especially after abuse of the term by the media). There's nothing wrong with a good troll and it looks like you caught a big one.
Re: (Score:3, Informative)
You might have noticed the three characters at the very end of my post. A colon, followed by a hyphen, followed by a closing parenthesis. You might want to inform yourself about the meaning of this letter combination. Here's a hint: It is known as smiley. Google for it. You might get enlightened.
See also: Whoosh.
Re: (Score:2)
As I said, I found it hilarious. The smiley was insufficient to its task, though. You could just as easily been happy about garbage collection. A lot of people are -- why do you think it's infested so many languages?
Re: (Score:3)
They are only non-deterministic if you don't know what you're doing.
Re: (Score:2)
Re: (Score:2)
Dude, 70ies are over, you are not writing software on a teletype. There is nothing wrong with verbosity.
And no, the languages you have listed are not particularily difficult to write code in, the only difficult thing to do is writing ugly hacks.
Re: (Score:2)
Oh, no generic types in Pascal so it's impossible. Ok, try at least a polymorphic version - again not easy because of braindeadness of pointers in Pascal. BTW, there's no garbage collector in Standard Pascal.
Advocating braindead languages from 60-s is not even funny now.
Re: (Score:2)
Well, standard Pascal is obsolete, no argument about it. But that is true for any language from the sixties.
By the way, if I remember correctly, there are generic types in modern Pascal dialects like Delphi or Free Pascal.
Re: (Score:3)
Re: (Score:2)
Dude, if the language is Turing-complete you can write anything in anything. If you can't think of a way, that's your failure of imagination, not a failure of the language.
Hell, the Apollo spacecraft were programmed in the languages of the 60s ... if not the 50s.
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Sorry to burst your bubble, but some of Pascal's syntactic features were conceived by people who demonstrably had no idea how people -- the software developers -- actually process information. You see, programming languages are first and foremost tools for people. The code must be readable long after it has been written. Pascal's arbitrary separation of variable declarations from the first point-of-use is positively unergnomic and runs against the need for locality-of-reference as an aid to understanding. T
Re: (Score:2)
Pascal was designed to be (a) easy to teach and (b) easy to compile (not necessarily in that order).
Declaration at point of first use can break programs in block-scoped languages like Pascal or Algol. It's easier to teach a newbie to just declare everything in the outermost block (of a given procedure/function) than worry about whether something will still be in scope a few 'end' delimiters later.
If your procedures are of reasonable length (no more than a few dozen LOC), that's locality of reference enough
Seems straightforward (Score:3, Insightful)
All of you eggs, meet your basket. (Score:5, Insightful)
I started my career in nuclear engineering before moving into software development.
There were three really important principles: Redundancy (having several of everything); Diversity (having different implementations i.e. different designs from different manufacturers) and Segregation (keeping things physically separate and firewalled off from each other).
I'm a bigger Linux fan than many here. I've been using it since 1995 and I'm a die-hard Slackware user, but having everything running on the same OS seems like an accident waiting to happen. Yes, I know that it's great that you can have one piece of code that you can compile and run anywhere, and that's easier if you're only using one OS.
However, one of the great things about Open Standards and Open Source was (is) that for many years software was portable so that it could be compiled and run on big- or little-endian 32- and 64-bit POSIX-like systems on a wide variety of CPU architectures.
That may have been "expensive" in terms of software maintenance, but as I learned when working for a now-defunct very large UNIX company, writing your software to be portable across those systems exposes (and forces you to fix) many subtle bugs that otherwise would not have been found until deployment.
Also, relying on just one OS puts you at the mercy of any latent bug in that specific system. Having a diversity of OSes in use mitigates that problem.
The state of Software Engineering in general is still pretty primitive. I'm still amazed at the poor quality of a lot of "professional" code and the cavalier attitude towards testing...In the land of the blind, the one-eyed man is king.
Re: (Score:1)
All of what you say is done at Airbus. They generally have zero issues with software, except in rare cases where the specification is already less than perfect. The same is with NASA's shuttle software. I assume you can say the same with Boeing 777s and 787s.
Software != Software
In beancounting, software quality matters remotely. In avionics, a lack of quality will surely kill.
Re: (Score:1)
Bean counters will be angry if your software doesn't calculate and round up money correctly.
Re: (Score:2)
Re: (Score:1)
Well, I guess they'll just run your salary through your miscalculating application, or your tax, depending on to which side the errors go (salary: too low, tax: too high).
Re: (Score:2)
China AIrlines Flight 140. Software fighting humans. Software wins.
Re: (Score:2)
That wasn't a case of software fighting humans, that was an issue of humans doing something incorrectly and not correcting properly - all they had to do was disconnect the autopilot and they would have had full command authority, but they didn't.
Funny that - they gave the autopilot full command authority and then tried to fight it. And failed.
Re: (Score:1)
Airbus is well known for their blue screams of death. Just look at AF-400. There was NO feedback on the stick so the senior pilot did not know that the international pilot with minimal experience was pulling back on the stick.
And,
Re: (Score:2)
Which was a pilot error.
The captain had only slept a couple of hours, the two co-pilots had been out partying, doing drugs and alcohol the evening before.
Re: (Score:2)
Which was not a software error, as was aluded to by the post I replied to.
The software itself warned that the readings were incorrect.
Re: (Score:1)
JPL agrees with you:
lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf
LOC-1: Language Compliance
Rule 1 (language)
All C code shall conform to the ISO/IEC 9899-1999(E) standard for the C
programming language, with no reliance on undefined or unspecified
behavior. [MISRA-C:2004 Rule 1.1, 1.2]
The purpose of this rule is to make sure that all mission critical code can be compiled
with any language compliant compiler, can be analyzed by a broad range of tools, and
can be understood, debugged, tested, and maintained by any competent C programmer. It
ensures that there is no hidden reliance on compiler or platform specific behavior that
may jeopardize portability or code reuse. The rule prohibits straying outside the language
definition, and forbids reliance of undefined or unspecified behavior.
The other rules in the JPL coding guidelines are also interesting.
Re: (Score:2)
Yeah, this is something I try to hammer into the students I occassionally mentor:
Do not write your code to be written for a specific compiler(Linux and its GCC'isms being a specific example) or even a hard reliance on a library, if it can be avoided. Oh, sure, you can't build a program like a kid builds with Duplo Lego in quite the same way any more, but when done properly it's more likely to be solid.
Re: (Score:2)
That quote you show is a bit like a sign reading "when smoking at a gas station, make sure no hot ashes touch the ground".
Re: (Score:2)
The reason they insist on C is for two reasons:
(1) It's more portable than most of those safer languages. When every design you build is custom hardware, you want to have easy code reuse for standard functionality and signal processing.
(2) VxWorks (not sure about 6xx, but previous versions this really stands) is built to be a front-end for C - it adds a multi-tasking thin OS (limited memory protection), debugging and logging tools, and that's about it. You get a marginal Tornado IDE supporting C and C++,
Re: (Score:2)
If you are writing stuff for a spacecraft, why not spend some cash in a decent computer? You are spending a megabuck on hardware, but have to live with 1k of stack? Sounds like bad engineering to me.
Re: (Score:3)
Having a diversity of OSes puts you at the mercy of random and subtle bugs in one OS that require customizing either the OS or the code. (Which customization carriers further hazards of it's own.) Relying on one OS mitigates that problem.
TANSTAAFL.
Re: (Score:2)
Or crossing that OS off the list. Or having the bug fixed.
Also: Automotive Radar/Breaking (Score:1)
I work for a major European, high-end automotive company and we use Linux for radar applications. Now that this works nicely, we can do stuff like braking automagically (you still need to slightly tip the pedal) with the lowest possible deceleration which will assure you don't hit the vehicle in front of you.
That means, even if you are trailed by a somewhat sleepy person that car has the maximum reaction time to also hit the brakes.
The excellent quality of Linux now shows in all sorts of applications from a
Re: (Score:2)
imagine you would have to use Qt instead of MFC, how bad would that be
LOL!
Why mention Linux? (Score:1, Insightful)
Linux is mentioned twice in the summary. Is there a reason why? We all know Linux has major use in the embedded and scientific world, this isn't a secret. Are people still desperate to mention Linux anywhere as some form of validation that they chose the right decision to stick with it or something?
I thought Linux had "won", or something. If that was the case then it shouldn't be necessary to circlejerk the name anymore.
Re:Why mention Linux? (Score:5, Funny)
"Linux is mentioned twice in the summary. Is there a reason why?"
2013 will be the year of Linux on spaceships.
Re: (Score:1)
Actually hard realtime Linux exists for a while -- just in different forms.
RTAI
Xenomai
PREEMPT_RT
There are some hardware architectures (actually one very popular hardware architecture) that usually have unpredictable crap running in background through OS-inaccessible interrupts, but that's the problem with irresponsible BIOS vendors, and it breaks realtime on all operating systems. Once that crap is disabled, even that architecture allows hard realtime -- I participated in a project that used just that.
Re: (Score:1)
That was actually very easy to believe in 1993. What was hard to believe is that THERE WILL BE SO FEW SPACECHIPS IN 2013, dammit!!!
Re: (Score:2)
What was hard to believe is that THERE WILL BE SO FEW SPACECHIPS IN 2013, dammit!!!
The salt and crumbs tend to get into the controls. Also, you don't even want to talk about space salsa.
Re: (Score:2)
Keeping score much?
Re: (Score:2)
Re: (Score:1)
SpaceX is "Linux, Linux, Linux", he said.
It's a nice word, can't say it too many times.
SpringX (Score:1)
[snarktag]The solution is clear, keep parameters in an XML file. Use base class reference objects and inject the proper objects at runtime
Re:Max Length (Score:5, Informative)
Okay, somebody ban this guy...
"You must be new here".
Do you actually believe that trolls are "banned" at Slashdot?
That's what the moderation system is for.
Slashdot is not like other "forums" in that it is *not* "moderated" by "super users", but rather regular users like you and I who are occasionally gifted with "mod points".
The "offending" post is never removed, it is just pushed below most users viewing threshold.
Seriously, "ban this guy"? You *MUST* be new here...
Mod System: Down == Disagree, Up == Agree (Score:2, Interesting)
No, no. You must be new here. The moderation system is there so you can impose your opinion on otherwise reasonable posts. That's what it's used for, that's what it's best at, Slashdot refuses to change it (I'd say fix it, but it appears to be working as intended) and so... one must conclude that is what it was designed for.
Re: (Score:2, Offtopic)
The moderation system is there so you can impose your opinion on otherwise reasonable posts. That's what it's used for, that's what it's best at, Slashdot refuses to change it (I'd say fix it, but it appears to be working as intended)
Oh, woe is you.
The slashdot system is so bad it must be the worst [*] site on the internet. Oh so sad. :( :( :( :( :(
And the editors refuse to fix it because they're so biased. I'll bet they ignored all of your brilliant suggestions.
Oh the humanity. :( :( :( :( :(
So what is yo
Re: (Score:2)
haha.
Well, there is a certain delicious irony in my defense of the moderation system getting modded down. :)
Re: (Score:2)
A moderator totally disagreed with you, or perhaps your tone. Welcome to the machine. :)
Now aren't you glad I read at -1? I saw your post despite the moderation: not because of it.
Re: (Score:2)
I would suggest a mod system where everyone can moderate, all the time. Mod up and mod down are supported, and tracked separately (thumbs up, thumbs down.) Each moderation is signed; if I had modded a post, you could both see my id, and which way I modded it. You can, at that time, check a box that says, essentially, "don't listen to this guy." After you did that, my mods would no longer count in scoring posts for your review.
In this way, you could tune your experience for moderation by people who earn your
Re: (Score:2)
Well OK.
Almost everyonw whining about slashdot has absoloutely nothing of value to add.
It seems you are an exception.
Re: (Score:1)
You're right of course, but I'm getting REALLY over the same guy spamming with the same post in so many threads. At SOME point, getting everything you post instantly modded to -1 (especially when the content is near identical) should incur some greater punishment.
Re: (Score:2)
Okay, somebody ban this guy, or if you can't do that, then impose a maximum length restriction for postings.
As much as the guy annoys me this is the prime example of where free speech is important and basically he has the freedom of speech to communicate that he is a jerk. Now his message might be very important however, whatever message this fellow has is lost on annoying the audience because their mental filters kick in and block his message. I'm actually amazed at the energy he puts into the post, oooppps, he OR she.
I cannot stand what this guy is going on about, but I will defend his right to say it even if
Where is NASA ? (Score:5, Insightful)
NASA sent out spaceships to moons and brought them back to earth, back in the 1960's
They did that without using Linux, or Unix, or any type of "nixes"
If there is a real need to learn a "lesson", methinks the best lessons we can learn are from those who develop original programs for NASA
Nobody else even comes close
Re:Where is NASA ? (Score:5, Interesting)
And they spent years, millions of dollars and thousands of man-hours doing so. I'm sure that the folks at NASA are pretty happy with modern toolsets. My father worked on the Saturn V instrumentation - Op Amps the size of cigarette boxes, telemetry transmitters the size of breadboxes with 300 baud max speeds. Graph paper. Slide rules. Simple changes requiring weeks of rework.
Linux and associated bits and pieces are a big step in the right direction.
Learning from NASA's experience ... (Score:4, Insightful)
Back in those days, the problem with "random flip bit", brought on by space radiation was already present
Apparently NASA has successfully dealt with that phenomenon - or a lot of NASA's spacecrafts would have spun out of control
Since NASA has decades of experiences dealing with fascinating problems like that it's UTTER FOOLISHNESS if we do not learn from NASA
Re:Learning from NASA's experience ... (Score:5, Informative)
And obviously you think -- I find it a common misconception -- that SpaceX is reengineering everything from scratch, including the engineering process itself. Well, here's a wakeup call for you: they employ plenty of people with lots of legacy space mission experience. The choice of the kernel is a minor thing in the grander scheme of things.
Re:Obligatory... (Score:2)
So now we've got rockets that run Linux... I'm shocked, shocked! that no one has stooped low enough to say this yet, so let me be the first to stoop...
Can you imagine having a Beowulf Cluster of THESE?! [spacex.com]
Re:Actually they employ a lot of people (Score:4, Insightful)
They have to, the pool of experienced aerospace people is small and shrinking due to old age. There is only one way to produce more people with experience in aerospace and it does not involve H1-B or wishing really hard.
Re: (Score:2)
Just hook everything up to a Android phone and you are set, because it runs Linux. And Java.
What could go wrong?
Re:Learning from NASA's experience ... (Score:4, Informative)
Back in those days, the problem with "random flip bit", brought on by space radiation was already present
Apparently NASA has successfully dealt with that phenomenon - or a lot of NASA's spacecrafts would have spun out of control
Except that the problem with "random bit flips" was relatively minor, because:
1) The AGC CPU's were much larger and less vulnerable to high energy particles, using basically the first generation of Intel NOR gate circuits (I believe there were two NOR gates in a single IC package back then) with large feature size.
2) The RWM was realized in form of a matrix of magnetic cores, which are completely unaffected by high energy particles,
3) The ROM was realized in form of "ropes" of huge magnetic cores (which, again, are completely unaffected by high energy particles) threaded with wires in an elaborate way.
The actual threading of the wires was done by elderly women with skilled hands, according to a translation of the machine code to instructions for the women as to how to thread the wires. I guess you can imagine the turnaround rate. Punch cards are a lightning-fast way of programming computers, compared to this.
Given the extent to which active feedback control has replaced a lot of hardware in modern high technology, and how everything is so much more self-diagnostic, self-tuning, and remotely monitored these days (look at car ECUs, for example), I *seriously* hope you're not trying to say that we should build and program a modern spacecraft control computer this way.
We have different electronic technology nowadays, and we have different manufacturing technology to build the electronic components, and we have to use completely different approaches to deal with different problems (or rather, different solutions for the old problems) that this brings.
You sound like the people who keep asking "Why don't we just build the Saturn V again?" The answer is still the same: "We can't, even if we had all the plans, Saturn V was designed to be built by hand with power tools and toolings that nobody uses anymore, we have to design a CNC-buildable rocket and it's simply going to be different." In the case of electronics, the answer is analogical, we have to re-solve the old problems in terms of new electronic components.
Mod this up (Score:2)
Re: (Score:2)
Or we can accept that things have improved and go from there.
If you accept the later, than it makes good sense to use an OS that can have DO-178C ratings amongst other issues.
Re: (Score:1)
Re: (Score:2)
Stack-based appreciation is frowned upon. You will express your feelings in algebraic terms or be severely modded down.
Re:Cruel and unusual punishment (Score:5, Interesting)
In his team, they have a full-size Justin Bieber cutout that gets placed facing the team member who broke the build. They found that "100% of software engineers don't like Justin Bieber", and will work quickly to fix the build problem.
You see, that's why you have overflowing prisons. This would easily reduce the crime rate by a factor of ten!
Doubtful. At my $lastjob we had a rule that if you broke the nightly build you bought doughnuts for everyone. And the project lead would rip you a new one.
Despite my admonitions to not check stuff in at the end of the day we had two guys that just couldn't figure it out. One of them worked in St. Petersburg (Russia, not Florida) and he'd check stuff in at the end of his day and go home, meaning we'd be stuck with the dirty job of backing his stuff out so that we could proceed.
And the local guy would whine and cry about how it wasn't his fault, it worked in his tree, yada yada yada. Well, his tree was usually a few days out of date by the time he was ready to check his stuff in, and he just couldn't get the knack of rebasing his tree and building before committing to the master. Sheesh. This stuff isn't rocket science. And as I said, he insisted on doing this at the end of the day – every time. Eventually it cost him his job.
So no, I don't believe the threat of being stared at by a full size cutout of the Biebs would solve crime either.
Re:Cruel and unusual punishment (Score:5, Funny)
So no, I don't believe the threat of being stared at by a full size cutout of the Biebs would solve crime either.
Then maybe you just need to up the ante. How about have the real Bieber sitting naked on the edge of his desk seductively blowing him kisses?
Re: (Score:2)
Why the fuck wasn't the build done automatically before a commit would go through? I mean, what the heck?
Re: (Score:1)
Or both?
Re:Emacs (Score:5, Funny)
Re: (Score:1)
Sledgehammer random components until it works?
Re: (Score:1)
T.A.R.D.I.S: the only hardware where bogosort is the most optimal sorting algorithm.