How do automotive firmware engineers do their jobs?
April 9, 2011 12:00 PM Subscribe
Are you an automotive engineer? Do you design engine control module firmware? Do you know or love someone who does? How is that firmware actually written?
I'm interested in learning how automotive engineers actually perform development when they're creating the firmware that runs on a car's ECU. I've got a background in software development and am familiar with drafting specifications and turning them into a delivered product. I've also got some background in embedded systems programming and a know a bit about controls. That is, I know enough to know just how complex something like a car's ECM can be given the huge number of variables involved.
So how do they do it? I can't imagine that there's a team of programmers sitting around hacking Erlang or C... right? I've played with MATLAB and Simulink, and know that you can somehow turn rigorous simulations into code but my impression is that it's not easy or straightforward. Is that what they do? What tools and techniques are the current state of the art at somewhere like Siemens or Bosch or Toyota?
I'm interested in learning how automotive engineers actually perform development when they're creating the firmware that runs on a car's ECU. I've got a background in software development and am familiar with drafting specifications and turning them into a delivered product. I've also got some background in embedded systems programming and a know a bit about controls. That is, I know enough to know just how complex something like a car's ECM can be given the huge number of variables involved.
So how do they do it? I can't imagine that there's a team of programmers sitting around hacking Erlang or C... right? I've played with MATLAB and Simulink, and know that you can somehow turn rigorous simulations into code but my impression is that it's not easy or straightforward. Is that what they do? What tools and techniques are the current state of the art at somewhere like Siemens or Bosch or Toyota?
Odds are, they use some sort of Hardware Description Language to specify and simulate the ECU itself.
This is really just a guess. I have no experience in this arena. Odds also are that there are proprietary systems and languages developed for this exact purpose. However, if I was designing a complicated embedded system, I'd probably prototype it on an FPGA before manufacturing it into silicon.
Xilinx, a popular FPGA maker used to produce some ECU dev kits, which suggests that my guess is not completely outlandish.
posted by schmod at 12:52 PM on April 9, 2011
This is really just a guess. I have no experience in this arena. Odds also are that there are proprietary systems and languages developed for this exact purpose. However, if I was designing a complicated embedded system, I'd probably prototype it on an FPGA before manufacturing it into silicon.
Xilinx, a popular FPGA maker used to produce some ECU dev kits, which suggests that my guess is not completely outlandish.
posted by schmod at 12:52 PM on April 9, 2011
Response by poster: zsazsa, Tesla is, imo, an interesting edge case - I can imagine the controls for a Tesla powertrain to be quite a bit simpler than for an internal combustion engine. DC motors are pretty simple, after all, even if they are very, very high power. Modern ICEs have all sorts of dynamic inputs and outputs (cam timing, valve lift, spark timing, dwell, etc) that massively complicate things. I do think the MISRA link is really interesting, though, and I'll certainly dig into that further.
posted by TheNewWazoo at 12:53 PM on April 9, 2011
posted by TheNewWazoo at 12:53 PM on April 9, 2011
I've been tempted to pitch an O'Reilly-esque book for 'Hacking your car' after playing around with the VAG folks years ago (bet liability would kill it) but share a number of the same questions you ask- but overall, basically just EEPROMs and microcontrollers, and have become pretty easy to access at a relatively high level with commodity hw/sw.
It would be interesting to chat up the aftermarket tuner community... i.e. the folks who can re-flash your ECU to remap a/f, turbo psi etc., and I feel the homebrew OBDII exploiting crowd have been doing some very interesting things as well.
For example:
- Exploring your 1.8t
- Defcon:Hack your car for Boost and Power
Also, some good research papers are starting to emerge in response to 'cyber' fears (I blame that terrible Diane Lane movie)
- Car Hacking Fears Go Wireless
Cheers!
posted by oldefortran at 12:54 PM on April 9, 2011 [2 favorites]
It would be interesting to chat up the aftermarket tuner community... i.e. the folks who can re-flash your ECU to remap a/f, turbo psi etc., and I feel the homebrew OBDII exploiting crowd have been doing some very interesting things as well.
For example:
- Exploring your 1.8t
- Defcon:Hack your car for Boost and Power
Also, some good research papers are starting to emerge in response to 'cyber' fears (I blame that terrible Diane Lane movie)
- Car Hacking Fears Go Wireless
Cheers!
posted by oldefortran at 12:54 PM on April 9, 2011 [2 favorites]
Response by poster: I've done a fair bit of aftermarket hacking of various CPUs, and to respond to schmod, most modern ECMs are based on standard general-purpose microcontrollers. I don't think I've ever seen anything based on an FPGA or CPLD or an ASIC. Hacking on an existing system is usually pretty easy, especially the VAGCOM stuff, which is basically just finding where the various maps live in memory and poking new values into them. I'd expect some sort of HDL too, but... how do you turn that into, for example, 68HC12 code?
That's more what I'm wondering. How do the pros do it in between specification and firmware upload?
posted by TheNewWazoo at 12:59 PM on April 9, 2011
That's more what I'm wondering. How do the pros do it in between specification and firmware upload?
posted by TheNewWazoo at 12:59 PM on April 9, 2011
Firmware, in general, gets made in lots of quality levels. ECM is one of the higher ones. I'd be surprised beyond belief that any transportation motor control code was made in anything but the most formal manner... standard development procedures, software specs, hardware specs, implementation specs, test plans, qualification testing, yada yada ... the whole banana. Also, I'd expect super slow change cycles and lots of configuration control stuff of the type you'd see in military products and man-rated hardware.
I have written code for ELECTRIC motor controls used in a mobility vehicle. It was very informal, though. I developed it in assembly language on an 8-bit superchip (for the time). I've worked on pieces of remote vehicle software/firmware, and seen what "informal" can get you first hand. The mobility controller is still in production, and was pretty cool for its time (early 1990's).
Last year, I got into the ECM specs for my Honda Insight during some research for repairing a problem elsewhere. The quantity, variety of IO was staggering. The obvious generality of the thing was also right there to see. My impression of that was that there is a lot of stuff going on in those beasts, and if you want to swim in that pond, the first step would be to find an engine manufacturer and look there. Large equipment outfits, such as Caterpillar (for example), would be high on my list of prospects. You can probably get a start on small engines that are not used for transport (chainsaws, generators, etc.) at a lot more places.
posted by FauxScot at 1:34 PM on April 9, 2011
I have written code for ELECTRIC motor controls used in a mobility vehicle. It was very informal, though. I developed it in assembly language on an 8-bit superchip (for the time). I've worked on pieces of remote vehicle software/firmware, and seen what "informal" can get you first hand. The mobility controller is still in production, and was pretty cool for its time (early 1990's).
Last year, I got into the ECM specs for my Honda Insight during some research for repairing a problem elsewhere. The quantity, variety of IO was staggering. The obvious generality of the thing was also right there to see. My impression of that was that there is a lot of stuff going on in those beasts, and if you want to swim in that pond, the first step would be to find an engine manufacturer and look there. Large equipment outfits, such as Caterpillar (for example), would be high on my list of prospects. You can probably get a start on small engines that are not used for transport (chainsaws, generators, etc.) at a lot more places.
posted by FauxScot at 1:34 PM on April 9, 2011
Former (?) MeFite Steven Den Beste spent most of his professional career in embedded software. I'm sure he'd be happy to answer questions.
posted by dhartung at 1:35 PM on April 9, 2011
posted by dhartung at 1:35 PM on April 9, 2011
While I have not written code for the ECMs, I have written code for other nodes on the bus such as electronic shifters and steering angle sensors (GMLAN, no Ford as yet) and for diagnostic tools that temporarily get on the bus.
The comment about MISRA C is spot on. The development team is heads down, working in C.
Other items on the bus could possibly be done in MatLab or LabView.
The spec for the GM ECM is hundreds of pages. And that is just the "how-to" for interfacing to the ECM and communicating with it. Each and every bit (I/O and packet communications) is accounted for along with 'future expansion' items.
posted by Drasher at 1:57 PM on April 9, 2011
The comment about MISRA C is spot on. The development team is heads down, working in C.
Other items on the bus could possibly be done in MatLab or LabView.
The spec for the GM ECM is hundreds of pages. And that is just the "how-to" for interfacing to the ECM and communicating with it. Each and every bit (I/O and packet communications) is accounted for along with 'future expansion' items.
posted by Drasher at 1:57 PM on April 9, 2011
IANAAE, but I did take embedded programming and design classes with engineers who did write ECUs for their Baja Buggy. When it comes to embedded programming, as I understand it, simplicity is at a premium. So they roughly design the firmware as a couple of large lookup tables and an infinite loop. Like, instead of approximating sin() (or what have you) in real time, precalculate it and store it in the firmware.
Modern ICE's are complicated, but a large portion is still mechanical. Rather than put spark plug timings in the ECU, you have a distributor cap that rotates and forms connections with spark plug wires in proper sequence. That distributor is linked to the drive shaft, so sparks trigger lockstep with your engine RPM. Last I looked ECM is basically coded as:
fuelTable = { ... values };
while (true) {
oxygen = oxygenSensorReading();
fuel = fuelTable[oxygen];
mix(fuel);
}
In all likelihood oxygen is just one factor among many that determine mixture. There may be busy loops for timing, or interrupt driven systems instead of infinite loops, and likely there's a ton of diagnostics and reporting over the wire to other components. There's also likely to be a ton more for dealing with sensor failure and such.
As for programming them, it's really easy. You write some C code, and find a compiler that targets your microcontroller. I assume they're using MISRA C restrictions, since they went to the trouble of inventing it. Just as an example, here's a GNU toolchain for m68hc11. There's countless more. Programming them varies by board. Hardware wise I doubt they use FPGA or ASIC because FPGA is expensive and ASIC is risky. If you were particularly crazy, I imagine you could operate a car engine by Arduino.
posted by pwnguin at 2:17 PM on April 9, 2011
Modern ICE's are complicated, but a large portion is still mechanical. Rather than put spark plug timings in the ECU, you have a distributor cap that rotates and forms connections with spark plug wires in proper sequence. That distributor is linked to the drive shaft, so sparks trigger lockstep with your engine RPM. Last I looked ECM is basically coded as:
fuelTable = { ... values };
while (true) {
oxygen = oxygenSensorReading();
fuel = fuelTable[oxygen];
mix(fuel);
}
In all likelihood oxygen is just one factor among many that determine mixture. There may be busy loops for timing, or interrupt driven systems instead of infinite loops, and likely there's a ton of diagnostics and reporting over the wire to other components. There's also likely to be a ton more for dealing with sensor failure and such.
As for programming them, it's really easy. You write some C code, and find a compiler that targets your microcontroller. I assume they're using MISRA C restrictions, since they went to the trouble of inventing it. Just as an example, here's a GNU toolchain for m68hc11. There's countless more. Programming them varies by board. Hardware wise I doubt they use FPGA or ASIC because FPGA is expensive and ASIC is risky. If you were particularly crazy, I imagine you could operate a car engine by Arduino.
posted by pwnguin at 2:17 PM on April 9, 2011
Response by poster: pwnguin, the approach you take is kind of the "canonical" one for hobbyist ecus, though I'd add that almost all modern engines use direct sequential control of ignition coils, and no more distributors. I'm very familiar with such approaches, and even with automotive engineering in general, thus wondering how the big boys do it.
posted by TheNewWazoo at 2:22 PM on April 9, 2011
posted by TheNewWazoo at 2:22 PM on April 9, 2011
Response by poster: That is to say, I've written code that does alpha-n and speed-density fuel injection, but as I added IO it got really, really complex to the point where it was easier to parallelize the operation, and I wasn't even really doing the seriously complex stuff like accounting for manifold wall wetting or knock control, but lots and lots of it boiled down to "for n parameter set, do output y at time t" but for really, really big permutations of n. If the pros do it in C, well, I'm impressed, but it seems like there would be a better way to get reliable results, given how easy such big simulations are in other engineering packages.
posted by TheNewWazoo at 2:26 PM on April 9, 2011
posted by TheNewWazoo at 2:26 PM on April 9, 2011
I'm not so sure it is germane to your specific question but here is a document that outlines the software development requirements for avionics code: [pdf] FAA certification of a MEMS AHRS, section 6.2. Note - I'm not a coder, but I had to understand it to make certification go forward. And a whole lot of this code is in C.
posted by jet_silver at 2:48 PM on April 9, 2011
posted by jet_silver at 2:48 PM on April 9, 2011
TheNewWazoo: "pwnguin, the approach you take is kind of the "canonical" one for hobbyist ecus, though I'd add that almost all modern engines use direct sequential control of ignition coils, and no more distributors. I'm very familiar with such approaches, and even with automotive engineering in general, thus wondering how the big boys do it"
Ah. Last engine I've really had to look at was my 91 Honda civic. I haven't the initiative or garage to tear apart my new car engine. I kinda wondered if they'd gone that route but assumed not given how unpopular CAN proved to be.
I do find papers on Google about model based approaches to ECU software design, but if experience is anything like the rest of software engineering fields, industry ignores it and insists that it's "impractical and requires a PhD thesis to understand." Keep in mind also that compilers for microcontrollers generally add language features to support interrupts concisely, so you can do stuff like I/O in parallel. As long as the overhead is predictable and you're able to apply rate monotonic rules, you're likely able to find a CPU fast enough to do the job. If you haven't taken a look at FreeEMS, it should be instructive.
If you don't get the in depth answer you're looking for, more formal theory "prove this real time system can work" stuff, there's grad school classes you can take. Internships with John Deer, Catepillar or one of the auto companies might also give you direct experience with that question.
posted by pwnguin at 5:41 PM on April 9, 2011
Ah. Last engine I've really had to look at was my 91 Honda civic. I haven't the initiative or garage to tear apart my new car engine. I kinda wondered if they'd gone that route but assumed not given how unpopular CAN proved to be.
I do find papers on Google about model based approaches to ECU software design, but if experience is anything like the rest of software engineering fields, industry ignores it and insists that it's "impractical and requires a PhD thesis to understand." Keep in mind also that compilers for microcontrollers generally add language features to support interrupts concisely, so you can do stuff like I/O in parallel. As long as the overhead is predictable and you're able to apply rate monotonic rules, you're likely able to find a CPU fast enough to do the job. If you haven't taken a look at FreeEMS, it should be instructive.
If you don't get the in depth answer you're looking for, more formal theory "prove this real time system can work" stuff, there's grad school classes you can take. Internships with John Deer, Catepillar or one of the auto companies might also give you direct experience with that question.
posted by pwnguin at 5:41 PM on April 9, 2011
Don't ECUs run on top of some kind of RTOS software? I'm surprised no one has mentioned anything like that.
posted by kenliu at 6:05 PM on April 9, 2011 [1 favorite]
posted by kenliu at 6:05 PM on April 9, 2011 [1 favorite]
I have no direct knowledge of automotive firmware development, but I do write firmware for a living, and as far as I have heard nobody, anywhere, writes any firmware in anything but C. We certainly don't - a couple of the newer projects use C++, but only for namespaces and the occasional non-virtual instance method.
posted by Mars Saxman at 7:19 PM on April 9, 2011 [1 favorite]
posted by Mars Saxman at 7:19 PM on April 9, 2011 [1 favorite]
I know some people who do avionics firmware. It's all in Simulink, which lets you build control systems directly out of block diagrams. It spits out a huge chunk of unreadable C code, and you then write a wrapper that calls the Simulink code and sends the inputs and outputs to the right places.
posted by miyabo at 8:05 AM on April 10, 2011 [1 favorite]
posted by miyabo at 8:05 AM on April 10, 2011 [1 favorite]
I think you *might* be overcomplicating it. One team will design the hardware and firmware. Get the data from the sensors to the actuators correctly, on time and without errors. Then another team (or at least phase of the project) takes the available inputs and outputs and figures out the logic. ("We have 100mhz of speed, so we can sample the Gizmo three times a second, and we have 128k of memory, so our lookup tables can have four bits of resolution.") THEN they figure out how to populate the lookup tables using a dyno and info like wall-wetting and all that.
I read somewhere where the real time systems in cars tend to be like ATM in communications networks, where "packets" of data or processing power are hierarchically and strictly divided up. Unlike something like a router's firmware, where it can slow down if it is under heavy usage, each "thing" has guaranteed "space". If something fails to do its thing within its prescribed time, a workaround is used and an error is thrown. Like someone above said, in an infinite loop, with the restriction that it is paced by time events. (Unlike normal programming, where events take as long as they take, each event or signal here must only take the amount of time it is allotted. A malfunctioning [something] won't take down the whole system. The system notes the error or lack of signal and does what it has to do.) (Also, the event might not be time, it might be angles of crankshaft._
They also do more and more "black box" style of systems design. They specify functions and protocols, and leave the nasty bits to some other team designing the gas pedal or injector controller. (Or the same thing from the other end: they buy an ABS controller that takes certain inputs, and they figure out how to give it what it needs.)
In other words, they run the project like any other engineering project. Strict requirements, adherence to standards and splitting the project into small enough problem domains that near-perfection can be had.
(In answer to your concern about C- they use it because it is as close to a perfect multipurpose language yet conceived, and because its performance depends on the hardware and the compiler. The language is just a way to manipulate tokens and juggle tasks, and they could probably use any language. But C has inertia, so that's what they use.)
posted by gjc at 9:45 AM on April 10, 2011
I read somewhere where the real time systems in cars tend to be like ATM in communications networks, where "packets" of data or processing power are hierarchically and strictly divided up. Unlike something like a router's firmware, where it can slow down if it is under heavy usage, each "thing" has guaranteed "space". If something fails to do its thing within its prescribed time, a workaround is used and an error is thrown. Like someone above said, in an infinite loop, with the restriction that it is paced by time events. (Unlike normal programming, where events take as long as they take, each event or signal here must only take the amount of time it is allotted. A malfunctioning [something] won't take down the whole system. The system notes the error or lack of signal and does what it has to do.) (Also, the event might not be time, it might be angles of crankshaft._
They also do more and more "black box" style of systems design. They specify functions and protocols, and leave the nasty bits to some other team designing the gas pedal or injector controller. (Or the same thing from the other end: they buy an ABS controller that takes certain inputs, and they figure out how to give it what it needs.)
In other words, they run the project like any other engineering project. Strict requirements, adherence to standards and splitting the project into small enough problem domains that near-perfection can be had.
(In answer to your concern about C- they use it because it is as close to a perfect multipurpose language yet conceived, and because its performance depends on the hardware and the compiler. The language is just a way to manipulate tokens and juggle tasks, and they could probably use any language. But C has inertia, so that's what they use.)
posted by gjc at 9:45 AM on April 10, 2011
This thread is closed to new comments.
Why can't you imagine that? Because it's hard to write bug-free C code? Here's a job listing at Tesla Motors to "Develop, enhance and debug new and existing low-level real-time firmware in C." That listing mentions MISRA C, "a software development standard for the C programming language developed by MISRA (Motor Industry Software Reliability Association)." Looks like MISRA is what you follow when you're writing C for stuff that absolutely cannot fail, ever: "MISRA has evolved as a widely accepted model for best practices by leading developers in sectors including aerospace, telecom, medical devices, defense, railway, and others."
posted by zsazsa at 12:48 PM on April 9, 2011 [2 favorites]