e m be dde d.co m http://www.embedded.

co m/design/o perating-systems/4006638/Po rting-an-existing-embedded-system-to -Linux

Porting an existing embedded system to Linux

David Knuth and Daniel Daly, Intellibot Robotics, LLC April 11, 2006 inShare0 David Knuth and Daniel Daly, Intellibot Robotics, LLC Every embedded system has the potential to reach a limit where requirements cannot be implemented without compromise to the design, or perhaps at all. When that time comes, the company's management and technical staf f must agree on what to do. T his usually reduces to two choices. T he f irst is to make the minimal necessary changes to maintain the stability and momentum of the development. T his may work f or a while, but will likely f orce the company to embrace the other choice. T hat choice is to retire part or all of the system and rebuild it on a f oundation that will, at the minimum, meet the current and near f uture requirements, but optimistically, be a prof itable f oundation f or a signif icant time into the f uture. T his paper describes the journey taken by engineers at Intellibot Robotics to port the f unctionality of an autonomous robotic f loor scrubber (ref erred to simply as "the robot" f or the rest of the paper) f rom one hardware platf orm without an operating system to another running Linux. Our intention is to share the hands-on knowledge gained and to provide examples usef ul f or making inf ormed business decisions about using Linux f or an embedded system. Knowing the capabilities that Linux has will guide this choice. T his paper illustrates some of the challenges we f aced in discovering those capabilities. Background on "the robot" T he development project that produced the robot dates back to the early 90's. To appreciate the migration, f irst an understanding of the old robot is necessary as this is the platf orm that we replaced with one running Linux. A 68000 main processor and a handf ul of peripheral microcontrollers controlled the machine. To drive down cost and add some headroom f or more f eatures, later generations migrated to a 68332. T he architecture was f airly typical of small microcontroller-based systems of the time: it had no operating system, the code base was relatively small and f or what it had to do, it was adequately f ast. It didn't use an operating system mainly because there wasn't a need f or one. T he code was single-threaded, and any OStype services needed were easily written without concern of what the rest of the system was doing. Although the product was technically f unctional, it lacked f eatures to satisf y market needs. Over time, the purpose of the robot grew f rom simply being an intelligent f loor cleaner to becoming an autonomous machine that needed even less human intervention. T his growth directly translated to more technical requirements. T here were more sensors to manage, more data to collect, more computations to complete in the same amount of time. With the f irst f ew generations of the robot, the addition of new f eatures implemented in sof tware was largely successf ul. In late 1999, however, the requirements put on the machine began to exceed what the platf orm could provide. T here f ollowed an episode involving the introduction of a large amount of sof tware to this system to meet these new requirements. Not long into that ef f ort, we discovered that the processor could not run the new sof tware and still meet the requirements of controlling the robot in real time. In embedded systems development, this is what we like to call a "dismal f ailure". T he product development stalled on that f ront; entrance into that particular

new market was put on hold. Af ter that, requirements were changed to enhance how the machine perf ormed in existing markets, since there was not time to improve the hardware. T his new f eature set kept the robot a viable product while our engineering team worked at squeezing more f unctionality out of the existing hardware, but that ef f ort was quickly approaching the processing limits of the platf orm. Making the transition In late 2003, with the company under new ownership, the requirement to enter new markets became a priority again. Several discussions of the current capabilities versus the desired capabilities quickly showed the necessity of migrating the robot to a new platf orm. Not knowing whether this would be a triumph or a disaster, our management simply stated that the product should be moved to Linux. Much of the robotic development in academia is happening on Linux and our owner wanted to leverage this. T hough we were f amiliar with Linux on workstations and servers, we hadn't used Linux as an RT OS. T here were many questions we had to answer to make the transition successf ul. However, with the direction now specif ied by our management and with the appropriate f inancial backing, the decision to proceed with Linux was made without any of the research that would typically occur when making such a signif icant engineering change. T here was no comparison with or evaluation of other RT OSes. T here was no investigation if Linux was even appropriate f rom a GNU licensing point of view. It was known that Linux was f ree but that's all. Our management was comf ortable taking the risk that Linux would meet our needs. T hough we can gladly report that the migration was a success, we recommend that a complete investigation by both engineering and management be undertaken to answer these questions in advance. T he rest of this paper exposes some of these questions and the answers we f ound while attending the School of Hard Knocks. Finding the right hardware platf orm Shortly af ter the decision was made to move to Linux, we needed to f ind a suitable hardware platf orm capable of running Linux that would also f it the needs of the robot. T he current motherboard of the robot was an inexpensive custom design ideallysuited f or the robot, but lacked support to run Linux. T heref ore, the search began f or a low-cost single-board computer (SBC) that could run Linux. T hough we pref erred a PowerPC processor, the SBCs we f ound were f ar more expensive than equivalent x86based counterparts. Using the existing subsystems of the robot required an SBC with an SPI bus. Yet, x86 processors --the target where Linux is most mature --generally do not have an SPI port. Internet searches did not yield anything. By chance, we spotted an ad in Circuit Cellar Inc. that showed an x86 board with an SPI port, which the vendor claimed ran Linux. T he cost of the board was about half of our existing custom motherboard, too. T he migration was now on its way. Looking back, here are some things to investigate when looking f or a platf orm to run Linux. If the hardware expertise does not exist in-house to design a cheaper target board, then an of f -the-shelf SBC is probably the most cost-ef f ective. Also, look f or a vendor who not only advertises that Linux runs on their products, but has the technical support available to help get your application working with Linux. T he vendor we chose of f ers unlimited f ree technical support, a bootable Linux f ile system on a CompactFlash, extensive downloads and support on their web site. Early on in powering up the system, we discovered a BIOS problem and their tech support quickly f ixed the problem. Without this, we would have added a signif icant delay in getting our problems resolved.

A nice side ef f ect of the vendor we chose was that they are now of f ering SBCs using a f aster ARM processor at a lower cost than an x86. With the product now ported to Linux, the migration to that board -or any other SBC that supports Linux --should entail little more than a recompilation, a signif icant benef it not yet realized. Because the SBC had Linux pre-installed, it was simply a matter of applying power and connecting its serial port to a computer. T he sof tware support on the stock Linux f ile system was reasonably complete, and readily provided a login prompt on one of the SBC's serial ports, as well as card support and network drivers f or the PCMCIA wireless Ethernet adapter. In very little time, the SBCs were communicating with other computers on both wired and wireless networks. T he next important step was to block out a period of time to explore the capabilities of Linux on the SBC. Understanding how it behaves in a normal situation is imperative bef ore unf ortunate situations arise. Getting started Up to this point, the new SBC was indistinguishable f rom other Linux systems on the network: it responded to ping, telnet, and f tp. T he f ile system layout was typical of a Linux system. It knew the correct time and date. At this point, a key f eature of moving to Linux was made apparent and became a cornerstone to the entire development process. T he f ile system of our Linux server could be NFS-mounted on to the SBC's f ile system. T hat meant that f iles could be ef f ortlessly transf erred between the development server and the robot. T hough this mount worked over the wired Ethernet connection, the real benef it was that it worked over a WiFi connection, too. T his would mark the f irst time that the development of the mobile robot would become untethered. T he f oundation had been laid, upon which the f irst level, a development environment, would be built. T he more development in an embedded environment resembles native development, the f aster and easier development in that embedded environment will be. Cross development adds at least one step to the build process, and cross debugging is tricky and cumbersome in comparison to native development. Despite this, we had established a workable, though tethered, cross-development and cross-debugging environment in previous generations of the robot. Moving away f rom this f amiliar environment to another was a risk. However, the cross development and debugging system set up in Linux turned out to be equally as powerf ul and simpler to get working -and it was wireless. T he f ile system that came with the SBC f it on a 32 MB Compact Flash with some room to spare. Complete as it was f or getting the system up and running, it had somewhat limited support in the way of tools and run-time libraries. T he f ile system included no development tools. T he version of the libraries on the f ile system was of an older version than that of our other Linux systems. T he lack of on-board development tools was not a limiting f actor, since it was not in the plan to build applications on the SBC; our Linux servers were designated f or that. Programs would not have to get very big bef ore building on the target would be time-and space-prohibitive, and a Linux workstation provided much better f acilities f or editing source code. T he build process quickly reduced to building the application on the server, copying it into place on the target SBC via an NFS mount, and running it f rom the SBC. T his was much more convenient than programming ROMs, as was done in the past. Another dif f erence moving f rom a no-OS platf orm to Linux was that there were now libraries of code that could be utilized by the application. T he challenge was how to use these in a resource-ef f ective way. Because the initial set of run-time libraries on the SBC was limited and of an older version than that on the server, the programs either had to be so simple that they only used common run-time libraries between the

SBC and the server, or programs had to be statically linked. Statically linking the application A statically-linked application can run on the SBC regardless of the libraries present on the f ile system, since such an application contains all the runtime support it requires. T his makes even small statically-linked applications huge compared to the same programs, dynamically linked. T here is merit to having little or no run-time support on the f ile system if the system is small with a limited number of applications and tools. In such an arrangement, all applications (including the command shell and widely used tools such as ls, cp, and mv) would have to be statically linked, each containing its own copy of the libraries it requires to run. Resources are quickly consumed when there are many applications each with their own copy of the identical libraries. In contrast, dynamic libraries save RAM because each statically-linked application loads its own run-time support into RAM at runtime. Dynamic libraries are loaded into memory once regardless of the number of applications loaded at any given time. Linking to the libraries on the SBC, rather than the server, though possible, was not considered because they lacked complete support f or POSIX threads. Although the robot application was single-threaded, there were already thoughts of splitting it apart into many threads, so POSIX-complete libraries were important. Considering all these f actors, we decided to make the libraries on the SBC match what we had on our servers. T his was simply a matter of copying the libraries f rom the server to the SBC. Since the SBC's runtime environment could easily be a subset of that of the server's, only the libraries that were needed were copied f irst. Mainly using ldd, a utility that determines the shared-library dependencies of a given application, a good f irst guess as to the contents of this minimal set was determined. As various required tools were put onto the f ile system, accompanying run-time library support was also determined and added, until the support was so complete that it was no longer necessary to check the dependencies. Libraries, tool chains and building blocks T hat set of libraries is what resides on the f ile system today, providing a very complete and convenient environment f or development, debugging, and deployment. Applications can be built on the server, dynamically linked, copied to the target, and run. Debugging tools such as gdb were deployed in exactly the same way; the very same version that runs on the server also runs on the target. Another advantage of this approach is that it was not necessary to rebuild the development tool chain in order to build applications f or the target. T he tool chain already installed on the server builds applications f or the target just as easily as it does f or itself . With this arrangement as a baseline, we could, with reasonable conf idence, rebuild the tool chain on this platf orm or some other, and continue development f rom there. T he vendor's initial f ile system layout was helpf ul f rom the beginning, as it f acilitated quick startup early on, and provided the basis f or conf iguration changes and improvements made since then. T his incremental approach to modif ying the layout and contents of the f ile system kept the number of variables low, which was key to maintaining stability of the system. Changing one thing at a time, we developed and maintained a high level of comf ort with the system as a whole. Also benef icial in getting the development process started was having another Linux system readily available f or building applications, and as a repository f or developing and storing source code. On a more capable target system, the development could be done completely natively, editing, compiling, building, and running earlyapplications all completely on the target. In our case, the target was not suf f icient to carry all those responsibilities, nor did it have to be. T he server was a base camp, f rom which many reconnaissance missions into new territory on the target were launched and successf ully carried out. Having a PCMCIA slot on the SBC meant one thing above all else: our product had the ability to enter the world of wireless networks by simply inserting a standard 802.11 card. Many embedded systems are

physically stationary. For development, they sit on the workbench, and the network comes to them in the f orm of an RJ-45 jack. Bef ore the Linux port, the only line of communication with the robot was an RS-232 port and a radio modem to break the physical connection. Breaking the physical tether was an important part of protecting a company's investment in development equipment: laptop computers in motion tend to stay in motion, regardless of any sudden stop on the part of the robot on which they're riding. Kernels, modules, and device drivers Linux is large compared to a typical RT OS, but it is organized so that understanding it does not require swallowing it whole. Like an onion, Linux can be peeled away in layers and understanding each layer can be accomplished without the need to dig deeper. Unless your name f requently comes up in the Linux kernel source tree, you're quite likely still several layers up f rom the core. T here's nothing wrong with that; we are nowhere near all the way down. T he important thing is that you understand just enough to get your job done, and this is possible with Linux. Linux is widely regarded as a complete operating system, including a kernel, a f ile system layout, a command shell, tools, and sometimes even a graphical user interf ace. T his may be true, but Linux ref ers primarily to the innermost part, the kernel, originally written by Linus Torvalds. T he kernel is what boots up the computer, makes it run, and provides the operating-system services that applications need to make the computer do usef ul things. Whether or not one modif ies the kernel (as we have done), porting a product to Linux at least initially involves the kernel, as it f orms the basis f or all other work required to get the system running. Depending upon the hardware vendor, the kernel may already be installed, which is what we recommend f or a f irst project. Linux is developed and released under the GNU General Public License. Its source code is f reely available to everyone. T his open-source nature of Linux allows anyone to view, modif y, apply, and extend it f or their own purposes. Chances are, if a person adds to Linux, they want to make their work accessible under the same License, and as a result, Linux gets better. For each piece of hardware supported by Linux, it is almost certain that somebody somewhere has written a device driver to make the hardware work with the system. Device drivers are distinct pieces of sof tware that enable a certain hardware device to respond to a well-def ined internal programming interf ace, hiding how the hardware device works behind that interf ace. In such an arrangement, users (and applications) do not have to understand how the hardware works in order to use it [1]. If there is specialized hardware in your system, you will have to decide whether to control that hardware completely in your application, or write a device driver to do it. A later section discusses this decision at length. Whether the goal is a device driver or not, the main (and easiest) approach to building onto the Linux kernel involves no changes to the kernel itself : making a kernel module. Unlike an application, which perf orms a specif ic task f rom beginning to end, a module attaches to the kernel and registers itself with the kernel in order to serve f uture requests [2]. Modules can be built completely separately f rom the kernel source tree. T he kernel can load and unload them easily; there is a wellestablished mechanism inside the kernel to do this. Modules are simply a vehicle to dynamically add f unctionality to the kernel. T hese two constructs, modules and device drivers, are of ten related, but are not synonymous. A module may contain a device driver. A device driver may be composed of one or more modules. While a device driver may exist in source code f iles that f orm one or more modules, that device driver may also be built right into the kernel (it then becomes a part of the "kernel proper"), eliminating the need to load the module af ter the

system has booted. Loadable modules provide f lexibility and changeability at the cost of having additional f iles on the f ile system, which must be loaded bef ore the device driver can be used. Building the device driver as part of the kernel eliminates the need to load the module, but requires a rebuilding of the kernel (and reinstallation of the kernel image)when the device driver is modif ied. While a device driver (or other addition to the kernel) is in development, it can take the f orm of loadable modules until it gains a level of maturity. T his can make module and device driver development easier since the kernel source can remain unmodif ied. Writing device drivers Not all of the old robot was to be redesigned or replaced. A self -imposed requirement was to have the new SBC communicate seamlessly with several existing peripherals on an SPI bus. New SPI peripherals would also be added. T he problem of controlling this interf ace easily reduced to two parts: writing sof tware to control the new processor's SPI port, and writing sof tware to implement the communication protocols employed by the peripherals. T he latter naturally f ell into the purview of the application. Protocols do (and did) change. Doing this work in the application allowed f or support of existing protocols and the ability to add new ones easily. T he f ormer presented a f undamental decision: should the control of the SPI port occur in the application or in a device driver? In an embedded system, it's common practice to roll all the code required to control the entire system into one piece of application-level sof tware. While this is of ten the only choice on bare metal, when an operating system f orms the basis of the sof tware, this all-in-one approach rejects many of the benef its of building an application on that operating system. Most notably, it ignores the ability of the kernel to arbitrate demands f or access to system resources, especially input/output devices such as the SPI port. Knowing that others had successf ully implemented support f or this type of interf ace completely in application space, it was tempting to code up control of the SPI port completely within the application. Early on, this is how the hardware was tested and f rom this we gained understanding of running the SPI port on this new platf orm. In the long run, however, controlling the SPI port entirely in application space would not meet the requirements. T his approach did not make good use of the operatingsystem benef its. For instance, the original, single-threaded control system was about to become multithreaded (the current system spawns over twenty threads, several of which access the SPI). Managing the SPI in the application meant having to develop elaborate thread-saf e sof tware to arbitrate accesses of the interf ace. In addition, since a Linux-based application cannot respond directly to hardware interrupts, it is resigned to polling,delaying, and looping. T he operating system already does this low-level work, and is very ef f icient at it; all the programmer must do is provide, in a device driver, the details of how to operate this particular device. With a device driver, multithreaded, thread-saf e, application-level access to the device becomes ef f ortless. Also, accessing hardware through a device driver af f ords the system designers a convenient separation of mechanism (working the device) f rom policy (using the device) [3]. T he application doesn't need to know anything about how to operate the device. T he application is insulated by the operating system f rom changes at the level of the device. In theory, the interf ace, or the way it is operated, could completely change without requiring any change in (or rebuilding of ) the application. Having seen the benef its of device drivers, the time came to learn how to implement them. Rubini & Corbet's book Linux Device Drivers (O'Reilly) became required reading, and remains the pref erred ref erence

book. First attempt at a kernel module A f irst attempt at a kernel module containing a device driver f or the SPI was completed in a week. In retrospect, this f irst module was a small step, essentially moving the application-level, polled SPI support into the kernel. Small step or giant leap, the entire robot was now beingcontrolled using standard open(), write(), read()and close()system calls f rom application space to access the hardware. T he application now contained none of the nuts and bolts of handling the interf ace. T hough the application now had nothing to do with controlling the hardware of the SPI port, the read()and write()calls were not any more ef f icient at runtime than the app-level code they replaced. T he SBC's processor provides rich hardware support f or SPI. To use this support to its f ull potential would require redesigning the driver to exploit the ability of the processor's internal SPI hardware to interrupt the CPU. T his would remove any polling and looping in the SPI driver code, f reeing up more CPU cycles f or the application, and making SPI communication as execution-ef f icient as it could be. While not especially easy, the task of rewriting the driver to f it an interrupt-driven model was made as easy as it could be, thanks again to Rubini and Corbet. Not only did they lay out all the elements necessary to make a design like this work (and work it does),they also f ocused attention on almost all the pitf alls we were to encounter while writing the device driver. T his inf ormation enabled us to preempt many of the typical problems accompanying development in this space, especially those involving race conditions and resource contention that are so common when f ielding multiple requests f or hardware and common memory. T his experience repeated a message that emerges again and again when one works with open-source sof tware at any level: support f or what you want to do almost certainly exists. It may take the f orm of a book, a HowTo, a Google search, a newsgroup or mailing list, a phone call to a colleague, a visit to a local User Group, or inspection of the source code itself . T he independent variable becomes your ability and willingness to reach the inf ormation you need. Releasing control One of the biggest changes going f rom the old embedded system to the SBC running Linux was powering down the machine. In the past, turning the key switch to OFF killed power to the robot and the hardware shut down properly. Doing the same to the new robot with Linux running on the SBC was dangerous. T he f ile system could easily become corrupt because the kernel would not have an opportunity to write cached data to the f ile system. Since the application was no longer running on bare metal, the application should no longer directly control the hardware. T he saf est approach is to allow the kernel to perf orm all of its shutdown tasks, such as unmounting f ile systems and taking down network interf aces, bef ore removing power. In cases where the f ile system is mounted read-only, it may be saf e to simply cut power. If write caching is disabled on a read-write f ile system, it can be made saf e, provided the f ile system is allowed to become quiescent bef ore power is removed. Since the application must relinquish all control of the system long bef ore the kernel shuts down, we needed the kernel's cooperation to solve the problem. T he Linux kernel did not disappoint. Within a day, including research (the Linux kernel source, and Rubini & Corbet again), we had written a 41-line C module that registers with the kernel to be notif ied upon halt or shutdown. A startup script loads the module. When notif ied by the kernel, the module exercises the I/O line connected to the relay disconnecting power to the machine only af ter the kernel f inishes all its shutdown tasks. T his provides a clean shutdown of the system.

Kernel timing Once the interrupt-driven SPI device driver was implemented and the application was using it to communicate with all the peripherals to control the robot, it was time to determine the character of the traf f ic on this multiplexed SPI bus. Af ter all, this was a stock Linux kernel, not a f lavor of Linux enhanced f or optimum real-time perf ormance. Hard timing requirements could not be imposed as in a typical embedded system. Had the interrupt-driven model really allowed the multiplexing of the SPI bus as we had envisioned? What did the timing between SPI transf ers look like? What was the overall throughput of the SPI bus? Was this arrangement going to make it easier f or the application to accomplish all the communication tasks it had, quickly enough to satisf y the sof t real-time requirements of the robot? In an embedded system, one does not have to get very f ar down into the code bef ore sof tware tools become cumbersome in determining the real-time perf ormance of the system. Gauging the ef f icacy of the SPI device driver at the level of clock cycles, individual transf ers, and interrupt service latency solely with sof tware tools would be like determining a person's f acial f eatures using a bowling ball as a probe. Fortunately, much of the hardware outside the SBC was of our own design. T he signals of the SPI bus were easily accessible to a logic analyzer, which handed us measurements on a platter. Af ter adding some "instrumentation" in the f orm of toggling digital outputs at critical points in the application and the driver, hard evidence showed that the application was perf orming transactions with the peripherals through the device driver, and the driver was interleaving those transactions on the SPI bus almost as expected. Peripherals with a microcontroller of their own required a delay between transf ers to allow the microcontroller to service their transf er-complete interrupt and prepare f or the next transf er. T his delay is typically 200 microseconds. T his magnitude of delay, we reasoned, should be an easy matter f or the application, in which the involved thread would sleep af ter initiating a transf er f or however long the peripheral needed, f reeing up the main CPU f or other things, waking up af ter the delay to begin the next transf er. We theref ore expected to see a transf er with one peripheral, then a 200-microsecond delay during which other transf ers could occur on the SPI bus, then another transf er with the peripheral. What we f ound instead were delays on the order of 10 milliseconds! Not only did this completely go against our intention of dense, bursty utilization of the bus, but it also made transactions which ought to take 2 milliseconds (approximately ten transf ers separated by a delay of 200 microseconds) take over 100ms to complete. While it's true that our utilization of the CPU had greatly improved, the throughput of the system was abysmal; at some point, the application still had to wait f or communication with peripherals to f inish to complete its processing, and this could not happen and meet even the sof t real-time needs. What was causing this long delay, almost two magnitudes greater than we had intended? How could a usleep(200) call (theoretically putting a thread to sleep f or 200us) put of f the task f or a whole 10ms? As with previous device driver work, assistance was waiting in the typical places where one f inds help when working with open-source sof tware. In this case, it was in the kernel source, along with, yet again, Rubini and Corbet. T he answer lied in granularity of timing of the kernel. T he kernel's main timer runs, by def ault, at 100Hz on a typical x86 system. T his value is determined by a macro, aptly named HZ , which is def ined inside include/asm-i386/param.h f or an x86 target. T he kernel timer, with its 10ms period, is responsible f or much of the timing of the system, including task switching. It became clear that the best granularity we could hope f or, regardless of the value we passed to usleep(), was 10ms unless the def inition of this macro was changed. To prove the signif icance of this setting, we changed the value of HZ to 1000, recompiled and installed the

kernel, and ran the test again. T he delays decreased by a f actor of ten. Granularity of 1ms was better, but not good enough to achieve 200us timing between transf ers. T he bus was still strangled providing only one-f if th the required throughput. <>A suprise discovery In a surprise discovery, we solved the problem another way using gettimeof day(),which interestingly provides the date and time down to the microsecond. A busy-wait loop never giving up the CPU f or other threads, was patently unacceptable. Implementing and testing a busy-wait loop. However, in implementing and testing this loop, the surprise was that gettimeof day() did in f act release the CPU to other threads in a way that seemed to allow the thread to run again bef ore the beginning of the next timer interval. Digging into the kernel source code conf irmed that it did in f act yield to the processor. T he new delay not only gave the f ine timing needed, but satisf ied the requirements of allowing multithreaded operation even during multibyte transactions on the SPI bus. Even though HZ no longer needed to be set to 1000 f or SPI timing, the value was kept in our kernel to provide 1ms-grained delays using usleep() throughout the rest of the application. One may wonder about the about the run-time consequences of multiplying by ten the duty cycle of the system timer. In the f inal analysis, this change appeared to have an insignif icant ef f ect on CPU loading, and an immeasurable ef f ect on the run-time characteristics of our system. T his did not come as much of a surprise, considering the eminent ef f iciency of the Linux kernel timer code, and the speed of the CPU. T hat said, increasing HZ was not without consequence. At some point long af ter the SPI timing problem was consigned to the pages of engineering notebooks, a new WiFi card was installed. Poor network perf ormance and f requent network f ailures suddenly ensued. We traced the problem down to timeouts in the device driver f or the new interf ace, which, while dependent upon the kernel timer, were not expressed in the source code in terms of HZ . As a result, the timeouts were expiring in one-tenth the time they would have bef ore HZ was increased. By changing HZ to improve the Linux kernel timing granularity, we had broken the device driver. Repairing the driver was simply a matter of correcting the length of the timeouts, and recompiling the module. T he experience taught us three lessons. First, such a f undamental change as HZ can and of ten will have f ar-reaching ef f ects, some subtle and misleading. While neither new to us nor unique to Linux, it was important f or us to receive this lesson. Second, we saw once again that digging into the kernel source code is not only allowed, but also encouraged (as shown by our success). In this case, we saw an error message on the console when we tried to use the device. By searching the kernel source tree f or this error message, we were able to locate the driver source f ile and zero in on the problem relatively quickly. T hird, we learned that in writing kernel modules, clear, concise, unique run-time error messages are vital to drilling in on the cause of a problem: given a cryptic, non-unique error message in the case of the drivertiming problem, it would have taken f ar longer to get to the cause of the problem, were we able to f ind it in a reasonable amount of time at all. Dealing with SPI transf er timing For SPI transf er timing, we could have built precise delays right into the SPI device driver to provide the right amount of time between transf ers. T his was never a viable option. First, looping f or delay in the driver goes against the intent of interrupt-driven design, whose beauty lies in its ability to initiate a transf er then return right back to the application, to gather the results of the transf er through an interrupt, again consuming minimal CPU cycles. Waiting in the driver would rob the application of these CPU cycles.

Second, doing so would violate Rubini & Corbet's separation of concerns (mechanism and policy), unduly f orcing upon the driver protocol-related constraints regarding how multiple transf ers are to be carried out. T he driver's job is to serve the interf ace to the application in a simple, straightf orward way; it's the application's job to worry about the interf ace's use and purpose. T he astute reader will notice that there exist multiple answers to questions of implementation, mostly in the f orm of kernel patches, allowing real-time extensions to Linux. T hese extensions take many f orms, and promise millisecond or microsecond granularity, some without busy-wait loops. Also, our processor has f ine-grained timing hardware built in, which may help solve this problem. In the end, the gettimeof day()-loop approach was chosen because it required no change to the kernel, and because it was easy to implement. T here was also some comf ort inherent in staying with a stock kernel as much as possible. Finally, time pressure on the project f orced a quick resolution. Implementing this delay completely in the application also provided f lexibility to change the approach easily whenever needed. As the Linux kernel continues to improve with each release, we will likely revisit and improve upon our solution to this timing problem. However, the current solution works and meets today's needs. When a better solution is required, we will once again ply the market and the Internet f or solutions to this real-time problem. With our embedded Linux experience deeper now than when we f irst wrote of this problem in our notebooks, we may be able to arrive at a more elegant solution. Overall impressions You get what you pay f or. T his phrase applies almost universally, but as with statements like it, context is signif icant. What are you paying f or? A product that works as expected. What are you paying? To an embedded systems engineer, the currencies are money and time. Both are easy to spend. In a perf ect world, you spend one to save the other. In reality, that's a neat trick. Linux is f ree only if your time has no value. [4] Applying the perf ect-world time/money model, what you gain by not paying money f or Linux is lost in the time you spend bending it to your will. T he equation is the same, but the coef f icients change, depending on the objective, and on the person, organization, or culture involved in the ef f ort. How "f ree" Linux is depends on the value that entity places on its money and time. No matter what the project, technical support is required. Whether that support is paid f or, or is simply gathered f rom experience, it has to be paid f or with money or time. From our experience with Linux we don't f eel that we're missing anything by not having technical support only a phone call away. T he return on the time investment has been positive. T he f ramers of the C Programming Language said that C "wears well as one's experience with it grows." [5] T his is true of almost every tool that people use. Linux is no exception; as we continue to apply it to the control of the robot, our competence and comf ort level with it increases. It's unlikely that we would have been so willing to port this product to Linux without a high level of understanding and experience with it. T his statement is meant less as advice as it is an assessment of our sense of caution. A much more adventuresome reader may jump headf irst into such a task af ter having gained quite a bit less experience, and will likely be successf ul, having made the commitment to devote time and ef f ort to research and learn, and to solve problems as they arise. Our ef f ort to bring to lif e the new generation of autonomous f loor scrubber took about 22 months. Of that, two sof tware engineers spent about 2 months f ocused on device driver development and about 13 months f ine-tuning the application to take advantage of Linux and run on the new hardware with several new peripherals. T he cost in time yielded a tremendous gain of knowledge that may not have been gained through traditional technical support models. In retrospect, the only change that we would recommend would be to have more engineers f rom the start, but wouldn't we all want that?

Conclusion Linux is not the right tool f or every job; there will always be problems that are best solved by an 8-bit microcontroller running 8k of caref ully craf ted C or assembly language. Systems with stringent, microsecond-level timing requirements also may not be a good f it f or Linux, either. As embedded systems become more complex, and hardware becomes more capable, the sphere of sensible embedded applications of Linux grows. However, f or our robot, Linux was and is an excellent choice. It provides all the f lexibility one expects f rom open-source sof tware, meets the current real-time demands of the system, and leverages our Linux experience well. When problems appear, answers are f orthcoming through one or more of the many support channels. In most cases, the problems are solved more quickly than would be the case were a vendor's Technical Support department involved. Looking back over the journey we've taken, it is thrilling to see that the ef f ort of porting to Linux has been a success; the robot works as desired and is capable of beingimproved upon f or a long time. T he Linux f oundation is so comf ortable to us there is no consideration of moving the code to another RT OS -- yet, we have the f lexibility of quickly moving to another hardware platf orm. If your next embedded application qualif ies, we encourage you to consider using Linux. David Knuth is director of engineering and Daniel Daly is senior software engineer at Intellibot Robotics, LLC T his article is excerpted f rom a paper of the same name presented at the Embedded Systems Conf erence Silicon Valley 2006. Used with permission of the Embedded Systems Conf erence. For more inf ormation, please visit www.embedded.com/esc/sv. Ref erences [1] Rubini, A. & Corbet, J. (2001). Linux Device Drivers (2nd Ed.). O'Reilly, p. 1 [2] Rubini, A. & Corbet, J. (2001). Linux Device Drivers (2nd Ed.). O'Reilly, p. 16 [3] Rubini, A. & Corbet, J. (2001). Linux Device Drivers (2nd Ed.). O'Reilly, p. 2 [4] Z awinski, J. (1998, 2000). Mouthing Of f About Linux. http://www.jwz.org/doc/linux.html [5] Kernighan, B. & Ritchie, D. (1988). T he C Programming Language (2nd Ed.). Prentice Hall, p. x [6} T he Linux FAQ, linux.org, various entries. [7} T he Linux source code tree, http://www.kernel.org/, various entries inShare0