Application eXecute-In-Place (XIP) with Linux and AXFS
Sören Wellhöfer
soeren.wellhoefer@gmx.net
September 17, 2009
Abstract
XIP, or eXecute-In-Place when written out, is
a technique of directly accessing application
code and data in non-volatile flash memory
rather than transferring it to physical RAM
first in order for an execution to proceed. It is
frequently used in the contexts of embedded
computing.
The sheer notion, however, of using more
cost-intensive flash chips to run programs
from has been looked upon rather suspiciously
by many a system engineer in the past.
Moreover, in the world of Linux, running
applications in-place is far less common than
doing it for the kernel in a likewise fashion.1
For that matter, kernel XIP has already been
successfully used to improve boot-up time in a
number of cases. Application XIP, on the other
hand, is not yet as widely taken advantage of
by open source system designers.
This article is to demonstrate, in concept as
well as practice, how application eXecute-In-
Place basically works and to shed some light
on the argument that XIP might not be such an
aberrant idea after all.
Alas, free, comprehensive and practical re-
sources on the subject matter are relatively
scarce; it is hoped that with this short article
this condition can at least be somewhat reme-
diated.
In its first part, this article gives a broader
view on the concepts of eXecute-In-Place (XIP)
for user applications with specific references
to Linux. The second part focuses on AXFS
(Advanced XIP File System) and begins by ex-
plaining its basic ideas, followed by concrete
instructions on how to set up a Linux box with
it.
1
Finally, this article is complemented by pre-
senting and analyzing the results of perfor-
mance tests that have been conducted on
two embedded systems to compare AXFS and
JFFS2 in terms of execution speed.
1 Regular program execution
A program, when stored on disk or flash, is
in essence nothing else but binary data, that
is, code that gets executed and program data
that will be used by the program during exe-
cution. Linux uses a specific and very flexible
format for this purpose called ELF (Executable
and Linkable Format). It is widely accepted and
has become the official de-facto standard in the
Unix world.
When a user decides to execute a program,
the shell he uses invokes the Linux system call
fork(). By doing so, a new process with
a unique PID (Process Identification) is created.
This process initially is an exact clone of the
shell process itself; all the executable code as
well as the process data is merely copied.2
The next thing that happens is that a sys-
tem call of the exec() family of functions is
invoked which receives the path name of the
file that the user wishes to execute as one of its
arguments. The exec()-like call now recog-
nizes the ELF format and attempts to replace
sections of the currently running process with
those found in the file. While doing so, most of
the program code as well as the program data
1. Here, application XIP denotes the fact that user-
space programs are to be executed from flash memory.
2. Actually, processes data is not blindly copied but
both parent and child process share the same region of
memory unless writes occur; then the needed sections
are really copied. This technique is often called copy-on-
demand.
is replaced with that of the now to be newly
executed program.
2 Application XIP
What has been described in the previous sec-
tion was the normal-fashioned way of doing
things. In-place-execution - or also abbreviated
XIP - takes a slightly different approach in
that the actual program code as well as the
program’s static data (the .text and .data
sections in an ELF file respectively) are never
copied to RAM and neither will they be copied
when a process forks itself by calling fork().
When a program is to be executed in-place,
the only things actually made space for in
RAM is the .bss section3 , meaning the unini-
tialized data that a program will be using,
as well as the program’s stack that grow dy-
namically as execution proceeds4 . The .text
and .data sections of an ELF file do not get
transferred to RAM since execution directly
proceeds from non-volatile flash memory by
setting the execution pointer to a memory lo-
cation on the flash itself.
This is potentially useful if a tight limit on
RAM resources constrains its utilization such
as is often the case for embedded systems and
smaller devices; with XIP, no fetching of pages
from flash and copying them to RAM is neces-
sary whatsoever.
Because of its nature XIP can be perceived as
a form of shared memory access in that multi-
ple processes executing the same code would
all share it in flash memory without maintain-
ing a unique copy somewhere in RAM just for
themselves.
A method commonly found in the Linux
world is to use XIP in connection with com-
pressed read-only file systems such as cramfs
or SquashFS. There are XIP extensions (patches)
for those file systems available that aim to min-
imize flash storage utilization through com-
pression whilst providing XIP capabilities.
This is usually achieved by storing application
2
binary data unchanged in flash memory while
still applying compression to everything else
on the file system image.
Within the virtual memory address space
of a process, the executable code and static
data (.text and .data) are directly mapped
to flash memory so that a paging5 of these
sections becomes unnecessary.
Note that for XIP data, it cannot be possible
to be compressed as this would thwart the
very purpose of in-place execution itself; data
would have to be inflated and moved to RAM
first in order to execute it - this is precisely
what is being tried to avoid.
Again, not making use of XIP means that
program data (the ELF .data and .text sec-
tions) are stored in compressed form in flash
memory just as everything else on a com-
pressed file system. It is then decompressed
and loaded into RAM when needed during the
execution flow, that is, when misses in the page
cache lookup performed by the kernel occur.6
3. The .bss (Block Started by Symbol) of an ELF file
describes all the uninitialized data of a program. It does
not occupy any actual space on disk but gets allocated in
RAM upon execution and is filled with all zeros initially.
By contrast, the .data section does already occupy
memory on disk; it contains all the non-changeable data
which is hard-coded into the application. An example for
this would be a statically assigned string constant in a C
program.
4. Note that all this is only true for read-only file sys-
tems. If data is to be also modified, the .data section
does also get (partially) copied to RAM so that it can be
changed in before committing it back to the disk or flash.
5. Paging, as performed by the operating system and
in the the applied context, is the proccess of retrieving
data segements called pages (usually 4kB in size) from
and external media, such as hard disks or flashes, and
loading them into RAM. This allowes the memory space
of a process to be built up when execution begins. But
because the memory as it is seen by the process is not
truly continuous, but rather is in itself made up of many
fragmented portions residing at some location in physi-
cal RAM, this method is often called virtual memory, or
VM. With most file systems that apply compression, it
becomes in addition to merely copying pages also nec-
essary to decompress them whilst paging; this naturally
takes up time as well.
6. The kernel is, on most architectures, aided in this
This method of pulling them in only when
truly needed is often called demand-paging be-
cause a loading only occurs when they are
absolutely wanted.
One disadvantages of XIP is that read ac-
cess to flash memory is relatively slow when
compared to RAM. Whereas retrieving a page
of memory in random access fashion from
regular SRAM takes about 25 ns, this amount
increases to about 100 ns for an average NOR
flash chip. Although recent developments
have dramatically reduced this number down
to even 70 ns for some rather expensive NOR
chips, flash memory access can never be as fast
as SRAM due to its intrinsic technical make-
up.
It is important to note that eXecute-In-Place
is only truly possible with NOR flash memory
without applying some sort of emulation layer.
Unlike with NOR, NAND flash memory is
not directly memory-mappable and can only
be access on a per-block basis of usually 512
kB. This generally yields a faster data rate,
however it renders NAND unsuitable for XIP
where it must be possible to singly access and
read individual bytes; this is necessary for a
program execution.
Another disadvantage is posed by the fact
that RAM is generally available at a lower cost
than flash memory. For XIP to be used benefi-
cially, one must consider whether the advan-
tages gained in RAM preservation outweigh
the disadvantages of higher flash memory uti-
lization and slower memory access and render
the trade-off worthwhile.
In a test case constructed by engineers at
Intel, it has been shown that when running an
application and reducing the available RAM
at fixed successive intervals, system perfor-
mance will drop rapidly at a certain point
where memory saturation has been reached.
As the system runs of memory, the operating
system is excessively burdened with the task
of swapping7 out pages to an external storage
medium; this, of course, degrades system per-
3
formance tremendously.8
On the contrary, conducting the same test
with the application being XIP, this drop in
performance is much more gradual and not as
abrupt. At very low levels of free memory
space available (less than 5%) where regular
execution from RAM would result in a com-
plete system freeze, XIP is capable of still keep-
ing up a fairly good system performance and
overall responsiveness.
Moreover, the differences in speed men-
tioned above, between executing from NOR
flash and RAM can be alleviated by employing
larger-scaled CPU instruction caches that con-
tinuely pre-fetch soon to be executed chunks of
code from flash memory.
For instance, the Alchemy Au1100 MIPS32-
based processor, one that is part of one of the
systems used for the benchmarks described in
the last part of this article, is endowed with
a 16 kB instruction cache. As the percentage
of cache hits begins to increase at a steady
rate (relative to the number of cache flushes),
the differences between RAM and NOR access
speed begin indeed to smooth out, especially
for higher system loads. Performance is, of
course, not the same nor can it be; however,
instruction caches can have a noticeable impact
on the speed of execution from flash memory.
In summary it can be said that if a ma-
jor point of emphasis when making design
choices is placed on minimizing RAM utiliza-
tion, XIP can definitely be said to fulfill this
requirement at large. As having huge amounts
of RAM available does also always mean to
put up with a higher power consumption,
XIP can bring about an improvement because
task by an MMU (Memory Management Unit), a hardware
component to speed up page table look-ups.
7. If physical memory becomes exhausted, it is com-
mon to apply swapping. The principle is to temporally
store data from RAM on disk (“swap out”) as to forth-
with create space for other currently more exigent data.
8. The condition of high “memory pressure”, keeping
the system from doing anything useful, has generally
become to be known as thrashing.
an excessive amount of RAM becomes quite
unnecessary. Thus, it is especially useful for
applications that involve low-power consump-
tion and portability, such as cell phones, PDAs
and other portable media devices.
Furthermore, the concern that storing XIP
data uncompressed does necessarily imply to
use up more flash memory is only partially
justified. Making use of XIP in conjunction
with compressed read-only file systems can
also keep flash utilization relatively low if the
balance between compressed segments and
XIP segments on the flash is carefully kept and
adjusted according to the specific requirements
of the system.
Conclusively, XIP must be acknowledged to
be a viable solution that can be safely taken
into account by system engineers designing
embedded systems and smaller devices.
3 AXFS
Although XIP patches for cramfs (called “linear
cramfs”) have by now been literally used for
years, it had never been put to the Linux kernel
mainline, nor has it ever been cleanly imple-
mented in any well-defined way.
By designing and implementing AXFS (Ad-
vanced XIP File System) the author intended
to create a stand-alone file system that ele-
gantly supports application XIP while decreas-
ing RAM footprint and speeding up the execu-
tion of certain time critical code.
In a paper describing the motivations behind
AXFS, the author furthermore laments the fact
that in the past, Linux kernel developers seem
to have misperceived the notion of flash mem-
ory in that they have created abstraction layers
that allowed them to essentially treat flashes
like block devices, without making proper use
of the intricacies of flash as memory. In this
sense, the author speaks of a “Flash-as-block
paradigm”[6] that was to be overcome with
AXFS.
The main feature of AXFS, however, is that
4
individual pages of an application’s binary
can be marked for in-place execution, mean-
ing that only those pages will directly be run
from flash memory instead of being copied
(paged) to RAM first. This is a clear advantage
when compared to patched versions of cramfs,
where only applications in their entirety can,
by means of setting a “sticky bit”, be marked
for in-place execution.
A user usually decides which pages (or
chunks in AXFS terminology) should be
marked for XIP by applying a specific method
called profiling. By doing so, it is possible
to determine the most frequently called upon
portions of a program’s binary; these can then
be marked for XIP. Being able to mark pages
individually conveniently allows the user to
specifically pinpoint the hot-spots where exe-
cution time is spent most.
This has the advantage of offering true
eXecute-In-Place capabilities while still being
able to partially offer the benefits associated
with a compressed file system. In plain
words, space is saved through compression,
but where appropriate, no compression is ap-
plied in favor of in-place executability.
All other portions of a program that are
not marked for XIP remain compressed on the
flash and are only inflated and copied to RAM
when necessary; here, precisely the notion of
demand-paging which has been mentioned in
the previous section applies. Eventually, XIP
regions from flash and uncompressed blocks
now residing in RAM are used to piece to-
gether an application’s virtual memory space.
Figure 1 is intended to illustrate how the linear
virtual address space of an example process
might be composed.
Application code and data marked for XIP,
as was seen, is not, unlike with the unpatched
versions of cramfs or SquashFS, stored in com-
pressed form. Although this increases the size
9. In the schematic, assume the size of each shown
chunk of data to be a multiple of a the system’s default
page size.
 NOR Flash memory RAM Virtual memory
Low memory
XIP Code
Compr. Code .text
XIP
Code
XIP Uncompr.
Data
Compr. .data
Uncompr. Data

.bss

High memory

Figure 1: This diagram shows how elegantly AXFS combines its inherent eXecute-In-Place capabilities with
the effectiveness of file system compression. XIP regions and compressed blocks alike reside in
flash memory9 , here shown on the left-hand side. As indicated by the blue arrows, XIP regions
are directly mapped into the program’s virtual address space shown to the far right; a storing in
RAM of those is avoided entirely. This, however, is not so for compressed blocks; when required
during the program flow, page faults and are generated and the necessary pieces are loaded into
RAM (demand-paging). Once there, they are used to complete the virtual memory layout. It should
furthermore be noted that it does not matter in what section of a the process’s memory map XIP
regions eventually show up - they can be either, .text or .data.

of data residing in flash memory slightly, the memory.”[4]

author of AXFS argues for this to be an actual
advantage.
By running and profiling a particular pro-
gram it is indeed very likely to discover and
identify often executed chunks within the ap-
plication binary; for most programs, execution
time tends to cumulate very distinctively at
least in a few places.
The author’s main arguments for his file
system are based on the fact that with JFFS2,
SquashFS, and even cramfs lots of memory
space gets effectively wasted, and that with
AXFS, this is avoided altogether:
“Consider 2MB of file data being actively
used. It would exist both as 1MB com-
pressed in Flash (assuming 50% compression)
and 2MB uncompressed in RAM, 3MB of total
In spite of the severity of this argument,
imagine a system where many larger-sized
programs are to be executed at once. Regard-
ing this case, free memory space would dwin-
dle rather quickly because for each individual
application code and data sections would be
stored in RAM in their inflated state, thus
wasting a considerable amount of memory.
If, on the other hand, use of AXFS’ eXecute-
In-Place capabilities had been made, the many
program instances could have all been directly
executing from flash memory, thus circum-
venting the time-consuming process of read-
ing, decompressing and copying.
Proceeding in this line of reason of wasted
RAM space the author of AXFS assesses that
in the example from above “AXFS would use

5
2MB total memory instead of 3MB.” As AXFS
application code data exists uncompressed the
full 2MB would need to be stored in flash
memory. However, he concludes that “it uses
1MB extra flash and saves 2MB of RAM.” -
a definite advantage in the realms of limited
memory resources. In this sense the author
speaks of a 2:1 ratio by which AXFS always
beats its fellow contenders like cramfs.
The author of AXFS does also see another
advantage in the fact that with his file system,
paging, in certain cases, is practically unnec-
essary. This, he says, can boost up system
performance tremendously: “Eliminating the
overhead of paging (copying the compressed
data and decompressing) allows AXFS to be
much faster at booting systems and launching
applications.”
With all of the above said, it is safe to deem
AXFS a suitable file system in the context of
platforms that have to be contented with rather
low RAM resources while still having to be
able to maintain a level of good overall system
performance.
It should at last be mentioned that, as was
stated in the previous section, true in-place ex-
ecution is only possible with NOR flash chips.
To this end AXFS is luckily capable of operat-
ing on the basis of NAND flashes, too; this is
done by practically simulating execution from
a NOR flash in RAM. This happens as follows:
Pages, or chunks as they are called in AXFS
parlance, marked for XIP are silently copied
to RAM upon mounting the AXFS file system
image; this process may or may not incur a
noticeable latency period.
The actual execution of those XIP chunks
proceeds from RAM rather than from flash
memory itself. Naturally, this whole process
is much slower than doing true in-place exe-
cution using NOR flashes. Figure 2 aims to
illustrate the whole process just described.
One other feature AXFS offers and that
should at least be made mention of herein is
that it allows NOR and NAND flashes to be
6
combined in such a way as to have the first one
hold all the XIP data, and the latter to contain
the compressed payload.
4 Setting up AXFS
This section attempts to briefly outline the
steps required to successfully set up a Linux
box with AXFS.
First and foremost it is necessary to integrate
support for the AXFS file system into a rea-
sonably recent version of the Linux kernel. In
order to do so, a copy of the kernel must be
obtained; it could e.g. be downloaded from
http://www.kernel.org.
Having subsequently extracted the kernel
sources, the next step is to also download a
reasonably new version of AXFS. The easiest
way for this is to achieve is to do a check-
out from the developer’s SVN repository at
SourceForge:
svn co https://svn.sourceforge.net\
/svnroot/axfs
After having completed the download, one
can proceed from here on by entering the now
present branch directory. The ./patchin.pl
perl script10 that resides in one of the sub-
directories should be called with the appropri-
ate command line arguments to automate the
process of applying the patches that integrate
the AXFS sources into the present kernel tree.
./patchin.sh <path_to_kernel_source>
Fortunate is the one who does not run into
more deeply-rooted problems at this point.
However, fixing a few compiler complaints
by adjusting the source accordingly should in
many cases suffice to being the end of most
troubles.
The kernel should now be in a patched state
and is ready to be configured. This can be
10. Obviously, the perl interpreter must be installed on
the development host.
 NAND Flash memory RAM Virtual memory
Low memory
XIP XIP Code
Compr. XIP Code .text
XIP
Uncompr. Code
XIP
XIP Data

Compr. .data
Uncompr. Data

.bss

High memory

Figure 2: This schematic depicts how AXFS copes with NAND flashes which, in reality, are unsuitable for in-
place execution. Initially all, that is, compressed and uncompressed AXFS data structures alike (so-
called blocks and XIP regions), reside in flash memory, here shown on the left. Upon mounting the
AXFS file system image, those chunks marked for in-place execution (the XIP regions) are copied
to RAM, indicated by the blue lines. When then, during program execution, page faults occur,
the compressed blocks still in flash memory, are decompressed and loaded to RAM (indicated
by the red lines); together with the XIP chunks already loaded, they are used to complement the
process’s virtual memory layout show on the right-hand side. Note in all this that compared
to figure 1 (NOR), much more overall RAM is used-up, yielding practically no positive effects
towards memory preservation.

achieved by running make menuconfig in
the kernel source top-level directory. To enable
AXFS
Later on, when enough profiling informa-
tion have been gathered, the kernel can be
recompiled with profiling support disabled; in
fact, it is highly recommended to do so in order
to permanently improve performance for the
production release.
Finally, the new kernel can be compiled
with make prepare && make. After this, it
should now be deployable to the target device.
If module compilation of AXFS was opted for,
a quick make modules install should turn
up the module axfs.ko at the proper location;
it can then be copied and loaded into the run-
ning kernel on the target system.
The next step now is to compile and use the
mkfs.axfs utility that is used to create an
AXFS file system image. Having entered the
appropriate SVN directory and having com-
piled the tool, a directory tree that is populated
with a typical root file system structure as well
as all the relevant tools should be procured.
Invoking the AXFS mkfs.axfs utility in the
following way produces a valid file system
image:
mkfs.axfs <root_fs_dir> <img_name>
Here, the first argument denotes the root
directory mentioned above and the second one
is meant to be the name of the output file the
file system image will be written to.

7
 Once the new kernel is up and running, it is
about time to try out if it is capable of recog-
nizing and using the AXFS file system image
previously created. This is done by simply
mounting it.
First off, the image file should be made ac-
cessible on the target platform through means
like directly copying it to the device, NFS,
or similar methods.11 Before now actually
mounting it, it becomes necessary to create a
loop device12 node first; that is so that the
AXFS driver within the kernel can associate
itself with the image file when mounting later
on. The following command should be entered
in order to create the new device node:
mknod /dev/loop0 b 7 8
Eventually, the actual mounting may take
place. This can be achieved through the fol-
lowing command:13
mount -t axfs -o loop \
<axfs_root_img> <mountpoint>
Here, the last argument indicates the path to
a directory that serves as the immediate mount
point for the file system; it can practically be
any arbitrary location.
Having, with the previous step, confirmed
that the file system is indeed working and
useable, it is now time get the image deployed
to flash. It is very likely that the reader uses the
MTD (Memory Technology Device) subsystem,
which provides an abstraction layer for access-
ing flash memory as device blocks.14 The tool
eraseall, which is part of the mtdutils tool
suite, should be used to make sure that the
mtd device is fully erased before using it for
the AXFS file system image. After this step
is taken, the image can be simply copied onto
it. The following listing gives the necessary
commands:
eraseall /dev/mtdX
cp <axfs_root_img> /dev/mtdblockX
8
Here X is the number given to the mtd de-
vice.
As the last finalizing step it should be
made sure that the kernel command line con-
tains the right parameter to actually find
the new root file system. Therefore, one
should be certain that the kernel command
line contains something very similar to this:
root=/dev/mtdblockX.
Now, re-booting the platform, the AXFS root
file system is used. Note that at this stage,
everything is compressed; only the next step of
profiling reveals the segments on of the image
that should be marked for XIP. After this, the
file system image can be recreated using the
information gathered and to enable XIP.
5 AXFS Profiling
In order to make use of AXFS’ eXecute-In-Place
capabilities, it becomes necessary to profile one
or many target applications to see where most
of the execution cycles get spent; once identi-
fied, they can be easily marked for XIP. Recall
again that with AXFS, the XIP mechanism op-
erates on the basis of individual pages instead
of entire executables as a whole as it is done for
linear cramfs.
What basically happens on the file system
level when profiling is that the AXFS kernel
driver records the number of page faults that
occur for a specific location within the file sys-
tem memory map.
11. Note that for the purpose of simply mounting the
image, it is not yet required for it to be located on the
flash.
12. Loop devices allow file system images to be mounted
as if they were block devices. Block devices are most
frequently associated with hard disks.
13. Note that the mount utility that is supplied with
busybox might have problems to recognize the -t flag.
It is better to obtain and compile a full-fletched version
of mount that is part of the util-linux tool suit. It can
be downloaded from http://www.kernel.org/pub/
linux/utils/util-linux/.
14. For more information, see http://www.
linux-mtd.infradead.org/.
 Profiling a particular applications in practice
means to simply run in on the target platform
and to let the AXFS profiler do its work.15 It
will collect data and write the information it
could gather to /proc/axfs/volume0.
Profiling data therein is presented line-wise;
each line reveals how often which page was
called for, starting at the time the file system
got mounted. The format used specifies these
information separated by a comma. So for
instance, a line in the profiling file could look
similar to this:
/sbin/init, 2412, 8
This would mean that in the executable
saved as /sbin/init, page 2412 was up until
now called exactly 8 times. Calling the ap-
plication to be scrutinized a few times should
eventually yield a fairly clear picture of where
the hot spots are.
Having obtained this information, it now
becomes possible to regenerate the AXFS file
system (as previously described in section 4),
but this time with the desired pages marked
for XIP. The mkfs.axfs tool can receive an
additional argument using the -i flag which
points to a user-created file in the XML format;
it indicates which pages the user desires to
have marked for XIP within the file system
image.
An XML file created for this purpose does
typically have a format similar to the one pre-
sented in the following excerpt from such a file:
<xipfsmap>
<file>
<name>/bin/sed</name>
<chunk>
<size>4096</size>
<offset>0</offset>
</chunk>
<chunk>
<size>4096</size>
<offset>12512</offset>
</chunk>
...
</file>
9
...
</xipfsmap>
Multiple binary files and their XIP chunks
may be given in the XML file. For each chunk
specified, the size must be explicitly given.
This number usually defaults to the page size
typical for the current system which in most
cases defaults to 4 Kilobytes (4096 Bytes).
To all other pages not listed in the input XML
file, AXFS applies its compression algorithm;
the pages marked for XIP, however, remain
unaltered (uncompressed) altogether.
When subsequently executing an applica-
tion, the kernel internally decides which the
right method to be invoked for reading and
handling a specific page of the program is,
depending on whether it is compressed or
marked for XIP (uncompressed).
To give some last insight at this point, the fol-
lowing C-code snippet from the AXFS kernel
driver is shown; it initializes a structure that
contains pointer references to functions that
give the appropriate operation for retrieving
a page from flash, depending on whether the
page is XIP or not.
static struct
address_space_operations axfs_aops =
{
.readpage = axfs_readpage,
.get_xip_page = axfs_get_xip_page
};
After having successfully created the new
file system image, the steps outlined in the pre-
ceding section should be redone, only that the
kernel should now be recompiled with AXFS
profiling disabled.
6 Benchmarks
In the sections above it was freely stated that it
is possible to reduce memory footprint and to
speed up program execution times by marking
15. Note that for this to work the kernel must have
gotten compiled with AXFS profiling enabled as has been
described in the section prior.
certain sections of it for in-place execution.
That now to be shown by proof-of-concept and
by presenting the results of actual tests that
have been conducted.
The general idea of testing was to not only
measure run time (which is frequently done for
XIP benchmarks), but to also put the data ob-
tained for raw execution time into relation with
the sizes of the binary files that were executed.
Doing this, it is possible to get a much better
feel for how execution time changes as file size
grows.
In the succeeding section, the basic ideas and
principles behind this approach are explained.
6.1 Procedure
First, assume the following snippet of C-code
which in itself is a small program. Its unspec-
tacular functionality does nothing else but to
decrementally step through an array of 4 byte
integers while copying single data fields from
it.
#include <stdint.h>
const int32_t a[0x40000<<(SIZE)] =
{1};
int main()
{ done
register int i = \
sizeof a / sizeof(int32_t);
int32_t v;
while(--i >= 0) v = a[i];
return 0;
}
While executing, the while-loop of this
nifty program traverses the integer array and
internally causes page faults as it goes. By
copying from the array it is ascertained that
each single piece of array data will indeed be
truly read.
For non-XIP execution, this forces the kernel
to load the required pages from flash memory
10
and to store them in RAM, applying decom-
pression algorithms if necessary; there they
can now be accessed by the program. For
XIP, however, no copying takes place as data is
directly accessed on the flash - “in-place”, so to
speak. It goes without saying that this is much
faster.16
Furthermore, by making the global array’s
total length variable at compile time, it is now
conveniently possible to alter the actual file
size of the resulting executable by specifying
different values for SIZE. This can be achieved
by passing the compiler flag -DSIZE=X to
GCC, where X denotes the desired value17 .
Using this method, files of considerable size
can be produced.
If X equals zero, the binary’s size after com-
pilation is about 1 Megabyte. When increased
sequentially, the file size increases exponen-
tially with a base of two; thus, the file’s final
size is always at about 2X Megabytes.
Now, using the following simple shell script
it is easily possible generate a number of pro-
grams with identical functionality whose file
sizes increases rather rapidly.
#!/bin/sh
for i in $(seq 0 $1); do
gcc -DSIZE=$i -o $2$i $2.c
Assuming that the C-program from above
got saved as bmark.c and that the shell script
got saved as gen.sh, one may now invoke
the script as follows and thereby create eight
executables from which the largest on is about
27 = 128 Megabytes, that is half of the quarter
16. Notice that this is holds only true for the very first
round of execution. Upon subsequent invocations, the
virtual memory manager will have kicked in, aiding the
program with its caches that still contain the uncom-
pressed program data from previous executions; running
the program now literally proceeds from RAM rather
than from flash which is faster by nature.
17. Note that cross-compiler version of GCC for the
target platform was used
of a Gigabyte, in size.18
./gen.sh 8 bmark
After a short while, the results can be eas-
ily verified by issuing the command ls -1hs
that should result in an output similar to this:
1.1M bmark0 33M bmark5
2.1M bmark1 65M bmark6
4.1M bmark2 129M bmark7
8.1M bmark3
17M bmark4
As can be seen, each program’s size is about
as much as was predicted.19 Eventually, a pool
of ancillary programs that can be utilized for
the purpose of execution speed benchmarking
has been obtained.
To now gauge the definite run time of each
program, the Unix tool time is used which
is capable of measuring the elapsed real-time
and of displaying the result in seconds after the
process has finished running. For this to work,
the following command was run on the target
platform:
time -p ./bmarkX
Here, the X indicates the size of the program
according to the conventions devised above.
Waiting out the time the program takes to
complete, an output similar to this is given by
time:
real 0.21
user 0.02
sys 0.19
The interesting part here being the topmost
line, which is the elapsed real-time in seconds;
herein, this is taken as the measure of execu-
tion speed.
In all this, it is worth pointing out that be-
tween each successive step in the series of run-
ning the next larger program, it was made sure
that no data remained in the read buffers of the
11
kernel’s virtual memory manger by making
sure the page caches were cleared each time.
This was accomplished by running the follow-
ing command sequence:
sync && \
echo 3 > /proc/sys/vm/drop_caches
Even if it is unlikely that cached pages from
one application should be reused for another
one at all, clearing the page cache has at least
the effect of providing each program with an
empty starting ground; furthermore, the ker-
nel is now definitely forced to go through the
complete sequence of reading, decompressing,
and copying each page from flash memory
anew.
6.2 Results and evaluation
The testing procedure delineated in the section
prior was in this exact form applied to two
embedded systems which will subsequently be
referred to as system A and B.
Both systems are equipped with an RMI
Alchemy Au1x00 processing core which is
based on the MIPS32 processor architecture.
Whereas system A is endowed only with an
Au1100 CPU clocked at about 350 MHz, sys-
tem B is geared up with the newer succeeding
model, namely the Au1200, that runs at core
frequency of about 500 MHz. Both systems
have an on-board SRAM of 128 Megabytes in
size integrated into them. System A as well
as system B were running a Linux kernel of
18. 128 Mb was chosen as a limit because more RAM
was not available on the systems used for testing. Had an
even larger file size than this be used, the system would
have been likely to be in the condition of thrashing; that
is, most of the system’s resources would have been used
up to copy pages back and forth between swap space and
physical RAM.
19. The slight overhead that is discernible is probably
due to inaccuracies in measurement, compiler alignment,
and general program overhead as the array is not the
only piece of data that makes up a program as a whole.
version 2.6.22 with AXFS patched into to; pre- Size JFFS2 AXFS Delta Cached
emption20 was enabled and the slab allocator 1 0.08 0.05 0.03 0.02
was opted for. 2 0.15 0.10 0.05 0.04
The other file system besides AXFS that was 4 0.30 0.21 0.09 0.08
made use of for the purpose of obtaining com- 8 0.61 0.40 0.21 0.15
parable data was JFSS2 which is fully log- 16 1.20 0.83 0.37 0.29
structured (journaled) and zlib-compressed 32 2.42 1.59 0.82 0.60
file systems that is built on top MTD (Memory 64 4.91 3.00 1.91 1.23
Technology Device) kernel subsystem; it is gen- 128 9.66 5.91 3.75 2.50
erally accepted to be a well-performing read-
Table 2: Results for system B
/write file system for flashes.
By now successively running the bench-
marking application series bmarkX, the fol- one, execution time does also roughly appear
lowing two tables (1 and 2) of values, for to double in general. However, crucial to
system A and B respectively, were obtained. recognize is that, although the initial differ-
A more speaking representation is given by ences in speed may not seem too large for an
figures 3 and 4. executable of 1 Megabyte, the gap becomes
Note that in addition to the values for the rapidly wider as file size increases; this is easily
two file systems, a third set of data was gath- discernible in both of the bar charts. The rate of
ered along the way; it represents the run-times “chasm growth” (the delta change) settles also
of the programs shortly after the first round of roughly at about two.
execution for JFFS2 had been completed. By AXFS does not only beat JFFS2 at an increas-
that time, most of the application code and ingly better rate regarding execution time, but
data had already been cached in RAM so that one must also bear always in mind that for
swapping did in essence not occur, thus the JFFS2 this exact amount memory is actually
column “Cached”. The other fourth column used up in physical RAM, whereas for AXFS
termed “Delta” just shows the differences in none is consumed at all. Moreover, JFFS2 does
time between JFFS2 and AXFS. also occupy a considerable amount of flash
memory to store the exact same data, only this
Size JFFS2 AXFS Delta Cached time with compression applied.
1 0.11 0.05 0.06 0.02 So for instance, taking the case with an ex-
2 0.20 0.12 0.08 0.05 ecutable file size of 32 Megabytes and esti-
4 0.39 0.25 0.14 0.09 mating JFFS2’s applied zlib compression with
8 0.77 0.51 0.26 0.17 50%21 , there is 16 Megabytes stored com-
16 1.50 1.03 0.47 0.35 pressed on flash as well as 32 Megabytes stored
32 2.99 2.01 0.98 0.69 uncompressed in RAM adding up to a total
64 5.91 4.06 1.85 1.37 of 48 Megabytes. Compared to AXFS, 16
128 12.31 8.11 4.22 2.83 Megabytes got thus wasted.
Table 1: Results for system A 20. Preemption allows the kernel to interrupt the cur-
rently running task at any given point in favor of another
more predominant one. Under usual circumstances,
execution is guaranteed to return to this point; however,
Taking a first casual glance at the data at
this may happend after an indeterminable period of time
hand unsurprisingly reveals that with each has elapsed.
consecutive execution of the next larger file 21. In most real cases this ratio stays somewhat below
that is twice as much in size as the previous 50%.

12
 12

8
10

JFFS2
8

AXFS
Execution Time (sec)

Execution Time (sec)

6
Cached

JFFS2
AXFS
6

Cached

4
4

2
2
0

0
2 4 8 16 32 64 128 2 4 8 16 32 64 128

Executable Size (Mb) Executable Size (Mb)

Figure 3: Speeds on System A Figure 4: Speeds on System B

Therefore, the gain from using AXFS for
larger files is twofold, once in execution speed
and once in the amount of RAM saved.22
Stated quantitatively, AXFS has, for both
system tested, brought a gain of about 35%
in overall execution speed compared to JFFS2.
Nevertheless, once cached, AXFS is certain to
be at the short end; here it is approximately
60% slower than SRAM. In section 2 it was
indirectly expressed that NOR flash access
speeds are around 75% slower than they are for
SRAM. The ratio observed here is not quite as
bad but it comes directionally near it.
One finding that comes as no big surprise is
that system B is generally faster than system
A when overlooking the range of data more
broadly. The ratio at which system B beats A
is roughly 20%, relative to system B; this does
not appear to change much for either of the file
systems, nor does the executable file size seem
to matter much in this regard.
Interestingly enough, the execution speeds,
that for RAM are directly correlatable with
memory access speeds, appears to differ only
marginally, leaning a little more towards the
faster system B.
All that was examined in the above discus-
sion is conducive to the conclusion that in cases
where program images to be executed become
larger, the benefits for execution speed when
using AXFS begin to show more distinctively.
Adding in the factor that multiple instances
of the same larger-sized program might be
running at once, or that more than only one
memory-heavy application is run, makes the
mentioned benefits appear even clearer.
7 Prospects
Currently everything makes the appearance as
if NAND will continue to push NOR flashes
further out of the market sector. NAND
is available at lower “price per bit”, has
faster access speeds for bulky data, and can
22. One might bring up the argument that file sizes of 64
Mb and more are beyond reality in embedded systems;
however, there exist such memory-intensive applications
even on smaller devices.

13
generally store more data.23 Values for
power-consumption are relatively comparable.
NAND is deemed faster for writing, whereas
NOR is generally believed to be more effective
for reading.
Notwithstanding these facts, there have
been various attempts endeavoring to redesign
NAND chips in such a way as to render them
palatable for XIP mechanisms to take place;
this, however, is a difficult task in and of itself.
Most of the solutions proposed and imple-
mented involve at least some form or another
of temporary buffering.24 Plainly asserted, re-
garding a NAND chip’s internal organization,
there simply is no effortlessly feasible way for
linear reads and random access to occur; this,
however, is a fundamental requirement neces-
sary for eXecute-In-Place.
An imaginable future for XIP is advertised
by exploiting the expedient synergy that might
result in the combination of NOR and NAND
flashes in one and the same device; if NAND
- because of its cost-effectiveness, speed, and
larger allowable storage volume - and NOR
- because of its faster reading speeds and in-
herent XIP capabilities - were to be dually
incorporated, the combined power of both
could be harnessed by finding the right bal-
ance between XIP and data storage require-
ments. AXFS has gone an important step into
this direction by making exactly this one of its
features. Perhaps, adding other technologies
like PSRAM (Pseudo-Static RAM) into the mix
might even be worth a consideration, too.
One pro-argument for XIP might further-
more be that many devices, especially cell
phones, PDAs and some routers, are already
equipped with NOR flashes by design. If flash
memory is available in any case it might as
well be utilized for purposes besides exclu-
sively storing persistent data.
After all, the final reckoning remains with
the engineers who are trusted to make the right
design choices for their specific requirements -
if, in the future, XIP turns out to be useful in
14
other places than cell phones it will likely be
seen around more often.
AXFS specifically is still rather fresh and
documentation for it other than the source
code itself is scarce if at all present. Luckily, it
is sufficient simple and can be understood by
almost anybody having a good proficient level
of understanding in C and some experience in
kernel programming related topics.
With the advent of AXFS, some good ideas
were put into practice, especially regarding ap-
plication XIP. If propagated more in the future,
it might likely change how embedded open
source developers judge the possible benefits
gained from using XIP.
Acknowledgements
First of all, much thanks must gratefully be
given to the Ultratronik Entwicklungs GmBH
for supporting the efforts of this article by pro-
viding the embedded boards used for testing.
Thanks does also go to Jared Hubert, the
author of AXFS, for kindly answering a few
questions.
References
[1] Jared Hulbert, Justin Treon, “Creating opti-
mized XIP systems”, Intel, 2006
[2] “Execute-in-place for file mappings”,
http://www.mjmwired.net/kernel/
Documentation/filesystems/xip.
txt, 2009
[3] “NAND vs. NOR Flash Memory - Technol-
ogy Overview”, Toshiba America Electron-
ics Components, Inc., 2008
23. As of this writing, common-use NAND flashes are
capable to hold data up 16 Gigabyte, whereas for NOR,
it is at about 4 Gigabyte.
24. Key terms often associated with these approaches
are i.e. code-shadowing, sliding-window data retrieval, or
fast-paging. The OneNAND technology is, for instance,
one that makes use of some of these concepts.
[4] Jared Hulbert, “[RFC] Advanced XIP File
System”, http://lwn.net/Articles/
182337, 2006

[5] Jonathan Corbet, “AXFS: a compressed,

execute-in-place filesystem”, http://lwn.
net/Articles/295545, 2008

[6] Jared Hulbert, “Introducing the Advanced

XIP File System”, Proceedings of the Linux
Symposium, 2007

15
Copyright
Copyright (c) 2009 Sören Wellhöfer
Permission is granted to copy, distribute and/or modify this document under the
terms of the GNU Free Documentation License, Version 1.3 or any later version
published by the Free Software Foundation; with no Invariant Sections, no Front-Cover
Texts, and no Back-Cover Texts. A copy of the license is can be obtained from
http://www.gnu.org/copyleft/fdl.html

16