Professional Documents
Culture Documents
and Others
Revision History Revision v1.0.3 Minor Fixes Revision v1.0.2 RPM Build Revision v1.0.1 Major updates Revision v1.0 Minor updates Revision v1.0 At last Revision v1.0 RC 1 Major Cleaning Revision v0.95 Replaced ClumpOS by PlumpOS Revision v0.94 Patches by Mirko Caserta Revision v0.93 Extra features and fixes Revision v0.92 Revision v0.91 Revision v0.90 18 june 2004 29 july 2003 19 july 2003 09 july 2003 11 may 2003 07 may 2003 04 april 2003 25 february 2003 16 february 2003 21 january 2003 27 september 2002 03 september 2002
Revision v0.71 26 August 2002 Spleling Fexis Revision v0.70 22 August 2002 Stripped out empty parts, replaced Mosixview with openMosixView Revision v0.50 6 July 2002
First openMosix HOWTO Revision v0.20 Latest Mosix HOWTO (for now) Revision v0.17 Revision v0.15 Revision v0.13 Revision ALPHA 0.03
5 July 2002 28 June 2002 13 March 2002 18 Feb 2002 09 October 2001
"The best way to become acquainted with a subject is to write a book about it." (Benjamin Disraeli)
Table of Contents
Chapter 1. Introduction ......................................................................................................................................1 1.1. openMosix HOWTO .........................................................................................................................1 1.2. Introduction.......................................................................................................................................1 1.3. Disclaimer.........................................................................................................................................1 1.4. Distribution policy............................................................................................................................2 1.5. New versions of this document.........................................................................................................2 1.6. Feedback...........................................................................................................................................2 Chapter 2. So what is openMosix Anyway ?....................................................................................................3 2.1. A very, very brief introduction to clustering....................................................................................3 2.1.1. A very, very brief introduction to clustering...........................................................................3 2.2. The story so far.................................................................................................................................5 2.2.1. Historical Development ..........................................................................................................5 . 2.2.2. openMosix...............................................................................................................................6 2.2.3. Current state............................................................................................................................6 2.2.4. Which applications work.........................................................................................................7 2.3. openMosix in action: An example....................................................................................................7 2.4. Components......................................................................................................................................7 2.4.1. Process migration....................................................................................................................7 2.4.2. The openMosix File System (oMFS)......................................................................................7 2.4.3. Direct File System Access (DFSA).........................................................................................8 2.5. openMosix Test Drive .......................................................................................................................8 2.6. Pros of openMosix............................................................................................................................8 2.7. Cons of openMosix...........................................................................................................................8 II. Installing openMosix ....................................................................................................................................10 Chapter 3. Requirements and Planning ..........................................................................................................11 3.1. Hardware requirements...................................................................................................................11 3.2. Hardware Setup Guidelines............................................................................................................11 3.3. Software requirements....................................................................................................................11 3.4. Planning your Cluster.....................................................................................................................11 3.5. Classrooms......................................................................................................................................12 Chapter 4. Distribution specific installations.................................................................................................13 4.1. Installing openMosix......................................................................................................................13 4.2. Before getting openMosix ...............................................................................................................13 4.3. Getting openMosix.........................................................................................................................13 4.4. openMosix General Instructions.....................................................................................................14 4.4.1. Kernel Compilation...............................................................................................................14 4.4.2. Syntax of the /etc/openmosix.map file..................................................................................15 4.4.3. oMFS.....................................................................................................................................16 4.5. Red Hat and openMosix ..................................................................................................................18 4.6. Suse and openMosix.......................................................................................................................19 4.7. Debian and openMosix...................................................................................................................19 4.8. openMosix and Gentoo...................................................................................................................20 4.9. Other distributions..........................................................................................................................20
Table of Contents
Chapter 5. Autodiscovery.................................................................................................................................21 5.1. Easy Configuration.........................................................................................................................21 5.2. Compiling autodiscovering ...........................................................................................................22 5.3. Troubleshooting autodiscovery .......................................................................................................22 Chapter 6. PlumpOS .........................................................................................................................................24 6.1. What Plump/OS is..........................................................................................................................24 6.2. How does it work?..........................................................................................................................24 6.3. Requirements..................................................................................................................................24 6.4. Getting Started................................................................................................................................25 Chapter 7. Cluster Installation........................................................................................................................27 7.1. Cluster Installations........................................................................................................................27 7.2. DSH, Distributed Shell...................................................................................................................27 III. Administrating openMosix........................................................................................................................29 Chapter 8. Administrating openMosix...........................................................................................................30 8.1. Basic Administration......................................................................................................................30 8.2. Configuration..................................................................................................................................30 8.3. the userspacetools.........................................................................................................................31 8.4. Cluster Mask...................................................................................................................................33 Chapter 9. Tuning Mosix.................................................................................................................................35 9.1. Introduction.....................................................................................................................................35 9.2. Creating a "Master" node................................................................................................................35 9.3. Optimizing Mosix...........................................................................................................................35 9.4. Channel Bonding Made Easy ..........................................................................................................36 9.5. Updatedb.........................................................................................................................................36 9.6. openMosix and FireWire................................................................................................................36 Chapter 10. openMosixview.............................................................................................................................37 10.1. Introduction...................................................................................................................................37 10.2. openMosixview vs Mosixview.....................................................................................................37 10.3. Installation....................................................................................................................................38 10.3.1. Installation of the RPMdistribution ...................................................................................38 10.3.2. Installation of the sourcedistribution .................................................................................38 10.3.3. Automatic setupscript.......................................................................................................38 10.3.4. Manual compiling ................................................................................................................38 10.3.5. Hints....................................................................................................................................39 10.4. using openMosixview...................................................................................................................39 10.4.1. main application..................................................................................................................39 10.4.2. the configurationwindow..................................................................................................40 10.4.3. advancedexecution............................................................................................................41 10.4.4. the commandline...............................................................................................................42 10.4.5. the hostchooser..................................................................................................................42 10.4.6. the parallel hostchooser.....................................................................................................42 10.5. openMosixprocs............................................................................................................................43 ii
Table of Contents
Chapter 10. openMosixview 10.5.1. intro.....................................................................................................................................43 10.5.2. the migratorwindow..........................................................................................................43 10.5.3. managing processes from remote........................................................................................44 10.6. openMosixcollector......................................................................................................................45 . 10.7. openMosixanalyzer.......................................................................................................................46 10.7.1. the loadoverview...............................................................................................................46 10.7.2. statistical informations about a clusternode......................................................................47 10.7.3. the memoryoverview .........................................................................................................47 10.7.4. openMosixhistory................................................................................................................48 10.8. openMosixmigmon.......................................................................................................................49 10.8.1. General................................................................................................................................49 10.8.2. Tooltips:..............................................................................................................................50 . 10.8.3. Drag'n Drop! ........................................................................................................................50 10.9. openmosixview FAQ....................................................................................................................50 10.10. openMosixview + ssh:................................................................................................................51 Chapter 11. Other openMosix related Programs ...........................................................................................53 11.1. Introduction...................................................................................................................................53 11.2. openMosixView............................................................................................................................53 11.3. openMosixapplet...........................................................................................................................53 11.4. wmonload ......................................................................................................................................53 11.5. openMosixWebView....................................................................................................................53 Chapter 12. Common Problems......................................................................................................................54 12.1. Introduction...................................................................................................................................54 12.2. My processes won't migrate ..........................................................................................................54 12.3. I don't see all my nodes.................................................................................................................57 12.4. I often get errors: No such process...............................................................................................57 12.5. DFSA ? MFS ?..............................................................................................................................57 12.6. Python Troubles............................................................................................................................58 Chapter 13. Hints and Tips..............................................................................................................................59 13.1. Locked Processes..........................................................................................................................59 13.2. Choosing your processes ...............................................................................................................59 13.3. Java and openMosix ......................................................................................................................59 13.4. openMosix and Hyperthreading ....................................................................................................59 13.5. openMosix and Firewalls..............................................................................................................59 Chapter 14. (stress)Testing your openMosix installation..............................................................................61 14.1. A small Test Script.......................................................................................................................61 14.2. Perl Proggie by Charles Nadeau...................................................................................................61 14.3. the openMosix stresstest.............................................................................................................64 14.3.1. General description ..............................................................................................................64 14.3.2. Detailed description .............................................................................................................65 14.3.3. Installing the strestest suite ..................................................................................................65 14.3.4. Running the tests.................................................................................................................66
iii
Table of Contents
IV. Running Applications on openMosix........................................................................................................67 Chapter 15. Improving Compiling Performance...........................................................................................68 15.1. Introduction...................................................................................................................................68 Chapter 16. Imaging with openMosix.............................................................................................................69 16.1. Introduction...................................................................................................................................69 16.2. Povray...........................................................................................................................................69 Chapter 17. BioInformatics and openMosix..................................................................................................73 17.1. Introduction...................................................................................................................................73 17.2. Blast..............................................................................................................................................73 V. openMosix Development ..............................................................................................................................74 Chapter 18. Getting started with openMosix internals.................................................................................75 18.1. Introduction...................................................................................................................................75 VI. FAQ..............................................................................................................................................................76 Chapter 19. the openMosix FAQ.....................................................................................................................77 19.1. General..........................................................................................................................................77 19.2. Getting, building, installing and running openMosix...................................................................78 19.3. Kernel Questions ...........................................................................................................................79 19.4. File Systems..................................................................................................................................80 19.5. Programming openMosix.............................................................................................................80 19.6. Resources......................................................................................................................................81 19.7. Miscellaneous...............................................................................................................................81 Chapter 20. PlumpOS FAQ.............................................................................................................................84 20.1. Frequently Asked Questions.........................................................................................................84 Appendix A. How to produce openMosix's Kernel RPM files ......................................................................86 A.1. How to produce openMosix's Kernel RPM files...........................................................................86 Appendix B. More Info.....................................................................................................................................88 B.1. irc ....................................................................................................................................................88 B.2. Further Reading..............................................................................................................................88 B.3. Translations....................................................................................................................................88 B.3.1. Chinese.................................................................................................................................88 B.3.2. Spanish ..................................................................................................................................88 B.3.3. Russian ..................................................................................................................................88 B.4. Links...............................................................................................................................................88 B.5. Mailing List....................................................................................................................................89 Appendix C. Credits.........................................................................................................................................90
iv
Table of Contents
Appendix D. GNU Free Documentation License...........................................................................................92 0. PREAMBLE......................................................................................................................................92 1. APPLICABILITY AND DEFINITIONS..........................................................................................92 2. VERBATIM COPYING....................................................................................................................93 3. COPYING IN QUANTITY...............................................................................................................93 4. MODIFICATIONS............................................................................................................................94 5. COMBINING DOCUMENTS ...........................................................................................................95 6. COLLECTIONS OF DOCUMENTS................................................................................................95 7. AGGREGATION WITH INDEPENDENT WORKS.......................................................................96 8. TRANSLATION ................................................................................................................................96 9. TERMINATION................................................................................................................................96 10. FUTURE REVISIONS OF THIS LICENSE ...................................................................................96 How to use this License for your documents.........................................................................................97 Index......................................................................................................................................................97
Chapter 1. Introduction
1.1. openMosix HOWTO
In the beginning there was Mosix, then came openMosix, in my opinion a more interesting project. Not only from a technical point of view but also due to the more correct license. I made the decision to focus this HOWTO on openMosix rather than on Mosix, mainly based on the fact that openMosix has a bigger userbase. (Moshe Bar states that about 97% of the old Mosix community has switched over to openMosix.) (20020705) Given the above, lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest release of the Mosix HOWTO, containing info about both Mosix and OpenMosix will be 0.20 My intention is to focus on the openMosix HOWTO, however not neglecting the Mosix users. More info on http://howto.ipng.be/MosixHOWTO/
1.2. Introduction
This document gives a brief description of openMosix, a software package that turns a network of GNU/Linux computers into a computer cluster. Along the way, some background to parallel processing is given, as well as a brief introduction to programs that make special use of openMosix's capabilities. The HOWTO expands on the documentation as it provides more background information and discusses the quirks of various distributions. Since the creation of this HOWTO some people of the Mosix team created openMosix (more info later), initially both openMosix and Mosix were discussed in this HOWTO. Although lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest relase of the Mosix HOWTO, containing info about both Mosix and OpenMosix will be 0.20 and can be found on http://howto.ipng.be/MosixHOWTO/MosixHOWTO/ Kris Buytaert got involved in this piece of work when Scot Stevenson was looking for somebody to take over the Job: this was during February 2002. While initially we discussed both Mosix and openMosix, this version of the HOWTO now mainly focuses on openMosix. Please note that the document often still mentions Mosix where it should read openMosix. You will notice that some of the headings are not as serious as they should be. Scot had planned to write the HOWTO in a slightly lighter style, as the world (and even the part of the world with a burping penguin as a mascot) is full of technical literature that is deadly. Therefore some parts still have these comments.
1.3. Disclaimer
Use the information in this document at your own risk. I disavow potential liability for the contents of this document. Use of these concepts, examples, and/or other content of this document is entirely at your own risk. All copyrights are owned by their respective owners, unless specified otherwise. Use of a term in this document should not be regarded affecting the validity of any trademark or service mark. openMosix is Copyright (c) by Moshe Bar. Mosix is Copyright (c) by Amnon Barak. Linux is a Registered Trademark of Linus Torvalds. openMosix is licensed under version 2 of the GNU General Public License as published by the Free Software Foundation. Naming of particular products or brands should not be seen as endorsements. Chapter 1. Introduction 1
The openMosix HOWTO You are strongly recommended to take a backup of your system before major installation and backups at regular intervals.
1.6. Feedback
Currently this HOWTO is being maintained by Kris Buytaert. Please do send remarks and updates for the howto to him. If you have a technical question about openMosix/Mosix itself, please post them on the more appropriate mailing list.
Chapter 1. Introduction
The openMosix HOWTO 2.1.1.2. Supercomputers vs. clusters Traditionally Supercomputers have only been built by a selected number of vendors: a company or organization that required the performance of such a machine had to have a huge budget available for its Supercomputer. Lots of universities could not afford the costs of a Supercomputer by themselves, therefore other alternatives were being researched by them. The concept of a cluster was born when people first tried to spread different jobs over more computers and then gather back the data those jobs produced. With cheaper and more common hardware available to everybody, results similar to real Supercomputers were only to be dreamed of during the first years, but as the PC platform developed further, the performance gap between a Supercomputer and a cluster of multiple personal computers became smaller. 2.1.1.3. Cluster models [(N)UMA, PVM/MPI] There are different ways of doing parallel processing: (N)UMA, DSM, PVM and MPI are all different kinds of Parallel Processing schemes. Some of them are implemented in hardware, others in software, others in both. (N)UMA ((Non)Uniform Memory Access), machines for example have shared access to the memory where they can execute their code. In the Linux kernel there is a NUMA implementation that varies the memory access times for different regions of memory. It then is the kernel's task to use the memory that is the closest to the CPU it is using. DSM aka Distributed Shared memory, has been implemented in both software and hardware , the concept is to provide an abstraction layer for physically distributed memory. PVM and MPI are the tools that are most commonly being used when people talk about GNU/Linux based Beowulfs. MPI stands for Message Passing Interface. It is the open standard specification for message passing libraries. MPICH is one of the most used implementations of MPI. Next to MPICH you also can find LAM, another implementation of MPI based on the free reference implementation of the libraries. PVM or Parallel Virtual Machine is another cousin of MPI that is also quite often being used as a tool to create a Beowulf. PVM lives in user space so no special kernel modifications are required: basically each user with enough rights can run PVM. 2.1.1.4. openMosix's role The openMosix software package turns networked computers running GNU/Linux into a cluster. It automatically balances the load between different nodes of the cluster, and nodes can join or leave the running cluster without disruption of the service. The load is spread out among nodes according to their connection and CPU speeds. Since openMosix is part of the kernel and maintains full compatibility with Linux, a user's programs, files, and other resources will all work as before without any further changes. The casual user will not notice the difference between a Linux and an openMosix system. To her, the whole cluster will function as one (fast) GNU/Linux system. openMosix is a Linuxkernel patch which provides full compatibility with standard Linux for IA32compatible platforms. The internal loadbalancing algorithm transparently migrates processes to other Chapter 2. So what is openMosix Anyway ? 4
The openMosix HOWTO cluster members. The advantage is a better loadsharing between the nodes. The cluster itself tries to optimize utilization at any time (of course the sysadmin can affect the automatic loadbalancing by manual configuration during runtime). This transparent processmigration feature makes the whole cluster look like a BIG SMPsystem with as many processors as available clusternodes (of course multiplied with X for Xprocessor systems such as dual/quad systems and so on). openMosix also provides a powerful optimized File System (oMFS) for HPCapplications, which unlike NFS provides cache, time stamp and link consistency.
GNU/Linux was chosen as a development platform for the 7th incarnation in 1999. Early 1999 Mosix M06 Beta was released for Linux 2.2.1 At the end of 2001 and early 2002 openMosix, the open version of Mosix was born (more in the next paragraph). Chapter 2. So what is openMosix Anyway ? 5
2.2.2. openMosix
openMosix is in addition to whatever you find at mosix.org and in full appreciation and respect for Prof. Barak's leadership in the outstanding Mosix project. Moshe Bar has been involved for a number of years with the Mosix project (www.mosix.com) and was coproject manager of the Mosix project and general manager of the commercial Mosix company. After a difference of opinions on the commercial future of Mosix, he has started a new clustering company Qlusters, Inc. and Prof. Barak has decided not to participate for the moment in this venture (although he did seriously consider joining) and held long running negotiations with investors. It appears that Mosix is not any longer supported openly as a GPL project. Because there is a significant user base out there (about 1000 installations worldwide), Moshe Bar has decided to continue the development and support of the Mosix project under a new name: openMosix and under the full GPL2 license. Whatever code in openMosix comes from the old Mosix project is Copyright 2002 by Amnon Barak. All the new code is Copyright 2002 by Moshe Bar. There could (and will) be significant changes in the architecture of the future openMosix versions. New concepts about autoconfiguration, nodediscovery and new userland tools are being discussed in the openMosix mailing lists. Most of these new functionalities are already implemented while some of them, such as DSM (Distributed Shared Memory) are still being worked on at the moment I write this (march 2003). To approach standardization and future compatibility the procinterface has changed from /proc/mosix to /proc/hpc and the /etc/mosix.map was changed to /etc/hpc.map. More recently the standard for the config file has been set to be located in /etc/openmosix.map (this is in fact the first config file the /etc/init.d/openmosix script will look for). Adapted commandline userspace tools for openMosix are already available on the webpage of the project. The openmosix.map config file can be replaced with a nodeautodiscovery system which is called omdiscd (openMosix auto DISCovery Daemon) about which we will discuss later. openMosix is supported by various competent people (see openmosix.sourceforge.net) working together around the world. The main goal of the project is to create a standardized clusteringenvironment for all kinds of HPCapplications. openMosix has also a project webpage at http://openMosix.sourceforge.net with a CVS tree and mailinglists for developers as well as users.
2.4. Components
2.4.1. Process migration
With openMosix you can start a process on one machine and find out it actually runs on another machine in the cluster. Each process has its own Unique Home Node (UHN) where it gets created. Migration means that a process is splitted in 2 parts, a user part and a system part. The user part will be moved to a remote node while the system part will stay on the UHN. This systempart is sometimes called the deputy process: this process takes care of resolving most of the system calls. openMosix takes care of the communication between these 2 processes.
The openMosix HOWTO Shared memory issues (an alpha release of DSM should be available as of late march 2003). There are issues with Multiple Threads not gaining performance. You won't gain performance when running one single process such as your web browser on an openMosix Cluster: the process won't spread itself over the cluster. Except of course your process will migrate to a more performant machine.
10
11
The openMosix HOWTO In a Singlepool configuration all the servers and workstations are used as a single cluster: each machine is a part of the cluster and can migrate processes to each other existing node. This of course makes your workstation a part of the pool. In an environment that is called a Serverpool, servers are a part of the cluster while workstations aren't part of it, they don't even have openMosix kernel. If you want to run applications on the cluster you will need to specifically log on to these servers. However your workstation will also stay clean and no remote processes will migrate to it. A third alternative is called an Adaptivepool configuration: here servers are shared while workstations join or leave the cluster. Imagine your workstation being used during daytime by yourself but, as soon as you log out in the evening, a script tells the workstation to join the cluster and start crunching numbers. This way your machine is being used while you don't need it. If you need the resources of the machine again just run the openmosix stop script and your processes will stay away from the cluster and viceversa. Practically this means that you will change the role of your machine by using mosctl.
3.5. Classrooms
Although it might seem a good idea to convert your classroom into an openMosix cluster at night, you'll have to consider training your end users not to pull the power switch of those machines when they want to use them again. More recent machines support automatic shutdowns when hitting the power button, but with older machines you might loose some data when this actually happens.
12
13
At the password prompt, just type enter since you're doing an anonymous login. Please take care that CVS trees DO BREAK now and then and that it might not be the easiest way to install openMosix ;)
In the rare case you don't have "zcat" on your system, do:
mv /root/openMosix2.4.202.gz /usr/src/linux2.4.20 cd /usr/src/linux2.4.20 gunzip openMosix2.4.202.gz cat openMosix2.4.202 | patch Np1
If the even more weird case you don't have a "cat" on your system (!), do:
mv /root/openMosix2.4.202.gz /usr/src/linux2.4.20 cd /usr/src/linux2.4.20 gunzip openMosix2.4.202.gz patch Np1 < openMosix2.4.202
The "patch" command should now display a list of patched files from the kernelsources. If you feel adventurous enough, enable the openMosix related options in the kernelconfiguration file, e.g.
... CONFIG_MOSIX=y # CONFIG_MOSIX_TOPOLOGY is not set CONFIG_MOSIX_UDB=y # CONFIG_MOSIX_DEBUG is not set # CONFIG_MOSIX_CHEAT_MIGSELF is not set CONFIG_MOSIX_WEEEEEEEEE=y CONFIG_MOSIX_DIAG=y CONFIG_MOSIX_SECUREPORTS=y CONFIG_MOSIX_DISCLOSURE=3 CONFIG_QKERNEL_EXT=y
14
However, it's going to be pretty much easier if you configure the above options using one of the Linuxkernel configuration tools:
make config | menuconfig | xconfig
The above means you have to choose one of "config", "menuconfig", and "xconfig". It's a matter of taste. By the way, "config" is going to work on any system; "menuconfig" needs the curses libraries installed while "xconfig" needs an installed Xwindow environment plus the TCL/TK libraries and interpreters. Now compile it with:
make dep bzImage modules modules_install
After compilation install the new kernel with the openMosix options within you bootloader; e.g. insert an entry for the new kernel in /etc/lilo.conf and run lilo after that. Reboot and your openMosixclusternode is up!
or
1 2 3 4 1 192.168.1.1 192.168.1.2 192.168.1.3 192.168.1.4 192.168.1.1
or with the help of the rangesize both of the above examples equal to: openMosix "countsup" the last byte of the ipaddress of the node according to its openMosixNode_ID. Of course, if you use a rangesize greater than 1 you have to use ipaddresses instead of hostnames. If a node has more than one networkinterface it can be configured with the ALIAS option in the rangesize field (which equals to setting the rangesize to 0) e.g. Chapter 4. Distribution specific installations 15
Here the node with the openMosixNode_ID 4 has two networkinterfaces (192.168.1.4 + 192.168.10.10) which are both visible to openMosix. Always be sure to run the same openMosix version AND configuration on each of your Cluster's nodes! Start openMosix with the "setpe" utility on each node :
setpe w f /etc/openmosix.map
Execute this command (which will be described later on in this HOWTO) on every node in your openMosix cluster. Alternatively, you can grab the "openmosix" script which can be found in the scripts directory of the userspacetools, copy it to the /etc/init.d directory, chmod 0755 it, then use the following commands as root:
/etc/init.d/openmosix stop /etc/init.d/openmosix start /etc/init.d/openmosix restart
4.4.3. oMFS
First of all, the CONFIG_MOSIX_FS option in the kernel configuration has to be enabled. If the current kernel was compiled without this option, then recompilation with this option enabled is required. Also the UIDs (User IDs) and GIDs (Group IDs) on each of the clusters' nodes filesystems must be the same. You might want to accomplish this using openldap. The CONFIG_MOSIX_DFSA option in the kernel is optional but of course required if DFSA should be used. To mount oMFS on the cluster there has to be an additional fstabentry on each node's /etc/fstab. in order to have DFSA enabled:
mfs_mnt /mfs mfs dfsa=1 0 0
the syntax of this fstabentry is: After mounting the /mfs mountpoint on each node, each node's filesystem is going to be accessible through the /mfs/[openMosixNode_ID]/ directories. With the help of some symbolic links all clusternodes can access the same data e.g. /work on node1
16
Now every node can read+write from and to /work ! The following special files are excluded from the oMFS: the /proc directory special files which are not regularfiles, directories or symbolic links (e.g. /dev/hda1) Creating links like:
ln s /mfs/1/mfs/1/usr
or
ln s /mfs/1/mfs/3/usr
is invalid. The following system calls are supported without sending the migrated process (which executes this call on its home (remote) node) going back to its home node: read, readv, write, writev, readahead, lseek, llseek, open, creat, close, dup, dup2, fcntl/fcntl64, getdents, getdents64, old_readdir, fsync, fdatasync, chdir, fchdir, getcwd, stat, stat64, newstat, lstat, lstat64, newlstat, fstat, fstat64, newfstat, access, truncate, truncate64, ftruncate, ftruncate64, chmod, chown, chown16, lchown, lchown16, fchmod, fchown, fchown16, utime, utimes, symlink, readlink, mkdir, rmdir, link, unlink, rename Here are situations when system calls on DFSA mounted filesystems may not work: different mfs/dfsa configuration on the clusternodes dup2 if the second filepointer is nonDFSA chdir/fchdir if the parent dir is nonDFSA pathnames that leave the DFSAfilesystem when the process which executes the systemcall is being traced if there are pending requests for the process which executes the systemcall Next to the /mfs/1/ /mfs/2/ and so on files you will find some other directories as well.
Table 41. Other Directories /mfs/here /mfs/home /mfs/magic The current node where your process runs Your home node The current node when used by the "creat" system call (or an "open" with the "O_CREAT" option) otherwise, the last node on which an oMFS magical file was successfully created (this is very useful for creating temporaryfiles, then immediately unlinking them) /mfs/lastexec The node on which the process last issued a successful "execve" systemcall. /mfs/selected Chapter 4. Distribution specific installations 17
The openMosix HOWTO The node you selected by either your process itself or one of its ancestor's (before forking this process), writing a number into "/proc/self/selected". Note that these magic files are all ``per process''. That is their content is dependent upon which process opens them. A last not about openMFS is that there are versions around that return faultive results when you run "df" on those filesystems. Don't be surpised if you suddenlty have about 1.3 TB available on those systems.
and edit your /etc/openmosix.map if you don't wish to use the autodiscovery daemon (omdiscd). Since this seems to be a problem for lots of people, let's go with another example. Say you have 3 machines: 192.168.10.220, 192.168.10.78 and 192.168.10.84. Your openmosix.map will look like this.
[root@oscar0 root]# more /etc/openmosix.map # openMosix CONFIGURATION # =================== # # Each line should contain 3 fields, mapping IP addresses to openMosix nodenumbers: # 1) first openMosix nodenumber in range. # 2) IP address of the above node (or nodename from /etc/hosts). # 3) number of nodes in this range. # # Example: 10 machines with IP 192.168.1.50 192.168.1.59 # 1 192.168.1.50 10 # # openMosix# IP numberofnodes # ============================ 1 192.168.10.220 1 2 192.168.10.78 1 3 192.168.10.84 1
Now by rebooting the different machines with the newly installed kernel you will get one step closer to having a working cluster. Most RedHat installations have one extra thing to fix. You often get the following error:
[root@inspon root]# /etc/init.d/openmosix start Initializing openMosix... setpe: the supplied table is wellformatted, but my IP address (127.0.0.1) is not there!
This means that your hostname is not listed in /etc/hosts with the same ip as in your openmosix.map. You might have a machine called omosix1.localhost.org in your hostfile listed as
18
If you modify your /etc/hosts to look like below, openMosix will have less troubles starting up.
[root@inspon root]# /etc/init.d/openmosix start Initializing openMosix... [root@inspon root]# /etc/init.d/openmosix status This is openMosix node #2 Network protocol: 2 (AF_INET) openMosix range 11 begins at 192.168.10.220 openMosix range 22 begins at inspon.localhost.be openMosix range 33 begins at 192.168.10.84 Total configured: 3
If you would like to use more bleeding edge patches, you can always opt for the src rpm and run rpmbuild rebuild on it. This will install the source for you and create an initial config file. From there you can go further applying patches to openMosix A tutorial on how to build your own openMosix RPM's can be found in the Appendixes. As new RedHat versions come out, they might be supported out of the box so, feel free to drop the author a note and help him keeping this information updated.
19
You now will need to edit your /etc/openmosix.map. Please follow the instructions given in the ``Syntax of /etc/openmosix.map'' part of this HOWTO. After rebooting with this kernel and a configured /etc/openmosix.map, you should then have a cluster of openMosix machines that talk to eachother and that do migration of processes. You can test that by running the following small script:
awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}'
a couple of times, and monitor its behaviour with "mosmon" where you will see that it spreads the load between the different nodes. We also setup openMosixView on the Debian machine:
aptget install openmosixview
In order to be able to actually use openMosixView you will need to run it from a user who can log in to the different nodes as root. We suggest you set this up using ssh. Please note that there is a difference between the ssh and ssh2 implementations. If you do have an identity.pub file, ssh will check authorized_keys, while if you do have an id_dsa.pub you will need authorized_keys2! openMosixView gives you a nice interface that shows the load of different machines and gives you the possibility to migrate processes manually. A detailed discussion of openMosixView can be found elsewhere in this document.
20
Chapter 5. Autodiscovery
5.1. Easy Configuration
The autodiscovery daemon (omdiscd) provides a way to automatically configure an openMosix cluster hence eliminating the need of a /etc/mosix.map or similar manual configurations. Autodiscovery uses multicast packages to notify other nodes that it is an openMosix node. This way adding an extra node to your mosix cluster means that you just have to start the omdiscd on your machine and it will join the cluster. However there are some small requirements, Like with any openMosix cluster , you need to have networking configured correctly. mainly the routing. Without a default route, you must specify an interface to omdiscd with the i option. Otherwise omdiscd will exit with an error like.
Aug 31 20:41:49 localhost omdiscd[1290]: Unable to determine address of default interface. This may happen because there is no default route configured. Without a default route, an interface must be: Network is unreachable Aug 31 20:41:49 localhost omdiscd[1290]: Unable to initialize network. Exiting.
Flags U U UG
Metric 0 0 0
Ref 0 0 0
Use 0 0 0
Basically from now on everything will get easier. Just start And have a look at your logfiles you should see something similar to this
Sep Sep Sep Sep Sep Sep 2 2 2 2 2 2 10:00:49 10:00:49 10:00:49 10:00:49 10:00:49 10:00:49 oscar0 oscar0 oscar0 oscar0 oscar0 oscar0 kernel: kernel: kernel: kernel: kernel: kernel: openMosix openMosix openMosix openMosix openMosix openMosix configuration changed: #2780 is at IP address #2638 is at IP address #2646 is at IP address #2627 is at IP address #2634 is at IP address
This is openMosix #2780 (of 6 con 192.168.10.220 192.168.10.78 192.168.10.86 192.168.10.67 192.168.10.74
Congratulations , your openMosix cluster is now working. omdiscd has some other options that you can use. You can either run omdiscd as a daemon (default) or in the foreground where output goes to the screen (standard output) omdiscd n . An interface can be specified with the i option. Now lets still have a short look at the other tool , it's showmap. This tool will show you the newly auto generated openMosix map.
[root@oscar0 root]# showmap My NodeId: 0x0adc Base NodeId 0x0adc 0x0a4e Address 192.168.10.220 192.168.10.78 Count 1 1
Chapter 5. Autodiscovery
21
Autodiscovery has some other features not listed here such as a routing mechanism for clusters with more than one network. More detailed information can be found in the README and DESIGN files in the userland tools source tree. More recent versions of the openMosix rc scripts will first verify wether an /etc/openmosix.map file or similar exists before trying to use autoconfiguration.
You will need to put this in comment. If you want to have some more logging available to you you should edit main.c to show log_set_debug(DEBUG_TRACE_ALL); (somewhere around line 84) now run
% make clean % make
Aug 31 20:45:58 localhost kernel: openMosix configuration changed: This is openMosix #98 (of 1 co Aug 31 20:45:58 localhost kernel: openMosix #98 is at IP address 10.0.0.98Aug 31 20:45:58 localho openMosix Aug 31 20:45:58 localhost kernel: Received an unauthorized information request from 10
What you should to then is try to force your NIC into promiscuous and/ or multicast mode manually.
ifconfig ethx promisc or ifconfig ethx multicast
which will have the same effect but you will now also be able to see the packages yourself. On some Layer 3 switches you other configs might be required. An openMosix user found out that on his Switch Summit48Si (Extreme Networks) he had to run
disable ipmcforwarding (to deactivate the routing of multicast paquets) disable igmp snooping
before he was the different omdiscd's were able to see eachother, other switches might require similar configs. Chapter 5. Autodiscovery 22
If you see the simulated you have probably forgotten to put the
#define ALPHA
in comment. I have also noticed that autodiscovery does not work with FireWire based network cards.
Chapter 5. Autodiscovery
23
Chapter 6. PlumpOS
6.1. What Plump/OS is
PlumpOS is a CDbased GNU/Linux/openMosix minidistribution designed to allow users to quickly, or temporarily, add nodes to an openMosix cluster; as I write this in march 2003 the version (release 6.9 RC1) is a 16.7M ISO download. This chapter is a quick hack up by Peter Willis of a very similar chapter contributed by JeanDavid Marrow (who is the author of Clump/OS and inspirer of PlumpOS props to JeanDavid for his fine work on the departed Clump/OS).
6.3. Requirements
As the purpose of PlumpOS is to add nodes to a cluster, it is assumed that you already have a running openMosix cluster or perhaps only a single openMosix node from which you will be initiating jobs. All machines in the cluster must conform to the following requirements: PlumpOS Machine(s) 586+ CPU bootable CDROM drive Network Interface Card Between 32M and 128M of RAM. Because of some oddities in the openMosix kernels at the time of this writing, simply passing the size of the ramdisk is not working so one has to actually use a ramdisk premade for a specific size. There should be several ramdisks avaliable at PlumpOS mirrors which can be placed in the disks' root directory on the actual cdrom ISO (or rootdisk/disks/ if you're inside the PlumpOS package directory). Master Machine(s) GNU/Linux/openMosix kernel (same version as all the PlumpOS kernel you are booting on the Child/Slave PlumpOS Machine(s)) Chapter 6. PlumpOS 24
The openMosix HOWTO Network Environment Running DHCP server (if you don't, or won't, run DHCP, you can still manually configure your system: simply go to each PlumpOS machine and enter in your desired configuration information with the supplied networking tools. Using DHCP is highly recommended however, and will greatly simplify your life in the long run. The following network modules are present in most if not all of the supplied kernels' modules tarball (in /kernels/KERNELNAME/modules.tgz), although not all support autoprobing; if you don't see support for your card in this list, then PlumpOS will not work for you. 3c501.o, 3c503.o, 3c505.o, 3c507.o, 3c509.o, 3c515.o, 3c59x.o, 8139cp.o, 8139too.o, 82596.o, ac3200.o, acenic.o, aironet4500_card.o, aironet4500_core.o, aironet4500_proc.o, arlanproc.o, arlan.o, at1700.o, bsd_comp.o, cs89x0.o, de4x5.o, depca.o, dgrs.o, dl2k.o, dmfe.o, dummy.o, e100/e100.o, e1000/e1000.o, e2100.o, eepro.o, eepro100.o, eexpress.o, epic100.o, eth16i.o, ewrk3.o, fealnx.o, hamachi.o, hpplus.o, hp.o, hp100.o, lance.o, lp486e.o, mii.o, natsemi.o, ne.o, ne2kpci.o, ni5010.o, ni52.o, ni65.o, ns83820.o, pcmcia, pcnet32.o, ppp_async.o, ppp_deflate.o, ppp_generic.o, ppp_synctty.o, pppoe.o, pppox.o, sis900.o, sk98lin/sk98lin.o, slhc.o, smcultra.o, smc9194.o, starfire.o, strip.o, sundance.o, sungem.o, sunhme.o, tc35815.o, tg3.o, tlan.o, tokenring/{3c359.o abyss.o ibmtr.o lanstreamer.o olympic.o smctr.o tms380tr.o tmsisa.o tmspci.o}, tulip/tulip.o, viarhine.o, wavelan.o, wd.o, winbond840.o, wireless/{airo.o airo_cs.o hermes.o orinoco.o orinoco_cs.o orinoco_pci.o orinoco_plx.o}, yellowfin.o Please also note that PlumpOS may not work on a laptop: it definitely doesn't support PCMCIA cards (yet), and will probably not configure openMosix properly if your machine contains multiple connected Ethernet adapters. This is a temporary limitation of the configuration scripts, and should be resolved in future releases.
and enter the new directory "plumpos6.9rc1/". Now you have several options as to how you go about setting up PlumpOS. First let's familiarize with the directory structure here. There should be 3 directories: rootdisk, which contains the layout of the iso to be created; scripts, which contains small programs used to help in the creation of the iso; final, which will contain the ISO generated by the whole process. There should also be a file called install which you will run when you are done configuring your system. If you intend on using your own kernel with PlumpOS, there are several steps you must follow in order for it to boot properly. First you must make a new directory in the rootdisk/kernels/ directory that will be in the old and often blamed DoS 8.3 character format and only contain letters and numbers (and a single '.'). In that directory you should put a file named bzImage which is your kernel and a file named modules.tgz which is a gzipped tarball of your kernel's modules (with the relative path of lib/modules/ so it can be extracted from the root dir). Optionally you can also provide the System.map and config file for your kernel. When this is done you may simply run the install program and it will detect and install your kernel(s) for use with the generated ISO. Chapter 6. PlumpOS 25
The openMosix HOWTO Now, say you want to include a 3rd party addon package. This is done very simply: create a gzipped tarball containing all the files you want to include in your package (relative to "/") and put these packages into the rootdisk/packages/ directory. Then edit the file rootdisk/packages/list and add the path relative to / on the ramdisk where the package can be located (in other words for a package named "openssl.tar.gz", add the line "/cdrom/packages/openssl.tar.gz" to the file rootdisk/packages/list). Optionally you can use a line such as "cdrom:openssl.tar.gz" which will automatically search the cdrom's packages directory for the package "openssl.tar.gz". In future releases this will be useful for things like nfs and boot floppies, so for now don't worry about it ;)
Chapter 6. PlumpOS
26
Repeat the process for the dsh package. Say we have a small cluster with a couple of nodes. To make life easier we want type each command once but have it executed on each node. You then have to create a file in $HOME/.dsh/group/clusterwname that lists the ip's of your cluster. eg.
[root@inspon root]# cat .dsh/group/mosix 192.168.10.220 192.168.10.84
As an example we run ls on each of these machines We use g to use the mosix group (this way you can create subsets of a group with different configurations)
[root@inspon root]# dsh r ssh g mosix ls 192.168.10.84: anacondaks.cfg 192.168.10.84: id_rsa.pub 192.168.10.84: install.log 192.168.10.84: install.log.syslog 192.168.10.84: openmosixkernel2.4.17openmosix1.i686.rpm 192.168.10.84: openmosixtools0.2.01.i386.rpm 192.168.10.220: anacondaks.cfg 192.168.10.220: id_dsa.pub 192.168.10.220: id_rsa.pub 192.168.10.220: openmosixkernel2.4.17openmosix1.i686.rpm 192.168.10.220: openmosixtools0.2.01.i386.rpm 192.168.10.220: oscar1.2.1rh72 192.168.10.220: oscar1.2.1rh72.tar.gz
Note that neither of the machines ask for a password. This is because we have set up RSA authentication between the different accounts. If you want to run commands with multiple parameters you will either have to put the command between quotes.
[root@inspon root]# dsh r ssh g mosix "uname a" 192.168.10.84: Linux omosix2.office.be.stoneit.com 2.4.17openmosix1 #1 Wed May 29 14:32:28 CEST 2002 i686 unknown 192.168.10.220: Linux oscar0 2.4.17openmosix1 #1 Wed May 29 14:32:28 CEST
27
28
29
8.2. Configuration
The values in the flat files in the /proc/hpc/admin directory presenting the current configuration of the cluster. Also the administrator can write its own values into these files to change the configuration during runtime, e.g.
Table 81. Changing /proc/hpc parameters echo 1 > /proc/hpc/admin/block blocks the arrival of remote processes echo 1 > /proc/hpc/admin/bring bring all migrated processes home ...
Table 82. /proc/hpc/admin/ (binary files) config (flat files) block bring dfsalinks expel gateways lstay mospe nomfs overheads quiet decayinterval slowdecay fastdecay speed stay the main configuration file (written by the setpe util) allow/forbid arrival of remote processes bring home all migrated processes list of current symbolic dfsalinks sending guest processes home maximum number of gateways local processes should stay contains the openMosix node id disables/enables MFS for tuning stop collecting loadloadbalancing informations interval for collecting informations about loadbalancing default 975 default 926 speed relative to PIII/1GHz) enables/disables automatic process migration
30
The openMosix HOWTO Table 83. Writing a 1 to the following files /proc/hpc/decay/ clear cpujob iojob slow fast clears the decay statistics tells openMosix that the process is cpubound tells openMosix that the process is iobound tells openMosix to decay its statistics slow tells openMosix to decay its statistics fast
Table 84. Informations about the other nodes /proc/hpc/nodes/[openMosix_ID]/CPUs /proc/hpc/nodes/[openMosix_ID]/load /proc/hpc/nodes/[openMosix_ID]/mem /proc/hpc/nodes/[openMosix_ID]/rmem /proc/hpc/nodes/[openMosix_ID]/speed /proc/hpc/nodes/[openMosix_ID]/status /proc/hpc/nodes/[openMosix_ID]/tmem /proc/hpc/nodes/[openMosix_ID]/util how many CPU's the node has the openMosix load of this node available memory as openMosix believes available memory as Linux believes speed of the node relative to PIII/1GHz status of the node available memory utilization of the node
Table 85. Additional Informations about local processes /proc/[PID]/cantmove /proc/[PID]/goto /proc/[PID]/lock /proc/[PID]/nmigs /proc/[PID]/where /proc/[PID]/migrate /proc/hpc/remote/from /proc/hpc/remote/identity /proc/hpc/remote/statm /proc/hpc/remote/stats reason why a process cannot be migrated to which node the process should migrate if a process is locked to its home node how many times the process migrated where the process is currently being computed same as goto remote processes the home node of the process additional informations about the process memory statistic of the process cpu statistics of the process
31
[openMosi
setdecay interval
Table 86. more detailed stay nostay lstay nolstay block noblock quiet noquiet nomfs mfs expel bring gettune getyard getdecay whois getload getspeed status isup getmem getfree getutil setyard setspeed no automatic process migration automatic process migration (default) local processes should stay local processes could migrate block arriving of guest processes allow arriving of guest processes disable gathering of loadbalancing informations enable gathering of loadbalancing informations disables MFS enables MFS send away guest processes bring all migrated processes home shows the current overhead parameter shows the current used Yardstick shows the current decay parameter resolves openMosixID, ipaddresses and hostnames of the cluster display the (openMosix) load shows the (openMosix) speed displays the current status and configuration is a node up or down (openMosix kind of ping) shows logical free memory shows physical free mem display utilization sets a new Yardstickvalue sets a new (openMosix) speed value
32
The mosrun command can be executed with several more commandline options. To ease this up there are several preconfigured runscripts for executing jobs with a special (openMosix) configuration.
Table 87. extra options for mosrun nomig runhome runon cpujob iojob nodecay slowdecay fastdecay
setpe
runs a command which process(es) won't migrate executes a command locked to its home node runs a command which will be directly migrated and locked to a node tells the openMosix cluster that this is a cpubound process tells the openMosix cluster that this is a iobound process executes a command and tells the cluster not to refresh the loadbalancing statistics executes a command with a slow decay interval for collecting loadbalancing statistics executes a command with a fast decay interval for collecting loadbalancing statistics
manual node configuration utility syntax: setpe w f [hpc_map] setpe r [f [hpc_map]] setpe off
w reads the openMosix configuration from a file (typically /etc/hpc.map) r writes the current openMosix configuration to a file (typically /etc/hpc.map) off turns the current openMosix configuration off tune openMosix calibration and optimizations utility. (for further informations review the tuneman page)
Additional to the /proc interface and the commandlineopenMosix utilities (which are using the /proc interface) there is a patched "ps" and "top" available (they are called "mps" and "mtop") which displays also the openMosixnode ID on a column. This is useful for finding out where a specific process is currently being computed. This actually summarised the command line tools, but have a look at openMosixview which is a GUI for the most common administration tasks, and which ill be discussed in a future chapter.
The openMosix HOWTO Here is how it works: /proc/[pid]/migfilter enable/disable the capability of filter migration. /proc/[pid]/mignodes is a bitlist of nodes. The bit position of a node is calculated as 2^(PE1). PE is node number. /proc/[pid]/migpolicy is the policy of the filtering: 0=DENY: the process can migrate in all nodes except when the relative bit on mignodes is 1 1=ALLOW: the process can migrate in all nodes where the relative bit on mignodes is 1 We are shortly going to release also a simple userland tool to set the node mask, but I would like you guys to give it a try asap before we release it as openMosix 2.4.203.
34
where n should be much lower than the speed of the other nodes Processes will move/migrate away fast. You can get the speed of a node with :
mosctl getspeed
which, if everything went right, will give you a listing of your /etc/mosix.map. If things did not go right, try
setpe w f /etc/mosix.map
to see if your child processes are locked in your mode (1) or can migrate (0). If for some reason you find your processes are locked, you can change this with
echo 0 > /proc/$$/lock
until you fix the problem. Repeat the whole configuration scheme for a second computer. The programs tune_kernel and prep_tune that Mosix uses to calibrate the individual nodes do not work with the SuSE distribution. However, you can fake it. First, bring the computer you want to tune and another computer with Mosix installed down to single user mode by typing
init 1
as root. All other computers on the network should be shutdown if possible. On both machines, run the following commands: Chapter 9. Tuning Mosix 35
This fakes prep_tune and the first parts of tune_kernel. Note that if you have a laptop with a pcmcia network card, you will have to run
/etc/init.d/pcmcia start
instead of "/etc/init.d/network start". On the computer you want to tune, run tune_kernel and follow instructions. Depending on your machines, this can take a while if you have a dog, this might be the time to go on that long, long walk you've always promised him. tune_kernel will create a program called "pg" in /root for testing reasons. Ignore it. After tuning is over, copy the contents of /tmp/overheads to the file /etc/overheads (and/or recompile the kernel). Repeat the tuning procedure for each computer. Reboot, enjoy Mosix, and don't forget to brag to your friends about your new cluster.
Channel bonded networks can connect to standard networks via a router or bridge that supports channel bonding( I just use an extra NIC and portforwarding in the head node).
9.5. Updatedb
Updatedb in combination with mfs can cause some issues, you might want to add /mfs to the PRUNEFPATHS or mfs to the PRUNEFS in your /etc/updatedb.conf to disable updatedb from indexing this mountpoints.
36
37
10.3. Installation
Requirements QT library root rights ! rlogin and rsh (or ssh) to all clusternodes without password the openMosix userlandtools mosctl, migrate, runon, iojob, cpujob ... (download them from the www.openmosix.org website) On a RH 8.0 you will need at least the following rpm's qt3.0.517, libmng1.0.4, XFree86MesalibGLU4.2.0, glut3.7 etc ... Documentation about openMosixview There is a full HTMLdocumentation about openMosixview included in every package. You find the startpage of the documentation in your openMosixview installation directory: openmosixview/openmosixview/docs/en/index.html The RPMpackages have their installation directories in: /usr/local/openmosixview
38
10.3.5. Hints
(from the testers of openMosixview/Mosixview who compiled it on different linuxdistributions, thanks again) Create the link /usr/lib/qt pointing to your QT2.3.x installation e.g. if QT2.3.x is installed in /usr/local/qt2.3.0
ln s /usr/local/qt2.3.0 /usr/lib/qt
then do the same in the subdirectory openmosixcollector, openmosixanalyzer, openmosixhistory and openmosixviewprocs. Copy all binaries to /usr/bin
cp cp cp cp cp openmosixview/openmosixview /usr/bin openmosixviewproc/openmosixviewprocs/mosixviewprocs /usr/bin openmosixcollector/openmosixcollector/openmosixcollector /usr/bin openmosixanalyzer/openmosixanalyzer/openmosixanalyzer /usr/bin openmosixhistory/openmosixhistory/openmosixhistory /usr/bin
39
The openMosix HOWTO openMosixview displays a row with a lamp, a button, a slider, a lcdnumber, two progressbars and some labels for each clustermember. The lights at the left are displaying the openMosixId and the status of the clusternode. Red if down, green for available. If you click on a button displaying the ipaddress of one node a configurationdialog will pop up. It shows buttons to execute the most common used "mosctl"commands. (described later in this HOWTO) With the "speedsliders" you can set the openMosixspeed for each host. The current speed is displayed by the lcdnumber. You can influence the loadbalancing of the whole cluster by changing these values. Processes in a openMosixCluster are migrating easier to a node with more openMosixspeed than to nodes with less speed. Sure it is not the physically speed you can set but it is the speed openMosix "thinks" a node has. e.g. a cpuintensive job on a clusternode which speed is set to the lowest value of the whole cluster will search for a better processor for running on and migrate away easily. The progress bars in the middle gives an overview of the load on each clustermember. It displays in percent so it does not represent exactly the load written to the file /proc/hpc/nodes/x/load (by openMosix), but it should give an overview. The next progressbar is for the used memory the nodes. It shows the currently used memory in percent from the available memory on the hosts (the label to the right displays the available mem). How many CPUs your cluster have is written in the box to the right. The first line of the main windows contains a configuration button for "allnodes". You can configure all nodes in your cluster similar by this option. How good the loadbalancing works is displayed by the progressbar in the top left. 100% is very good and means that all nodes nearly have the same load. Use the collector and analyzermenu to manage the openMosixcollector and open the openMosixanalyzer. This two parts of the openMosixviewapplication suite are useful for getting an overview of your cluster during a longer period.
40
The openMosixconfiguration of each host can be changed easily now. All commands will be executed per "rsh" or "ssh" on the remote hosts (even on the local node) so "root" has to "rsh" (or "ssh") to each host in the cluster without prompting for a password (it is well described in a Beowulf documentation or on the HOWTO on this page how to configure it). The commands are:
automigration on/off quiet yes/no bring/lstay yes/no exspel yes/no openMosix start/stop
If openMosixprocs is properly installed on the remote clusternodes click the "remote procbox"button to open openMosixprocs (procbox) from remote. xhost +hostname will be set and the display will point to your localhost. The client is executed on the remote also per "rsh" or "ssh". (the binary openmosixprocs must be copied to e.g. /usr/bin on each host of the cluster) openMosixprocs is a processbox for managing your programs. It is useful to manage programs started and running local on the remote nodes and is described later in this HOWTO. If you are logged on your cluster from a remote workstation insert your local hostname in the editbox below the "remote procbox". Then openMosixprocs will be displayed on your workstation and not on the clustermember you are logged on. (maybe you have to set "xhost +clusternode" on your workstation). There is a history in the combobox so you have to write the hostname only once.
10.4.3. advancedexecution
If you want to start jobs on your cluster the "advanced execution"dialog may help you.
41
Choose a program to start with the "runprog" button (fileopenicon) and you can specify how and where the job is started by this executiondialog. There are several options to explain.
Table 101. how to start no migration run home run on cpu job io job no decay slow decay fast decay parallel start a local job which won't migrate start a local job start a job on the node you can choose with the "hostchooser" start a computation intensive job on a node (hostchooser) start a io intensive job on a node (hostchooser) start a job with no decay (hostchooser) start a job with slow decay (hostchooser) start a job with fast decay (hostchooser) start a job parallel on some or all node (special hostchooser)
42
10.5. openMosixprocs
10.5.1. intro
This processbox is really useful for managing the processes running on your cluster.
You should install it on every clusternode! The processlist gives an overview what is running where. The second column displays the openMosixnode ID of each process. 0 means local, all other values are remote nodes. Migrated processes are marked with a green icon and non movable processes have a lock. By doubleclicking a process from the list the migratorwindow will popup for managing e.g. migrating the process. There are also options to migrate the remote processes away, send SIGSTOP and SIGCONT to it or to "renice" it. If you click on the "manage procs from remote" button a new window will come up (the remoteprocs windows) displaying the process currently migrated to this host.
43
The openMosixviewmigrator window displays all nodes in your openMosixcluster. This window is for managing one process (with additional statusinformation). By doubleclicking on an host from the list the process will migrate to this host. After a short moment the processicon for the managed process will be green, which means it is running remote. The "home"button sends the process to its home node. With the "best"button the process is send to the best available node in your cluster. This migration is influenced by the load, speed, CPU's and what openMosix "thinks" of each node. It maybe will migrate to the host with the most CPU's and/or the best speed. With the "kill"button you can kill the process immediately. To pause a program just click the "SIGSTOP"button and to continue the "SIGCONT"button. With the reniceslider below you can renice the current managed process (20 means very fast, 0 normal and 20 very slow)
44
The TabView displays processes that are migrated to the local host. The procs are coming from other nodes in your cluster and currently computed on the host openMosixview is started on. Similar to the two buttons in the migratorwindow the process is send home by the "goto home node"button and send to the best available node by the "goto best node"button.
10.6. openMosixcollector
The openMosixcollector is a daemon which should/could be started on one clustermember. It logs the openMosixload of each node to the directory /tmp/openmosixcollector/* These history logfiles analyzed by the openMosixanalyzer (as described later) gives an nonstop overview of the load, memory and processes in your cluster. There is one main logfile called /tmp/openmosixcollector/cluster Additional to this there are additional files in this directory to which the data is written. At startup the openMosixcollector writes its PID (process id) to /var/run/openMosixcollector.pid The openMosixcollectordaemon restarts every 12 hours and saves the current history to /tmp/openmosixcollector[date]/* These backups are done automatically but you can also trigger this manual. There is an option to write a checkpoint to the history. These checkpoints are graphically marked as a blue vertical line if you analyze the history logfiles with the openMosixanalyzer. For example you can set a checkpoint when you start a job on your cluster and another one at the end.. Here is the explanation of the possible commandlinearguments:
openmosixcollector openmosixcollector openmosixcollector openmosixcollector openmosixcollector d k n r //starts the collector as a daemon //stops the collector //writes a checkpoint to the history //saves the current history and starts a new one //print out a short help
You can start this daemon with its initscript in /etc/init.d or /etc/rc.d/init.d. You just have to create a symbolic link to one of the runlevels for automatic startup. Chapter 10. openMosixview 45
The openMosix HOWTO How to analyze the created logfiles is described in the openMosixanalyzersection.
10.7. openMosixanalyzer
10.7.1. the loadoverview
This picture shows the graphical Loadoverview in the openMosixanalyzer (Click to enlarge)
With the openMosixanalyzer you can have a nonstop openMosixhistory of your cluster. The history logfiles created by openMosixcollector are displayed in a graphically way so that you have a longtime overview what happened and happens on your cluster. The openMosixanalyzer can analyze the current "online" logfiles but you can also open older backups of your openMosixcollector history logs by the filemenu. The logfiles are placed in /tmp/openmosixcollector/* (the backups in /tmp/openmosixcollector[date]/*) and you have to open only the main history file "cluster" to take a look at older loadinformations. (the [date] in the backup directories for the logfiles is the date the history is saved) The start time is displayed on the top and you have a fullday view in the openMosixanalyzer (12 h). If you are using the openMosixanalyzer for looking at "online"logfiles (current history) you can enable the "refresh"checkbox and the view will autorefresh. The loadlines are normally black. If the load increases to >75 the lines are drawn red. These values are openMosixinformations. The openMosixanalyzer gets these informations from the files /proc/hpc/nodes/[openMosix ID]/* The Findoutbutton of each nodes calculates several useful statistic values. Clicking it will open a small new window in which you get the average load and mem values and some more statically and dynamic informations about the specific node or the whole cluster.
46
If there are checkpoints written to the loadhistory by the openMosixcollector they are displayed as a vertical blue line. You now can compare the load values at a certain moment much easier.
This picture shows the graphical Memoryoverview in the openMosixanalyzer With Memoryoverview in the openMosixanalyzer you can have a nonstop memory history similar to the Loadoverview. The history logfiles created by openMosixcollector are displayed in a graphically way so that you have a longtime overview what happened and happens on your cluster. It analyze the current "online" logfiles but you can also open older backups of your openMosixcollector history logs by the filemenu. The displayed values are openMosixinformations. The openMosixanalyzer gets these informations from the files
/proc/hpc/nodes/[openMosixID]/mem. /proc/hpc/nodes/[openMosixID]/rmem.
47
If there are checkpoints written to the memoryhistory by the openMosixcollector they are displayed as a vertical blue line.
10.7.4. openMosixhistory
displays the processlist from the past openMosixhistory gives a detailed overview which process was running on which node. The openMosixcollector saves the processlist from the host the collector was started on and you can browse this logdata with openMosixhistory. You can easy change the browsing time in openMosixhistory by the timeslider. openMosixhistory can analyze the current "online" logfiles but you can also open older backups of your openMosixcollector history logs by the filemenu. The logfiles are placed in /tmp/openmosixcollector/* (the backups in /tmp/openmosixcollector[date]/*) and you have to open only the main history file "cluster" to take a look at older loadinformations. (the [date] in the backup directories for the logfiles is the date the history is saved) The start time is displayed on the top/left and you have a 12 hour view in openMosixhistory.
48
10.8. openMosixmigmon
10.8.1. General
The openMosixmigmon is a monitor for migrations in your openMosixcluster. It displays all your nodes as little penguins sitting in a circle. > nodescircle. The main penguin is the node on which openMosixmigmon runs and around this node it shows its processes also in a circle of small black squares. > main processcircle
49
The openMosix HOWTO If a process migrates to one of the nodes the node gets an own processcircle and the process moved from the main processcircle to the remote processcircle. Then the process is marked green and draws a line from its origin to its remote location to visualize the migration.
10.8.2. Tooltips:
If you hold your mouse above a process it will show you its PID and commandline in a small tooltipwindow.
50
The openMosix HOWTO The openMosixviewclient is executed per rsh (or ssh which you can configer whith a checkbox) on the remote host. It has to be installed in /usr/bin/ on each node. If you use RSH try: "xhost +hostname" "rsh hostname /usr/bin/openMosixview_client display your_local_host_name:0.0" or if you use SSH try: "xhost +hostname" "ssh hostname /usr/bin/openMosixview_client display your_local_host_name:0.0" If this works it will work in openMosixview too. openMosixview crashes with "segmentation fault"! Maybe you still use an old version of openMosixview/Mosixview ? in the mosix.mapparser (which is completly removed in openMosixview !!) (the versions openMosixview 1.2 and Mosixview > 1.0 are stable) 10.9.5. Why are the buttons in the openMosixviewconfiguration dialog not preselected? (automigration on/off, blocking on/off......) I want them to be preselected too. The problem is to get the information of node. You have to login to each clusternode because these information are not clusterwide (to my mind). The status of each node is stored in the /proc/hpc/admin directory of each node. Everybody who knows a good way to get these information easy is invited to mail me.
That is why it is a bit tricky to configure SSH. SSH is secure even if you use it to login without being prompted for a password. Here is a (one) way to configure it. At first a running secureshell daemon on the remote site is required. If it is not already installed install it! (rpm i [sshd_rpm_packeage_from_your_linux_distribution_cd]) If it is not already running start it with:
/etc/init.d/ssh start
Now you have to generate a keypair for SSH on your local computer whith sshkeygen.
sshkeygen
You will be prompt for a passphrase for that keypair. The passphrase normally is longer than a password and may be a whole sentence. The keypair is encrypted with that passphrase and saved in
/root/.ssh/identity //your private key and /root/.ssh/identity.pub //your public key
Do NOT give your privatekey to anybody!!! Now copy the whole content of /root/.ssh/identity.pub (your publickey which should be one long line) into /root/.ssh/authorized_keys on the remote host. (also copy the content of /root/.ssh/identity.pub to your local /root/.ssh/authorized_keys like you did it with the remotenode because openMosixview needed passwordless login to the localnode too!) If you ssh to this remote host now you will be prompted for the passphrase of your publickey. Giving the right passphrase should give you a login. What is the advantage right now??? The passphrase is normally a lot longer than a password! The advantage you can get using the sshagent. It manages the passphrase during ssh login. Chapter 10. openMosixview 51
The sshagent is started now and gives you two environmentvariables you should set (if not set already). Type:
echo $SSH_AUTH_SOCK and echo $SSH_AGENT_PID
to see if they are exported to your shell right now. If not just cut and paste from your terminal. e.g. for the bashshell:
SSH_AUTH_SOCK=/tmp/sshXXYqbMRe/agent.1065 export SSH_AUTH_SOCK SSH_AGENT_PID=1066 export SSH_AGENT_PID
With these variables the remotesshddaemon can connect your local sshagent by using the socketfile in /tmp (in this example /tmp/sshXXYqbMRe/agent.1065). The sshagent can now give the passphrase to the remote host by using this socket (it is of course an encrypted transfer)! You just have to add your publickey to the sshagent with the sshadd command.
sshadd
Now you should be able to login using ssh to the remote host without being prompted for a passwod! You could (should) add the sshagent and sshadd commands in your loginprofile e.g.
eval `sshagent` sshadd
Now it is started when you login on your local workstation. You have done it! I wish you secure logins now. openMosixview There is a menuentry which toggles using rsh/ssh with openMosixview. Just enable this and you can use openMosixview even in insecure networkenvironments. You should also save this configuration (the possibility for saveing the current config in openMosixview was added in the 0.7 version) because it gets initial data from the slave using rsh or ssh (just like you configured). If you choose a service wich is not installed properly openMosixview will not work! (e.g. if you cannot rsh to a slave without being prompted for a password you cannot use openMosixview with RSH; if you cannot ssh to a slave without being prompted for a password you cannot use openMosixview with SSH)
52
11.2. openMosixView
openMosixview is the most used and the best known applet for openMosix administration, you can read more about it in the openMosix adminstration chapter.
11.3. openMosixapplet
The openMosixApplet lets you watch the realtime load of your openMosix cluster. It consists of a local daemon which listens for connections by applets. The applet uses chart2D to provide a goodlookin' feeling.
11.4. wmonload
wmomload is a simple, but handy and small dockapp for overviewing the load of cluster nodes in a small openMosixbased cluster.
11.5. openMosixWebView
openMosixWebView Produces web charts for monitoring an openMosix cluster. openMosixWebView is a PHP script for monitoring an openMosix cluster via the WEB. It uses openMosixview's openMosixCollector logs. Download now the last release openmosixwebview0.2.12.tar.gz (16 Feb 2003) See openMosixWebView screenshots and running :) Released under the GNU General Public License (GPL). See README and FAQ files.
53
where $PID is the processid of the process in question. Now listen to what Moshe himself has to say about this topic. Often people have the same kernel but on a different distribution, say a mixed environment of RedHat and Debian ,rc scripts from different distros tend to start openmosix differently. Some implementations completely modify /etc/inittab to start all daemons (and their children) with
mosrun h
. So that they won't migrate. Therefore all these processes have a 1 in /proc/pid/lock when you start. You can force them to migrate by writing a 0 to this file . Ok, this simple program should always migrate if launched more times than number of local CPUs. So for a 2way SMP system, starting this program 3 times will start migration if the other nodes in the cluster have at least the same speed like the local ones:
int main() { unsigned int i; while (1) { i++; } return 0; }
On a Pentium 800Mhz CPU it takes quite a while to overflow. This sample program with content like this will never migrate:
#include <sys/types.h> #include <sys/ipc.h> #include <sys/shm.h>
54
MODIFIED Programs using pthreads since version 2.4.17 don't migrate, however they don't segfault anymore.
// // Very simple program demonstrating the use of threads. // // Commandline argument is P (number of threads). // // Each thread writes "hello" message to standard output, with // no attempt to synchronize. Output will likely be garbled. // #include <iostream> #include <cstdlib> // has exit(), etc. #include <unistd.h> // has usleep() #include <pthread.h> // has pthread_ routines // declaration for function to be executed by each thread void * printHello(void * threadArg); // Main program int main(int argc, char* argv[]) {
55
Programs using all kinds of file descriptors, including sockets do migrate (sockets are not migrated with the process however, files are migrated if using oMFS/DFSA) (all above code is by Moshe as Moshe Bar or by Moshe as CTO of Qlusters, Inc.) Please also refer to the man pages of openMosix , they also give an adequate explanation why processes don't migrate. If for some reason your processes stay locked while they shouldn't. You can try to allow locked processes to migrate by simply putting
# tell shells to allow subprocs to migrate to other nodes echo 0 > /proc/self/lock
56
The openMosix HOWTO into "/etc/profile" Warning : this fix will allow all process to migrate not just the ones you want. To only allow specific process to migrate use 'mosrun l' to unlock only the desired process.
You might also need to set up a routing table, which is a whole different subject. Maybe you used different kernelparameters on each machine? Especially if you use the 'Support clusters with a complex network topology' option you should take care that you use the same value for the also appearing option 'Maximum networktopology complexity support' on each machine.
what does this mean ? The above line meas that the shell you were using has acutallly migrated to another node ? This printout from bash is caused by a bug in old version of openmosix, but a fix has been commited. (Muli BenYehuda mulix@actcom.co.il)
The openMosix HOWTO have openMosix working, this is not true, however it can make things easier. With DFSA enabled, system calls will be executed on the remote node withouth migrating the process back to it's home node. This behaviour (direct filesystem access) causes processes migratiing to the data and not the other way around (which is common). If DFSA is not enabled MfS is "just" a noncaching networkfilesystem. Very generally speaking, if you don't have DFSA turned on, each and every I/O will go to the home node for execution. With DFSA turned on, if the file happens to be residing on the node where the process finds itself then the I/O will happen locally. A very common error is that people mix kernels with DFSA enabled and disabled. So one has to have a way to find out wether DFSA is actually enabled. This information can be obtained by typing
cat /proc/hpc/admin/version
58
However, you should make an effort to find out what the problems is
59
The openMosix HOWTO converted to decimal, they are: 4660 and 5428 resp. (convert 1234, and not 3412, 'cos of the networkhost byte ordering conversions.. Read the IP/TCP/UDP RFCs for info) the mig_daemon port is a tcp port, the info_daemon port is udp. Hence tcp/4660 and udp/5428, Matt also mentions tcp/723 somewhere.
60
I've saved it as test_mosix, now when I want to see If everything works I start op mosmon and I launch this script for a zillion times as in
for i in `ls /etc/` ; do ./test_mosix ; done
Now watch openMosix kicking in after a few seconds ... Just pkill awk on the home node to kill'em all ;)
61
#Portfolio.pl, version 0.1 #Perl program that simulate a portfolios for various stock composition for a given period of time #We run various scenarios to find the mix of assets that give the best performance/risk ratio #This method is base on the book "The intelligent asset allocator" by William Bernstein #Can be used to test an OpenMosix cluster #This program is licensed under GPL #Author: CharlesE. Nadeau Ph.D., (c) 2002 #Email address: charlesnadeau AT hotmail DOT com use Parallel::ForkManager; #We use a module to parallelize the calculation
#Available at http://theoryx5.uwinnipeg.ca/mod_perl/cpansearch?new=Search;filetype=%20distributi srand; #We initialize the random number generator #Initializing constant $NumberOfSimulation=$processes; #Number of simulation to run $NumberOfMonth=100000; #Number of month for wich to run the simulation $NumberOfStock=6; #Number of different stocks in the simulation #Portfolio to simulate #TODO: Read the stock details from a file $Stock[0][0]="BRKB"; #Stock ticker $Stock[0][1]=0.01469184; #Stock average monthly return $Stock[0][2]=0.071724934; #Stock average monthly standard $Stock[1][0]="TEST "; #Stock ticker $Stock[1][1]=0.01519; #Stock average monthly return $Stock[1][2]=0.063773903; #Stock average monthly standard $Stock[2][0]="SPDR"; #Stock ticker $Stock[2][1]=0.008922718; #Stock average monthly return $Stock[2][2]=0.041688404; #Stock average monthly standard $Stock[3][0]="BRKB"; #Stock ticker $Stock[3][1]=0.01469184; #Stock average monthly return $Stock[3][2]=0.071724934; #Stock average monthly standard $Stock[4][0]="TEST "; #Stock ticker $Stock[4][1]=0.01519; #Stock average monthly return $Stock[4][2]=0.063773903; #Stock average monthly standard $Stock[5][0]="SPDR"; #Stock ticker $Stock[5][1]=0.008922718; #Stock average monthly return $Stock[5][2]=0.041688404; #Stock average monthly standard
deviation
deviation
deviation
deviation
deviation
deviation
62
#We initialize the array that will contain the results @Results=(); for $i (0..$NumberOfSimulation1){ for $j (0..$NumberOfStock+3){ $Results[$i][$j]=0.0; #Equal to 0.0 to start } } for $i (0..$NumberOfSimulation1){ #Loop on the number of simulation to run $Results[$i][0]=$i; #The first column of each line is the number of the simulation $pm>start and next; #Start the thread $TotalRatio=1; #The sum of the proprtion of each stock
#Here we calculate the portion of each investment in the portfolio for a given si for $j (0..$NumberOfStock2){ #We loop on all stock until the second to last one #TODO: Replace rand by something from Math::TrulyRandom $Ratio[$j]=rand($TotalRatio); $Results[$i][$j+1]=$Ratio[$j]; #We store the ratio associated to this sto $TotalRatio=$TotalRatio$Ratio[$j]; } $Ratio[$NumberOfStock1]=$TotalRatio; #In order to have a total of the ratios equ $Results[$i][$NumberOfStock]=$Ratio[$NumberOfStock1]; #We store the ratio associ
$InvestmentValue=1; #Initially the investment value is 1 time the initial capital my $stats=new Statistics::Descriptive::Discrete; #We initialize the module used t for $j (1..$NumberOfMonth){ #Loop on the number of months
$MonthlyGrowth[$j]=0.0; #By how much did we grow this month. Initially, n #We loop on each stock to find its monthly contribution to the yield for $k (0..$NumberOfStock1){
$MonthlyGrowth[$j]=$MonthlyGrowth[$j]+($Ratio[$k]*((gaussian_rand()*$Stoc }
$stats>add_data($MonthlyGrowth[$j]); #Add the yield for this month so we $InvestmentValue=$InvestmentValue*(1+$MonthlyGrowth[$j]); #Value of the I
} $Results[$i][$NumberOfStock+1]=$stats>mean(); #Calculate the average monthly gro $Results[$i][$NumberOfStock+2]=$stats>standard_deviation(); #Calculate the stand $pm>finish; #Finish the thread } $pm>wait_all_children; #We wait until all threads are finished #Printing the results print "Simulation "; for $j (0..$NumberOfStock1){ print "Ratio$Stock[$j][0] ";
63
if($Results[$i][$NumberOfStock+2]!=0) { printf "%5.4f %5.4f %5.4f\n", $Results[$i][$NumberOfStock+1], $Results[$i][$Nu } else { printf "%5.4f %5.4f } }
#Subroutines #Subroutine to generate two numbers normally distributed #From "The Perl Cookbook", recipe 2.10 sub gaussian_rand { my ($u1, $u2); my $w; my ($g1, $g2); do { $u1=2*rand()1; $u2=2*rand()1; $w=$u1*$u1+$u2*$u2; } while ($w>=1 || $w==0); #There was an error in the recipe, I corrected it here $w=sqrt(2*log($w)/$w); $g2=$u1*$w; $g1=$u2*$w; return wantarray ? ($g1,$g2) : $g1; }
64
This will install the required perl modules and compile the testprograms. The installation of these modules requires rootprivileges. Later you can run the openMosix stresstest also as a regular user. Chapter 14. (stress)Testing your openMosix installation 65
The openMosix HOWTO (you maybe have to delete old temporary files from root's testruns in /tmp because you won't have the permission to overwrite it as regular user) You are ready to run the test now with the command:
./start_openMosix_test.sh
using the RPMpackage There are some requirements to be met when installing the omtest.rpm, you will need e.g expect and compatlibstdc++7.32.96.110 (If you are on RH 8.0) Just install the omtest.rpm with the following command:
rpm ihv omtest.rpm
Now you can start the openMosix stress test with the command:
start_openMosix_test.sh
(The RPMpackage will be installed in /usr/local/omtest) (Please not that the RPM wil allso install some perl modules).
starting the openMosix stress test now! the results will be saved in : /tmp/openMosixstresstestreport03/16/200311:27:02.txt oMFS is not mounted at /mfs! oMFStest will be disabled. Please mount oMFS before running this openMosixtest You will find instructions how to configure and use oMFS at: http://howto.ipng.be/openMosixHOWTO/x222.htm#AEN243
66
67
68
16.2. Povray
The Persistence of Vision Raytracer is a highquality, totally free tool for creating stunning threedimensional graphics. Raytracing is a rendering technique that calculates an image of a scene by simulating the way rays of light travel in the real world. However it does its job backwards. In the real world, rays of light are emitted from a light source and illuminate objects. The light reflects off of the objects or passes through transparent objects. This reflected light hits our eyes or perhaps a camera lens. Because the vast majority of rays never hit an observer, it would take forever to trace a scene. These kind of applications can be easily made parrallel by using pvmpovray. Pvmpovray expects to working on a Beowulf style cluster and spread it's load to other nodes using pvm. The openMosix way of doing this is the same, however we just do this on 1 machine and have openMosix do the load spreading work fo you ! A GREAT Howto on PVM Povray will show you how to setup PVMPovray. Below is a small summary.
$ cd pvmpov3_1g_2 $ tar xfz ../povuni_s.tgz $ tar xfz ../povuni_d.tgz $ ./instpvm Trying to apply the patch. Searching for rejected files $
Now compile with aimk ( which is a wrapper script provided by the pvm rpm , buth which probably won't be in your path.(some of the readers might remember aimk from other platforms / applications ) If you are on a RH 8.0 box id moved libpng and zlib to .notused .. This in order to prevent version issues .. with other libpng and zlib versions.
export PATH=$PATH:/usr/share/pvm3/lib export PVMROOt=/usr/share/pvm3
I then run aimk newunix. Then we start pvm and quit it. The daemon stays active.. And a last thing that is not known by a novice pvm user is that pvm does use its own paths, and you have to put pvmpov either in that path or launch it with the complete pathname.
[root@dhcp71 povray31]# /usr/local/bin/pvmpov L
69
70
PVM Task Distribution Statistics: host name [ done ] [ late ] late ] dhcp71.office.be.stoneit.com [ 5.21%] [ 0.00%]dhcp71.office.be.stoneit.com [ 7.81%] dhcp71.office.be.stoneit.com [ 8.85%] [ 1.17%]dhcp71.office.be.stoneit.com [ 4.69%] dhcp71.office.be.stoneit.com [ 8.85%] [ 0.98%]dhcp71.office.be.stoneit.com [ 4.17%] dhcp71.office.be.stoneit.com [ 5.21%] [ 0.00%]dhcp71.office.be.stoneit.com [ 8.33%] dhcp71.office.be.stoneit.com [ 5.21%] [ 0.00%]dhcp71.office.be.stoneit.com [ 5.73%] dhcp71.office.be.stoneit.com [ 7.29%] [ 2.73%]dhcp71.office.be.stoneit.com [ 4.17%] dhcp71.office.be.stoneit.com [ 5.21%] [ 0.00%]dhcp71.office.be.stoneit.com [ 6.77%] dhcp71.office.be.stoneit.com [ 4.69%] [
host name
[ done ] [
71
POVRay statistics for finished frames: skyvase.pov Statistics (Partial Image Rendered), Resolution 1024 x 768 Pixels: 303104 Samples: 303104 Smpls/Pxl: 1.00 Rays: 1192710 Saved: 0 Max Level: 0/5 Ray>Shape Intersection Tests Succeeded Percentage Cone/Cylinder 1842227 900504 48.88 CSG Intersection 2742731 323346 11.79 CSG Union 1801008 521692 28.97 Plane 20223278 11233348 55.55 Quadric 1801008 1196533 66.44 Sphere 1801008 461786 25.64 Bounding Object 1842227 900504 48.88 Calls to Noise: 1201944 Calls to DNoise: 2108954 Shadow Ray Tests: 2856188 Succeeded: 85620 Reflected Rays: 889606 Smallest Alloc: 9 bytes Largest: 20508 Peak memory used: 5643343 bytes Time For Trace: 0 hours 1 minutes 7.0 seconds (67 seconds) Total Time: 0 hours 1 minutes 7.0 seconds (67 seconds)
As you can see the application was splitted into differen parts and run separatly, openMosix then did the job of balancing the load to other machines. I had good results with 2 to 3 times the the number of cpu's I had available
72
73
V. openMosix Development
Table of Contents 18. Getting started with openMosix internals 18.1. Introduction
V. openMosix Development
74
75
VI. FAQ
Table of Contents 19. the openMosix FAQ 19.1. General 19.2. Getting, building, installing and running openMosix 19.3. Kernel Questions 19.4. File Systems 19.5. Programming openMosix 19.6. Resources 19.7. Miscellaneous 20. PlumpOS FAQ 20.1. Frequently Asked Questions
VI. FAQ
76
77
The openMosix HOWTO 19.1.6. Who is the copyright holder of openMosix?: All MOSIX code is copyright by Professor Amnon Barak of Hebrew University of Jerusalem. All openMosix code is copyright by Moshe Bar, Tel Aviv. The openMosix system does not contain any nonGPL (i.e. MOSIX) code. 19.1.7. Is openMosix a fork of MOSIX?: Originally, openMosix was a fork of MOSIX, but it has evolved into an advanced clustering platform. The openMosix system no longer contains any nonGPL (i.e. MOSIX) code. Compared to MOSIX, a number of features were added: A port to the UML (Usermode Linux) architecture New and cleaner migration code A better load balancer Much reduced kernel latencies Support for Dolphin and IA64 A greatly simplified installation processes that uses RPM packaging A wealth of documentation 19.1.8. Why did openMosix split from the MOSIX group?: The principal issue was that MOSIX was not licensed with an Open Source license.
78
The openMosix HOWTO 1. Start by unpacking both the Linux kernel sources and the corresponding openMosix distribution in a directory, say /usr/src. 2. Then
$ cd /usr/src; tar xzf linux2.x.xx.tar.gz ;gunzip openMosix2.x.xx.gz
3. Apply the openMosix patches to the pristine Linux kernel sources with
$ patch p1 openMosix2.x.xxx
The directory /usr/src/linux2.x.xx now contains the 2.x.xx kernel sources with the openMosix patches. Compile and install the resulting kernel as usual. 19.2.4. What are userland tools?: Userland tools are a collection of administrative tools used to examine and control an openMosix node.
79
The openMosix HOWTO 19.3.4. I installed a Linux distribution and it says that its kernel is x.x.xx. The openMosix README says not to mix kernel versions. Does that mean that the openmosixx.x.xy RPM will not work on my machine?: No. It means is that if you install openMosix on your cluster, all your machines should have the openmosixx.x.xy kernel installed. You should not mix kernels which have different kernel versions, i.e. do not mix openmosixx.x.zx, and openmosixx.x.xy, etc. 19.3.5. What does the phrase the same kernel on every machine mean? Does it mean the same kernel version, or the same kernel image?: It means the same kernel version. You can build different kernel images of the same source version to meet the hardware/software needs of a given node.
The openMosix HOWTO 19.5.1. Generally, how do I write an openMosixaware program?: Write your programs as you normally would. Any processes that you spawn are candidates for migration to another node. 19.5.2. Can I write openMosix programs in perl?: Yes. Use the Parallel::ForkManager available from CPAN or directly from http://www.cpan.org/authors/id/D/DL/DLUX/ParallelForkManagerx.x.x.tar.gz.
19.6. Resources
19.6.1. Where can I find out technical details about openMosix?: 19.6.2. What other resources are available?: 19.6.1. Where can I find out technical details about openMosix?: Here are some links: Brian Pontz's openMosix source code browser is at http://www.openmosix.org/lxr/source. openMosix Internals: How openMosix Works is at http://sourceforge.net/docman/display_doc.php?docid=10390&group_id=46729. LTSP + openMosix: A Quick HowTo is at http://sourceforge.net/docman/display_doc.php?docid=10431&group_id=46729. Distributed OSs: General description of openMosix is at http://sourceforge.net/docman/display_doc.php?docid=9562&group_id=46729. More links at the link section of the howto. 19.6.2. What other resources are available?: Here are some: The goal of the Parallel Execution Framework is to create a standalone device that allows easy clustering of standard PCs. See http://parallel.sourceforge.net.
19.7. Miscellaneous
19.7.1. I don't see all my nodes. What's happening?: 19.7.2. Whats the difference between /etc/mosix.map , /etc/hpc.map , /etc/openmosix.map: 19.7.3. setpe: the supplied table is wellformatted, but my IP address (127.0.0.1) is not there!: 19.7.4. I want to install openMosix but I am afraid my machines are too weak for this: 19.7.5. Under what conditions does VMWare work with openMosix: 19.7.6. What architectures besides x86 (e.g. SPARC, AXP, PPC...) are supported by openMosix?: 19.7.7. Is there a parallel make tool for openMosix such as MPmake?: 19.7.1. I don't see all my nodes. What's happening?: When you run 'mosmon', press 't' to see the total number of machines running. Does it warn you that Chapter 19. the openMosix FAQ 81
The openMosix HOWTO openMosix is not running? If it does, then make sure your machine's IP address is included in /etc/openmosix.map. Don't use 127.0.0.1. If you do, you will probably have problems with your DHCP server or your DNS nameserver. If it does not, then see what machines show up. Do you see only your machine? If yes, then your machine is most likely running a firewall and is not letting openMosix through. If not, then the problem is most likely with the machine that doesn't show up. Also: Do you have two NIC cards on a node? If so, you have to edit /etc/hosts to have a line that has the following format
<noncluster ip> <clusterhostname>.<clusterdomain> <clusterhostname>
You might also need to set up a routing table, which is an entirely different subject. Maybe you used different kernelparameters on each machine? Especially if you use the 'Support clusters with a complex network topology' option you should take care that you use the same value for the also appearing option 'Maximum networktopology complexity support' on each machine. 19.7.2. Whats the difference between /etc/mosix.map , /etc/hpc.map , /etc/openmosix.map: They represent three stages of Mosix/openmosix growth. The file /etc/mosix.map is the orginal Mosix map name, The file /etc/hpc.map was an early openMosix map name (and 'hpc' is still used for the /proc files in openMosix). The current map name is /etc/openmosix.map. 19.7.3. setpe: the supplied table is wellformatted, but my IP address (127.0.0.1) is not there!: You'll need to modify your /etc/hosts file. On Red Hat machines mostly the /etc/hosts file includes a line like
127.0.0.1 hostname.domain.com localhost
If hostname.domain.com has an IP address of 192.168.10.250, and if you looked up hostname.domain.com you might get 127.0.0.1 as an answer. However, if you put
192.168.10.250 hostname.domain.com 127.0.0.1 localhost
in your /etc/hosts, openMosix won't complain. 19.7.4. I want to install openMosix but I am afraid my machines are too weak for this: A machine is never too weak: I have three P200s (64MB each) and two P166s (one with 48MB and one with 192MB). Two of them are on 10BaseT and the other three on 100BaseT. Even with these antiquated machines and "heterogenous" network, I get perfect load balancing to run simulation programs that I write in Perl. (Look at our ProgramToTestACluster"). Don't be held back by the fact your machines are old. To us this is a nice feature of openMosix: you can add newer machines to an existing cluster as they become available. Chapter 19. the openMosix FAQ 82
The openMosix HOWTO And you do not need to have all identical machines. That's fantastic! However, a 100BaseT network is recommended! Contributed by Charles Nadeau. 19.7.5. Under what conditions does VMWare work with openMosix: If you intend to run VMWare under openMosix so that openMosix would loadbalance several instances of that (yes, that works). But, if you want to run openMosix in several VMWare instances and let these instances load balance (that fails). The first case works. The latter case does not work because VMware has a bug in its Pentium emulation that makes VMware crash (not openMosix, but the VMware binary) on the first migration. 19.7.6. What architectures besides x86 (e.g. SPARC, AXP, PPC...) are supported by openMosix?: Only IA32 is currently supported. The port of openMosix to the Intel(r) Itanium(tm) IA64 Processor Family is complete. Project plans for openMosix' second year include porting to the 64bit AMD Opteron(tm) processor. 19.7.7. Is there a parallel make tool for openMosix such as MPmake?: You can use regular gcc make. just use make j #, where the # represents how many child proccesses to spawn.
83
The openMosix HOWTO 20.1.6. The penguins are gonna eat the poor halibut in the logo! Yes, but not to worry: no halibuts were harmed in the making of the logo. The halibut was found deceased on an ice drift after having suffered a heart attack. Also, any resemblance the halibut may have to another operating system's mascot of choice is purely coincidental.
85
5. Make a symbolic link to the directory where you've checked out the linuxopenmosix module, for instance:
# ln s /home/mcaserta/src/linuxopenmosix/linuxopenmosix \ /usr/src/linuxopenmosix
6. copy the .spec file and all the .config files which are found in this directory under /usr/src/redhat/SOURCES, ie:
# cp /usr/src/linuxopenmosix/configs/openmosixkernel.spec \ /usr/src/redhat/SOURCES # cp /usr/src/linuxopenmosix/configs/*.config \ /usr/src/redhat/SOURCES
7. Now it's time to check the version numbers before we make the patch file: make sure the very first lines in
/usr/src/linuxopenmosix/Makefile and /usr/src/redhat/SOURCES/openmosixkernel.spec have the correct
kernel version and openMosix revision number 8. Good, time to make the patch (suppose we're releasing a patch for the 2.4.20 Linux Kernel and the 3rd release of openMosix):
# cd /usr/src # diff Naur exclude=CVS exclude=configs \ linux2.4.20 linuxopenmosix > \ /usr/src/redhat/SOURCES/openMosix2.4.203 # bzip2 /usr/src/redhat/SOURCES/openMosix2.4.203
10. Now you only need to rpmbuild the whole thing what I usually do is:
openmosixkernel.spec openmosixkernel.spec openmosixkernel.spec
86
The openMosix HOWTO but you can easily build all rpms by calling:
# rpmbuild ba target all_x86 openmosixkernel.spec
11. After rpmbuild has done its job, you should have the following files under the /usr/src/redhat directory:
a) RPMS/i386/openmosixkernel2.4.20openmosix3.i386.rpm b) RPMS/i686/openmosixkernel2.4.20openmosix3.i686.rpm c) RPMS/i686/openmosixkernelsmp2.4.20openmosix3.i686.rpm d) RPMS/athlon/openmosixkernel2.4.20openmosix3.athlon.rpm e) RPMS/athlon/openmosixkernelsmp2.4.20openmosix3.athlon.rpm f) RPMS/i386/openmosixkernelsource2.4.20openmosix3.i386.rpm g) SRPMS/openmosixkernel2.4.20openmosix3.src.rpm h) SOURCES/openMosix2.4.203.gz
where:
a) binary kernel package for i386 UP* machines b) binary kernel package for i686 UP* machines c) binary kernel package for i686 SMP** machines d) binary kernel package for athlon UP* machines e) binary kernel package for athlon SMP** machines f) source kernel package for any x86 machine (basically this is useful if you need to have the openMosix kernel headers) g) source kernel package (see point 11) h) kernel patch file compressed with gzip
12. The magic spell to obtain all the files back from the .src.rpm file is: # rpm2cpio openmosixkernel....src.rpm | cpio di Special thanks to Martin Hy for the help while I was trying to get the whole thing together. I hope you find this document useful. At least it is for me since I tend to forget things a few minutes after I've accomplished them :) * UP = UniProcessor (i.e. one CPU) ** SMP = Symmetric Multi Processing (i.e. more than one CPU)
87
B.3.1. Chinese
Ding Wei has written some documents in Chinese, you can read them at http://software.ccidnet.com/pub/disp/Article?columnID=732&articleID=25795&pageNO=1 Here is a local copy to the Chinese doc in PDF
B.3.2. Spanish
Together with some collegues Miquel Cataln Cothas been working on a spanish translation of the HOWTO http://w3.akamc2.net/
B.3.3. Russian
Dmitry Katusubo translated the openmosix website and together with Yuri Prushinsky he also translated the openMosix HOWTO http://www.openmosix.org.ru/docs/openMosixHOWTOmulti/
B.4. Links
openMosix on Sourcefourge the openMosix HOWTO the openMosix FAQ the openMosix WIKI the openMosix Community Contributions openmosixview Mosix Debian HOWTO K12 LTSP Mosix HOWTO Mosix Mandrake Linux Terminal Server Project openMOSIXVIEW, a GUI for managing openMosixCluster User Mode openMosix, a virtual openMosix cluster running in Usermode Appendix B. More Info 88
The openMosix HOWTO RxLinux, Web Interface for central configuration and management LTSP+OpenMosix: A Mini HowTo FuBAR: An openMosix cluster at Texas AM Corpus Christi Charting the Land of Elliptic Curves using openMosix openMosixWebView Italian experiences with openMosix clusters Links to several papers presented at the conference. These papers cover many recurring questions seen on the openMosix mailing list: Osservatorio Astronomico Cagliari (pdf)Securing an openMosix cluster in an untrusted networking environment. D.T.I. dell'Universit di Milano, Polo didattico e ricerca, Crema (pdf) Experiences with Java and C programs on openMosix. Riello Group (pdf) Use of openMosix for parallel I/O balancing on storage (backup) INFM & Dept. of Physics University of Napoli (pdf) Documents a number of small to large clusters and their uses. (Includes many references to software and projects.) Conecta srl (pdf) Contains more information on Kraken, the openMosix based game server. Included is how they monitor the cluster's temperature via lmsenors and how they improved internode communications using iproute2 queue controls. openMosix with Diskless client Use networked Linux systems to solve your computing challenges by Daniel Robbins
89
Appendix C. Credits
The list of people who deserve credits for this HOWTO is long, I actually lost track of all the people that should be in here. I often add their names right next to the parts they have contributed. If you feel that your name is missing here do not hesitate to contact me and I'll gladly put your name into the list. Scot W. Stevenson I have to thank Scot W. Stevenson for all the work he did on this HOWTO before I took over. He made a great start for this document. Assaf Spanier worked together with Scott in drafting the layout and the chapters of this HOWTO. and now promised to help me out with this document. Matthias Rechenburg Matthias Rechenburg should be thanked for the work he did on openMosixview and the accompanying documentation , which we included in this HOWTO. JeanDavid Marrow is the author of Clump/OS, he contributed the documentation on his distribution to the HOWTO. Bruce Knox is the maintainer of the openMosix website, he helps where he can and gives a lot of feedback ! Evan Hisey for putting a lot of effort into putting extra documentation in the WIKI Charles Nadeau for putting a lot of effort into putting extra documentation in the WIKI Louis Zechter Moshe Bar For writing the code he wrote and helping out with the docs wherever he knows the answers ! Amit Shah for getting started with the openMosix internals Mirko Caserta Appendix C. Credits 90
The openMosix HOWTO For sending in huge patches to this howto Ramon Pons for proofreading the howto and sending in some advice
Appendix C. Credits
91
Copyright (C) 2000 Free Software Foundation, Inc. 59 Temple Place, Suite 330, Boston, MA 021111307 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
0. PREAMBLE
The purpose of this License is to make a manual, textbook, or other written document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others. This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software. We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.
92
The openMosix HOWTO A "Transparent" copy of the Document means a machinereadable copy, represented in a format whose specification is available to the general public, whose contents can be viewed and edited directly and straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup has been designed to thwart or discourage subsequent modification by readers is not Transparent. A copy that is not "Transparent" is called "Opaque". Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standardconforming simple HTML designed for human modification. Opaque formats include PostScript, PDF, proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machinegenerated HTML produced by some word processors for output purposes only. The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.
2. VERBATIM COPYING
You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3. You may also lend copies, under the same conditions stated above, and you may publicly display copies.
3. COPYING IN QUANTITY
If you publish printed copies of the Document numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: FrontCover Texts on the front cover, and BackCover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects. If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages. If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machinereadable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a publiclyaccessible computernetwork location containing a complete Transparent copy of the Document, free of added material, which the general networkusing public has access to download anonymously at no charge using publicstandard network protocols. If you use the latter option, you must Appendix D. GNU Free Documentation License 93
The openMosix HOWTO take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public. It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.
4. MODIFICATIONS
You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version: A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission. B. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has less than five). C. State on the Title page the name of the publisher of the Modified Version, as the publisher. D. Preserve all the copyright notices of the Document. E. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices. F. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below. G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice. H. Include an unaltered copy of this License. I. Preserve the section entitled "History", and its title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence. J. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission. K. In any section entitled "Acknowledgements" or "Dedications", preserve the section's title, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein. L. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles. M. Delete any section entitled "Endorsements". Such a section may not be included in the Modified Version. N. Do not retitle any existing section as "Endorsements" or to conflict in title with any Invariant Section. If the Modified Version includes new frontmatter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these Appendix D. GNU Free Documentation License 94
The openMosix HOWTO sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles. You may add a section entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various partiesfor example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard. You may add a passage of up to five words as a FrontCover Text, and a passage of up to 25 words as a BackCover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of FrontCover Text and one of BackCover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one. The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.
5. COMBINING DOCUMENTS
You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice. The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work. In the combination, you must combine any sections entitled "History" in the various original documents, forming one section entitled "History"; likewise combine any sections entitled "Acknowledgements", and any sections entitled "Dedications". You must delete all sections entitled "Endorsements."
6. COLLECTIONS OF DOCUMENTS
You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects. You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.
95
8. TRANSLATION
Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License provided that you also include the original English version of this License. In case of a disagreement between the translation and the original English version of this License, the original English version will prevail.
9. TERMINATION
You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.
96
Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with the Invariant Sections being LIST THEIR TITLES, with the FrontCover Texts being LIST, and with the BackCover Texts being LIST. A copy of the license is included in the section entitled "GNU Free Documentation License". If you have no Invariant Sections, write "with no Invariant Sections" instead of saying which ones are invariant. If you have no FrontCover Texts, write "no FrontCover Texts" instead of "FrontCover Texts being LIST"; likewise for BackCover Texts. If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.
Index
97