OBC design and integration experiences

The story of the birth of the On Board Computer for SSETI Express

KarlKaasLaursen 2004

Preface
ThisdocumentiswrittenasaremindertomyselfandothersofthingstodoandespeciallyNOTdo whendesigningandintegratingacomplexsystemlikeanonboardcomputerforspaceapplications. Thefirstsectionsarememoiresofmoreorlesssignificantincidentsandgeneraldocumentationofthe workdonebymyteamandmefromthebeginningofExpressuntilnowwherethespacecraftisclose tocompletion.Thispartofthedocumentisarrangedintosmall,easytodigestsnacksoftheStory ofTheOBC.Laterinthedocumentareshortproblemdescriptionsleadingtolessonslearnedwhich Iregardasworthtakingaquicklookat.AllpeoplementionedinthisdocumentareREALliving peoplewiththoughtsandfeelingsoftheirown(well,atleastmostofthemhavethoughts)andifyou areoneofthesepeople,don'tfeeloffendedifyournameappearsinthesamesentenceasanItalian youmightbealright,anyway. KarlKaasLaursen,October2004,Jason'scleanroomatESA/ESTEC

Part I
The Story of the OBC From AAUSAT to SEx
ItallstartedinDecember2003whenLarsAlminde,AAU,camebackfromESTEC withaproposalformyseventhsemestergroupatAalborgUniversity:Wecouldhave theOBCthatweweredesigningforAAUSATIIflownonajointEuropeanstudent satellitecalledSSETIExpress(SEx).Theproposalwasgladlyacceptedwithout furtheradueandwecontinuedourdailylifedoingotherthingsastheseventh semesterOBCprojectwasalreadyfinishedandwewerecaughtupinourprojectsfor thenextsemester. BythewordfinishedIdon'tmeanOBCinaboxaliveandkickingbutrather semesterendedandwegotboredandwantedtogohome.WespenttheOBC designsemestercomingupwithanonboardcommunicationprotocolknownas INSANE(InternalSatelliteAreaNetwork),andthemainprocessorandoperating systemforitwerechosenleadingtoaframeworkforOBCforAAUSATII.Butno hardwaredesignwasaccomplishedasweregardedthattobesomewhattrivialand quietlydecidedtopostponethatuntilwefeltthesourbreathofthedeadlinesonthe backofournecks(youfeelthatjustmomentsbeforehearingthewhooshingsound theymakeastheygoby). Tworepresentativesfromthegroup(MikeandDanny)weresenttoESTECforthe SSETIExpressJanuaryworkshopthatinitiatedthedesignofthesatellite.Alongwith themwentLarsAlmindeandMortenBisgaard(fromhereonknownastheGrumpy OldMennooffence,ofcourse)whoalsohadsubsystemstostickontoExpress, namelytheACDS(attitudecontrolanddetermination)andCAM(camerapayload).
2

AttheworkshopitbecameclearthattheAAUSATIIcomputerwouldnotexactlyfit Expressinitsoriginalformbuttherewerewaysaroundthat.

Designing the On Board Computer for SSETI Express

WedecidedtodesignacomputerthatwouldbemuchliketheoneforAAUSATII anddevelopaninterfacecardforittocomplywiththeinterfacesinExpress.Ithad beendecidedtouseaCANbusforallinternalcommunicationinAAUSATII becauseofitsrobustnessandthesimpleelectricalpropertiesthatenablecompact design.Unfortunately,onlyoneothersubsystemonExpresswouldusetheCAN interfacesotheinterfacecardwastobefittedwithfiveRS232connectionsforEPS, UHF,SBAND,CAMandACDS.Inaddition,theinterfacecardshouldhaveanalogue inputsforsamplingthermistorsinTCS,thethermalcontrolsystem,andforreading valuesofsunsensorsintheACDS.Also,digitaloutputlineswereneededinorderto controltheelectromagneticattitudecontrolactuators,themagnetorquers. Thedesignofthecomputerandtheinterfacecardwasprogressingquiteslowly.Some ofusweremorefocusedondevelopingattitudecontrolalgorithmsforAAUSATII andtestingthemonthe7thStudentParabolicFlightCampaigninthesummer2004 andothersweredoingnonspacerelatedprojectsanddidnotpayattentiontoExpress atall.AsthedesignworkshopinMayrapidlyapproachedwemadethefirstdraftfor theoverallarchitectureoftheOBCwhichIpresentedattheworkshopandpeople seemedtoacceptit. ThedesignofthemainboardincludedanAtmelARM7mainprocessor(goodplan) fortheonboarddatahandlingsoftwareandasmaller(andlesspowerful)utility processorforinterfacingtheARMtoCANbus(verybadidea).Theutilityprocessor waschoseninsteadofadedicatedCANcontrollerbecauseitwouldprovideuswitha seeminglycleverwayofmakingsoftwareuploadtoanonvolatilerewritableFLASH memory.Thestoryoftheutilityprocessorshouldhaveitsownchapter...maybelater, TheRS232connectionsontheinterfacewasareallyannoyingfeaturetoimplement becauseitrequiresalotofhardwarebesidethecompulsorymicrocontroller.I decidedtomultiplextheRS232connectionssuchthatweonlyneededtohaveone UART(universalasynchronousreceiver/transmitter)whichanymicrocontroller wouldhaveinternally.Thisideademandedtheneedforhardwareflowcontrol(good idea)onallsubsystemscommunicatingviatheseconnectionsandatsomepoint duringtheworkshopthemultiplexerthingwasdroppedandreplacedbyfive independentUARTsthatallhadtoworkinparallelwithoutflowcontrol(verybad idea).Atthetimeitlookedlikeadoablesolutionandquickcalculationsdoneinmy headconvincedmethatwecouldeasilyaccommodatethedataflowdemandsfrom
3

thefiveRS232connectionsandthesingleCANconnection.Ishouldcometoregret this.

Interfaces
ThedesignworkshopinMaywastheplaceforinterfacediscussionsandintheperfect worlditwouldhavebeenthelastplaceforinterfacechanges.Theworldisnot perfect.Thestoryofthechanginginterfacesistoolongtofitintothisdocumentso onlytheconclusionisincluded:Defineallinterfacesandwhentheyhavebeenagreed ondon'tchangethemwithoutallpartiesinvolvedbeingtotallyawareofthechange. Otherwise,therewillbemisalignedinterfacesthatarenotdiscovereduntilafter someone'sflighthardwareisalreadyfinished.

Thedesign
ThefinaldesignoftheOBCwascarriedoutinaweekortwoinJuly2004.Danny andIwereattheparabolicflightwhiletherestofthegroupspentthreedayssketching aschematicofthemainboard.WereturnedfromBordeauxtoahalffinisheddesign anddiscoveredquicklythatsomeonehadforgotthattheutilityprocessor(likeany otherprocessorinspace)alsohadtouseonetimeprogrammablecodememory.The chipswehadacquiredforutilityandinterfacecardprocessorsweretheAtmel AT89C51CC03withinternalFLASHmemoryforcodeandoptionforexternalcode memory.ThatmeantthatwehadtopluganexternalOTPROM(onetime programmablereadonlymemory)ontoeachoftheprocessorsasOTPROMismuch moreresistanttoradiationthantheFLASHmemory.Havingthreeofthese89C51 processorswewould,ofcourse,havetohavethreeexternalPROMseachwitha datalatchforaddress/databusmultiplexing. Now,itsuddenlyoccurredtousthatwehadfourprocessorsinourdesignwithboth FLASHandPROM,sowhynotdesignthehardwareinsuchawaythatitwouldbe possibletouploadnewsoftwaretoanyofthematanygiventime(nonono!)?We addedexternallatchestotheprocessorsenablingthemtoswitchbetweenFLASHand PROMcodememorybysendingitcommandswhenexecutingfromPROM(default codememory).Theideawasthatatsomepointweshouldprogrammeabootloader thatcouldreceivenewsoftwarefromsomewhere,storeitinFLASHandthen executethecode.ButthesomepointnevercameIamstillwaitingforit. Afterafewintensivedaysofcomputerdesigntherewerepagesofschematicsbutno timetoprototypeanythinginjusttendayswehadtogotoESTECtointegratethe OBCflighthardware(andsoftware)withalltheothersubsystems.Therewasonly onesensiblethingtodothen:LayouttheflightPCBsinajiffy,soChristianandIsat downforthirtyhoursandlaidoutthemainboardandinterfaceboard.Christiandid
4

themainandIdidtheIF.Thirtyhourspassedandwehadacompletelyfinished hardwaredesignofamainboardwithanARM7CPU,an89C51,twoRAMchips, threePROMs,aFLASH,somelatches,anRS232linedriverfordebugginginterface, aCANtransceiver,twocrystalsandlotsofaccessorycomponentslikedecoupling capacitors,pullup/downresistorsandpowerupresetcircuits.Andwehadaready designoftheIFboardwithtwo89C51CPUs,twoPROMs,threelatches,twodual UARTSforRS232,threedualRS232linedrivers,twoCANtransceivers,twenty eightprotectiondiodesforanalogueanddigitalIO,twocrystalsandallthelittle thingsplusaswitchingpowerconverter,coil,capacitorsandadiodeforconverting the28Vregulatedbusvoltagedownto3.3Vthatmostelectronicsuse;includingthe onboardcomputer. Coincidentally,wehadgotaverynicedealwiththePCBmanufacturerElprintA/S whowantedtohelpusbyproducingtheflightboardsfreeofchargesavingus thousandsofEuros.ThePCBGerberfilesweresenttoElprintwhilewewerebusy programmingallthesoftwareneededtooperatenolessthanfourprocessorsinour distributedcomputerarchitecture.Thousandsoflineofcodewerewritteninafew dayswithoutanyprototypehardwaretotestiton(don'tdothat)wecallthis procedureopenloopprogramming.

ESTEC here we come

TimecaughtupwithusandthePCBshadstillnotarrivedfromElprintbutwehadan appointmentwithDr.Melville(youmaycallhimNeil)tocometoESTECwiththe OBCflighthardwaresoweleftAalborg.Danny,MikeandJakobinaLupo.Christian andIonaplane.ArrivingatESTECwecouldtellNeilthatourstuffwasonthe wayandjusthadtobeputtogetherbeforewecouldstarttesting.Confidencewas stillhigh.WhilewaitingforthePCBswecontinuedwritingsoftwareinopenloopand Jakobstartedprogrammingtheonboarddatahandlingsystemtorunonthemain processorontopoftheeCosoperatingsystem. PCBsarrivedandwewerethrilledtoseethequalityoftheboardsfromElprint.It wouldbeapleasuretosolderthoseboardssoweimmediatelysetupasoldering laboratoryinNeil'sandMarie'soffice(don'tdothatitisnotantistatic)andsoldered the(first)engineeringmodelsofthemainboardandtheinterfacecard.Eventhough weusedaniceoldWellersolderingiron,itwasstillliketryingtouseacowfor solderingthe.5mmpitchpinsontheprocessors,UARTsandtheFLASH.Butit seemedtoworkoutalright,andaftermanyhourswewerereadytopowerupthe boards. FivePROMswithfirstdraftsoftwarewereburned(usingourillegallyacquiredROM programmer)andplacedinthefivePLCCchipcarriers(btw,forflighthardware,
5

don'tusePLCCsocketswithpingridarraypackagethatisnotcompatiblewiththe footprintofthechipsthatgointhesockets).

Themainboard
Themainboardwaspoweredupanddebuggingthehardwarecouldbegin.The numberofproblemspresentingthemselvesinthehours,days,weeksandmonthsto comedoesn'tfittheformatofthisdocument.Inthebeginning,therewereproblems withdeadRAMpossiblycausedbystaticelectricityinNeil'ssolderinglab.They wereforsuredeadandhadtobereplaced.Aftersometimewesucceededingetting themainboardtoshowsignsoflifeonthedebugginginterfaceandJakobcouldstart testingthedatahandlingsoftwareinitsrealenvironmentinsteadofusingtheARM7 evaluationboard.Butbyaccident,aselfdestructversionofhissoftwarefoundits wayontothePROMandkilledtheRAMsagain. Whenmorememorychipsaresharingadatabus(RAM,ROMandFLASH),you mustensurethatonlyonechiphasadefinedelectricalpotentialonitsdataoutputsat anygiventime,otherwisethechipswillsourcecurrentfromeachother'soutputsuntil onlyoneofthecompetingchipsisleftalive.Thistime,theFLASHchipwonand bothRAMdiedfromamistakeinthememorylayoutthattheARM7wasinstructed tofollowafterthesocalledremapinstruction. Theerrorwascorrectedbuttheboardwasinapoorshapeafterthepreviousde solderingoftheRAMsandcouldn'tsurviveanotherrefittingofRAM.Another engineeringboard(tobeknownasthepreflightboard)wasmadeafewdayslaterin thecleanroom. Meanwhile,theinterfacecardwasbeingtestedinparallelwiththemainboardafter thecompletionofsolderinginNeil'slab.Poweron.On?Off.Onagain.Nothing smellingfunny(whichisgood)butnotmuchfunhappeningatall.Off.On.Andsuch beginsthestoryofexternalcodefetchontheAT89C51CC03fromAtmelCorp.

TheStoryofExternalCodeFetchOntheAT89C51CC03
ThestoryofexternalcodefetchontheAT89C51CC03islongbutI'llmakeitshort ish.The89C51processorcanexecutecodefromeitherinternalFLASHmemoryor externalanythingmemory,inourcaseitwasPROM.Itsayssointhedatasheetfor thechip.Butwhenweconfiguredthechipforexternalcodefetchusingtheexternal addressingswitchtheprocessorjustpretendedtoexecutetheexternalcode.Using ourlogicanalyserwecouldobservetheCPUaddressingthePROMjustasexpected fromaddresszeroandcountingupwardsandthebytesofcodereturnedfromthe PROMtotheCPUlookedjustlikethecodewehadwritten.Ataddresszerowas instruction0x02followedbytwobytesparameterwhichisalongjumptothe
6

beginningofthecode.SoafteraddressingthefirstthreeaddressestheCPUshould jumptotheaddresspointedtobythelongjumpandstartourprogram.Butthe addressesjustcycledupwardsinbinarycountingsequentialorder.Thatmeantthatno longjumpwasperformed(thisanalysistookseveraldays).Strange. Backtothedatasheet.Everythingwiredupcorrectly?Yes.Correctinstructionformat generatedbythecompiler?Yes.TimingoftheaddressingandthePROMoutput correct?Yes.Anythingouttheordinaryanywhere???No.Dayswentbystaringatthe logicanalyserandtryingtohookituptodifferentprocessors.Nothing.Wecalled Jens,oursupervisor,onaSundaynightandhestartedstudyingthedatasheettoseeif hecouldfindsomethingweirdthatwewouldneverthinkof.Thisleadtothe discoveryofthebootloaderjumpbitwhichmightnotbesetcorrectly.Theboot loaderjumpbitisonebitinaregisterintheCPUthattellsitwhetherornotitshould executeitspreprogrammedinternalbootloaderinsteadoftheuserapplication regardlessofthehardwarecondition(likeexternaladdressingswitch).Thedatasheet seemedabitfuzzyonthispointsowehadtofindawaytogoinandgetthatjumpbit out. TherearetwowaysofalteringthecontentsthearegisterintheCPU:Viaanexternal parallelprogrammingdeviceorbyinteractionwithaprogramrunningonCPU.The firstoptionwasnotreallypossibleasthechipswerealreadysolderedtotheboardand wedidn'twanttoremovethemtodothat(verydamagingtotheboardandtotally destructivetothechipitself).Sowehadtofindawaytocommunicatewiththeboot loaderresidentininternalFLASH.ThisparticularbootloaderusesCANinterfaceto communicatewiththeworld,butbeingwithoutanyCANinterfacecardforanyofour computerswewouldhavetobuildoneoutofthecomponentsinourpockets. Luckily,weareengineers(ifbydiplomayet,thenbyspirit,anyway)whichmeans thatwehavelotsofcomponentsinourpocketssowefoundaPIC18F458micro controllerwhichhasaCANinterfaceandanRS232interfaceandquicklywiredupa circuitonapieceofcardboard.DannywrotesomesoftwareforthePIC,Christian wrotesomelowleveldriverforitforaPCandIdesignedapieceofsoftwaretoparse hexfilesonthecomputerandcommunicateviatheprotocolofthebootloader.After afewdays(everythinghappensinsequencesofafewdays)wehadmadeaCAN interfaceforaPCwithanRS232connections(standardserialline)andwecouldchat withthebootloaderandflipthedarnjumpbit. TheCANinterfacesystemcomprisingthePICplusotherhardwareonapieceof cardboardandaprogramforLinuxwouldbeknownasCanTermanditisthebest inventionoftheentireOBCproject.Ithassimplyproventobeinvaluablefor developingasystemwithCANinterfaceandithasbeenmodified/expandedmany timessinceitsbirthinasmellyofficeatESTEC.
7

Thedaysofthenastyjumpbitwereoverandwepoweredupthesysteminexternal addressingmodeandwaitedforsomethingtohappen.Waiting,Isaid.Nothing. Now,thingsreallystartedtolookscarybecause,accordingtothedatasheet, everythingwasbeautiful.Only,itwasn't.Noinstructionswereexecutedfromthe externalmemorysowecontactedthechipmanufacturer,Atmel.Aftersomedaysthey answeredusandtheanswercontainedthewordswearesorryfor....Itturnedout thattheprocessorscouldnotexecutecodefromexternalmemorybecauseofafactory settingofahardwaresecuritybytethatdisabledexternalcodefetch.Atmelapologised fornotwritingthatfactloudandclearinthedatasheetwhenexternaladdressingwas describedasafeatureoftheCPU. Therewasonlyonewayofresettingthesecuritylevelofthechipstoenableexternal fetchandthatwasbyinsertingthechipsintoaparallelprogrammerandperformafull chiperase.Chipshadtobedesolderedwhilenewchipswerebeingerasedand reprogrammedwiththebootloaderwhichwehadgrownfondofbecauseitmakesit possibletouploadandexecutesoftwareontheprocessorswithoutburninganew PROMeachtimethereisachange.AndCanTermwasthenumberonetoolfordoing so. Onechipwasdesolderedandreplacedbyafreshlyerasedbrotheranditwasthe momentoftruth.WhilewaitingforanswersfromAtmelwehaddonesoftware developmentandtestingontheCPUsusingCanTermandhadsomeworking software.AworkingpieceoftestsoftwarewasburnedintoaPROMthatwasplaced inthesocketforPROM1forCPU1ontheinterfacecard.Powerup.RS232cable connected.OnebytetransmittedtotheCPU.Anditanswered.Externalcodefetch wasnowworkingafterthreeweeksatESTEC.Withouthesitationweerasedabunch ofchipstobeusedontheflightboardsandrushedtothecleanroomwhereJason awaitedus.

Em111b
ThisisthestoryofJasonPage'scleanroom,roomEm111b.Likemostotherstories, thiscouldpotentiallybecomealongstory.Jason'scleanroomistheplacewhere SSETIExpressisbeingintegratedrightnow(thisdocumentisbeingwritteninthis room)andJason'swordsarethelegislationinhere.Andthisiswherewebroughtour PCBsandhundredsofcomponentstointegratetheOBCflighthardwareusing properequipment. Thecleanroomisequippedwithmicrowavesolderingequipment,microscopesforuse whensoldering,3Dmicroscopeforinspectionofsolderjoints,allkindsofcleaning equipmentandallchemicalsneededforthat,componenttinningbaths,harness manufacturingtoolsandwires/connectorsandaninfinitenumberofinfinitelyclever
8

toolsforeverysituation.TheDconnectorpinextractiontoolisanexampleofa remarkablyeffectiveandstunninglysimpletoolthatIhaveaspecialplaceforinmy heart(yes,Ihaveaheartsomewhere). HavingreceivedinstructionsbyJasonwestartedsolderingaboardthatweclassified asapreflightboardtoreplacethedeadengineeringboard.Thiscouldhavebeena flightboardbutweconsidereditagoodideatopractisesolderingskillsonsomething nonflightbeforemakingtheflighthardware.Christianstartedtheworkonthepre flightboardandafteronedayofSSETIGeneralAssembly,Itookover.ForaweekI wastheguyatthemicroscope,visiblefromthewebcam,apparentlynotmovingone centimetreduringawholeday.Afteraboutfivedayslikethatthepreflightandthe flightboardswerecompletedexceptforafewcomponentsthathadreplaced componentsthatsufferedacrueldeathontheengineeringboards. Solderingusingtheequipmentinthecleanroomisatotallydifferentstorythan solderinginNeil'slabusingacow.Even.5mmpitchpinscanbesolderedperfectly withoutanydangerofcomingintocontactwithadjacentpinsorsolderpads.A solderingcourseisoutsidethescopeofthisdocumentbutitisatleastworth mentioningthatusingtherighttoolsandknowingwhatyouaredoingisessentialtoa goodresult.Followingthesolderingguidelines,howeveroverprecocioustheymight sound,isinstantlyrewardinginthattheboardsworkthefirsttimeandcontinuesto workforeveriftheyareproperlysoldered. Thecleanroomiscleanandthereforeeverythinginitshouldalsobecleanatalltimes includingPCBs,sowhensolderingonPCBsyoualwayskeepitcleanusingIPA everytenminutesorso.Thisnotonlymakesiteasiertoworkwithbutalsohasthe advantagethateverytenminutesyoutakeawayresidualfluxandotherkindsofmore oflessopaquesubstances/particlesthatmayconcealerroneoussolderjoints.

Autumn activities no life

AugustpassedatESTECandSeptemberluredaroundthecornerwithabrandnew semesterandlotsoflecturestofollowbackhomeattheuniversity.Threeoutofsix teammembersdecidedthatthelecturesandtheirsemesterprojectswerefarmore importantthanastudentsatelliteprojectintheNetherlandssoMikeandIdidn'tsee muchofthemforthemonthstocomeandtheOBCwasstillnotready.Theopen loopsoftwaredevelopmentthroughoutthesummerhadresultedinthousandsoflines ofcodewithhundredsofseeminglyunresolvablebugsthattendedtodriveuscrazy.

ReturningtoEm111b
ThedealwasthatIshouldgobacktoESTECafterfinishingtheOBCsoftwareback homeinlittleoveraweek.Theflightwasbookedanddeparturedaycame.The
9

softwarelookedprettygoodandIflewtoA'damwithMikebackingupfromAalborg sittingintheSatLabtherewithourengineeringmodeladdingthelastcoatsofchrome tothesoftwarewhileIwasworkinginthecleanroomsolderingthelastfew componentsandsolderingredundantwiresontotheboards. Itturnedoutthatthedaysofthehundredbugsinthesoftwarewerenotoveryetand withmeinESTECandMike,theonlyremainingteammember,inAalborgitwas almostimpossibletodoefficientdebugging.SoLarsmadeaquickdecisionfromhis recentlyacquiredmanagementchair(PhDstudentposition).MikeleftforESTEC withtwelvehoursnoticeandheandIdidsomeintensivedebuggingdownhere.But strangesoftwarebugsjustseemedtoreplacethemselvesbyevenstrangerbugswhen theEMwasmoved700kmsouthwestofAalborg. TwodayslaterMikeflewbacktoAalborgwiththeEMandmoremanpowerhadto bebroughtin.ThismeantdraggingtheGrumpyOldMenawayfromtheirresearch anddowntothecosySatLabwherewelearnedtoliveourlivesincompleteisolation fromtheotherstudentsinourdepartmentthoughmostofthemwerebasedjustnext door(includingthethreedeserters). AftertenmoredaysatESTECtheOBCflighthardwarelookedprettymuchreadyto flyexceptfortheemptyPLCCchipcarriersthatjustwaitedforfivePROMswith workingsoftwaretobepluggedintothem.Butthesoftwarewasnotatallreadyyet whenIflewbacktoAalborg.

Aalborgagain
BackinAalborgIimmediatelyreturnedtotheSatLabtoassistinthesoftware debugging.Meanwhile,OtherJacobhadbeenenrolledbyMiketohelpwiththe software.OneofthemajorproblemswasgettingtheCANdriveronthe89C51to workproperly.Theoriginalprogrammerofthedriverhadlefttheprojectandthebug countwasscarysoOtherJacobdecidedtocompletelyredothedriverfromscratchin amorelogicalandthoroughwaywhichturnedouttobeabrilliantidea. Jakob(notOtherJacob)hadtoldmethattheonboarddatahandlingsystemwas finishedandhadnoknownbugs.Ireadthroughmostofthecodeandfoundhuge partsmissing(youcan'treallyseethatamissingpartishugeunlessyouhaveanidea ofhowitshouldbeimplementedIdid)andthepartsthatweren'tmissingwere prettymuchcoveredinbugsleavingclosetonobuglessfunctionalityleft.Withclose totenthousandlinesofcodetodebugIwaslookingforwardtomanylongdaysinthe SatLab. Istarteddebuggingthebuggypartsandfillinginthemissingparts(likee.g.telemetry andpicturedownloadandcommunicationwiththemodem)andthedatahandling
10

systemwastakingshape.OneofmyfavouritegameswasFindingtheFLASHBug oftheDaywhichIcouldcontinuedoingformorethanaweek.Andstill,threeweeks later,IfindsmallbugsintheFLASHstoragecodethataremissioncriticalrendering telemetrystorageimpossible(nowitallseemstowork,though).Isn'tthatfun?!? AftertwoweeksoflivinginSatLabthesoftwarewasclosetoareleasecandidate.By theway,don'ttrytoausefixedbaudratefortheCANbootloaderinthe AT89C51CC03.Bydefault,isusesautomaticbaudratedetection,whichdoesn'twork thatwellwhenmorethantwoCANnodesareconnectedtothenetwork,butsetting thebaudratetoafixedvalueonCPU2onourinterfacecardtheCPUstayedforever silent.Thebaudrateissetbysettingthreeinternalregistersviathebootloaderby sendingitcertainCANmessagesdescribedinthedatasheet.ButIguessthedatasheet hadgotthenumberingofthoseregistersbackwardsorsomethingbecausesilence. So,wehadtosolderaCPU2ontoanotherinterfacecardandhookituptotheother board.TheOBCengineeringmodelnowconsistedofthree,nottwo,boards.Furry muff... Anyway,wehadadesignreviewforAAUSATIIthedaybeforemythirdtripto ESTECwiththecompletedflightOBCandweweretopresentourfinaldesignof theOBCforAAUSATIIwhichwehadnotcomearoundtothinkaboutyet.Wejust knewforsurethatwedidn'twantmorethanoneprocessor!TheutilityCPUhad proventobearealmenaceandanARM7withbuiltinCANcontrollerhadreached themarketjustafewmonthsagosoifthatcouldbeacquiredforAAUSATIIsoonit wouldbemuchbetter.Also,weconsideredtouseanARM7withinternalRAMand FLASHinordertocompletelyeliminatetheriskofoutputenablingmorethanone memoryunitatatimecausingcrueldeathofRAMsandFLASHchips.Then,we wouldonlyneedtoplugtheexternalPROMstotheCPUandthatwouldcomprisea completeOBC. OnthemorningofthedesignreviewwehadHolgerEckardt(UHF,SSETIExpress) visitingSatLabtohelpusoutwiththeSSETIExpressKISSTNCmodemengineering modelthatdidn'tseemtoworkwhenweattemptedtointegrateitwiththeonboard computer.Afterawhilewedeclaredthemodemdeadandtheintegrationofthatpiece ofhardwarewasinterrupted.Afterthereviewweburnedthefirstfiveflight PROMsfortheOBCthatIwastobringtoESTECthenextmorning.

ESTEConceagain
Onceagain,IwokeupearlyinthemorninginanaeroplaneonmywaytoA'damvia Copenhagen.Thiscouldonlymeanonething:IwasonmywaytoESTECwithyet anotherflightreadyOBCasusual.IarrivedatESTECandrushedtothecleanroom (mysecondhome).Here,IspentanhourortwosolderingtheACDSflightboardfor
11

Larsandhelpedmommyanddaddyscrubbingdownhoneycombpanelsbeforethey hadtogototheDutchmarinestobesprayedblackfortheeverimportantthermal reasons.Hairychuff... Thenextdaywentbywithalittlebitofsolderingandwirecrimpingandendedwith anOBCinaboxaliveandkicking.Thistime,theOBCwasveryclosetoflight status:Go!TheonlybutwasthattheutilityprocessorsoftwareonthePROMsin thePLCCsocketwascompiledwithoutoutARMRESEToptionwhichmeansthatit didn'tperformpowerupresetofthemainprocessorasitwasintendedto.Notafatal mistake,yetveryannoying,becauseitmeantthatIhadtosolderaresetwiredirectly ontothemainboardinordertobeabletostartuptheOBCandbegintestingit.And soIdid.

TheSelfDestructWire
ThisisthestoryoftheSelfDestructWireoftheSSETIExpressOnBardComputer flightmodel00.Theresetwire,thatIhadsobeautifullyandcarefullysolderedonto themainboardenablingmetostarttheARMCPUup,turnedagainstmeinamoment ofunawareness,Iguess.BeingconnecteddirectlytotheresetpinoftheCPUthiswire wastheperfectpointofattackforanyhighvoltageterroristwantingtopermanently disabletheOBC.Asthewirewasfirmlysecuredandthoroughlyelectricallyisolated fromharmfulpowersources,anactofwhat'shisfacecausedanunfriendlyelectric potentialtofinditswaytothetipofthepinontheendoftheSelfDestructWire causinginstantselfdestructionofsomeinternalcircuitsofthemainCPU.Atfirst,the CPUpretendedtocontinueexecutingcodebutnothingdeterministiccameoutofit. Nowelcomegreetingatpowerup. Halfanhouroffrantichardwaredebuggingwentby.Thissimplycouldnotbetrue! Thecomputerwasworkingperfectlyafewminutesago!Approximatelythirty minutesaftertimeofimpacttheFLASHresignedinapuffofbluesmoke(notquitea puff,butoneshouldtrytokeeptechnicaldocumentationasanimatingaspossible).I rememberedthatfaultjustalittlebittoowellmorechipshadbeenoutputenabledat thesametimemeaningthatnochipsonthedatabus(CPU,RAM,FLASHand PROM)couldbetrustedanymore.Solution:Makeanewflightboard.Lesson learned:Don'tsolderselfdestructwiresontoflightPCBs.

YetAnotherFlightBoard
Somepeoplecallmeluckybastardafterthisincident.Normally,itwouldtakedays orweekstoordernewcomponentsforacomputerlikethisnottomentionhavinga newflightPCBmanufacturedbyElprint.But,fortunately,wewerealsobuildingan OBCforaRussianstudentsatellite,theBaumanetz,andnewimprovedboardsand
12

componentsforthreecompletecomputerswerekickingaroundinSatLabbackhome, andtheGrumpyOldMenhadalreadyscheduledavisittoESTECthenextevening. Fantastic,Ithought,theycouldbringnewstuffandIcouldsolderanewflight boardthefollowingday.AndsoIdid. FortyeighthoursafterselfdestructionabrandnewOBCflightmodelwasgreeting thenotsounhappyAalborgianswiththegoodold +SSETIExpressPlatform:AtmelAT91/AAUOBC(ARM7TDMI) Copyright(C)2000,2001,2002,RedHat,Inc. RAM:0x030000000x03200000,[0x0300bf680x03200000]available RedBoot> whichisthewelcomescreenofthebootloaderforourARM7whichenablessoftware tobedownloadedintoRAMatruntimewithoutburninganewPROM.Withthenew boardbeingasfinishedastheoldone,theonlyremainingtaskwassolderinga thermistorontosomewiresandgluingitontothesurfaceoftheARMprocessor.This meanthavingwiresroutedontopoftheboardfromtheplatedholestotheCPU,and thereisonlyonecorrectwayofdoingthis:TheJasonway. TheJasonwayofmodifyingaboard(usingspacequalifiedteflonisolatedwire,of course)isbyroutingthewiresusingonlyninetydegreeorfortyfivedegreeangles whilekeepingthewiresparallelandstraight.Icantellyouthisisatedioustaskthat coulddrivemostpeoplecrazy(toolateforme).Thewireswereroutedonaboard withoutcomponents(easiertoworkwith)andbentintoshapeheldinplacewith smallpiecesofcaptontapeandthenmovedtotheflightboardwhereitwassoldered andthenglueddowninthreeplaceswithepoxy.Finally,theideawastogluethe thermistoritselfontotheprocessorusingthermallyconductiveglue(thermalbound), buttheThermalboundhadsufferedsomekindofcrueldeathandwasnobetterat gluingthermistorsontoprocessorsthanaduck.(Whichinreturndoesn'tspeakmuch German,likemostItaliansdon'tdo,either.Therefore,theThermalboundmustbe Italian.) Beforeallthisthermistorfun,whichIdidonaquietdayinthecleanroomafterthe GrumpyoldMenhadleftagain,theactualelectricalintegrationofsubsystemshad begunwiththesuccessofthenewOBCflightboard.

Integration Camera
NowthefuncouldbeginasMorten,cameraguyandGrumpyOldMan,waspresent inthecleanroomwithhis,atthattime,fullyfunctioningcameraforthesatellite.We
13

managedtointegratetheOBCandcamerawithoutmajorproblemsandapicturewas takenviaatelecommand,storedintheonboardcomputerFLASHmemoryand downloadedontoalaptopwithoutlossofdata.Andthenthecamerasufferedacruel death(readmoreaboutthatincidentintheReportoftheGrumpyOldMen).

AttitudeControlandDeterminationSystem
WiththecameraoutofthewayIhadtimetoengageinintegrationactivitieswiththe magnetometertheprimarysensoroftheACDS.Naturally,thatdidn'tresultina holeinonebutafterhavingmodifiedtheACDSsoftwareontheOBCtocomplywith themagnetometeritwasplayingthetunethatacertainGrumpyOldManwas expecting.Done.TimeforrewiringthepinsoftheDsub9connectorsforACDSas thisratherconfusinginterfacehadcontinuedtoconfusemeevenafterIhadmadea sketchofallthetranslations:ADC_sun_12>Ch0>ADC3>Pin8>Sun1D2, andsoonforallthedifferentinputsandoutputsconnectedtoACDS.

ElectricPowerSystem
TheOBChastwointerfacestotheEPS:Poweranddata.ThepowerinputtotheOBC hasnotyetbeenfedbyEPSasitisnotyetcapableoffeedingpower.Thedata interfacetothepowerdistributionunithasbeensuccessfullytestedwiththeEPS flightPDUandalltelecommandshavebeentriedindifferentsequences.TheOBC receivedtelemetryfromthePDUandprocessedthekeepalivepingseverytwenty seconds.Forsomestrangereason,afewpingswerereceivedwrongontheOBCand thereforenotprocessed.ThisresultedinkeepalivetimeoutinthePDUwhichin returnissuedashutdowncommandtotheOBCthatgracefullyshoutdownallits threadandwaspowercycledbyEPS(thepowercyclewassimulatedwithlight emittingdiodes).Thesourceofthismissingpingflukeisyettobedetermined.

TheMAGICbox
TheMAGICboxthatcontrolsthepropulsionsystemistheonlysubsystemusinga CANinterfacetocommunicatewithOBC.ThebaudrateoftheCANbusis1Mbps offeringgreatchancesfornodesonthebustooverruntheutilityprocessorwithlarge burstsofdata.Thisfactwasthesourceofsomeworriedthoughtsbeforetestingwith theMAGICboxasitiscapableoftransmittingratherlargeamountsofinternal telemetryonthebus. AfterunifyingtheOBCCANprotocolandtheMAGICCANprotocoltoCAN2.Aby uploadingnewsoftwaretoMAGICthetwosystemswerehookedupandthestartup message,anITMpacketonmeasurementID0x7D,wasreceivedbyOBCandstored inthetelemetryqueue.AlltelecommandstoMAGICboxweretestedwithdifferent parametersandafteronlyafewhoursoftestingtheOBC<>MAGICinterfacecould
14

bedeclaredintegrated.NoteventhehugeamountsoftelemetryfromMAGIC showedanysignsofproblems.

UHFcommunicationsystem
NexttoEPS,UHFmustbeconsideredthemostvitalsubsystemonthesatellite.Being constructedbyaGermanradioamateurratherthanastudent,thissubsystemwas consideredtobeamongthemostreliablesystemsonExpressdespiteitscomplexity. TheinterfacebetweentheOBCandUHFisanRS232connectionutilisingtheKISS TNCprotocol(Ihope).Becauseofthespecialgroundingschemeoftheradio transceiverallinterfacestoUHFbuttheantennamustbegalvanicallyseparatedfrom theradio. InMay,itwasdecidedthatthegalvanicseparationontheRS232lineshouldbe implementedwithtwooptocouplersontheUHFmodemandthattheUHFRxline wouldusestandard5VnoninvertedlogicwhileTxwouldbeanopencollector outputrequiringapullupresistorontheOBC.Later,thisdecisionwasoverruledby thedecisionthatallRS232connectionshouldusestandardRS232levelsenablinga standardPCtobeusedfortestingtheinterfaces.Thisdecisionwasnotcommunicated insuchawaythatHolger,responsiblefortheUHFsystem,wasawareofitwhich resultedinhimfollowingtheoldinterfacerenderingcommunicationbetweenOBC andUHFimpossiblewithoutheavymodificationstooneofthesystems.Thisproblem revealeditselfwhentheUHFengineeringmodelarrivedatESTECtobetestedwith OBCanditwasdecidedtochangetheinterfaceonUHFtostandardRS232levelsfor theflightmodelmeaningthatnewPCBsweretobemanufactured. HavingmodifiedtheengineeringmodeltoconformwithstandardRS232Icouldtest themodemwhichwassupposedtousetheKISSTNCprotocolat19k2baud.I hookedtheOBCandmodemtogetherandsendaKISSTNCencodedAX.25frameto themodemwhilemeasuringtheoutputfromthemodemtotheradio.Themodemis supposedtosenda2VFSKmodulatedaudiosignaltotheradiouponreceptionofa validKISSdataframebutnothinghappenedontheoutput.Theproblemcouldbe withOBCsoItookitoutoftheloopandusedalaptoptomanuallysendKISS packagestothemodem.Itriedtomostsimpledataframeconsistingof <FEND>0x00DATA<FEND> WhereFENDisaframedelimiterequalto0xC0.The0x00inthefirstbyteofthe payloadoftheKISSframeindicatesthattheframesisadataframe,notacommand frame.Thedatawasonebyte,0x41whichissimplyacapital'A'.Itriedtosendthis packageatthecorrectbaudratewithoutseeinganyreactiononthemodemaudio output.Itriedthesamepackageatallbaudratesavailableonthelaptop.Nothing.Had thismodemsufferedacrueldeath,somehow?Ordoesituseadifferentprotocolthan
15

KISSTNC?SMACK,maybe?OrFlexNet? AftertalkingtoHerrGttner,themanwhodesignedthemodem,Igotholdofthe documentationforthemodemandreadthroughit(allinGerman,ofcourse). Accordingtothatdocument,themodemshouldbeusingKISSTNCbydefaultand runat38k4baudontheRS232line,soItesteditwith38k4onceagain.Nothing. Timetotossthetowelandleavethefateofthemodeminthehandsofsomeone capableoftalkingtoit. Noproblem,IjustwaitforacommunicationsexperttoshowupatESTECtosortout theproblemsandgettheUHFintegratedwithOBCbeforeIgobackhome.This meantthesecondpostponeofmyreturnflight.

Part II
Lessons Learned Lesson1:Designandimplementationtakestime
FromthebeginningoftheOBCprojectweregardeditasrathertrivialtodotheactual hardwaredesignandimplementationofacomputer.Wehadalltriedthatbefore.But whentimecametodothedesignwefoundourselvescoveredinproblemsthathadto solvedwithquickdecisionswhichseeninretrospectwasn'talwaystheoptimal decisions.

Lesson2:Centralisedarchitecturebeatsdistribution
Theprimarybottleneckconstantlycausingusproblemsduringthesoftwaredesign wasthenumberofprocessorshavingsocommunicatetogether.Thecommunication betweentheprocessorswasmeanttobeverysimplyandfailsafebutfailsafeand simplearenotnecessarilycompatiblerequirementsinadistributedsystem. Therefore,itslowlybecamequitecomplicatedtoprogrammethesoftwaretoensure robustcommunicationusingthesimpleprotocolswehadspecified. HadthefiveRS232channelsbeenimplementedviaUARTsconnectorsdirectlytothe mainprocessorandmemorymappedtoitsaddressspacetheincomingdatacouldbe expeditedmuchfasterwithlesschanceofpackageloss.Thisleadstolessonthree.

Lesson3:Flowcontrolondatalinesisnecessary
Loosingpackagesondatachannelsisnotacceptableforanonboardcomputer.But withoutflowcontrolonthedatalinesgoingfrommultiplesubsystemsintoa computerviaacommonserialcommunicationschannel,inthiscasetheinternalCAN
16

busoftheOBC,itcannotbeguaranteedthateverybytesenttothecomputeractually getstherebeforeitisoverrunbythenextbyte. Theutilityprocessorisaclassicexampleofabottleneckinanetworkbecauseit theoreticallycannotprocessesallincomingdatafrombothsides(CANandARM7)if thedataflowofthesecommunicationchannelsreachacertainlimitbelowthe physicallimits.Thislimitcanbereachedjustbylettingallsubsystemstransmitatthe sametimeforawhileuntilthelocalbuffersintheCPUsontheinterfacecardandin theutilityprocessorarefull. Atthispointthebestthingtodoistostopthedataflowfromtheperipheraldata sourcewhichcanbedonebymeansofhardwareflowcontrolonRS232usingDTR (DataTerminalReady).ThisisimplementedfortheRussianBaumanetzsatellitethat featuresamodifiedversionoftheSExOBC.

Lesson4:ListentoJason
Thislessondidnotcomefrombitterexperienceratherthanfromthejoyofseeingthe qualityofthehardwarewhenithasbeenimplementedtheJasonway.Itdoesn'ttake longtolearntheJasonwayifyouhaveanopenmindandacceptthatflighthardware hastobeperfect.

Lesson5:Followlesson4 Lesson6:Makeascheduleandmultiplyby
Thislessonisverydifficulttolearnformostpeople(includingme).Stuffsimply takesmoretimethanyouwouldexpect.Whengettinganengineeringassignmentlike developingasystemforasatellite,makeaschedule,useworstcasedurationforeach taskintheschedule,addabithereandthere,thenmultiplythewholelotby3.14and thatistheamountoftimeyouwillactuallyspendontheassignment.Ontothenext Lesson.

Lesson7:BuildingsatellitesmakesyouaGrumpyOldMan
IfyouarenotyetaGrumpyOldManthenitmeansthatyoudidnotbuildasatellite. Whenventuringintothechallengingworldofspaceengineeringyouprobablywantto buildacompletelyautonomousintelligentspacecraftwithmorefancyfeaturesthanan electronictoaster.Butafterawhileyoudiscoverthatoftenitisnotnecessaryfora toastertobeintelligentnorautonomous.Infact,youmightevenfinditchallenging enoughtobuildtheheatingelementofthetoastersuchthatitcomplieswiththeJason way. Havinghadthisexperienceyoufindyourselfstrugglingtoconvincetheless
17

experiencedonesthataspacecraftiscomplicatedenoughinitselfyoudon'tneedto addextracomplexityjustforthefunofit.Ittakesjustaboutfourmonthstobecomea GOM(GrumpyOldMan).

Lesson8:Don'timplementselfdestructwire
IfyouthinkitisaverycleverideatosolderaresetwireontoyouflightPCBwith connectiondirectlytoyoumainprocessorthendon't.It'snotacleveridea!

Lesson9:FLASHisbad.Youshouldn'tdoFLASH.Or?
AccordingtoDanishspacehardwareexpertsattheDanishSpaceResearchInstituteit isforbiddentorelyonFLASHmemoryforstoringprogramcodethatasystemona satellitedependson.Thatis,FLASHcanbeusedtostoreonboardsoftwareaslongas thereisabootloaderresidinginontimeprogrammablenonvolatilememory, OTPROM,whichloadstheprogramcodefromFLASHandcorrectsanybitflipsthat haveoccurredinthecodebymeansoferrorcorrectingcodelikee.g.Hammingcode orotherblockcodeorcycliccodes. TheOBConSSETIExpressusedonlyOTPROMforallsoftwareonallprocessorsin itgivingnooptionforchangesintheflightsoftware.Thischoicewasmadeinorder toguaranteethatthesoftwarerunningonthespacesegmentis100%equaltoacopy runningontheEMonground. ButnotallexpertsagreethatFLASHisbad.Someoneoughttoperformaquantitative testontheprobabilityofbitflipsinFLASHmemoryperhapsaSSETIteam?

Lesson10:Don'tbuyreturnplanetickets
IfyougotoESTECtointegrateasubsystemwithlotsofnonexistentsubsystems, don'tbuyareturnticketyouwillnotbeonthereturnflight,anyway.

Lesson11:Exchangeprotocoldefinitions
Thislessonisquiteimportant.Whenyouagreeonacertaininterfacelikee.g.adata interfacebetweentwosystemsmakesurethatbothpartieshavethesamedefinitionof theinterfaceatalllevels:Physical,electricalandprotocolwise.Allofthesethree levelsmaybedifficulttochangeinthelastminutewhenintegratingsystems,andone mighthaveanotherdefinitionofaprotocolthantheother.Therefore,makesurethat allpartiesusethesameprotocolreferencedocuments.Ifonepersondecidesona particularprotocolthenthatpersonshouldmakeacompleteprotocoldescription availabletoanybodysharingthatparticularinterface.

18

Lesson12:Ifyouareapoorstudent,don'tbuildsatellites
Speaksforitself.Buildingsatellitesmeanstravelling.Travelreimbursementsdonot includelocalbuses,foodandwinewhichmeansthatyouwillspendapproximately 150Euroaweekinsteadof50Euro.Asastudentyoudon'thavemoneyforthat.

19

Alphabetical Index
89C51 3pp. AAUSATII 1p. Can 7 Jason 1,7 Microwavesoldering 7 Modifying Nothing 6p. 7 11 5p.,11,14

CAN 2pp.,6 CanTerm Clevertools Cow 4,8 7,10,12 4 1,3p.,6p. 3

OBC 1pp.,6p.,10 Parabolicflight Pinextractiontool PLCCsocket 4 PROM Prototype 3pp.,7 3 3 7

Crueldeath Elprint ESTEC

Externalcodememory GrumpyOldMen Interfaces IPA 8 3

RS232 2pp.,6p. Soldering 4,7

1,8,11p.

Solderingguidelines 8 TheJasonWay 11,15

20