3DReconstructionUsingTango

QuentinGautier,StevenLee,SungwookSon,JinYu
June9,2016

Abstract

Reconstructingmodelsinthreedimensionschangethewayhumansvisualize.Archaeologistsuse
visualizationtogetfurtherknowledgeontheirfindings.Manyarcheologiststodaystillusetheoldmethodof
pencilandpaper.ThisprojectallowsarchaeologiststouseGoogleTangotablettoreconstructtheirfindings
whileprovidingrealtimemappingandfeedback.TheGoogleTangoisaportabletabletusedinalltypesof
AugmentedRealityandVirtualRealityapplications.Ourimplementationof OnBoardSLAMgeneratesa
3DreconstructionthatefficientlyusesbothCPUandGPUtomapwhiledoingcalculation.After
realfieldtestinginGuatemala,wefixeddriftandlossissuesfoundonpreviousimplementation.

1. Introduction

Thegoalofthisprojectistoimplementtechnologicaladvancementsincurrentfieldofartand
archeologyforfurthervisualization.Thisvisualizationnotonlyallowsartistsandarcheologiststodetailtheir
findings,butalsoallowsthemtobuildahighlyscaledmodelwithmuchdeeperdepthanddetail.Eventoday,
manyoftheartistsandarcheologistsarelimitedfrommanyofthesemodernresources.Theseartistsand
archeologistsuseanoutdatedmethodofdrawinga3Dmodelonpaperwithpencil.Thisuseoflong
establishedmethod,whencomparedtotheuseofmoderntechnology,fallsbehindinmanyqualitiesasthe
sizeofmappinggetslarger,visualizationthroughtheuseoftraditionalmethodgetsobscure.Thus,with
biggerfindings,this3Dreconstructionprovidesbetteraccuracyandefficiency.Thisprojectaimstoprovide
artistsandarcheologistswithamobilesystemthatcanreconstructanundergroundenvironment,with
realtimepartialorfullfeedback.

The3DreconstructionGoogleTangoteamiscomposedofthreemembersandonesupervisor.In
ourteamwehaveStevenLeeasSLAMspecialist,JinWoongYuTangospecialist,andSungwookSonas
Tango/SLAMjackofalltrades.Ourteamleader,QuentinGautier,isacurrentmemberofEngineersfor
ExplorationorganizationatUniversityofCalifornia,SanDiego.Thisorganizationisinvolvedinmany
engineeringsolutionsthatgoesbeyondcurrenttechnology.Incollaborationwiththisorganization,weputan
efforttodevelop3Dreconstructioncapabilityonaportabledevice,whichwillbeusedtoreconstructthe
cavesofGuatemala.

ThemainpieceofhardwareistheGoogleTangohandhelddevicethatcomeswithanabundanceof
sensorsincludingbutnotlimitedto3Ddepthsensors,gyroscope,accelerometer,andmotiontracking
camera.IthasaNVIDIAmobileprocessorthatdoublesasaGPU.ProjectTangocanbeeasilyobtained
withapriceof$500.Itssuperbprocessingabilityandsensorsallowsustoreconstructtheareain3D.Tango
isverylightandportablethatitcouldbeusedanywhere.ForthisprojectwehaveusedprojectTangoto
mapthecavesinGuatemala,butthepossibilitiesareinfinite.Therearealreadytwodevicesinour
possession.

 WetriedtwodifferentimplementationsofSimultaneousLocalizationandMapping(SLAM).Thereis
notoneSLAMimplementationtofitallrequirements,andmoreover,wewantittomatchtheGoogleTango
capabilities.Throughvariousresearch,wecamedowntoRTABandOnBoardslam.Theadvantageof
implementingSLAMalgorithmsfor3Dreconstructionoveranyothermethodisthatasthemappingbegins,
highlyefficientSLAMalgorithmalsocalculatesthepositionrelativetotheworldandobjectsaroundthem.
Moreover,implementingSLAMalgorithmsonGoogleTangotabletattemptedtofixdriftandlossissues,
foundonpreviousKinfuapplication.

However,wewereonlyabletotogettheOnBoardslamworkingonGoogleTango.WhenQuentin
tookthisimplementationtoGuatemalaforrealfieldtesting,thisalgorithmwasonlyabletoholdsuch
amount.Whenmappingbiggerfindings,thisapplicationimplementedwithOnBoardslamalgorithm,
crashedaftermappingatmostfewminutes.Despitethis,whenwedidourtesting,wewereabletowalk
aroundthesecondflooroftheCSEbuildingwithoutmuchproblem.Oneproblemweencounteredwas
processingissue.Asthescalegetslarger,becauseGoogleTangohaslimitedpower,thealgorithmslows
downovertime.Moreover,whentryingtoextractthe.objfileaftersuccessfullymapping,thisprocesstooka
while.

OneissueweencounteredwhilemappingwithGoogleTangotabletwaslimitedbatterylife.
Becausethisapplicationdoessomuchcomputation,itdrainsthebatteryreallyfast.AstheSLAMalgorithm
furtherdevelops,thebatterymaybeanimpedimentforalargermapping.


2TechnicalMaterial

OurApproach

TheGoogleTangoisanAndroiddevicewithAPIsavailableinJava,C,andUnity.TheTango
devicehasoutoftheboxmotiontrackinganddepthsensingcapabilitiesduetoitsabundanceofsensors.
Someofthekeysensorsincludethemotionsensorcameraand3Ddepthsensor.ItusesaNVIDIATegra
processorwithCUDAsupportforgraphicscomputingapplications.
Theinitialgoaloftheprojectwastoimproveuponanexistingapplicationthathad3Dmapping
capabilities.ByscanninganareawiththeTangoscameraanddepthsensors,theTangocouldsavepoint
cloudinformationtobelaterexaminedasamodelfile.Theapplicationcouldthenbeusedby
ArchaeologiststomaptheirfindingsmorespecificallyGuatemalancavestheEngineersforExploration
organizationatUCSDwouldbeexploring.TheTangowouldhavetopossiblymaplongandwindingtunnels
withrockysurfaces.Theexistingapplicationdidnothavedriftcorrectionsooverlongperiodsoftime,the
motiontrackingwouldloseitspositionintheworld.Any3Dmappedmodelswouldnotbeaccuratedueto
thisdrift.Thesolutionwastointegrateasimultaneouslocalizationandmappingalgorithm(SLAM)that
wouldcorrectforthisdrift.
Theexistingapplicationrequiredagreatdealofhardwarelevelprogrammingduetoitsgraphics
processingdemands.TheAndroidOSwhichusestheJavalanguagehassupportforhardwarelevel
tinkeringthroughtheJavaNativeInterface(JNI)usingtheClanguage.Applicationscreatedwithsuch
supportarecalledJavaNativeApplications(JNA).Thisaddedagreatmanynewdifficultiestoourteam
whichwasunfamiliarwithJNAs.GoogleprovidesC,Java,andUnityAPIs.Weaddressedourweaknessby
developingintheUnityenvironment.
 Unityisaverypopular3Dgamedevelopmentprogram.Duetoitseaseofuse,ithasfoundaplace
amongtripleAcompaniesandindependentdevelopersalike.SomenotableproductscreatedwithUnity
includeBlizzardsHearthstoneandtheKerbalSpaceProgram.Unitysupportsmanydifferenttypesof
platformsaswellincludingourtargetofAndroid.Thegamecreationprocessexploitstheobjectorientated
natureofvideogames.Gameobjectshaveavarietyofdifferentcomponentsattachedtothemtodefine
theirbehaviors.ComponentscanrangefromSpriteRenderers,3DModels,Transforms,Rigidbodies,
Scripts,etc.ScriptsforUnitycanbewritteninC#orJavascript.WedecidedtogowiththeUnityfavoriteof
C#.ScriptsallowevenmorecomplexbehaviorstobeprogrammedintotheGameObjects.

3DMapping

The3Dmappingofourapplicationinvolvestwoseparatefeatures.

1. TheabilitytorecordandlaterexportdatainvolvingtheTangoandtherecordedenvironment
2. Theabilitytogiverealtimefeedbacktotheuseronwhathasandhasnotbeenrecorded

Thedatatorecordincludesposedata,cameraimages,depthdata,andmodeldata.Theposedata
istheorientationoftheTangodeviceinspaceatagivenmomentintime.Thecameraimagesarethe
imagesseenbythergbcamera.Thedepthdataisthesensordatafromthedepthcamerawhichistakena
rateof5Hz(muchslowerthantheactualcameradata).Themodeldataistheactualenvironment
representedasamesh.TheTangocomeswithconvenientAPIfunctionsthatarecallbackstoreceivepose
data,cameraimages,anddepthdata.Themodeldatacannotbecapturedassimplyandiscreatedthrough
ameshrenderer.
Themeshrendererisentirelycontainedinameshrenderingscript.Themeshisrepresentedasa
multitudeofinengineGameObjectswithshaderandtransformcomponents.Theshadergivesthemesh
thecoloroftheobjectbeingrepresented.Thetransformdeterminesthepositionandorientationofthemesh
intheapplicationworld.Thetransformisextremelyimportantasanaccuratetransformdeterminesthe
accuracyofanexportedmodel.
Themeshrenderingprocessinggoesthroughquiteafewsteps.Themeshrenderersubscribesto
theTangosdepthsensorcallback.TheTangofeedsthemeshrendererdepthdataallowingtherendererto
findflatsurfaces.Eachsurfaceisthentransformedintoagridinwhichasquareonthatgridcanthenbe
meshed.Dependingontheresolutiondeterminedbeforehand(wefound5cmstobeefficient),thegridwill
havedifferentlysizedsquares.Apointcloudisgeneratedforthepurposeofcoloringeachmeshtobetter
representwhatisbeingmapped.
Toimprovetheuserexperience,thecomputationallyheavymeshrenderingisdoneonaseparate
thread.Whenindicesareseenthatrequiremeshing,theyarepushedintoaqueuedatastructuretoawait
theirturn.Thisbacklogisprocessedforasmallportionofthecomputationcycleeverycycle.Ofcoursethis
meansthatmappingabrandnewareacreatesasurgeinmeshesthatneedtobeprocessed.Bykeeping
theTangofocusedonapreviouslyrenderedareaorbypausingtherenderingprocessthroughtheGUI,the
backlogcanbeprocessedmuchmorequickly.

SLAM

ManyoftheSLAMalgorithmsweresearchedincludingORBSLAM,RGBDSLAM,andLSDSLAM
cannotbecompiledthroughtheUnitysbuildprocess.ThismeantthatwehadtorelyonTangosonboard
SLAM.TheonboardSLAMnamedAreaLearningbyGoogle,functionsverysimilarlytotheseotherSLAM
algorithms.TheTangoisconstantlyviewingandrememberingspecificlandmarksinitsenvironment.As
theusertakestheTangoawayfromthestartinglocationandback,theTangowillrememberlandmarksfrom
thestartinglocation.Itthenrealizesthatithasformedaloopandtheloopcanbeclosedthrough
recalculatingpositionaldata.TheTangostorestheposedatafromwhenitbeganrecordingandwhena
loopisdetecteditcomparesthecurrentposedatatothestartingposedata.Bothposesshouldtechnically
bethesameastheTangohassimplyreturnedtothepositionithasstartedinbutmistakesinposition
accumulateovertimecreatingdrift.ThedifferencebetweentheposetheTangostartedwithandthepose
theTangoendedupwithcanbeusedtorecalculatealltheposesalongtheTangosroute.

Weaknesses

ThecurrentimplementationforArchaeologicalrecordingshassomeweaknessesthatneedtobe
addressed.TheapplicationisnotbuiltasaJNAthusitisnotusing100%ofitshardwarecapabilities.This
canbeseeninthedropsinframerateandbacklogprocessingrateoftheapplication.
TheapplicationusestheonboardSLAMalgorithmwhichcannotbeexaminedortunedby
developers.
TheapplicationisbuiltinUnitywhichislimitedinitscompilingandbuildingprocedures.

Strengths

TheUnityengineissimpletolearnandpowerfultouse.ByusingtheUnityAPIs,wedidnothaveto
spendmanpowerandresourceslearningJNA.
TheTangoislightandportableweighinginat0.82lb(370g).
Theapplicationmapsquicklyandaccuratelyforitsgivenhardware.

Thissubsectiondescribeshowtorunthe3Dreconstructionsystem.Thefollowingwilldemonstrate
requirementsandinstructionstosetupandbuildOnBoardSLAMonGoogleTangotablet.

2.1.1Requirements
Components:
GooglesProjectTango
Laptop

2.1.2SetUpInstructions

Environmentusedforthisbuild

Windows10/MacOSX
AndroidStudio2.1
Unityversion3.4.1.f
GoogleUSBDriver

Setup
1. InstallWindows10.Windows10isaverycommonoperatingsystemamongPCs.UsuallyWindows
10comeswiththecomputeritself.However,ifWindows10isnotavailable,registeranddownload.
Download: https://e5.onthehub.com/WebStore/ProductsByMajorVersionList.aspx?ws=94fd1fe3eb9b
e011969d0030487d8897&vsro=8
 2. InstallAndroidStudio2.1.AndroidStudioisaIDE(IntegratedDevelopmentEnvironment)
forandroid.
Download: https://developer.android.com/studio/index.html
3. InstallUnity( .Unityisatoolfor3Dgameengine.
3.4.1.f)viaUnity'sDownloadAssistant
GoogleTangohasUnitysupportforAR(AugmentedReality)andVR(VirtualReality).
MakesuretoselectAndroidBuildSupportandMicrosoftVisualStudioToolsforUnity
isrecommended.
Download: http://unity3d.com/getunity/download
4. InstallTangoUnityAPI.TangoUnityAPIallowsUnitytouseAPIsspecificallyforTango.
Download: https://developers.google.com/tango/downloads#tango_sdk_files
5. InstallGoogleUSBDriver.GoogleUSBDriverisadrivertolettheWindowscomputerto
recognizeandroiddevicessuchasTango.
Download: https://developer.android.com/studio/run/winusb.html

2.1.3BuildingApp
1. AddtheAndroidSDKpathtoUnitybyselectingUnity>Preferences>ExternalTools
2. ImporttheTangoSDKintoUnity,selectAssets>ImportPackage>CustomPackage,then
selectTangoSDK
3. Clone https://github.com/ssl024/TangoMapper.git
4. Buildapk

2.1.4RunningApp
1. TransfertheapktoTango
2. UsingFileViewerappinstalltheapk
3. HoldTangostillwhenopeningbecauseofthecalibration.
4. Typeinappropriatevaluesforthefield
5. Theappwillstartmappingthearea
6. Ifdesiredtoexport,pressexportandviewfilefrompathwrittenonstep4


2.2SystemUsage

3Milestones

Initialmilestone

Week Objective/Milestones Status Artifact/Comments

3 CompleteProjectSpecifications Comp Specifications

SetupTangoworkstation Comp SetupInstruction

4 CompleteProjectwebpage. Comp BitBucketWiki

Comp VideoDemoHelloWorld
 RunsimpleTangoprogramrunningsuch
asHelloWorld

5 OralUpdate. Comp PresentationSlides

UnderstandTangofunctionality. Comp VideoDemopointclouddemo

6 SLAMpreimplementation. Comp VideoDemomeshrendering

7 MilestoneReport. Comp This

SLAMimplementationcomplete. Comp VideoDemoSLAMimplementation

8 Tangotestinginacontrolledenvironment Comp

9 Tangotestinginrealworld Comp
Comp

10 Catchslips. Comp
Finalpresentation Comp
FinalVideo Comp

Bytheendofinitialmilestone(week5),
CompleteProjectSpecificationsandsetupTangoworkstations.
TosetupTangoworkstationsforuseanddownloadNDKandSDK,
https://developers.google.com/tango/apis/c/#install_the_ndk
CompleteProjectwebpageandasimpleTangotasksuchasHelloWorldprogram
AftersuccessfullysettingupTangoworkstation,wetestedmultiplesimpleprograms
 UnderstandTangofunctionality.SLAMresearch
AfterlookingatmultipleSLAMalgorithmson
https://openslam.org/
wecamedownto4
differentalgorithms:OnBoard,RGBD,RTAB,ORB

Byendofsecondmilestone(week7),
SLAMpreimplementation
WhilefinalizingwhichSLAMimplementationbestfitsourpurpose,wegotmeshrendering
implementedonTangousingUnity.
SLAMimplementationcomplete
Wecouldntfinishthisuntilbeginningofweek8.

Byendoffinalmilestone(week10),
ControlledTangotestingwithSLAM
AfterimplementingOnBoardSLAM,wemappedasmallareasecondfloorofCSEbuilding

TangotestinginrealworldSLAM

Notes:
NewObjectives:
ControlledTangotestingwithSLAM.
UsingORBSLAMwithUnityAPIformeshrendering.

Issuesencountered:

Atfirst,weworkedontwodifferentimplementationofSLAMalgorithm:RTABandOnBoard.Onourfirst
developmentofRTABSLAMalgorithm,wewereablecrosscompilealldependenciesandsuccessfullygot
theAPKfiletoputintheGoogleTangotablet.However,afterupdatingsomefeaturesandpullingfromthe
repository,crosscompilingfailedandfailedtoreproducetheAPK.

Also,downloadingthenewestversionofallsoftwaresandsourcesarenotthebestidea.While
implementingOnBoardSLAM,wehadlotsofdependencyissue.Stayingwithpriorversionmakeslife
easier.

4Conclusion

The3DReconstructionprojectusingGoogleTangotabletshowshowmodernartistsandarcheologistscan
usethisdeviceandapplicationtoeffectivelymaptheirfindings.Comparedtotheirpreviousmapping,which
reliedonhanddrawnstructures,thisprojectwilldefinitelybeanimpetusformanyartistsandarcheologists
tomoveonforward.Althoughthisprojecthaslimitationstomappingbigfindings,itcanstillmapmuchmore
efficientlyandshowshighdetailsduetouseofdepthsensor.Moreover,withtraditionalmethodofdrawing,
therearelimitationstohowmuchmappingcanbedone.Useofthis3Dreconstructionapplicationallowsthe
usertoextracttheirmappingandsaveitasan.objfilewhichthencanbeviewedoncomputerthrough3D
viewer,andcanevenbeprinted.Becausewedonothavestatisticstothisproject,wecannotsayhow
effectivetheuseofthisprojectwillbecomparedtootheralgorithmsorpreviouslyusedmethodstomapthe
findings.

Improvementscandefinitelybemade.Therearemanydifferentsimultaneouslocalizationapplication
mappingalgorithmsanditishardtoendupwithoneperfectalgorithm.BecauseouruseofOnBoardSLAM
algorithm,therearelimitations,aswecannottweakitasmuchforvariability.Onewaytoachievebetter
solutionwouldbetoresearchmoredifferentSLAMalgorithmandfindoutwhichalgorithmisgoodforlong
distanceandshortdistance.Becausearcheologistsliketomaplongdistancecaves,OnBoardSLAMwas
probablynotthebestidea.OurpriorimplementationofRealTimeAppearanceBasedMappingshowedhigh
detailsbutthisalgorithmisprobablynotthebestideaforlongdistancemapping,aswefoundoutthatthe
efficiencyofthisalgorithmdegradedovertime.

Comparedtoourotherteam,TeamKinect,theuseofGoogleTangowasmuchportable.However,dueto
itssmallsize,therewerecomputationallimits.ForTeamKinect,theyusedMicrosoftKinectwhichhasa
cameraandmultiplesensors.Moreover,thecomputationisdoneonalaptopusingitscentralprocessing
unit(CPU)andgraphicsprocessingunit(GPU).Fortesting,QuentinGautierusedalaptopwithi7processor
highendgraphicsprocessingunit,enablingsmoothrenderingandproducingrichqualityreconstruction.
AlthoughKinectteamproducedbetterresultsduringthetestatGuatemala,itsportabilitywasnotsogreat.
For3DreconstructiondonewithGoogleTangotablet,allcomputationandrenderingwasdoneonthetablet
while3DreconstructiondonewithKinecthadtocarryaroundpowersourceforkinectandalaptopfor
rendering.Overall,althoughnotperfect,bothteamproducedafullworking3Dreconstructionapplication.
Thus,webelieveourprojectwassuccessful.

References
https://openslam.org/
http://wiki.ros.org/rtabmap
https://developers.google.com/tango/hardware/tablet
https://unity3d.com/unity/editor
https://developers.google.com/tango/apis/c/#install_the_ndk