3DReconstructionUsingTango

QuentinGautier,StevenLee,SungwookSon,JinYu
June9,2016

Abstract

Reconstructingmodelsinthreedimensionschangethewayhumansvisualize.Archaeologistsuse
visualizationtogetfurtherknowledgeontheirfindings.Manyarcheologiststodaystillusetheoldmethodof
pencilandpaper.ThisprojectallowsarchaeologiststouseGoogleTangotablettoreconstructtheirfindings
whileprovidingrealtimemappingandfeedback.TheGoogleTangoisaportabletabletusedinalltypesof
AugmentedRealityandVirtualRealityapplications.Ourimplementationof OnBoardSLAMgeneratesa

3DreconstructionthatefficientlyusesbothCPUandGPUtomapwhiledoingcalculation.After
realfieldtestinginGuatemala,wefixeddriftandlossissuesfoundonpreviousimplementation.

1. Introduction

Thegoalofthisprojectistoimplementtechnologicaladvancementsincurrentfieldofartand
archeologyforfurthervisualization.Thisvisualizationnotonlyallowsartistsandarcheologiststodetailtheir
findings,butalsoallowsthemtobuildahighlyscaledmodelwithmuchdeeperdepthanddetail.Eventoday,
manyoftheartistsandarcheologistsarelimitedfrommanyofthesemodernresources.Theseartistsand
archeologistsuseanoutdatedmethodofdrawinga3Dmodelonpaperwithpencil.Thisuseoflong
establishedmethod,whencomparedtotheuseofmoderntechnology,fallsbehindinmanyqualitiesasthe
sizeofmappinggetslarger,visualizationthroughtheuseoftraditionalmethodgetsobscure.Thus,with
biggerfindings,this3Dreconstructionprovidesbetteraccuracyandefficiency.Thisprojectaimstoprovide
artistsandarcheologistswithamobilesystemthatcanreconstructanundergroundenvironment,with
realtimepartialorfullfeedback.

The3DreconstructionGoogleTangoteamiscomposedofthreemembersandonesupervisor.In
ourteamwehaveStevenLeeasSLAMspecialist,JinWoongYuTangospecialist,andSungwookSonas
Tango/SLAMjackofalltrades.Ourteamleader,QuentinGautier,isacurrentmemberofEngineersfor
ExplorationorganizationatUniversityofCalifornia,SanDiego.Thisorganizationisinvolvedinmany
engineeringsolutionsthatgoesbeyondcurrenttechnology.Incollaborationwiththisorganization,weputan
efforttodevelop3Dreconstructioncapabilityonaportabledevice,whichwillbeusedtoreconstructthe
cavesofGuatemala.

ThemainpieceofhardwareistheGoogleTangohandhelddevicethatcomeswithanabundanceof
sensorsincludingbutnotlimitedto3Ddepthsensors,gyroscope,accelerometer,andmotiontracking
camera.IthasaNVIDIAmobileprocessorthatdoublesasaGPU.ProjectTangocanbeeasilyobtained
withapriceof$500.Itssuperbprocessingabilityandsensorsallowsustoreconstructtheareain3D.Tango
isverylightandportablethatitcouldbeusedanywhere.ForthisprojectwehaveusedprojectTangoto
mapthecavesinGuatemala,butthepossibilitiesareinfinite.Therearealreadytwodevicesinour
possession.

WetriedtwodifferentimplementationsofSimultaneousLocalizationandMapping(SLAM).Thereis
notoneSLAMimplementationtofitallrequirements,andmoreover,wewantittomatchtheGoogleTango
capabilities.Throughvariousresearch,wecamedowntoRTABandOnBoardslam.Theadvantageof
implementingSLAMalgorithmsfor3Dreconstructionoveranyothermethodisthatasthemappingbegins,
highlyefficientSLAMalgorithmalsocalculatesthepositionrelativetotheworldandobjectsaroundthem.
Moreover,implementingSLAMalgorithmsonGoogleTangotabletattemptedtofixdriftandlossissues,
foundonpreviousKinfuapplication.

However,wewereonlyabletotogettheOnBoardslamworkingonGoogleTango.WhenQuentin
tookthisimplementationtoGuatemalaforrealfieldtesting,thisalgorithmwasonlyabletoholdsuch
amount.Whenmappingbiggerfindings,thisapplicationimplementedwithOnBoardslamalgorithm,
crashedaftermappingatmostfewminutes.Despitethis,whenwedidourtesting,wewereabletowalk
aroundthesecondflooroftheCSEbuildingwithoutmuchproblem.Oneproblemweencounteredwas
processingissue.Asthescalegetslarger,becauseGoogleTangohaslimitedpower,thealgorithmslows
downovertime.Moreover,whentryingtoextractthe.objfileaftersuccessfullymapping,thisprocesstooka
while.

OneissueweencounteredwhilemappingwithGoogleTangotabletwaslimitedbatterylife.
Becausethisapplicationdoessomuchcomputation,itdrainsthebatteryreallyfast.AstheSLAMalgorithm
furtherdevelops,thebatterymaybeanimpedimentforalargermapping.

2TechnicalMaterial

OurApproach

TheGoogleTangoisanAndroiddevicewithAPIsavailableinJava,C,andUnity.TheTango
devicehasoutoftheboxmotiontrackinganddepthsensingcapabilitiesduetoitsabundanceofsensors.
Someofthekeysensorsincludethemotionsensorcameraand3Ddepthsensor.ItusesaNVIDIATegra
processorwithCUDAsupportforgraphicscomputingapplications.
Theinitialgoaloftheprojectwastoimproveuponanexistingapplicationthathad3Dmapping
capabilities.ByscanninganareawiththeTangoscameraanddepthsensors,theTangocouldsavepoint
cloudinformationtobelaterexaminedasamodelfile.Theapplicationcouldthenbeusedby
ArchaeologiststomaptheirfindingsmorespecificallyGuatemalancavestheEngineersforExploration
organizationatUCSDwouldbeexploring.TheTangowouldhavetopossiblymaplongandwindingtunnels
withrockysurfaces.Theexistingapplicationdidnothavedriftcorrectionsooverlongperiodsoftime,the
motiontrackingwouldloseitspositionintheworld.Any3Dmappedmodelswouldnotbeaccuratedueto
thisdrift.Thesolutionwastointegrateasimultaneouslocalizationandmappingalgorithm(SLAM)that
wouldcorrectforthisdrift.
Theexistingapplicationrequiredagreatdealofhardwarelevelprogrammingduetoitsgraphics
processingdemands.TheAndroidOSwhichusestheJavalanguagehassupportforhardwarelevel
tinkeringthroughtheJavaNativeInterface(JNI)usingtheClanguage.Applicationscreatedwithsuch
supportarecalledJavaNativeApplications(JNA).Thisaddedagreatmanynewdifficultiestoourteam
whichwasunfamiliarwithJNAs.GoogleprovidesC,Java,andUnityAPIs.Weaddressedourweaknessby
developingintheUnityenvironment.

Unityisaverypopular3Dgamedevelopmentprogram.Duetoitseaseofuse,ithasfoundaplace
amongtripleAcompaniesandindependentdevelopersalike.SomenotableproductscreatedwithUnity
includeBlizzardsHearthstoneandtheKerbalSpaceProgram.Unitysupportsmanydifferenttypesof
platformsaswellincludingourtargetofAndroid.Thegamecreationprocessexploitstheobjectorientated
natureofvideogames.Gameobjectshaveavarietyofdifferentcomponentsattachedtothemtodefine
theirbehaviors.ComponentscanrangefromSpriteRenderers,3DModels,Transforms,Rigidbodies,
Scripts,etc.ScriptsforUnitycanbewritteninC#orJavascript.WedecidedtogowiththeUnityfavoriteof
C#.ScriptsallowevenmorecomplexbehaviorstobeprogrammedintotheGameObjects.

3DMapping

The3Dmappingofourapplicationinvolvestwoseparatefeatures.

1. TheabilitytorecordandlaterexportdatainvolvingtheTangoandtherecordedenvironment
2. Theabilitytogiverealtimefeedbacktotheuseronwhathasandhasnotbeenrecorded

Thedatatorecordincludesposedata,cameraimages,depthdata,andmodeldata.Theposedata
istheorientationoftheTangodeviceinspaceatagivenmomentintime.Thecameraimagesarethe
imagesseenbythergbcamera.Thedepthdataisthesensordatafromthedepthcamerawhichistakena
rateof5Hz(muchslowerthantheactualcameradata).Themodeldataistheactualenvironment
representedasamesh.TheTangocomeswithconvenientAPIfunctionsthatarecallbackstoreceivepose
data,cameraimages,anddepthdata.Themodeldatacannotbecapturedassimplyandiscreatedthrough
ameshrenderer.
Themeshrendererisentirelycontainedinameshrenderingscript.Themeshisrepresentedasa
multitudeofinengineGameObjectswithshaderandtransformcomponents.Theshadergivesthemesh
thecoloroftheobjectbeingrepresented.Thetransformdeterminesthepositionandorientationofthemesh
intheapplicationworld.Thetransformisextremelyimportantasanaccuratetransformdeterminesthe
accuracyofanexportedmodel.
Themeshrenderingprocessinggoesthroughquiteafewsteps.Themeshrenderersubscribesto
theTangosdepthsensorcallback.TheTangofeedsthemeshrendererdepthdataallowingtherendererto
findflatsurfaces.Eachsurfaceisthentransformedintoagridinwhichasquareonthatgridcanthenbe
meshed.Dependingontheresolutiondeterminedbeforehand(wefound5cmstobeefficient),thegridwill
havedifferentlysizedsquares.Apointcloudisgeneratedforthepurposeofcoloringeachmeshtobetter
representwhatisbeingmapped.
Toimprovetheuserexperience,thecomputationallyheavymeshrenderingisdoneonaseparate
thread.Whenindicesareseenthatrequiremeshing,theyarepushedintoaqueuedatastructuretoawait
theirturn.Thisbacklogisprocessedforasmallportionofthecomputationcycleeverycycle.Ofcoursethis
meansthatmappingabrandnewareacreatesasurgeinmeshesthatneedtobeprocessed.Bykeeping
theTangofocusedonapreviouslyrenderedareaorbypausingtherenderingprocessthroughtheGUI,the
backlogcanbeprocessedmuchmorequickly.

SLAM

ManyoftheSLAMalgorithmsweresearchedincludingORBSLAM,RGBDSLAM,andLSDSLAM
cannotbecompiledthroughtheUnitysbuildprocess.ThismeantthatwehadtorelyonTangosonboard
SLAM.TheonboardSLAMnamedAreaLearningbyGoogle,functionsverysimilarlytotheseotherSLAM
algorithms.TheTangoisconstantlyviewingandrememberingspecificlandmarksinitsenvironment.As
theusertakestheTangoawayfromthestartinglocationandback,theTangowillrememberlandmarksfrom

thestartinglocation.Itthenrealizesthatithasformedaloopandtheloopcanbeclosedthrough
recalculatingpositionaldata.TheTangostorestheposedatafromwhenitbeganrecordingandwhena
loopisdetecteditcomparesthecurrentposedatatothestartingposedata.Bothposesshouldtechnically
bethesameastheTangohassimplyreturnedtothepositionithasstartedinbutmistakesinposition
accumulateovertimecreatingdrift.ThedifferencebetweentheposetheTangostartedwithandthepose
theTangoendedupwithcanbeusedtorecalculatealltheposesalongtheTangosroute.

Weaknesses

ThecurrentimplementationforArchaeologicalrecordingshassomeweaknessesthatneedtobe
addressed.TheapplicationisnotbuiltasaJNAthusitisnotusing100%ofitshardwarecapabilities.This
canbeseeninthedropsinframerateandbacklogprocessingrateoftheapplication.
TheapplicationusestheonboardSLAMalgorithmwhichcannotbeexaminedortunedby
developers.
TheapplicationisbuiltinUnitywhichislimitedinitscompilingandbuildingprocedures.

Strengths

TheUnityengineissimpletolearnandpowerfultouse.ByusingtheUnityAPIs,wedidnothaveto
spendmanpowerandresourceslearningJNA.
TheTangoislightandportableweighinginat0.82lb(370g).
Theapplicationmapsquicklyandaccuratelyforitsgivenhardware.

Thissubsectiondescribeshowtorunthe3Dreconstructionsystem.Thefollowingwilldemonstrate
requirementsandinstructionstosetupandbuildOnBoardSLAMonGoogleTangotablet.

2.1.1Requirements
Components:
GooglesProjectTango
Laptop

2.1.2SetUpInstructions

Environmentusedforthisbuild

Windows10/MacOSX
AndroidStudio2.1
Unityversion3.4.1.f
GoogleUSBDriver

Setup
1. InstallWindows10.Windows10isaverycommonoperatingsystemamongPCs.UsuallyWindows
10comeswiththecomputeritself.However,ifWindows10isnotavailable,registeranddownload.
Download:
https://e5.onthehub.com/WebStore/ProductsByMajorVersionList.aspx?ws=94fd1fe3eb9b
e011969d0030487d8897&vsro=8

2. InstallAndroidStudio2.1.AndroidStudioisaIDE(IntegratedDevelopmentEnvironment)
forandroid.
Download:
https://developer.android.com/studio/index.html
3. InstallUnity(
3.4.1.f)viaUnity'sDownloadAssistant
.Unityisatoolfor3Dgameengine.
GoogleTangohasUnitysupportforAR(AugmentedReality)andVR(VirtualReality).
MakesuretoselectAndroidBuildSupportandMicrosoftVisualStudioToolsforUnity
isrecommended.
Download:
http://unity3d.com/getunity/download
4. InstallTangoUnityAPI.TangoUnityAPIallowsUnitytouseAPIsspecificallyforTango.
Download:
https://developers.google.com/tango/downloads#tango_sdk_files
5. InstallGoogleUSBDriver.GoogleUSBDriverisadrivertolettheWindowscomputerto
recognizeandroiddevicessuchasTango.
Download:
https://developer.android.com/studio/run/winusb.html

2.1.3BuildingApp
1. AddtheAndroidSDKpathtoUnitybyselectingUnity>Preferences>ExternalTools
2. ImporttheTangoSDKintoUnity,selectAssets>ImportPackage>CustomPackage,then
selectTangoSDK
3. Clone
https://github.com/ssl024/TangoMapper.git
4. Buildapk

2.1.4RunningApp
1. TransfertheapktoTango
2. UsingFileViewerappinstalltheapk
3. HoldTangostillwhenopeningbecauseofthecalibration.
4. Typeinappropriatevaluesforthefield
5. Theappwillstartmappingthearea
6. Ifdesiredtoexport,pressexportandviewfilefrompathwrittenonstep4

2.2SystemUsage

3Milestones

Initialmilestone

Week

Objective/Milestones

Status

Artifact/Comments

CompleteProjectSpecifications
SetupTangoworkstation

Comp
Comp

Specifications
SetupInstruction

CompleteProjectwebpage.

Comp
Comp

BitBucketWiki
VideoDemoHelloWorld

RunsimpleTangoprogramrunningsuch
asHelloWorld

OralUpdate.
UnderstandTangofunctionality.

Comp
Comp

PresentationSlides
VideoDemopointclouddemo

SLAMpreimplementation.

Comp

VideoDemomeshrendering

MilestoneReport.
SLAMimplementationcomplete.

Comp
Comp

This
VideoDemoSLAMimplementation

Tangotestinginacontrolledenvironment

Comp

Tangotestinginrealworld

Comp
Comp

10

Catchslips.
Finalpresentation
FinalVideo

Comp
Comp
Comp

Bytheendofinitialmilestone(week5),
CompleteProjectSpecificationsandsetupTangoworkstations.
TosetupTangoworkstationsforuseanddownloadNDKandSDK,
https://developers.google.com/tango/apis/c/#install_the_ndk
CompleteProjectwebpageandasimpleTangotasksuchasHelloWorldprogram
AftersuccessfullysettingupTangoworkstation,wetestedmultiplesimpleprograms

UnderstandTangofunctionality.SLAMresearch
AfterlookingatmultipleSLAMalgorithmson
https://openslam.org/
wecamedownto4
differentalgorithms:OnBoard,RGBD,RTAB,ORB

Byendofsecondmilestone(week7),
SLAMpreimplementation
WhilefinalizingwhichSLAMimplementationbestfitsourpurpose,wegotmeshrendering
implementedonTangousingUnity.
SLAMimplementationcomplete
Wecouldntfinishthisuntilbeginningofweek8.

Byendoffinalmilestone(week10),
ControlledTangotestingwithSLAM
AfterimplementingOnBoardSLAM,wemappedasmallareasecondfloorofCSEbuilding

TangotestinginrealworldSLAM

Notes:
NewObjectives:
ControlledTangotestingwithSLAM.
UsingORBSLAMwithUnityAPIformeshrendering.

Issuesencountered:

Atfirst,weworkedontwodifferentimplementationofSLAMalgorithm:RTABandOnBoard.Onourfirst
developmentofRTABSLAMalgorithm,wewereablecrosscompilealldependenciesandsuccessfullygot
theAPKfiletoputintheGoogleTangotablet.However,afterupdatingsomefeaturesandpullingfromthe
repository,crosscompilingfailedandfailedtoreproducetheAPK.

Also,downloadingthenewestversionofallsoftwaresandsourcesarenotthebestidea.While
implementingOnBoardSLAM,wehadlotsofdependencyissue.Stayingwithpriorversionmakeslife
easier.

4Conclusion

The3DReconstructionprojectusingGoogleTangotabletshowshowmodernartistsandarcheologistscan
usethisdeviceandapplicationtoeffectivelymaptheirfindings.Comparedtotheirpreviousmapping,which

reliedonhanddrawnstructures,thisprojectwilldefinitelybeanimpetusformanyartistsandarcheologists
tomoveonforward.Althoughthisprojecthaslimitationstomappingbigfindings,itcanstillmapmuchmore
efficientlyandshowshighdetailsduetouseofdepthsensor.Moreover,withtraditionalmethodofdrawing,
therearelimitationstohowmuchmappingcanbedone.Useofthis3Dreconstructionapplicationallowsthe
usertoextracttheirmappingandsaveitasan.objfilewhichthencanbeviewedoncomputerthrough3D
viewer,andcanevenbeprinted.Becausewedonothavestatisticstothisproject,wecannotsayhow
effectivetheuseofthisprojectwillbecomparedtootheralgorithmsorpreviouslyusedmethodstomapthe
findings.

Improvementscandefinitelybemade.Therearemanydifferentsimultaneouslocalizationapplication
mappingalgorithmsanditishardtoendupwithoneperfectalgorithm.BecauseouruseofOnBoardSLAM
algorithm,therearelimitations,aswecannottweakitasmuchforvariability.Onewaytoachievebetter
solutionwouldbetoresearchmoredifferentSLAMalgorithmandfindoutwhichalgorithmisgoodforlong
distanceandshortdistance.Becausearcheologistsliketomaplongdistancecaves,OnBoardSLAMwas
probablynotthebestidea.OurpriorimplementationofRealTimeAppearanceBasedMappingshowedhigh
detailsbutthisalgorithmisprobablynotthebestideaforlongdistancemapping,aswefoundoutthatthe
efficiencyofthisalgorithmdegradedovertime.

Comparedtoourotherteam,TeamKinect,theuseofGoogleTangowasmuchportable.However,dueto
itssmallsize,therewerecomputationallimits.ForTeamKinect,theyusedMicrosoftKinectwhichhasa
cameraandmultiplesensors.Moreover,thecomputationisdoneonalaptopusingitscentralprocessing
unit(CPU)andgraphicsprocessingunit(GPU).Fortesting,QuentinGautierusedalaptopwithi7processor
highendgraphicsprocessingunit,enablingsmoothrenderingandproducingrichqualityreconstruction.
AlthoughKinectteamproducedbetterresultsduringthetestatGuatemala,itsportabilitywasnotsogreat.
For3DreconstructiondonewithGoogleTangotablet,allcomputationandrenderingwasdoneonthetablet
while3DreconstructiondonewithKinecthadtocarryaroundpowersourceforkinectandalaptopfor
rendering.Overall,althoughnotperfect,bothteamproducedafullworking3Dreconstructionapplication.
Thus,webelieveourprojectwassuccessful.

References

https://openslam.org/
http://wiki.ros.org/rtabmap
https://developers.google.com/tango/hardware/tablet
https://unity3d.com/unity/editor
https://developers.google.com/tango/apis/c/#install_the_ndk