SEEING STARS Using technology to deliver an engaging app to capture - PowerPoint PPT Presentation

b e s p o k e A R f o r m o b i l e s SEEING STARS Using technology to deliver an engaging app to capture meteorite sightings, on Android and iOS

DESERT FIREBALL NETWORK Image credit: DFN 2

HTTP:// BIT.LY/FIREBALLSDL Or search App Store or Google Play for “Fireballs in the Sky” 3


 HELLO DAVID COLLS BRAD WARD NATHAN JONES Maths nerd Developer 
 Developer @davidcolls @the_nathanjones AR MATHS IOS ANDROID 5

FIVE * WHYS * Actual number may vary Image credit: National Library of Australia 6

1. WHY NATIVE? To describe a fi reball Words & numbers fall short, so 
 animated recreations were MVP. With particle systems Demanded performance beyond the reach of mobile web for the majority of devices. Meant 2 native apps Developed in parallel. 7

2. WHY AUGMENTED REALITY? AR not MVP, 
 but delightful And improved reporting � Option for Release 1 � Release 1 Implemented in Release 2 Release 2 8

3. WHY BESPOKE AR? A unique context No desire to license technology Based on sensors not camera image Camera view just black at night Very simple interaction Google Sky only Android Google Sky won’t subordinate And we had a Processing prototype 9

4. WHY PROTOTYPE IN PROCESSING? Fastest way to start… The maths guy knew Processing (visualisation IDE) Just HUD Rapid iteration to demonstrate we could do star maps (highest risk) Stars And planets! Support for tilt No dependencies … and fi nish Porting together to apps felt low risk Fully featured 10

AR MATHS * * Guaranteed to contain NO equations 11

APPROACH W h e r e a r e How do we draw this t h e (in a virtual window)? s t a r s ? Where Where Where are in the in the you looking? sky? universe? Where are you standing? 12

WHERE IN THE UNIVERSE? 8 mins 4 years Alpha Centauri circle to scale 13

WHERE ARE THE STARS? Infinitely How do the stars distant look from the earth’ s surface? “Fixed Stars” HYG Database Celestial sphere 14

WHERE IN THE SKY? Azimuth Elevation Time + Date Sidereal Time + Longitude LOCAL SIDERIAL TIME Terrestrial Celestial LATITUDE observer sphere 19

DRAWING IN A VIRTUAL WINDOW Known positions Screen View direction Choose an Known eye-screen positions distance Find where the line-of-sight Location hits the screen Perspective Projection 20

WHERE ARE YOU STANDING? API GPS satellites CLLocationManager LocationManager Cell towers WiFi access points 21

DEFINES LOCAL REFERENCE FRAME X - North Y - West Z - Up X - East Y - North Z - Up 22

WHERE ARE YOU LOOKING? API Magneto- meters SensorManager Device Register for updates & rotation getOrientation() WindowManager matrix… Device default orientation Accelero- …with respect meters to local reference CLLocationManager frame For heading CMMotionManager RefFrameXTrue NorthZVertical Gyroscopes 23

WHERE ARE YOU LOOKING? Azimuth Elevation Tilt Obtained from device rotation matrix 24

REVIEW Once per Once per Every Once per universe* session session frame [Azimuth & Device Device [RA & DECL] Elevation] rotation location (Fixed stars) (LST & Lat) matrix (APIs) (APIs) Where Where Where are in the in the you looking? sky? universe? Where are you standing? 25

IOS Sweet, fruity, and objective knowledge 26

IOS INGREDIENTS CoreLocation & CoreMotion CLLocationManager latitude - longitude CMMotionManager azimuth - elevation - tilt Using CMAttitudeReferenceFrameXTrueNorthZVertical reference frame The API-provided pitch , roll and yaw were not used. Used deviceMotion.attitude.rotationMatrix directly instead Reference frame ‘drifts’ over time, periodic resets resolve this 27

IOS INGREDIENTS Accelerate/vecLib library 
 hardware-accelerated vector maths vecLib uses the Advanced SIMD instruction set implemented by NEON on ARMv7 devices 2-10x performance bump over standard Objective C Avoided overhead of classes/GC in calculation code Work well with Accelerate library’s C interface Rendering code is Objective-C 28

IOS INGREDIENTS CoreGraphics CPU-based 2D rendering Minimal development e ff ort with reasonable fl exibility Was an expected (and realised) performance bottleneck ▫︎ OpenGLES would provide dramatically improved performance, at higher development cost. Final performance was good on iPhone5 devices � � 29

IOS IMPLEMENTATION So many options to improve performance… Optimise use of Accelerate library via bulk calculations OpenGLES (eg. Cocos2D or SpriteKit) for rendering Multi-threading Full GPU implementation of star-positioning calculations 30

ANDROID Developing for the bot with the lot 31

ANDROID INGREDIENTS Which API? SensorManager is the home for all sensors in Android ▫︎ Reference examples use deprecated ORIENTATION_SENSOR ▫︎ Discussion groups suggest hand-rolled sensor fusion of accelerometer and compass ▫︎ Take a look at the ROTATION_VECTOR Sensor � Adjust resulting vector for current and default orientation � 32

ANDROID INGREDIENTS How do I draw them? We use a regular SurfaceView We use a Timer targeting 60FPS instead of an explicit thread Draw on a regular 2D Canvas Not hardware accelerated � Room for improvement 33

ANDROID IMPLEMENTATION Coding Style Optimised vector math libraries not as mature Embrace some functional paradigms ▫︎ separate state and behaviour Multi-threading became an option Beware the garbage collector Pro fi le all the things, these are limited resources Many ways to skin a cat with vastly di ff erent performance 34

ANDROID - HERE BE DRAGONS Fragmentation exists Expect it and deal with it Eligible for installation on 3606 4508 devices Pick a baseline and work out what you are in for Don’t expect the API to be consistent Get some real devices Lowest and highest target OS versions Lowest and highest screen sizes - resolution and physical size Lowest performance - slow single-core phones Fall back to emulator only for sanity check on look and feel 35

BUILDING APPS On both Android and iOS Image credit: DFN 36

TESTING Use real devices Performance di ff ers from emulators Sensor data not available in emulators Test in the real world The acid test for an AR app 37

LOCATION Think about usage No wi fi or cell towers in the outback Don’t block the user While you fi nd their location Enough is enough Know the required accuracy, 
 and stop when you have it Stop wasting their battery (Android) Turn o ff location services on app hide 38

DEVICE ROTATION Di ff erences in APIs Android vs iOS Frame of reference True North or Magnetic North? Smoothing How to fi lter noise while preserving a responsive signal 39

THANK YOU For questions or suggestions: � njones@thoughtworks.com brward@thoughtworks.com dcolls@thoughtworks.com