FP7 Project Grant Number 288238 Brief introduction of WP4 10min - PowerPoint PPT Presentation

FP7 Project Grant Number 288238

» Brief introduction of WP4 10min ˃ General objectives and role in VENTURI ˃ Partners, tasks, deliverables, milestones ˃ Timeline » Main results (first year) 25min » Plans for second year 15min » Conclusions and questions 2 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

WP3: User Interface and Interaction Design WP7: Dissemination and Exploitation WP6 Technology Integration, WP1: Project Management Evaluation and Test-Cases WP5: Adaptive Content Harvesting, Creation and Delivery WP4: Context Sensing and Interpretation WP2: Architecture Definition and Development 3 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

4 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

On-board sensors: • vision: camera(s) • audio: microphone(s) • l ocation: GPS, … • relative orientation / motion: accelerometers, gyroscopes • absolute orientation: compass • other: pressure, proximity Data sources: • maps • DEMs • geo-tagged data: images, 3D models, etc. 5 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

6 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

7 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

8 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

9 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

Geographical localization Absolute orientation (yaw, (lat/lon/alt) (also indoor) pitch, roll, field of view, etc.) 10 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

• Scene reconstruction • Model the immediate surroundings of the mobile device 11 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

• User behavior understanding • What’s happening here? • Noise level / scene classification • Infer what the user is doing and what he could experience with his senses 12 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

Enable and explore innovative ways to interact with the environment Example: SixthSense (Fluid Interface Group at MIT Media Lab) 13 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

FBK 28MM ST-Italy 21MM Total WP4 MMs: 111 ST-France Fraunhofer HHI 19MM Sony 6MM INRIA 18MM metaio 16MM STE 3MM 14 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

T4.1 Hardware sensor interpretation T4.2 Auditory scene analysis T4.3 Visual analysis of the environment and geo-position refining T4.4 Physical surroundings modeling T4.5 What is, or could be, going on around me? 15 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

M18 = mar 2013 M30 = mar 2014 M12 = sep 2012 M24 = sep 2013 M36 = sep 2014 M15 = dec 2012 M6 = mar 2012 M1 = oct 2011 M9 = jun 2012 T4.1 T4.2 T4.3 T4.4 T4.5 first year 16 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

first year » D4.1 Synchronized dataset containing a dump of all on-board sensors of a real device, while being used in a real environment (STE, Mar 2012, M6, O-CO) » D4.2 Report on expected platform requirements of WP4 algorithms (FBK, Jun 2012, M9, R-CO) » D4.3 WP4 outcome definitions and API specification for inter- task / inter-WP communications (ST-Italy, Dec 2012, M15, R-CO) » D4.4 Prototype of and Report on multi-modal estimation of geo- location and orientation (FBK, Mar 2013, M18, P-CO / R-PU) » D4.5 In-the-lab prototype and Report on visual analysis and reconstruction of the scene (Fraunhofer, Sep 2013, M24, P-CO / R- PU) » D4.6 Final prototype integrating all WP4 outcomes (FBK, Mar 2014, M30, P-CO) 17 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

first year » MS9 First version of chain for geo-localization / orientation estimation, based on non-audio / non-video (FBK, Sep 2012, M12) » MS10 Final version of chain for geo-localization / orientation estimation (FBK, Mar 2013, M18) » MS11 WPC version of visual analysis and reconstruction of the scene, ready for integration (FBK, Sep 2013, M24) 18 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

M18 = mar 2013 M30 = mar 2014 M12 = sep 2012 M24 = sep 2013 M36 = sep 2014 M15 = dec 2012 M6 = mar 2012 M1 = oct 2011 M9 = jun 2012 D4.6 D4.1 D4.2 D4.3 T4.1 D4.4 T4.2 T4.3 D4.5 T4.4 T4.5 MS9 MS10 MS11 first year 19 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

» Brief introduction of WP4 10min ˃ General objectives and role in VENTURI ˃ Partners, tasks, deliverables, milestones ˃ Timeline » Main results (first year) 25min » Plans for second year 15min » Conclusions and questions 20 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

21 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

» Cross-partner / Cross-task results ˃ Shared tools and Android OS improvements ˃ Deliverables » Results showcased in Y1 demo ˃ Algorithms and their integration » Other results (relevant for Y2 and Y3 Use Cases) ˃ Analysis of state of the art ˃ Design, development, implementation of algorithms ˃ Evaluation and testing 22 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

» One of the main requirement elicited (in T2.1.2): accurate and common timestamping between camera and sensor domain ST9500-based platform: ST9540-based platform: Next (8680) platform: done done, under debugging taken into account from the beginning 23 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

Video: timestamping 24 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

Legend: ST-Ericsson Android Non-Android Third- proprietary origin FOSS party/other Android application Application Sensor HAL provides Sensor Timestamp here Android application framework Manager Class Sensor Manager JNI User space Android libraries Android Sensor Manager Sensor Service runtime library (client) (server) Android porting layer Sensor data are Time Sensor HAL stamped here Access Core Connectivity Multimedia services Accelero- Magneto- Sensor drivers (Linux) Gyro-scope Proximity Ambient meter meter not providing driver driver Light driver driver driver Timestamps Kernel space I2C Core Imaging Signal Processor I2C driver Hardware I2C Android ICS Vanilla Sensors

Legend: ST-Ericsson Android Non-Android Third- proprietary origin FOSS party/other Android application Application Sensor HAL provides Sensor Timestamp here Android application framework Manager Class Sensor Manager JNI User space Android libraries Android Sensor Manager Sensor Service runtime library (client) (server) Android porting layer Sensor HAL Access Core Connectivity Multimedia services Sensor data are Time Accelero- Magneto- Gyro-scope Proximity Ambient meter meter stamped here driver driver Light driver driver driver Kernel space IIO Framework I2C Core Time comes from here SIA I2C driver Hardware I2C VENTURI modification Sensors for 9500/9540 platforms

» Ongoing experiments to quantitatively evaluate the impact of accurate timestamping on actual WP4 algorithms. 27 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4

» FBKSequenceRecorder tool » It can capture and dump to disk timestamped data from all on-board sensors (raw video, audio, other sensors). » On STE boards, it exploits the new timestamping APIs (still debugging on STE9540). However, it works on any Android device. » Source code and APK shared with VENTURI partners. » Possibility to share it with general public. 28 First Review Meeting. Hosted by Fraunhofer HHI in Berlin, Germany on the 18 th of December 2012 WP4