Body-Mounted Cameras Claudio Fllmi foellmic@student.ethz.ch May - PowerPoint PPT Presentation

Body-Mounted Cameras Claudio Föllmi foellmic@student.ethz.ch May 28, 2013 Distributed Systems Seminar 1

Outline ● Google Glass ● EyeTap ● Motion capture ● SenseCam May 28, 2013 Distributed Systems Seminar 2

Introduction o ● Cameras have become small, light and cheap l o C - G E P ● We can now wear them constantly J - m e t s y s ● So what new things can we do with them? d 4 / g n i g a m ] I l - m / - t t h h . g 7 i L 7 / 1 s _ r o p s / p n i h e C S / - p 8 h 2 p 5 . V x e O d - n e i c / a t e f r n e . c t n e I l - e L y T s T a - e T i R a h A t U . w - w a r w e / m / : p a t C t h - r [ May 28, 2013 Distributed Systems Seminar 3

May 28, 2013 Google Glass Distributed Systems Seminar [ h t t p : / / c o m m o n s . w i k i m e d i a . o r g / w i k i / F i l e : G o o g l e _ G l a s s _ E x p l o r e r _ E d i t i o n . j p e g ] 4

Google Glass ● Like a head-worn smartphone ● Sensors, wi-fi, bluetooth, camera ● Input over voice commands, touchpad or phone app ● Runs Android ● App support May 28, 2013 Distributed Systems Seminar 5

Schematic ] 2 4 0 4 4 0 0 3 1 0 2 n o i t a c i l p p a t n e t a p S U [ May 28, 2013 Distributed Systems Seminar 6

Applications ● Lifelogging ● Looking up information on the internet ● Remote-controlling smartphone ● Getting notifications from smartphone May 28, 2013 Distributed Systems Seminar 7

Too futuristic? ] s o t o h p / 0 7 0 7 2 7 0 3 8 1 8 4 7 6 7 4 0 8 5 0 1 / m o c . e l g o o g . s u l p [ May 28, 2013 Distributed Systems Seminar 8

Meet Thad Starner and Steve Mann ] / g n i t u p m o c - e l b a r a e w / m o c . a i c n a t s i h . w w w / / : p t t h [ May 28, 2013 Distributed Systems Seminar 9

Steve Mann over the years ] 4 0 0 2 n n a M [ May 28, 2013 Distributed Systems Seminar 10

EyeTap ] 2 1 0 2 n n a M [ May 28, 2013 Distributed Systems Seminar 11

Schematic ] 3 1 0 2 n n a M [ May 28, 2013 Distributed Systems Seminar 12

How it works ● Incoming light is diverted onto camera ● Captured image is processed ● Processed image is projected into eye ● Functionally, the eye becomes a camera ● To the outside, the camera replaces the eye May 28, 2013 Distributed Systems Seminar 13

Applications ● Lifelogging ● Capture events from exact perspective of user ● Augmented / mediated reality ● Latency between capture and projection will disturb users (simulation sickness) ● Correction of visual impairment ● Not just long-/shortsightedness ● Increase visual dynamic range May 28, 2013 Distributed Systems Seminar 14

May 28, 2013 HDR filter Distributed Systems Seminar [ " R e a l t i m e H D R ( H i g h D y n a m i c R a n g e ) V i d e o f o r E 15 y e T a p W e a r a b l e C o m p u t e r s , F P G A - B a s e d S e e i n g A i d s , a n d G l a s s E y e s " , M a n n e t a l , 2 0 1 2 ]

Color blindness ● About 8% of all men are red-green colorblind ● Much of our everyday life is color-coded ● EyeTap could automatically ] s replace colors, or label colored s e n d n surfaces i l b r o l o C / i k i w / g r o . a i d e p i k i w . n e / / : p t t h [ May 28, 2013 Distributed Systems Seminar 16

Comparison with Glass EyeTap Glass ● More potential for ● Standard smartphone mediated reality functionality ● Can replace prescription ● Compact enough be worn glasses as an accessory May 28, 2013 Distributed Systems Seminar 17

Future development ● Size, weight and battery life will improve through normal technical progress ● More computing power will allow for more complex applications ● Latency is limiting factor of mediated reality ● Society's reaction to Glass will be very important ● Broad acceptance of Glass will make EyeTap more acceptable May 28, 2013 Distributed Systems Seminar 18

What if we film the user instead? May 28, 2013 Distributed Systems Seminar 19

May 28, 2013 Motion capture Distributed Systems Seminar [ h t t p : / / a v a t a r b l o g . t y p e p a d . c o m / a v a t a r - b l o g / 2 0 1 0 / 0 5 20 / b e h i n d - t h e - s c e n e s - l o o k - a t - t h e - m o t i o n - c a p t u r e - t e c h n o l o g y - u s e d - i n - a v a t a r . h t m l ]

Motion capture ● We have already seen a system using body-mounted cameras for capturing full body motion ● (April 9, talk by Antoine Kaufmann) ] 1 1 0 2 i r o t a r i h S [ May 28, 2013 Distributed Systems Seminar 21

Approach ● Capture the face using one camera and multiple light sources (Photometric stereo) ● Can be combined with other systems ● Many systems have a normal camera pointed at the face anyway ● The actor should be able to move freely ● Ambient light changes May 28, 2013 Distributed Systems Seminar 22

Why capture the face separately? ● Face is the most expressive part of body language ● Even small mistakes will stand out ● Traditionally, artists would correct generated images by hand May 28, 2013 Distributed Systems Seminar 23

How it works ] 1 1 0 2 s e n o J [ May 28, 2013 Distributed Systems Seminar 24

How it works ● Three different lighting directions ● One with ambient lighting only ● Capture 120 fps input for 30 fps output ] 1 ● Change lights at 360Hz to eliminate flickering 1 0 2 s e n o J [ May 28, 2013 Distributed Systems Seminar 25

How it works ● Assumes lambertian reflectance ● Surface will look equally bright from a ] f i g . range of angles 2 t r e b m a L : e l i F / i k i w / g r o . a i d e m i k i w . s n o m m o c / / : p t t h [ May 28, 2013 Distributed Systems Seminar 26

Equation I = L * N A ● I: image intensity (measured by camera) ● L: lighting direction (3x3 matrix, known) ● N: surface normal (vector that we want to know) ● A: albedo (scaling factor) ● Surface geometry can then be reconstructed from normals May 28, 2013 Distributed Systems Seminar 27

Corrections ● Closeness of both lights and camera leads to inaccuracies ● Lighting direction for each pixel depends on depth ● Use a generic smoothed face to initialize lighting directions ] 1 1 0 2 s e n o J [ May 28, 2013 Distributed Systems Seminar 28

Video demonstration “Head-Mounted Photometric Stereo for Facial Performance Capture” ] Q L http://www.youtube.com/watch?v=RGtFPLciFLQ F i c L P F t G R = v ? h c t a w / m o c . e b u t u o y . w w w / / : p t t h - 1 1 0 2 s e n o J [ May 28, 2013 Distributed Systems Seminar 29

Results ● Shadow artifacts show up as white albedo around nose ] 1 1 0 2 s e n o J [ May 28, 2013 Distributed Systems Seminar 30

Results ● Subtle movements are captured ● Exactly where non-camera-based approaches fail ● Using infrared light leads to more artifacts ● Computation of normals and geometry reconstruction can be done in real-time May 28, 2013 Distributed Systems Seminar 31

Comparison with more detailed method Jones et al, 2011 Beeler et al, 2011 May 28, 2013 Distributed Systems Seminar 32

Comparison with more detailed methods ● Actors can move freely ● Captures even small wrinkles ● Can handle changes in ● Needs multiple cameras ambient light and uniform lighting ● Actors must hold head still May 28, 2013 Distributed Systems Seminar 33

Next steps ● Use a picoprojector for lighting ● Use a customized input head model ● Arrange lights further apart ● Track shaking of camera ● Try out more faces ● Combine with other methods May 28, 2013 Distributed Systems Seminar 34

Lifelogging ] g p j . 3 0 9 0 0 0 0 0 / s e g a m i / m o c . e u v e r n o c i v . w w w / / : p t t h [ May 28, 2013 Distributed Systems Seminar 35

SenseCam ● Automatically triggered camera ● Lightweight ● Unobtrusive ● Battery life of at least 12 hours ● Capture interesting moments May 28, 2013 Distributed Systems Seminar 36

Internals ] 6 0 0 2 s e g d o H [ May 28, 2013 Distributed Systems Seminar 37

How it works ● Sensors judge if situation is interesting ● If yes, take picture ● Take picture after timeout ● Log reason why picture was taken ● Battery lasts over 24 hours when taking a picture every 30 seconds ] 6 0 0 2 s e g d o H [ May 28, 2013 Distributed Systems Seminar 38