Device-Free activity recognition Stephan Sigg Department of - PowerPoint PPT Presentation

Device-Free activity recognition Stephan Sigg Department of Communications and Networking Aalto University, School of Electrical Engineering stephan.sigg@aalto.fi Bad Worishofen, 10.07.2017

Stephan Sigg July 23, 2017 2 / 42

WiFi Fingerprinting Seifeldin et. al: Nuzzer: A Large-Scale Device-Free Passive Localization System for Wireless Environments, IEEE TMC 2013 Bong et. al: Reasonable Resolution of Fingerprint Wi-Fi Radio Map for Dense Map Interpolation, FRTA, 2014 Stephan Sigg July 23, 2017 3 / 42

Seifeldin et. al: Nuzzer: A Large-Scale Device-Free Passive Localization System for Wireless Environments, IEEE TMC 2013 Stephan Sigg July 23, 2017 4 / 42

Stephan Sigg July 23, 2017 5 / 42

Stephan Sigg July 23, 2017 6 / 42

Stephan Sigg July 23, 2017 7 / 42

Exploiting the RF-channel for environmental preception ◮ Multi-path propagation ◮ Reflection ◮ Signal superimposition ◮ Blocking of signal paths ◮ Scattering ◮ Doppler Shift ◮ Signal Phase ◮ Fresnel effects Stephan Sigg July 23, 2017 8 / 42

Aspects of the mobile radio channel j G a i n ) + γ i cos( ) t i + γ i ) f 2 π ϕ 2 π f t γ i ( ϕ ( j j i e e ϕ i 1 cos( ) i ϕ j(2 π f t + γ i ) e −δ i δ i Stephan Sigg July 23, 2017 9 / 42

Aspects of the mobile radio channel Superimposition of RF signals ◮ At a receiver, all incoming signals add up to one superimposed sum signal ◮ Constructive and destructive interference ◮ Normally: Heavily distorted sum signal Stephan Sigg July 23, 2017 10 / 42

Aspects of the mobile radio channel Superimposition of RF signals ◮ The wireless medium is a broadcast channel ◮ Multipath transmission ◮ Reflection ◮ Diffraction ◮ Different path lengths ◮ Signal components arrive at different times ι ◮ Interference � e j ( f i t + γ i ) � � ζ sum = ℜ i = 1 Stephan Sigg July 23, 2017 11 / 42

RF-based activity recognition Stephan Sigg July 23, 2017 12 / 42

Stephan Sigg July 23, 2017 13 / 42

Time-domain signal strength fluctuation ◮ Recognition of environmental situation (presence, movement (speed)) ◮ Non-intrusive ◮ Arbitrary antenna placement ◮ Pre-training possible ◮ Limited gesture recognition accuracy ◮ Noisy, information source Stephan Sigg July 23, 2017 14 / 42

Device-Free recognition (DFL / DFAR) Time domain features – Situation awareness Frequency domain features – Gesture recognition Fresnel effects DFAR on COTS hardware Stephan Sigg July 23, 2017 15 / 42

Aspects of the mobile radio channel relative speed between transmitter and receiver (v) Transmit node signal propagation α Movement direction Receive node Doppler Shift ◮ Frequency of received and transmitted signal may differ ◮ Dependent on relative speed between transmitter and receiver ◮ f d = v λ · cos ( α ) Stephan Sigg July 23, 2017 16 / 42

Whole-Home Gesture Recognition Using Wireless Signals, Q. Pu, S. Gupta, S. Gollakota, S. Patel, Mobicom’13 See Through Walls with Wi-Fi!, F Adib, D. Katabi, SIGCOMM’13 Stephan Sigg July 23, 2017 17 / 42

Micro doppler variations See Through Walls with Wi-Fi!, F Adib, D. Katabi, SIGCOMM’13 Stephan Sigg July 23, 2017 18 / 42

Micro doppler variations ◮ Recognition of fine-grained gestures ◮ Potentially directional recognition from multiple sources simultaneously ◮ Binary information (towards/away) ◮ Potentially also speed but noisy ◮ Accuracy dependent on direction of movement (towards Antenna) ◮ Requires non-standard hardware (e.g. software radios) Stephan Sigg July 23, 2017 19 / 42

Device-Free recognition (DFL / DFAR) Time domain features – Situation awareness Frequency domain features – Gesture recognition Fresnel effects DFAR on COTS hardware Stephan Sigg July 23, 2017 20 / 42

Stephan Sigg July 23, 2017 21 / 42

Human Respiration Detection with Commodity WiFi Devices: Do User Location and Body Orientation Matter?, Wang et al., Ubicomp 2016 Stephan Sigg July 23, 2017 22 / 42

. Human Respiration Detection with Commodity WiFi Devices: Do User Location and Body Orientation Matter?, Wang et al., Ubicomp 2016 Stephan Sigg July 23, 2017 23 / 42

Fresnel effets for DFAR ◮ Fine-grained centimer-scale accuracy ◮ Fragile instrumentation requirements ◮ Requires non-standard hardware (e.g. software radios) Stephan Sigg July 23, 2017 24 / 42

Device-Free recognition (DFL / DFAR) Time domain features – Situation awareness Frequency domain features – Gesture recognition Fresnel effects DFAR on COTS hardware Stephan Sigg July 23, 2017 25 / 42

Can we do this with standard hardware? RSSI Passive Stephan Sigg July 23, 2017 26 / 42

Measure signal strength on a phone ◮ Approx. 1 sample/sec Stephan Sigg July 23, 2017 27 / 42

Measure signal strength on a phone ◮ How to obtain this data on a phone? ◮ root access ◮ Firmware does not support such access Stephan Sigg July 23, 2017 27 / 42

Measure signal strength on a phone Stephan Sigg July 23, 2017 27 / 42

Measure signal strength on a phone Stephan Sigg July 23, 2017 27 / 42

Measure signal strength on a phone Stephan Sigg July 23, 2017 27 / 42

Measure signal strength on a phone Capturing Post processing Processing orange data raw data multidimensional mining toolkit .tab data points Radio grouped by sender signal Data Point Sender video analysis - timespan .pic video Sample - feature 1 kle - feature 2, ... matplotlib + tcpdump .pdf - windowing interactive .pcap raw data still access- receiver - feature calculation plot ible for visualisation .png ◮ http://www.stephansigg.de/DeviceFree/pcapTools.tar.gz Stephan Sigg July 23, 2017 27 / 42

Sampled RSSI over time RSSI samples over time −86 −87 −88 −89 RSSI [dBm] −90 −91 −92 −93 −94 −95 81.4 81.6 81.8 82 82.2 82.4 82.6 82.8 83 83.2 83.4 Time [seconds] ◮ Only use simple time-domain features ◮ Pre-processing? Stephan Sigg July 23, 2017 28 / 42

Which sample rate can we expect? University (ETH) Train station Packets/sec Packets/sec 1 10.45 g r o u n d 3 r d fl . 2 9.18 1 34.06 14.88 2 0.61 3 9.01 5.02 channel 4 21.91 3 4.45 0.19 channel 5 23.70 4 0.10 0.04 6 22.31 5 53.97 192.29 7 21.34 6 0.14 0.04 21.58 7 0.28 0.03 8 9 0.55 8 0.11 0.07 9 10 0.62 26.46 2.88 11 14.51 10 0.06 0.09 11 0.05 0.02 Café in center Dormitory Packets/sec Packets/sec 1 15.29 2 8.86 1 10.28 3 11.06 2 10.03 channel 4 1.41 3 12.13 channel 5 2.15 4 9.92 6 10.99 5 9.30 7 4.45 6 1.77 8 1.23 7 0.09 9 11.09 8 0.19 10 10.79 9 6.92 11 23.30 10 0.47 11 0.36 Suburban fl at Packets/sec 1 0.85 2 0.35 3 0.32 channel 4 0.20 5 0.85 6 3.10 7 2.59 8 11.85 9 4.46 2.05 10 11 9.61 Stephan Sigg July 23, 2017 29 / 42

Case studies Stephan Sigg July 23, 2017 30 / 42

Results Stephan Sigg July 23, 2017 31 / 42

Abdelnasser et. al: WiGest: A Ubiquitous WiFi-based Gesture Recognition System, INFOCOM, 2015 Stephan Sigg July 23, 2017 32 / 42

Abdelnasser et. al: WiGest: A Ubiquitous WiFi-based Gesture Recognition System, INFOCOM, 2015 Stephan Sigg July 23, 2017 33 / 42

RSSI-based ◮ COTS hardware ◮ Ubiquitously available ◮ low accuracy ◮ dependent on environmental traffic situation Stephan Sigg July 23, 2017 34 / 42

CSI-based DFAR Stephan Sigg July 23, 2017 35 / 42

The received vector y is expressed in terms of the channel transmission matrix H , the input vector x and noise vector n as y = Hx + n Stephan Sigg July 23, 2017 36 / 42

802.11n – CSI The CSI matrix The MIMO control field in the 802.11n Management frame (used to manage the exchange of MIMO channel state or transmit beamforming feedback information) contains a CSI cotrol field in which the CSI matrix for all carriers is stored. Example (3x3) – complex amplitude and phase: Stephan Sigg July 23, 2017 37 / 42

Open CSI tools Atheros CSI tool http://pdcc.ntu.edu.sg/wands/Atheros/ Intel 5300 tool https://dhalperi.github.io/linux-80211n-csitool/ Stephan Sigg July 23, 2017 38 / 42

CSI-based gait recognition Wang et. al: WiGest: Gait Recognition Using WiFi Signals, Ubicomp, 2016 Stephan Sigg July 23, 2017 39 / 42

CSI-based ◮ CSI phase fine-grained recognition of movement ◮ Available from COTS hardware ◮ Binary information ◮ Constant after change in distance conducted ◮ Recognition accuracy dependent on direction of movement wrt Rx antenna Stephan Sigg July 23, 2017 40 / 42

Device-Free recognition (DFL / DFAR) Time domain features – Situation awareness Frequency domain features – Gesture recognition Fresnel effects DFAR on COTS hardware Stephan Sigg July 23, 2017 41 / 42

Thank you! Stephan Sigg stephan.sigg@aalto.fi Stephan Sigg July 23, 2017 42 / 42