iType: Using Eye Gaze to Enhance Typing Privacy Zhenjiang Li 1 , Mo Li 2 , Prasant Mohapatra 3 , Jinsong Han 4 , Shuaiyu Chen 4 CityU 1 , NTU 2 , UC Davis 3 , XJTU 4
Wearables • Accelerometers • Gyroscope • Ambient light sensor • Hart rate sensor • Magnetometer • GPS • … Extend beyond timing daily life , e.g., fitness, exercise, business, etc. [1] https://www.iphones.ru/wp-content/uploads/2015/05/main.jpg
However Explicitly typing sensitive info. Continuously sense hand moves • • Password Accelerometers • • Personal data Gyroscope • • …. Security code • ….
Wait a moment … • Touch ID • But Account login Security code POS terminal Call support Explicit Textual-Input is unavoidable
Our idea for protection • Eye gaze for input • Front camera ***** • Secure • Back • A keyboard • Front • Difficult to distinguish • Keyboard layout may change
iType framework 3. Noises from device motions Gaze Engine Video Stream Gaze Tracker Frame Selector Front Camera iType Engine iType Group Centroid Estimator Keystroke Button xxxx _ Selector Detector Transitional Gaze Accelerometers 1 2 3 Remover 1. Unreliable mobile 4 5 6 7 8 9 gaze tracking Typing Error Virtual Flying Enhance 0 / ← Corrector Button Button Layer Joint Decoder Keyboard Password Assembler Rearranger 2. Lack of true text-entry value in error correction
Unreliable mobile gaze tracking • Problem statement: Gaze tracker training [2]: Output: gaze Input: image coordinates [2] “ ishadow: design of a wearable, real-time mobile gaze tracker”, in Proc. of ACM MobiSys, 2014.
Unreliable mobile gaze tracking • Problem statement: For mobile devices: Training
Unreliable mobile gaze tracking • Accuracy we need? Smartphone Tablet Error (degree) Error (degree)
Unreliable mobile gaze tracking ( a ) • Observations x x -axis ( b ) y -axis y Unreliable tracking
Unreliable mobile gaze tracking • Formal description Less Min. samples to achieve certain confidence? More • Solution overview ( n gaze points) At least (1- alpha)
Keystroke detection • When to start? Different! k = 4 Window size w = 12 G 1 1 2 3 4 5 6 7 8 k = 4 G 2 9 10 11 12 13 14 15 16 G 3 (b) (a) • KL divergence
Keystroke detection k = 4 Window size w = 12 • When to start: G 1 1 2 3 4 5 6 7 8 k = 4 • KL divergence G 2 9 10 11 12 13 14 15 16 G 3 (b) (a) • When to stop: • Approximation
Other modules 1 2 3 1 2 3 8 5 0 c 2 c 1 , • Input error correction 4 5 6 4 5 6 7 3 ,, 1 c 2 c 2 7 8 9 7 8 9 4 6 9 • Joint decoding 0 0 2 / ← / / ← ← (b) (c) (a) • Frame selection • Sensor-assisted
Evaluation • Overall performance Individual keystroke: • Accuracy • Static: 97% • Dynamic: 89% • Latency • Static: 2.0s • Dynamic: 2.6s
Takeaways 1. On-going trend a) More & powerful sensors 2. Dual aspects a) Beneficial to usage b) Potential privacy issue iType xxxx _ 1 2 3 3. Challenges for iType 4 5 6 7 8 9 0 / ← a) Unreliable mobile gazing b) Unknown ground truth c) Device motions
Recommend
More recommend
