My Research - The Problem: Computer Science Some people with Severe - PDF document

9/11/2012 Adaptable Interfaces for People with Disabilities John Magee Boston University / Clark University Computer Science Computer Science My Research - The Problem: Computer Science Some people with  Severe cerebral palsy  Traumatic brain injury  Stroke  Multiple sclerosis (MS)  ALS (Lou Gehrig’s) cannot communicate with traditional means:  Nonverbal  Limited voluntary motion Lack of communication ability ≠ lack of active minds! Communication technology: not available, or too expensive, inefficient, difficult, complex, etc. 1

9/11/2012 Talk Outline Computer Science  Observations and user studies with people with disabilities.  HAIL – Hierarchical Adaptive Interface Layout  Adaptive Mouse Control Functions  Study of automatic generation of online social networking messages. Team - Collaborators Computer Science Margrit Betke at Boston University Jim Gips at Boston College Many others, undergraduates and grad students: O. Al-Hinai, W. Akram, M. Chau, R. Cloud, C. Connor, S. Crampton, S. Epstein, C. Fagiani, I. Fedyuk, P. Fleming, M. Gorman, O. Gusyatin, K. Grauman, W.-B. Kim, C. Kwan, J. Lombardi, E. Missimer, W. Mullally, L. Tiberii, M. Urinson, M. Scott, M. Shugrina, B. Waber, E. Yu, Z. Wu Users  The Boston Home, Dorchester, MA Children’s Hospital, Communication Enhancement Center, Boston, MA   Campus School, Boston College  LGF Public Middle School, Boston, MA  McLaughlin Center, Connecticut  Hollybank Home, Yorkshire, UK  Several schools in Eastern Massachusetts & Northern Ireland 2

9/11/2012 Traditional Approaches Computer Science Binary Switch  Touch Switches (blue button)  Hit plate  Wobble stick  Grip handle  Pinch  Pull string  Photocell Switches  Sip or Puff Switches  Voice activated Switches The Camera Mouse Computer Science User with ALS  Camera-based tracking of body to enable control of a mouse pointer  Mouse replacement system: hands free, uses dwell time for clicking  Typical audience: Nonverbal users, people with quadriplegia, people with paralysis, etc Free Download – http://www.cameramouse.org Over 500,000 downloads 3

9/11/2012 The Camera Mouse Computer Science Camera Mouse Computer Science 4

9/11/2012 Observations Computer Science We’ve used educational programs, text entry, games, image editing, web browser…  Every user is different, different abilities  strong need to customize software (more than people without disabilities).  Some programs only work for people who can control pointer precisely. Our study included 12 people with moderate to severe motor disabilities. Applications Computer Science 5

9/11/2012 Two-level Text Entry Computer Science Block Escape Game Computer Science 6

9/11/2012 Some Lessons Learned Computer Science Designing Human-Computer Interface:  Applications need large, strategically placed areas for buttons to reduce the problem of false selection.  “Rest” areas needed (“Midas Touch Problem”) Using Human-Computer Interface:  Choice of tracking method should depend on application used.  Nose is most reliable feature for non-disabled users; allows fast and smooth motion.  Need more customizability. Assistive technology can be social! Challenging to evaluate systems with users with severe disabilities.  Limited session lengths. Inability to repeat many times. High variability . Different Users Computer Science Adults with stable conditions Learning curve, rehabilitation. Children Learning movement abilities. Adults with degenerative conditions: Movement ability may decrease. Should we design an individual interface for each user’s ability at a specific time? 7

9/11/2012 Talk Outline Computer Science  Observations and user studies with people with disabilities.  HAIL – Hierarchical Adaptive Interface Layout  Adaptive Mouse Control Functions  Study of automatic generation of online social networking messages. HAIL Computer Science Need to have hierarchical solution – allow interaction in a more principled way: The application should change and adapt to the user, rather than require the user to adapt to the program. 8

9/11/2012 Web browsing Computer Science Important application for accessing information and communication: But…  Complex layout  Interactive elements mixed with content  Too many buttons, accidental activations  Small links are hard to click Interface Idea Computer Science The configuration of the interface remains a manual setting. 9

9/11/2012 HAILtwitter Computer Science HAILbrowser Computer Science General purpose web browser application.  Basic features:  Select Next/Previous Link.  Follow Selected Link.  Back/Forward.  Page Up/Down.  Advanced features:  Bookmarks.  Text Entry / Keyboard.  Jump 10 Links.  Smart navigation.  Default to mobile web (?) HAILbrowser prototype 10

9/11/2012 Web 2.0 Computer Science Web pages provide both Web 2.0: Database driven web information and presentation: applications. User generated content. data html Application Browser View User’s View Specialized browsers can reinterpret, but the presentation Consumer decides how to present is decided by the provider. the information. Access data through APIs. HAILtwitter Computer Science  Specific-purpose application client for Twitter.  The web browser’s buttons navigate the interactive elements on a web page..  A specific application client can instead directly control interaction elements with the user interface.  Basic features:  Get more / Refresh.  Page Up/Down.  Select Next/Previous Msg.  Retweet (forward msg).  Advanced features:  See other timelines  Text Entry / Keyboard.  Trending topics.  Follow/Unfollow friends. 11

9/11/2012 HAIL – Lessons Learned Computer Science Task: Select left-top, right- top, and bottom-center buttons. - Size of buttons was not all that important. Edges create infinite width. - Some difficulty using dwell time to select buttons. Current and Future Work Computer Science • Add features to current prototype applications. • Continue to test applications with users with disabilities.  Incorporate further feedback. • Make applications publicly available. • Extend to other applications: • Facebook, Google+ • RSS Reader. • Email Client. • SMS Messaging. • Media/Video/Photo Center. • Model abilities for automatic adaptation. 12

9/11/2012 HAIL Conclusions Computer Science All users are different, and their abilities may change over time:  Interfaces should adapt to the user’s abilities, rather than forcing them to conform to a program. HAIL-based applications can address some of the problems faced with traditional programs.  Small buttons, “ midas touch”, different users, changing abilities. Designed for Camera Mouse, but could be used with other mouse replacements. Talk Outline Computer Science  Observations and user studies with people with disabilities.  HAIL – Hierarchical Adaptive Interface Layout  Adaptive Mouse Control Functions  Study of automatic generation of online social networking messages. 13

9/11/2012 User Capability Assumptions Computer Science  Cannot assume people with disabilities will be able to comfortably move their heads in a certain way. Observed in user studies. User Capability Assumptions Computer Science Also observed in interface tests:  Mouse target acquisition tests  Move the mouse to a highlighted target  Once reached, a new target is highlighted  Horizontal movements were difficult  Diagonal movements were easier  Small adjustments easier in diagonal motion Plots by: Emily Yu 14

9/11/2012 User Study Observation Computer Science Computer Science Adaptive Interfaces: The application should change and adapt to the user, rather than require the user to adapt to the program. 15

9/11/2012 Non-Adaptive System Computer Science Adaptive System Computer Science 16

9/11/2012 Mouse Motion Functions Computer Science Experimental Setup Computer Science 17

9/11/2012 Adaptive Function Result Computer Science Click Tester Computer Science Users moves mouse pointer to highlighted circle. Same sequence used for all subjects. 4 people with CP participated. 18

9/11/2012 Interface Tests Computer Science Camera Mouse Case Studies Computer Science User with CP who likes to lay on his couch: “It feels more natural” 19

9/11/2012 Case Studies Computer Science 3 Middle School Students All four were able to complete both EagleAliens and Clicktester tasks with and without adaptive mouse function. Other modalities: Stereo Thermal Video Computer Science 20

9/11/2012 Towards a Multi-Camera Camera Mouse Computer Science How can we improve the Camera Mouse?  We would like to address:  Lost Tracking  Improved mouse pointer response A Multi-Camera System could help:  Redundant input avoids single point of tracking failure.  Stereoscopic information may improve reliability.  3D Feature tracking may give better pointer response. Feature Motion Computer Science 21

9/11/2012 Trajectory Analysis Computer Science Feature Trajectory Computer Science 22