Cross-Device Interfaces: Existing Research, Current Tools, Outlook Michael Nebeling nebeling@umich.edu http://mi2lab.com
https://xduitutorial.wordpress.com/
Who Am I? Assistant Professor University of Michigan School of Information
Michigan Information Interaction Lab http://mi2lab.com
Who are you? (Name, affiliation, your most recent paper )
Tutorial Overview It’s a journey from single-device, to multi-device, to cross-device interfaces Mix of research and practical lessons on designing, programming, and testing cross-device interfaces Goal : understand the requirements and enabling techniques to support cross-device interfaces Challenges and requirements Design dimensions Concepts and basic implementation
Why do we want cross- device interfaces? (Applications, scenarios, settings, …)
Applications / Scenarios / Settings
Multi-Device Environments http://idea-garden.org/
Multi-Device Environments
Multi- Device Environments … Rooms (office, home, …) The User Equipped with various Nowadays carrying and devices and sensors to wearing multiple devices track users and their E.g., smartwatch, activities to support tasks smartphone, tablet E.g., Aura (CMU), iRoom Needs new techniques for (Stanford), WILD Room tracking users and their (INRIA), NiCE Discussion activities on the devices Room (Hagenberg) themselves Supported techniques are Existing techniques relying scalable with more sensors on external devices and sensors don’t apply
Multi-Device Use Multi-Device Use Serial Use Parallel Use Resource Related Unrelated Lending Parallel Use Parallel Use Tero Jokela, Jarno Ojala, Thomas Olsson: A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks ( CHI ’15)
Challenge 1: Multi-Device Interfaces
Challenge 2: Cross-Device Interaction
What are cross-device interfaces? (How do they differ from other kinds of interfaces? Characteristics, requirements, …)
Multi-Device Interfaces adapted to device in use E.g., layout based on viewing size a popular technique is responsive design using fluid grid layout flexible images media queries to determine device characteristics
Multi-Device Design Approaches Graceful Degradation Interface A Interface B Mobile-first / Progressive Enhancement Interface A Interface B Responsive / Adaptive Design Interface A Interface B
Cross-Device Interfaces distributed and synchronized between multiple devices support multiple interactive devices used sequentially or in parallel to support the same task support multiple users working on the same task
Cross-Device Interaction Cross- device interaction can be useful for… 1) combining devices for more screen real estate 2) making better use of devices with limited input/output 3) using one device to control the others remotely …
Issues Nowadays, many activities span multiple devices But : multi-device workflows have to be planned Two primary design issues 1) information and actions are tied to the device rather than the user 2) devices are not aware of other devices and their role in the user's larger task
Cross-device Distribution Adaptation Multi-device
What can be adapted? Presentation the perceivable aspects (including media and interaction techniques choice, layout, graphical attributes, …) Navigation (or “dynamic behavior”) including navigation structure, dynamic activation and deactivation of interaction techniques, … Content including texts, labels, images Various combinations of adaptation strategies are possible conservation, rearrangement, increase, reduction, simplification, magnification, …
What can be distributed? Input redirection of keyboard, mouse, touch, gesture, and voice input Output display or content redirection of text, graphics, audio, video Platform interface execution is distributed across different platforms (i.e., architectures, operating systems, networks, etc.) Space interface is distributed physically, or geographically, e.g., co- located or remote interactive spaces Time interface elements execute simultaneously (synchronously), or distributed in time (asynchronously) N. Elmqvist: Distributed user interfaces: State of the art. Distributed User Interfaces 2011
What can be synchronized? Input Events clicks (keypress, mouse, tap) input data changes (text fields, checkboxes, …) scrolling, zooming, … Output Events interface element focus interface manipulation (e.g., new element) interface redirection (different screen) Other Events device connections shared variables …
How do you program cross-device interfaces? (Frameworks, languages, tools you employ)
Frameworks, Languages, Tools
How We Program Interfaces
How We Will Program Interfaces
Paterno and Santoro ( EICS ’12)
Design Dimensions/Challenges Distribution, e.g. static vs. dynamic Migration, e.g. cross-platform transitions, task migration, data synchronization when switching between devices Granularity: manipulation of single/groups/all interface elements across various devices to support partial/total migration Trigger: system vs. user (or mixed); push vs. pull, depends on whether local or remote device is trigger Sharing: single vs. multi-user, sharing by moving vs. sharing by interaction Paternò and Santoro, “A Logical Framework for Multi -Device User Interfaces ” (EICS’12)
Design Dimensions/Challenges … Timing, e.g. immediate vs. deferred effect Modalities, i.e. mono vs. trans vs. multi-modality Generation, i.e. design-time vs. run-time Adaptation, e.g. scaling, transducing or transforming Architecture, e.g. client/server vs. peer-to-peer Paternò and Santoro, “A Logical Framework for Multi -Device User Interfaces ” (EICS’12)
Cross-Device Programming User Interface Design (Panelrama, XDStudio, Weave, XDBrowser, etc.) Application Development (HydraScope, MultiMasher, WatchConnect, etc.) Data Management (Firebase/Meteor/PubNub, XDSession, etc.)
What are hot topics in cross-device research? (Papers you’ve read, or papers you’d like to write )
Research Topics
Overview of XD Research On Cross-Device Working Need for sequential and parallel use of devices On Cross-Device Interaction Techniques Design space of cross-device interfaces is large On Cross-Device Interface Programming Many tools for prototyping, development, and testing
Cross-Device Cross-Device Working Techniques Cross-Device Anything Else Programming
GroupTogether (UIST’12) Oulasvirta&Sumari (CHI’07) Schmidt et al. (DIS’12) Dearman&Pierce (CHI’08) Seyed (ITS’12) Santosa&Wigdor (Ubicomp’13) Cross-Device Cross-Device Chen et al. (CHI’13) Kane et al. (INTERACT’09) Working Techniques Duet (CHI’14) Karlson et al. (CHI’10) Conductor (CHI’15) Jokela et al. (CHI’15) Raedle et al. (CHI’15) XDStudio (CHI’14) Damask (CHI’08) Panelrama (CHI’14) Gummy (AVI’08) XDKinect (EICS’14) Highlight (UIST’08) Cross-Device Fisher et al. (IJHCS’14) HuddleLamp (ITS’14) Anything Else Frosini&Paterno (EICS’14) Programming MultiMasher (WISE’14) Weave (CHI’15) XDBrowser (CHI’16) WatchConnect (CHI’15)
Audrey Sanctorum, Beat Signer: “Towards User -defined Cross- Device Interaction” (DUI’16)
Audrey Sanctorum, Beat Signer: “Towards User -defined Cross- Device Interaction” (DUI’16)
Family of Cross-Device Systems Studio [CHI’14] Kinect [EICS’14] Session [EICS’15] Browser [CHI’16]
Family of Cross-Device Systems Studio [CHI’14] Kinect [EICS’14] Session [EICS’15] Browser [CHI’16]
Recent Frameworks and Tools
XDBrowser M. Nebeling, A.K. Dey: XDBrowser: User-Defined Cross-Device Web Page Designs In Proc. CHI’16
XDBrowser M. Nebeling, A.K. Dey: XDBrowser: User-Defined Cross-Device Web Page Designs In Proc. CHI’16
XDBrowser Demo
Interfaces and Tasks Mail Media Inbox Compose Maps Article Slides 15 non-technical end -users 144 desirable multi-device designs
6 Cross-Device Interface Patterns (1) optimize for screen space (2) optimize for input (3) minimize device switching
6 Cross-Device Interface Patterns Async Patterns Move and Sync Patterns Copy and Sync Patterns
XDBrowser 2.0
XDAndroid …
New (Forms of) Devices (smartwatches/eye-wear, hybrid vs. shapeshifting vs. modular devices) Outlook Better Native Support More Conductors (prototyping, programming, (more cross-device debugging, etc.) designers & developers)
Recommend
More recommend
