Modelling and Synthesis of User Interfaces for Complex, Web-Based - PowerPoint PPT Presentation

Modelling and Synthesis of User Interfaces for Complex, Web-Based Modelling Environments Jacob Beard McGill University Modelling, Simulation and Design Lab (MSDL) 08/29/08

Overall Project Goal  The creation of a web-based meta- modelling environment for synthesizing domain- specific, web-based modelling environments.  Enable language- based multi-formalism modelling.

End-to-End Modelling of UI  Topical  Dave Raggett, W3C incubator group  Useful  Complex behaviour described well by statecharts  Meta-modelling allows easy testing and automation  Important concepts  Layering of Concrete Syntax, Abstract Syntax, and Semantics  Nested and Scoped UI  Statecharts for behaviour

Meta-Modelling the UI Two-Way Communication Between Each Layer HSMCanvas <<Abstract>> DChart Model Attributes: - name :: String Multiplicities: - To CanvasContains: 0 to N serviceLayoutRequest+ T: <layoutRequest> New Arrow Composite Basic CanvasContains Multiplicities: - To Toolbar: 0 to N Rollback* T: <ButtonPress-3> - From HSMCanvas: 0 to N - To PseudoCanvas: 0 to N T: [Done] Create DChart Entity+ T: <layoutRequest> T: [Done] Toolbar <<Abstract>> T: <Control-ButtonPress-3> Attributes: PseudoCanvas <<Abstract>> T: <Any-Motion> - name :: String Attributes: T: [Done] T: [Done] Multiplicities: - name :: String Drop Point* - From CanvasContains: 0 to N Multiplicities: - To ToolbarContains: 0 to N - From CanvasContains: 0 to N T: <Control-ButtonPress-1> - To PseudoCanvasContains: 0 to N T: (create)* T: <ButtonPress-1> Snap Points* Default Idle ToolbarContains T: <KeyPress-space> Multiplicities: PseudoCanvasContains - To HSMCompositeBtn: 0 to N Multiplicities: T: Reset - From Toolbar: 0 to N - To HSMComposite: 0 to N C_0 - To HSMBasicBtn: 0 to N - From PseudoCanvas: 0 to N T: <ButtonPress-1> T: <KeyPress-space> - To HSMBasic: 0 to N HSMCompositeBtn <<Abstract>> HSMBasicBtn <<Abstract>> T: <InteractionSelect>* Attributes: Attributes: T: <ButtonPress-1> - name :: String - name :: String HSMComposite <<Abstract>> HSMBasic <<Abstract>> Multiplicities: Multiplicities: Attributes: Attributes: - From ToolbarContains: 0 to N - From ToolbarContains: 0 to N HSMCompositeContains - name :: String - name :: String Multiplicities: Multiplicities: Drop Point2* Multiplicities: - To HSMBasic: 0 to N - From PseudoCanvasContains: 0 to N T: <Arrow Created> T: [Done] - From PseudoCanvasContains: 0 to N No Snap* - From HSMComposite: 0 to N - From HSMCompositeContains: 0 to N - To HSMCompositeContains: 0 to N - To HSMComposite: 0 to N - From HSMHyperEdge: 0 to N - To HSMHyperEdge: 0 to N B_0 - From HSMHyperEdge: 0 to N - To HSMHyperEdge: 0 to N - From HSMCompositeContains: 0 to N T: [Done] T: <Any-Motion> T: <ButtonPress-3> HSMHyperEdge Attributes: - name :: String - trigger :: String Rollback2* - action :: String Multiplicities: - To HSMBasic: 0 to N - From HSMComposite: 0 to N - To HSMComposite: 0 to N - From HSMBasic: 0 to N End UI Concrete Abstract Semantics Syntax Syntax Each Layer Can Itself Be Metamodelled ... Concrete Syntax Abstract Syntax Semantics

Benefits of this Architecture  Testing  At each layer, events can be recorded, later on played back within testing harness.  Allows headless automated testing of UI.  Each layer can be simulated, thus automated.  Collaboration  Centralized server = easy sync  Easy to hook multiple terminals to the same graphics server: many-to-one relationship

Nested and Scoped UI Buttons Formalism HSM Formalism Nested Buttons Formalism DCharts Formalism Nested Bounding Box Formalism

Statecharts for Behaviour DChart Model Selected Mode* Rotate Mode T: mousemove A: this.newEvent = this.params[0] T: mousedown this.rDelta = this.computeRDelta(this.eventStamp,this.newEvent); G: this.isTypeOfTRotationHandle(this.params[0].target) this.rotate(this.rDelta,this.cachedCenterPoint.x,this.cachedCenterPoint.y); A: this.stampFirst(this.params[0]) this.stampEvent(this.newEvent) this.params[0].stopPropagation(); this.updateTracker() //TODO: saturate on this Rotating T: mouseup Selected_R* Before Mode is Chosen T: mouseup A: this.parent.event('OBJ_FINISHED_TRANSFORM',[this]) T: mouseup A: this.tDelta = this.computeTDelta(this.firstEvent,this.eventStamp); T: mouseup Translating_R this.cacheCenterPoint() this.parent.event('OBJ_FINISHED_TRANSFORM',[this]) T: mousemove T: mousedown A: this.newEvent = this.params[0] G: this.isTypeOfTRect(this.params[0].target) Ready to Translate_R this.tDelta = this.computeTDelta(this.eventStamp,this.newEvent); A: this.params[0].stopPropagation(); T: mousemove this.translate(this.tDelta); A: this.stampFirst(this.params[0]) this.stampEvent(this.newEvent) this.updateTracker() //TODO: saturate on this T: mousemove T: mousedown A: this.stampFirst(this.params[0]) G: this.isTypeOfTRect(this.params[0].target) A: this.params[0].stopPropagation(); * H Scale Mode T: mousemove T: mousedown A: this.newEvent = this.params[0] G: this.isTypeOfTScaleHandle(this.params[0].target) this.scale(this.eventStamp,this.newEvent); A: this.stampFirst(this.params[0]) T: CANVAS_DESELECT this.stampEvent(this.newEvent) this.params[0].stopPropagation(); this.updateTracker() //TODO: saturate on this T: mouseup Scaling T: mouseup Selected_S* A: this.cacheCenterPoint() Idle* this.parent.event('OBJ_FINISHED_TRANSFORM',[this]) T: mousedown T: mouseup G: this.isTypeOfTCanvas(this.params[0].target) A: this.tDelta = this.computeTDelta(this.firstEvent,this.eventStamp); T: (create) this.cacheCenterPoint() T: mousedown A: this.cacheCenterPoint() this.parent.event('OBJ_FINISHED_TRANSFORM',[this]) G: this.isTypeOfTRect(this.params[0].target) A: this.params[0].stopPropagation(); Translating_S Ready to Translate_S Default T: mousemove T: mousemove A: this.newEvent = this.params[0] A: this.stampFirst(this.params[0]) this.tDelta = this.computeTDelta(this.eventStamp,this.newEvent); this.translate(this.tDelta); this.stampEvent(this.newEvent) this.updateTracker() //TODO: saturate on this

How This Comes Together in UI  Hierarchical State Machine Demo

Technical Challenges  What platform to use for web vector graphics? SVG/VML? Flash? Silverlight?  Libraries needed for complex, OO development with ECMAScript → Dojo  Improving performance.  Optimizing Statechart-to-ECMAScript compilation.  Saturating high-fidelity operations.  Other optimizations.  Determining best practices for development

Thank you for attending!