Robust Relational Layout Synthesis from Examples Pavol Bielik , - PowerPoint PPT Presentation

Robust Relational Layout Synthesis from Examples Pavol Bielik , Marc Fischer, Martin Vechev Department of Computer Science ETH Zurich

User Interface Design Today Designer Developer picture created by [Credits: Radu Cristian]

Designer

Materialize GroundworkCSS 2 Developer

User Interface Design Today Designer Developer

Our Goal Tool Designer Developer

Preventing Layout Errors Nexus 4 Alcatel Pop 3

Preventing Layout Errors Nexus 4 Alcatel Pop 3

Preventing Layout Errors Centering Asobimasu Overlaying Views Amigo Out of Screen Candid

Improving Performance Slow Layout #1 Layout #2 Fast Layout #3 [Credits: Sriram Ramani]

Code Porting Linear Layout (horizontal) Linear Layout (vertical) Constraint Layout Relative Layout >98% <2% Old / Deprecated

Automating End-User Design Designer Developer picture created by [Credits: Radu Cristian]

Machine Learning Approach Training </> Layout ✘ Syntactically Incorrect ✘ No Formal Guarantees ✘ Robustness ✘ User Feedback Design Specification

Programming Languages Approach Layout Robustness User Formalization Properties Feedback </> Layout Layout Synthesis Probabilistic List of Devices Model Design Specification

Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A A B margin B.left = A.right + margin

Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A A B margin B.left = A.right + 0 A B A B ✘

Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C margin B.left = A.right + margin A B A B ✘

Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C margin margin B.left = A.right + margin A B A B ✘

Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C m L margin bias = 0.25 B.left = A.right + margin A B A B ✘

Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C m L margin bias = 0.25 B.left = A.right + margin Center B in between right of A and left of C (Dynamic Size) A B A B C margin A B ✘

Relational Layout Constraints left right top View View baseline bottom Align Center of A to Center of B Center B in between right of A and left of C at an Angle + Distance A B C m L A r bias = 0.25 (1-b)*B.left + b*B.right = (1-b)*(A.right + m L ) + b*(C.left - m R ) α B Center B in between right of A and left of C (Dynamic Size) A B C margin B.left + B.right = 2r*sin( α ) + A.left + A.right B.left = A.right + m L ∧ B.right = A.left + m R

Relational Layout Constraints left right top View View baseline bottom Align Center of A to Center of B Center B in between right of A and left of C Formalized 26 types of constraints used by the at an Angle + Distance latest Android Constraint Layout A B C m L A r bias = 0.25 (1-b)*B.left + b*B.right = (1-b)*(A.right + m L ) + b*(C.left - m R ) α B Center B in between right of A and left of C (Dynamic Size) A B C margin B.left + B.right = 2r*sin( α ) + A.left + A.right B.left = A.right + m L ∧ B.right = A.left + m R

Overview Layout Robustness User Formalization Properties Feedback </> Layout Layout Synthesis Probabilistic List of Devices Model Design Specification

Layout Solving Goal Compute absolute position of all the views left right top bottom

Layout Solving Goal Compute absolute position of all the views A.left = ? B.left = ? v : Absolute Horizontal Positions A.right = ? B.right = ? A B 20 40 B.left = A.right + 40 c : Horizontal Position Constraints A.left = screen.left + 20 screen: Target Device screen.left = 0 ∧ screen.right = 240

Layout Solving Goal Compute absolute position of all the views A.left = 20 B.left = ? v : Absolute Horizontal Positions A.right = ? B.right = ? A B 20 40 B.left = A.right + 40 c : Horizontal Position Constraints A.left = 0 + 20 screen: Target Device screen.left = 0 ∧ screen.right = 240

Layout Solving Goal Compute absolute position of all the views A.left = 20 B.left = 140 v : Absolute Horizontal Positions A.right = 100 B.right = 220 A B 20 40 B.left = A.right + 40 c : Horizontal Position Constraints A.left = 0 + 20 screen: Target Device screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 s : Horizontal Size Constraints B.right - B.left = 80 v ⊧ ψ layout (screen, c , s )

Layout Solving vs Layout Synthesis Goal Goal Compute absolute position of all the views Synthesize position and size constraints A.left = ? B.left = ? A.left = 0 B.left = 120 A.right = ? B.right = ? A.right = 80 B.right = 200 A B A B 20 40 20 40 B.left = A.right + 40 ? A.left = screen.left + 20 ? screen.left = 0 ∧ screen.right = 240 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 ? B.right - B.left = 80 ? v ⊧ ψ layout (screen, c , s ) c ∧ s ⊧ ψ syn (screen, v )

Layout Solving vs Layout Synthesis Goal Goal Compute absolute position of all the views Synthesize position and size constraints Approach A.left = ? B.left = ? A.left = 0 B.left = 120 A.right = ? B.right = ? A.right = 80 B.right = 200 1. Encode the synthesis program as a logical formula ψ syn (screen, v ) A B A B 20 40 20 40 2. Find a satisfying assignment to B.left = A.right + 40 ? A.left = screen.left + 20 ? c ∧ s ⊧ ψ syn (screen, v ) screen.left = 0 ∧ screen.right = 240 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 ? B.right - B.left = 80 ? v ⊧ ψ layout (screen, c , s ) c ∧ s ⊧ ψ syn (screen, v )

Layout Synthesis ψ syn (screen, v ) = φ position (screen, v ) ∧ φ constraints ( v ) ∧ φ valid ( v ) ∧ φ acyclic ( v ) Input Constraints Ensure Constraints Specification Encoding are Valid

Layout Synthesis ψ syn (screen, v ) = φ position (screen, v ) ∧ φ constraints ( v ) ∧ φ valid ( v ) ∧ φ acyclic ( v ) Input Constraints Ensure Constraints Specification Encoding are Valid screen.left = 0 ∧ φ position ≝ screen.right = 240 ∧ A.left = 0 ∧ A.right = 80 ∧ B.left = 120 ∧ B.right = 200

Layout Synthesis ψ syn (screen, v ) = φ position (screen, v ) ∧ φ constraints ( v ) ∧ φ valid ( v ) ∧ φ acyclic ( v ) Input Constraints Ensure Constraints Specification Encoding are Valid

Layout Synthesis What are the possible constraints for view B? C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n }

Layout Synthesis What are the possible constraints for view B? C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n } A B B.left = A.right + m L (c 1 )

Layout Synthesis What are the possible constraints for view B? C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n } A B A B B.left = A.right + m L (c 1 ) B.left = screen.right + m L (c 2 )

Layout Synthesis What are the possible constraints for view B? C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n } A B A B B.left = A.right + m L (c 1 ) B.left = screen.right + m L (c 2 ) A B (1-b)*B.left + b*B.right = (1-b)*(A.right + m L ) + b*(screen.right - m R ) (c 3 )

Layout Synthesis What are the possible constraints for view B? O(|v| 2 ) C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n } A B A B B.left = A.right + m L (c 1 ) B.left = screen.right + m L (c 2 ) A B (1-b)*B.left + b*B.right = (1-b)*(A.right + m L ) + b*(screen.right - m R ) (c 3 )

Layout Synthesis What are the possible constraints for view B? O(|v| 2 ) C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n } Select exactly one constraint for each view A B A B B.left = A.right + m L (c 1 ) B.left = screen.right + m L (c 2 ) A B (1-b)*B.left + b*B.right = (1-b)*(A.right + m L ) + b*(screen.right - m R ) (c 3 )

Layout Synthesis What are the possible constraints for view B? O(|v| 2 ) C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n } Select exactly one constraint for each view g 1 ⇒ (B.left = A.right + m L ) (c 1 ) g 2 ⇒ (B.left = A.left + m L ) Boolean variables (c 2 ) denoting whether … constraint was selected g n ⇒ ((1-b)*B.left + b*B.right = … ) (c n )