sparkmap
play

Sparkmap Marsil Zakour, Sebastian Schlegel, Vladimir Yugay - PowerPoint PPT Presentation

Feb 07, 2024 •124 likes •919 views

Sparkmap Marsil Zakour, Sebastian Schlegel, Vladimir Yugay Technical University of Munich Department of Mathematics Data Innovation Lab Garching, 18. February 2020 Agenda Introduction & Objectives Related work Dataset

  1. Sparkmap Marsil Zakour, Sebastian Schlegel, Vladimir Yugay Technical University of Munich Department of Mathematics Data Innovation Lab Garching, 18. February 2020

  2. Agenda • Introduction & Objectives • Related work • Dataset • "Feature Vector" Approach • Solution Segmentation Network o Room Proposal Extraction o Algorithm for placing icons o • Results • Conclusion & limitations

  3. Introduction & Objectives For a lot of simple architectural tasks a person is needed One reason: Used data in the form of rasterized floorplan images Living room Bed- room Kitchen Bath Sebastian Schlegel 3

  4. Introduction & Objectives For a lot of simple architectural tasks a person is needed One reason: Used data in the form of rasterized floorplan images Living room Bed- room Goal of this Project: To place icons (furniture, facilities) on an "empty" floorplan Kitchen Bath Sebastian Schlegel 4

  5. Related work – Liu et al. 2017 A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach: CNN Sebastian Schlegel 5

  6. Related work – Liu et al. 2017 A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach: • Extracting geometric and semantic information independently • M geometries CNN semantics Sebastian Schlegel 6

  7. Related work – Liu et al. 2017 A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach: • Extracting geometric and semantic information independently • Merging both rule-based using Integer Programming • M Integer Programming geometries CNN semantics Sebastian Schlegel 7

  8. Related work – Liu et al. 2017 Geometric information is represented through "junction points" • 4 different junctions for "openings" (windows, doors): Sebastian Schlegel 8

  9. Related work – Liu et al. 2017 Geometric information is represented through "junction points" • 4 different junctions for "openings" (windows, doors): • 4 different junctions for icons: Sebastian Schlegel 9

  10. Related work – Liu et al. 2017 Geometric information is represented through "junction points" • 4 different junctions for "openings" (windows, doors): • 4 different junctions for icons: • 13 different junctions for walls: In total: 21 different junction types Sebastian Schlegel 10

  11. Related work – Liu et al. 2017 Network (modified ResNet 152) output: Geometric information: • 21 heatmaps – one regressed for every junction type Sebastian Schlegel 11

  12. Related work – Liu et al. 2017 Network (modified ResNet 152) output: Geometric information: • 21 heatmaps – one regressed for every junction type Semantic information: • One per-pixel classification for room types Sebastian Schlegel 12

  13. Related work – Liu et al. 2017 Network (modified ResNet 152) output: Geometric information: • 21 heatmaps – one regressed for every junction type Semantic information: • One per-pixel classification for room types • One per-pixel classification for icons, windows & doors Sebastian Schlegel 13

  14. Related work – Kalervo et al. 2019 ("CubiCasa") Modifying the approach of Liu et al. Differences: • Applying automatic weighting to the multi-task CNN of Liu et al. • Instead of Integer Programming a heuristic is used for merging geometries Heuristic CNN semantics Sebastian Schlegel 14

  15. Related work – Kalervo et al. 2019 ("CubiCasa") Merging geometric and semantic information • Split floorplan in a grid of rectangular cells using triplets of Junction points Sebastian Schlegel 15

  16. Related work – Kalervo et al. 2019 ("CubiCasa") Merging geometric and semantic information • Split floorplan in a grid of rectangular cells using triplets of Junction points • Label cells applying pixel-wise maximum voting Sebastian Schlegel 16

  17. Related work – Kalervo et al. 2019 ("CubiCasa") Merging geometric and semantic information • Split floorplan in a grid of rectangular cells using triplets of Junction points • Label cells applying pixel-wise maximum voting • Merge cells with the same label if no separating wall is in between Sebastian Schlegel 17

  18. Agenda • Introduction & Objectives • Related work • Dataset • "Feature Vector" Approach • Solution Segmentation Network o Room Proposal Extraction o Algorithm for placing icons o • Results • Conclusion & limitations

  19. CubiCasa Dataset Model Vladimir Yugay 19

  20. CubiCasa input Vladimir Yugay 20

  21. CubiCasa model generalization Model Vladimir Yugay 21

  22. CubiCasa model evaluation Vladimir Yugay 22

  23. CubiCasa label Vladimir Yugay 23

  24. Input generation Vladimir Yugay 24

  25. Label generation Vladimir Yugay 25

  26. Input generation Vladimir Yugay 26

  27. Input generation Vladimir Yugay 27

  28. Dataset randomization Each image has a random texture • Each text annotation has a random font size, weight and family • Each text annotation is rotated from 0 to 11 degrees • Vladimir Yugay 28

  29. Dataset statistics Number of room types were reduced from 65 to 8 • Only rooms with kitchen and bathroom were kept • In total 1881 images split in 1281/300/300 as train/validation/test • Vladimir Yugay 29

  30. Feature Vector Approach – Reusing Trained Model • Geometrical features (i.e. walls) are invariant to text removal • Detect Rooms, and Find their types using walls. Type IOU (Text) IOU (No Text) Wall 65.5% 64% Kitchen 63.9% 50.6% Living Room 73.1% 24.8% Marsil Zakour 30

  31. Feature Vector Approach – Building a Graph Marsil Zakour 31

  32. Feature Vector Approach – Features • What Set of nodes makes a room. • Wall Features: • Geometrical • Relational • Room Features: • Aggregation/summary of walls features. • Or better use Graph Convolutions Wall features height width # rooms # walls Marsil Zakour 32

  33. Feature Vector Approach - Stop Why we Stopped: • Rooms are not only cycles • More support for segmentation than for GCNs Marsil Zakour 33

  34. Agenda • Introduction & Objectives • Related work • Dataset • "Feature Vector" Approach • Solution Segmentation Network o Room Proposal Extraction o Algorithm for placing icons o • Results • Conclusion & limitations

  35. Solution Room type segmentation Classified polygons Icon Room placement polygons Openings information Vladimir Yugay 35

  36. Segmentation network details U-Net architecture • ResNet as a backbone • 25 millions of parameters • Dice loss • Vladimir Yugay 36

  37. Segmentation network Segmentation Prediction Label Network tensor tensor Vladimir Yugay 37

  38. Room type segmentation Predicted image Label image Input image Vladimir Yugay 38

  39. Openings segmentation Input image Predicted image Label image Vladimir Yugay 39

  40. Room Proposals – New Formulation Input Proposals Marsil Zakour 40

  41. Room Proposals - Training Marsil Zakour 41

  42. Room Proposals – Qualitative Results Labels Predictions Inputs Room Proposal Qualitative Results Marsil Zakour 42

  43. Predictions Fusion - Input Room Proposal input Room Type Marsil Zakour 43

  44. Predictions Fusion - Voting and Inpaint Align Vote Inpaint Marsil Zakour 44

  45. Predictions Fusion - Extract Polygons Polygons types Extract Polygons Marsil Zakour 45

  46. Predictions Fusion - Extract Polygon cont. (5,5) (0,0) - (5,5) 1,1,… 1 X > 3 X <= 3 2,2,… (3,0) - (5,5) (0,0) - (3,5) 1 y 2 y > 4 y <= 4 (3,0) - (3,4) (3,4) - (5,5) (0,0) x Marsil Zakour 46

  47. Predictions Fusion - Example Marsil Zakour 47

  48. Algorithm for placing icons The task of placing icons is rule-based : • "don´t place an icon in front of a door" Living room Bed- • "place a bath tube in a bath" room • … Kitchen Bath Sebastian Schlegel 48

  49. Algorithm for placing icons The task of placing icons is rule-based : • "don´t place an icon in front of a door" Living room Bed- • "place a bath tube in a bath" room • … "creative" (inconclusive) : • There is not one optimal solution Kitchen • Multiple constellations are possible • "Best choice" can depend on "taste" of a person Bath Sebastian Schlegel 49

  50. Algorithm for placing icons Information used for placing icons: door • Geometry of a room window • Type of a room • Location of windows and doors bedside table bed Sebastian Schlegel 50

  51. Algorithm for placing icons Information used for placing icons: • Distance from wall • If pixel in corner • Dist. From window • Dist. from door Sebastian Schlegel 51

Recommend

More recommend