Exteriors of Residential Buildings Lubin Fan Peter Wonka - PowerPoint PPT Presentation

A Probabilistic Model for Exteriors of Residential Buildings Lubin Fan Peter Wonka

Motivation B A C D E C B D E F A F 2

Goals • Learning • Learning The model can be learned from available remote The model can be learned from available remote sensing data. sensing data. • Sampling • Sampling The model should be generative . The model should be generative . • Hard constraints • Hard constraints The model can handle hard geometric constraints . The model can handle hard geometric constraints . ℎ 𝑥 3

Goals • Learning • Learning Model The model can be learned from available remote The model can be learned from available remote sensing data. sensing data. • Sampling • Sampling The model should be generative . The model should be generative . • Hard constraints • Hard constraints The model can handle hard geometric constraints . The model can handle hard geometric constraints . ℎ 𝑥 4

Challenges • How to represent a building model? • How to represent a building model? 0 Problem: each building has a different number of parameters. Problem: each building has a different number of parameters. • How to design a probabilistic model? • How to design a probabilistic model? Building Problem: training the model from a small dataset Problem: training the model from a small dataset representation • How to generate a building model? • How to generate a building model? Problem: generating a 3D building model from a high dimensional feature vector Problem: generating a 3D building model from a high dimensional feature vector Model 5

Related Work: Procedural Modeling CGA CGA++ Metropolis procedural modeling [Müller et al. 2006] [Schwarz and Müller 2015] [Talton et al. 2011] Learning Sampling Hard constraints 6

Related Work: Inverse Procedural Modeling Symmetry maximization Inverse procedural modeling Bayesian grammar induction Bayesian grammar learning [Zhang et al. 2013] [Wu et al. 2014] [Talton et al. 2011] [ Martinović and Van Gool 2013] Learning Learning Sampling Sampling Hard constraints Hard constraints 7

Related Work: Probabilistic Models Computer-generated building layouts Assembly-based 3D modeling Component-based shape synthesis [Merrel et al. 2010] [Chaudhuri et al. 2011] [Kalogerakis et al. 2012] no label ambiguity interior building model 8

Pipeline Database New buildings … training sampling Model 9

Pipeline Database Parametric representation Building generation training sampling Hierarchical graphical model training sampling Attribute-based Attribute-based representation representation 10

Building Representations A parametric An attribute-based A building model representation representation

A Parametric Building Representation Front Right ℎ Top Back Left 12

An Attribute-based Building Representation 𝑁 𝑑 Mass model attributes 𝑁 𝐝,𝑐𝑐𝑝𝑦 • Size of bounding box • 𝑁 𝐝,𝑐𝑑𝑠 Building coverage ratio • 𝑁 𝐝,𝑐𝑒𝑠𝑧 Boundary complexity • 𝑁 𝐝,𝑔𝑏𝑠 Floor area ratio • 𝑁 𝐝,𝑏𝑠𝑔 Area ratio of floors • 𝑁 𝐝,𝑕𝑏𝑠 Garage attributes • 𝑁 𝐝,𝑞𝑝𝑠 Porch attributes 𝐹 𝑒 𝐺 Element attributes Facade attributes 𝑑 𝐺 • • Roof styles 𝐹 𝑠𝑝𝑝𝑔 Window-to-wall ratio 𝐝,𝑥𝑥𝑠 • • 𝐺 Window styles 𝐹 𝑥𝑜𝑒 Symmetry 𝐝,𝑡𝑧𝑛 • • 𝐹 𝑡𝑞𝑓 Alignment between facade pieces 𝐺 Special building elements 𝐝,𝑏𝑚𝑗𝑕𝑜 13

Probabilistic Modeling Training data Hierarchical graphical model Samples

Design Choices 𝐷 1 𝐸 1 𝑇 𝑇 𝐷 2 𝐸 2 𝐸 𝐼 𝑗 𝐸 𝐷 𝑗 𝑘 𝑘 𝐽 𝐾 𝐸 𝐷 𝑗 𝐷 𝑗 𝑘 𝐾 𝐽 • 𝑇 depends on both • It requires large training data. continuous and discrete variables. • Defining a distance metric is difficult. 15

Hierarchical Graphical Model 𝑇 : the overall style of a building 𝑇 Level 1: discrete variable 𝑁 𝐞,𝑗 and 𝐺 𝐞,𝑘 : the styles of each 𝐺 𝐞,𝑘 𝑁 𝐞,𝑗 𝐹 𝐞,𝑙 Level 2: discrete variables attribute, respectively 𝐺 𝐝,𝑘 𝑁 𝐝,𝑗 Level 3: continuous variables 𝐽 𝐾 𝐿 mass facade element model variables variables 16

Hierarchical Graphical Model 𝑇 𝑁 𝐝,𝑐𝑒𝑠𝑧 𝐺 𝐞,𝑘 𝑁 𝐞,𝑗 𝐹 𝐞,𝑙 𝑁 𝐝,𝑐𝑑𝑠 𝑁 𝐝,𝑐𝑐𝑝𝑦 lateral edges 𝑁 𝐝,𝑔𝑏𝑠 𝑁 𝐝,𝑞𝑝𝑠 𝐺 𝐝,𝑘 𝑁 𝐝,𝑗 𝑁 𝐝,𝑏𝑠𝑔 𝑁 𝐝,𝑕𝑏𝑠 𝐽 𝐾 𝐿 𝑄 𝑪 = 𝑄 𝑇 𝑄 𝑁 𝐞,𝑗 |𝑇 𝑄 𝑁 𝐝,𝑗 |𝑁 𝐞,𝑗 , 𝜌 𝑁 𝐝,𝑗 ∙ 𝑗 𝑄 𝐺 𝐞,𝑘 |𝑇 𝑄 𝐺 𝐝,𝑘 |𝐺 𝐞,𝑘 , 𝜌 𝐺 𝐝,𝑘 ∙ 𝑄 𝐹 𝐞,𝑙 |𝜌 𝐹 𝐞,𝑙 , 𝑇 𝑪 = {𝑇, 𝑁 𝐞 , 𝑁 𝐝 , 𝐺 𝐞 , 𝐺 𝐝 , 𝐹 𝐞 } 𝑘 𝑙 17

Learning • Learning framework [Koller and Friedman 2009] Input: 𝑂 buildings 𝑃 = {𝑃 1 , 𝑃 2 , ⋯ , 𝑃 𝑂 } 𝑇 Output: • the structure 𝐻 𝑁 𝐞,𝑗 𝐺 𝐞,𝑘 𝐹 𝐞,𝑙 • the cardinality of the values of the hidden variables • the lateral edges 𝑁 𝐝,𝑗 𝐺 𝐝,𝑘 • the parameters Θ 𝐽 𝐾 𝐿 • Cheeseman-Stutz score [Cheeseman and Stutz 1996] 𝐷𝑇 𝐻 𝑃 = log 𝑄(𝐻) + log 𝑄(𝑃 ∗ |𝐻) + log 𝑄(𝑃|𝐻, Θ 𝐻 ) − log 𝑄(𝑃 ∗ |𝐻, Θ 𝐻 ) 18

Learning • Structure learning algorithm 𝑇 𝑇 Step 1: 𝑡 and {ℎ 𝑗 } 𝐸 𝐼 𝑗 𝑁 𝐞,𝑗 𝐺 𝐞,𝑘 𝐹 𝐞,𝑙 𝑘 𝐷 𝑗 𝑁 𝐝,𝑗 𝐺 𝐝,𝑘 𝐽 𝐾 𝐾 𝐿 𝐽 Step 2 𝐷 1 𝐷 𝐽 𝐷 2 𝐷 4 𝐷 3 19

Sampling 𝑇 Training data Samples ∗ 𝑁 𝑑 ∗ 𝑁 𝑑 ∗ 𝑁 𝐝 𝑁 𝐝 𝐺 𝐞,𝑘 𝑁 𝐞,𝑗 𝐹 𝐞,𝑙 ∗ 𝐺 training sampling ∗ 𝐺 𝑑 ∗ 𝐺 𝑑 𝐺 𝐝 𝐝 ∗ 𝐹 𝑒 ∗ 𝐹 𝑒 ∗ 𝐹 𝐞 𝐹 𝐞 𝐶 𝑗 𝑁 𝐝,𝑗 𝐺 𝐝,𝑘 𝐶 𝑗 𝑃 𝑗 𝐶 𝑗 𝐽 𝐾 𝐿 20

Building Generation Attribute-based Mass model and roof Facade generation representation generation

Mass Model and Roof Generation ∗ • Input: desired mass model attributes 𝑁 𝐝 • Output: building mass model 𝑁 • Mass model energy Attributes term: 𝑁 𝐝,𝑗 denotes the attributes of 𝑁 2 ∗ 𝑁 𝐝,𝑗 − 𝑁 𝐝,𝑗 𝐹 𝑛𝑏𝑡𝑡 𝑁 = + 𝐹 𝑢𝑝𝑞𝑝 𝑁 + 𝐹 𝑢ℎ𝑗𝑑𝑙 𝑁 2 ∗ 𝑁 𝐝,𝑗 𝑗 Topology term: Thickness term: All boxes should be Mass models should connected. not be too narrow. Top Top 22

Mass Model and Roof Generation • Algorithm: simulated annealing A B C Initialization Energy D E F A B C D E F Iterations Top views 23

Mass Model and Roof Generation F 24

Facade Generation ∗ , • Input: mass model 𝑁 , desired facade attributes 𝐺 𝐝 and the database (i.e., element set) • Output: facades of the building 𝐺 • Facade layout energy 𝑡 : right attributes of 𝐺 2 ∗ 𝐺 𝐝,𝑘,𝑡 − 𝐺 𝐝,𝑘,𝑡 𝐹 𝑔𝑏𝑑 𝐺 = ∗ 𝐺 𝒅,𝑘,𝑡 𝑡 𝑘 Facade pieces on the right side 25

Facade Generation 26

Results & Applications

Dataset • 200 buildings: building footprints + photographs Thumbnails of all buildings in our dataset. 28

Dataset • 200 buildings: building footprints + photographs garage back porch front porch 𝑁 𝐝 𝐺 𝐝 𝐹 𝐞 Thumbnails of all buildings in our dataset. 29

Model Structure: Hidden Variables 𝑇 𝐺 𝐞,𝑘 𝑁 𝐞,𝑗 𝐹 𝐞,𝑙 Style 1 Style 2 𝐺 𝐝,𝑘 𝑁 𝐝,𝑗 𝐽 𝐾 𝐿 Style 3 Style 4 Style 5 Style 6 Style 7 30

Model Structure: Lateral Edges 𝑇 𝐺 𝐞,𝑘 𝑁 𝐞,𝑗 𝐹 𝐞,𝑙 𝐹 𝑠𝑝𝑝𝑔 𝐹 𝑥𝑜𝑒 vs. 𝐺 𝐝,𝑘 𝑁 𝐝,𝑗 𝐽 𝐾 𝐿 𝑁 𝐝,𝑐𝑒𝑠𝑧 𝑁 𝐝,𝑐𝑑𝑠 vs. 𝑁 𝐝,𝑏𝑠𝑔 𝑁 𝐝,𝑐𝑐𝑝𝑦 𝑁 𝐝,𝑕𝑏𝑠 𝑁 𝐝,𝑞𝑝𝑠 31

Comparison with Other Models • Holdout validation method • a training set (80%) • a test set (20%) • Models • Our model • Model 1 : without learned edges between variables • Model 2 : without hidden variables on the second level • Model 3 : a directed graphical model without hidden variables • GMM : this model does not encode the relationship between building attributes 32

Application I: Building Synthesis 33

Application II: Building Completion 34

Application II: Building Completion 35

Application II: Building Completion Input Completion 1 Completion 2 Completion 3 Incomplete Backside of each building model building model 36

Discussions • Special building structures: garages and porches Limitations Limitations • Our model does not consider • Our model does not consider garage porch parcel • appearance of the building: color and texture • appearance of the building: color and texture • context of the building: parcel shape, parcel slope, etc. • context of the building: parcel shape, parcel slope, etc. • No guarantee to find a global minimum • No guarantee to find a global minimum 37