21st IEEE Symposium on High Performance Computer Architecture (HPCA) 2015-02-11 Augmenting Low-latency HPC Network with Free-space Optical Links Ikki Fujiwara National Institute of Informatics Michihiro Koibuchi Tomoya Ozaki Keio University Hiroki Matsutani Henri Casanova University of Hawai’i at Manoa
2 Story at a Glance • What if steerable wireless links appear on top of cabinets? FSO Terminal Laser Beam Cabinet Switch Cable Topology Efficient Power- Reduced cable optimization for aware On/Off length & latency diverse apps Link Regulation
• Motivation • How to make Free Space Optics (FSO) – FSO Terminal Devices – Layout of FSO Terminals • How to use FSOs in an HPC system – For Reduced Cable Length and Latency – For Improved Topology Embedding – For Power-aware On/off Link Regulation • Conclusion
4 Motivation 1/3: Cable Reduction Earth Simulator, 1st gen. (crossbar) K Computer (6-D mesh/torus) 83,200 cables 200,000 cables 2,400 km 1,000 km 140 tons FSO provides shorter cable length and lower link delay (c) kan-haru (c) Riken
5 Motivation 2/3: Topology Optimization • Diverse parallel applications have each different preferable topology Event-discrete simulation by SimGrid. 64 switches. Switch degree = 8. 1.8 1.6 Torus Random Better Relative performance 1.4 1.2 1 0.8 0.6 0.4 0.2 0 CG FT Graph500 NAS Parallel Benchmarks FSO provides a reconfigurable network
Motivation 3/3: Leveraging Power- 6 aware On/Off Link Regulation • Link consumes power regardless of workload e.g. Energy Efficient Ethernet • Turned-off links saves link power, but causes a negative impact on performance in HPC use [1] Performance loss is not acceptable for HPCs • Let’s turn off more links! – As long as the performance loss is compensated by replacing wired links with FSO-based shortcuts [1] Saravanan et al., “Power/performance evaluation of energy efficient Ethernet (EEE) for High Performance Computing”, ISPASS 2013
• Motivation • How to make Free Space Optics (FSO) – FSO Terminal Devices – Layout of FSO Terminals • How to use FSOs in an HPC system – For Reduced Cable Length and Latency – For Improved Topology Embedding – For Power-aware On/off Link Regulation • Conclusion
8 Free-space Optical Links Transceiver Optical Circulator Collimator Lens Optical fiber • 10 – 100 Gbps, 200m distance using commodity laser (e.g. 1310 nm) • Negligible interference enables high-density layout on top of cabinets • Terminal devices applicable to HPC use: Our prototype Hamedazimi’s [2] Arimoto’s [3] [2] Hamedazimi et al, “ FireFly: a reconfigurable wireless data center fabric using free-space optics”, SIGCOMM 2014 [3] Arimoto et al., “Wide field -of-view singlemode- fiber coupled laser communication terminal”, SPIE 2013
• Motivation • How to make Free Space Optics (FSO) – FSO Terminal Devices – Layout of FSO Terminals • How to use FSOs in an HPC system – For Reduced Cable Length and Latency – For Improved Topology Embedding – For Power-aware On/off Link Regulation • Conclusion
Line-of-sight Layout of FSO Terminals 10 Other FSO terminal FSO Transmitter FSO Receiver ! • No laser beam should be interrupted by the other terminals • Want to layout FSO terminals so as to minimize the interruption 2𝑀 Maximize the line-of-sight ratio (LSR) = 𝑂(𝑂−1) • 𝑂 = number of terminals • 𝑀 = number of terminal pairs with direct line of sight • Calculated using a ray tracer
11 Straight Layout (Naive) 1 ~60% LSR 0.9 Line-of-sight ratio 0.8 0.7 0.6 0.5 0 100 200 300 400 500 600 700 800 Number of cabinets = FSO terminals
12 Random Layout 1 0.9 Line-of-sight ratio 0.8 92.5% LSR at 800 cabinets 0.7 0.6 0.5 0 100 200 300 400 500 600 700 800 Number of cabinets = FSO terminals
13 Theater Layout 100% LSR up to 252 cabinets 1 0.9 Line-of-sight ratio 0.8 Scalability limited by the headroom 0.7 0.6 0.5 0 100 200 300 400 500 600 700 800 Number of cabinets = FSO terminals
14 Alternative Layout using a Mirror Mirror 100% LSR Unlimited scalability (ideally) • FSO beams can be reflected by a mirror • Similar idea is used for 60GHz wireless [4] • Hereafter we assume 100% LSR [4] Zhou et al, “Mirror mirror on the ceiling: flexible wireless links for data centers ”, SIGCOMM 2012
• Motivation • How to make Free Space Optics (FSO) – FSO Terminal Devices – Layout of FSO Terminals • How to use FSOs in an HPC system – For Reduced Cable Length and Latency – For Improved Topology Embedding – For Power-aware On/off Link Regulation • Conclusion
16 Physical Merits of FSO links • Reduced cable length • Lower end-to-end communication latency – At most 53% lower latency (in theory) Cable: 0.2 m/ns FSO: 0.3 m/ns Manhattan distance Euclidean distance (Go orthogonal) (Go diagonal) • Calculated using graph analysis – When replacing long cables with FSO links – 1,024 switches; 512 cabinets; 1, 2, 4 FSO terminals/cabinet
17 Reduced Cable Length 120 −23.0% 100 Total cable length [km] 80 −36.5% 60 −32.0% 40 −31.6% 20 0 0 2 4 0 2 4 0 2 4 0 2 4 #FSOs/cab 3D 5D Rand Rand Base topology Torus Torus deg=6 deg=10
18 Lower End-to-end Latency 4 Max. Max. Avg. −2.4% 3.5 Link delay Switch delay Zero- load latency [μs] 3 2.5 Avg. −8.9% −2.0% 2 −4.1% −4.6% 1.5 −8.1% −2.9% −5.1% 1 0.5 0 0 2 4 0 2 4 0 2 4 0 2 4 #FSOs/cab 3D 5D Rand Rand Base topology Torus Torus deg=6 deg=10
• Motivation • How to make Free Space Optics (FSO) – FSO Terminal Devices – Layout of FSO Terminals • How to use FSOs in an HPC system – For Reduced Cable Length and Latency – For Improved Topology Embedding – For Power-aware On/off Link Regulation • Conclusion
20 Topology Embedding • Many small jobs run simultaneously in an HPC • Want to efficiently allocate their preferable topology Graph embedding problem (NP-hard) • FSO largely alleviates the embedding problem FSO terminal Switch 2 × 4 mesh found in a random topology • Optimized using a genetic algorithm – So as to maximize the number of embedded topology
21 2 × 4 Tori Found 100 80 60 4 FSOs/cab leads to >80% of Coverage [%] nodes to be well allocated 4 FSOs/cab 40 2 FSOs/cab 1 FSO/cab 20 0 8 12 16 20 24 28 32 36 40 Degree of physical topology FSO opens a possibility for a better job allocation (random)
• Motivation • How to make Free Space Optics (FSO) – FSO Terminal Devices – Layout of FSO Terminals • How to use FSOs in an HPC system – For Reduced Cable Length and Latency – For Improved Topology Embedding – For Power-aware On/off Link Regulation • Conclusion
23 Power-aware On/Off Link Regulation • Our idea: let’s turn off more links! – As long as the performance loss is compensated by replacing wired links with FSO-based shortcuts depending on a given workload Deactivate Loop Compensate 1. Deactivate wired links less 2. Insert an FSO shortcut to contributing to avg path length remedy the avg path length
24 Power-aware On/Off Link Regulation • Evaluation results using flit-level simulator – 𝑞 percent of the wired links are replaced with FSO – 𝑟 percent of the links are deactivated 1.1 1.1 Average latency relative to p=0, q=0 Hop count relative to p=0, q=0 1 1 0.9 0.9 0.8 0.8 0.7 0.7 BT CG IS LU SP BT CG IS LU SP p=0, q=20 p=0, q=40 p=20, q=0 p=20, q=20 p=20, q=40 FSO works well with a power-aware link regulation
25 Comparable Technologies • 60 GHz radio wireless links – Larger interference than FSO [Zhou et al., SIGCOMM 2012] • Embedding using Optical Circuit Switches (OCS) – Wired links via an optical circuit switch can support partial reconfiguration – Its embedding capability is lower than FSO Only FSO realizes our three objectives
• Motivation • How to make Free Space Optics (FSO) – FSO Terminal Devices – Layout of FSO Terminals • How to use FSOs in an HPC system – For Reduced Cable Length and Latency – For Improved Topology Embedding – For Power-aware On/off Link Regulation • Conclusion
27 Conclusion • Augmenting Low-latency HPC Network with Free- space Optical Links, we get… Reduced cable Topology Efficient Power- length (−36%) & optimization for aware On/Off latency ( −9 %) diverse apps Link Regulation Torus FatTree Random FSO Terminal Laser Beam Cabinet Switch Cable
Recommend
More recommend
