Networking 7.2.6 Optical Links Ba Babak Abi DAQ FD D "D "Des esig ign" " Wor orkshop 2 26 Ja Jan 2018 Outline: • Data link Components and suggestion for current standard • Costs per link and Gb/s • Rough Estimation of number fibres needed 1
4 1 2 3 Timing interface? Trade of between fiber cost and Transceiver cost? • In case of 10G links RCE-FELIX need ~3000 TR module! • 1500 SNAP to CE
Data Links Options • Main questions : 1. What are the data links current and future options? • Link speed, form factor power consumption 2. Long-haul and short-haul links from the WIB to the DAQ 1. Technologies with coper for 30m till 40Gb 2. Long-haul 200m > should be optical links 3. Cost and trade-offs of each Ethernet Alliance Technology Roadmap 2014 paper • BUT Networking and high-speed solution and components have a very broad phase space since we are on the verge of a technological change in the landscape of data links. • The IEEE is proposing more Ethernet standards from 2015 to 2020 (five years) than its released in the last 40. There is even realistic talk of 10 TbE towards the end of the next decade. • Should we focus our designs on current or future standard ? • We need to review future standard since the hardware system design level would be effected if want to use future standard • We present the suggestion base on current standard 3
Optical MUX • Optical MUX Simplex MUX • Simplex MUX, needs two fibre, cheaper • Duplex MUX, needs ONE fibre, expensive • CWDM or DWDM? • Dense wavelength division multiplexing (DWDM) up to 80 (theoretically) channels can be multiplexed into on a single optical fiber. • CWDM (Coarse Wavelength Division Multiplexing) Up to 18 channel spacing 20 nm, 1270 nm to 1610 nm, Cost-effective • If we assume we fix design on SFP+ (10G) Duplex MUX • DWDM 40 links per fiber (400Gb) and CWDM 18 links per fiber (180Gb) • There are hacks for increase it like using qsfp+ in 1310nm line • Limitation on fibres we have only 96 DWDM 38Tb/s and CWDM 19Tb/s 4
CWDM and DWDM module • FMU 1U Rack Mount, LC/UPC, with Monitor Port , 3.0 to 6dB Typical IL • Could add a 22dBm Output Booster DWDM EDFA C-band 24dB Gain for long-haul • Price: • Retail price $1700 for 40 channels DWDM (4 APA if we use 10x10GbE ) • Retail price $710 for 18 channels CWDM ( almost 2 APA if we use 10x10GbE ) • BUT in mass order or FOB can be much cheaper • Mux muddles are fairly cheap but SFP module • CWDM SFP+ starts from $80 • DWDM SFP+ starts from $270 • But prices are falling down , need extrapolation for 2024? • Current cost for MUX end to end included TRX and MUX module but fibre : • DWDM ~$55 + 10 per Gb • CWDM ~$15 + 10 per Gb • DWDM 400Gb per fibre and CWDM 180Gb per fibre! • Fibre selection: Zero-water-peak fibres might be needed (backup slide) • Further options& considerations ? 10km 40km • BiDi SFP+ not widely used + we do not need downstream fibre?? • Enterprise Ethernet network switch ! (32*100GbE) 100GbE Spine/Core Layer for $8000 5
Estimation of number fibres needed • Number of fibres needed from RCEs for three data stream: 1) lossless data 2) lossy 1-TPC Lossless 4-Only TPC Total data (removed for now) 3) trigger primitives DATA stream from RCE to surf/FELIX Total single fiber RCE output links APA 10Kt totalDUNE RCE output links APA 10Kt totalDUNE stream. I am also assuming: 10Gbs links 8 1200 4800 10Gbs links 11 1650 6600 10Gbs links CWDM 0.44444 66.66667 266.666667 10Gbs links CWDM 0.611111 91.66667 366.66667 1- WDM is only supported on 10Gb/s FSP+ for 10Gbs links DWDM 0.2 30 120 10Gbs links DWDM 0.275 41.25 165 40Gbs links 2 300 1200 40Gbs links 3 450 1800 now. No WDM for 40/100Gb/s. 100Gbs links 1 150 600 100Gbs links 2 300 1200 2- One-sided traffic ratio less than 1:10 2-DATA stream from surf/FELIX to RCE RCE output links APA 10Kt totalDUNE 3- One COB per APA, No Compression, No link 10Gbs links 1 150 600 6- TPC + PD total sharing between COBS. 10Gbs links CWDM 0.05556 8.333333 33.3333333 Total single fiber 10Gbs links DWDM 0.025 3.75 15 RCE output links APA 10Kt totalDUNE 40Gbs links 1 150 600 10Gbs links 19 2850 11400 4- in PD since the links are 1Gbs we have to use 100Gbs links 1 150 600 10Gbs links CWDM 0.744444 111.6667 446.66667 the 10Gbs switch to reduce numbers of fibres 10Gbs links DWDM 0.335 50.25 201 3- Trigger Primitives ;bidirectional 40Gbs links 3.075 461.25 1845 and I used a factor 3 overhead for congestion RCE output links APA 10Kt totalDUNE 100Gbs links 2.03 304.5 1218 10Gbs links 2 300 1200 risk! 10Gbs links CWDM 0.11111 16.66667 66.6666667 10Gbs links DWDM 0.05 7.5 30 40Gbs links ? 0 0 0 ? 100Gbs links ? 0 0 0 ? 6
General conclusion • How much cost of different schemes (price Jan/2018) • DWDM $65 per Gb • CWDM $25 per Gb • DWDM 400Gb per fibre and CWDM 180Gb per fibre! • 100Gb $ 10-15 per Gb • QSFP+ , CFP/CFP2/CFP4/CXP • What is the recommended rate : • 10G, links if fiber is more expensive then Transceiver modules • 100Gb if We can get the signal integrities right! • We need at least 200/450 fibres DWDM/CWDM for TPC+PD • This is the worst case scenario, RCE in service cavern and FELIX is on Surface • If FELIX is in service room then triggered DATA needs even less fibres • So those number seems an upper limit of estimation • How much fibre cost? 7
Backup slides 8
water-peak fi fibres 9
Custom-made MTP(MPO) Module • Parallel optical link module to reduce the footprint of components to CE’s interface. • It is possible to make them in house, The knowledge of How-to exist already in wide range of experiments. (in case if they way expensive or difficult to find in future) Axel Roggenbuck ATLAS-Pixeldetektor Diplomarbeit, Universit¨at Siegen, 10
Link roadmap; speed EthernetAlliance_Roadmap_whitepaper_FINAL-032015-21 • TbE (Terabit Ethernet) vs. GbE (Gigabit Ethernet) • GbE link speeds to 100Gb and TbE links speeds from 100Gb • If we want to use TbE we need to design hardware now • CWDM and DWDM dominant transceivers are SFP and SFP+ 11
Recommend
More recommend