LLRF Embedded Development Brian Chase, Joshua Einstein-Curtis - PowerPoint PPT Presentation

LLRF Embedded Development Brian Chase, Joshua Einstein-Curtis Controls Modernization Group 28 September 2018

Introduction • LLRF is a high-speed, high-precision real time system. • For the FNAL accelerator infrastructure, have relatively high data rates – Similar to beam instrumentation – Necessitated builds of a Labview client for commissioning, debug, and diagnostics (Backdoor/custom) Example: CMTS has 4 ADC and 2 DAC channels per cavity. In addition to the raw data, there are also the baseband IQ and amplitude/phase signals, error, and controller intermediate stages 2 * (4*2 + 2 + 10) * 65 MSPS * 32-bit = 82 Gbps max 82/65 = 1.26 Gbps if 1 MSPS sampling 1.26 Gbps / 8 * 360 = 56 TB/hour 0.02 * 56 = 1.12 TB/hour if only 20 ms of 1MSPS data every second 2 LLRF Embedded Development 9/27/2018

Real-time Needs • Hard Realtime  LLRF (timescale-dependent, of course) – Failures not allowed • Bucket-resolution timestamping on data • Necessary to do extensive hardware testing or simulation (Qemu, ARM simulator) – Good debugging tools a necessity • Utilization and overloading – Need for optimizations and good compilers 3 LLRF Embedded Development 9/27/2018

vxWorks • MVME2400, MVME5500 with Kinetic Systems backplane • VxWorks 5.5, 6.1, 6.4, 6.9 – 5.5 running on MI, FAST, 6.4 on Recycler, 6.9 dev system provided to controls • Power PC 603e, 750 or 7457 • Standardized communication between front ends enables faster development. Shared libraries, etc. – https://www-bd.fnal.gov/controls/micro_p/mooc_project/welcome.html • Protocol Compiler – Custom for each device? Possibly leads to complexity and versioning issues – Each device has own receive stack, and each group develops their own 4 LLRF Embedded Development 9/27/2018

Current Model LLRF Frontend Spreadsheet sys_init.cpp Dennis’ Frontends Modification Python Scripts Scripts-to-dabbel Acnet and Front-end independently record and manage variables 5 LLRF Embedded Development 9/27/2018

SoC MFC 6 LLRF Embedded Development 9/27/2018

LLRF Control and Timing • VUCD – Developed in mid-nineties – No replacement for current systems available – Timing and data decoding using backplane triggers – Multi-timer system – No planned replacement for current/deployed systems • Bucket resolution and accurate timestamping needed for future diagnostics 7 LLRF Embedded Development 9/27/2018

Build Environment • nova build environment (esd tools) well developed – LLRF projects are all there, including configuration files for new SoC systems (spreadsheet register maps and defaults) • Need for a well-developed upgrade plan across departments • Need for supporting both legacy systems and new systems using same tools? • Significant increases in performance with each controller, but processor-specific programming can make porting difficult 8 LLRF Embedded Development 9/27/2018

Revision Control • Currently part of build toolchains on nova • One of the largest issues we’ve run in to is revision and versioning – Separate versions for firmware, software, libraries – Storing git commits in f/w and software works well – Xilinx allows for same bitfile from same files • routing/placement hash/seed is based on files included in project • Enforcing this across all project levels is a challenge • PLC versioning? Software versioning? HDL versioning? 9 LLRF Embedded Development 9/27/2018

Revision Control • LBL/LCLS-II and CERN have self-describing firmware/software – Register maps in control system and front-ends match • gzip’d JSON in memory on devices • CERN’s FESA and cheby • Acnet database • TFTP/remote image loading is crucial – Limit wear on flash (limited write) devices • Remote/central logging is crucial – See above • Verification and Continuous Integration are valuable tools 10 LLRF Embedded Development 9/27/2018

What is plan for future? • Responsibility of each department? • High-speed data and diagnostics – artDAQ? FTP? Something new? – DAQ analysis and storage – High-speed means what for each system? • Data lifetime? • Scope capabilities • Event identification • Design lifetime – 20 years? 5 years? 1 year? • Tool availability and automation – Build system for non-experts – Nova replacement? Command-line functionality? 11 LLRF Embedded Development 9/27/2018

Looking Forward • What will be available in 2 years? 5 years? 10 years? • Need a long-term plan • Due to few resources at developer level, really need process and procedure directed from managment • Hardware - should there be a standard developed across groups? – Shared FPGA/Processor boards? – Crate standards? – Physical interfaces? 12 LLRF Embedded Development 9/27/2018

Xilinx Zynq Ultrascale+ RFSoC • Designed for cell base stations, radar, military • Single-chip includes all necessary for an RF system, but might not meet performance requirements of modern accelerators. • Separate application and real-time processors • Shared memory makes for very powerful, high- bandwidth toolset https://www.xilinx.com/products/silicon-devices/soc/rfsoc.html 13 LLRF Embedded Development 9/27/2018

Conclusion • Concerns for future – DAQ and timing/precision – Data retention and high-speed data management – Revision Control and Build Environment are significant concerns – Ease of integrating in to control system: shared protocols, development architectures, etc • Need a multi-year, division-wide path forward – While still supporting legacy systems • Thanks! 14 LLRF Embedded Development 9/27/2018

Extra Slides 15 LLRF Embedded Development 9/27/2018

Switching to Fiber • Fiber benefits outweigh slightly increased cost • Decreased power • Higher bandwidth • Low cost per bandwidth • Relative ease of implementation given available tools 16 LLRF Embedded Development 9/27/2018

Communication Proposals • 9.027 MHz RF synchronous frame rate – Local clock source needed on each device • 1300 MHz harmonic as data rate • Need known frame size (ie 128-bit) • Previous State, Events, Beam pattern for next group • Single-source star • Send as timing/events -- Return line as MPS – Data concentrators? • Throw out proposal • 3 links – DAQ, timing/MPS, control 17 LLRF Embedded Development 9/27/2018

18 LLRF Embedded Development 9/27/2018

Intel Cyclone V and Arria 10 19 LLRF Embedded Development 9/27/2018

SoC MFC Test DAQ System Courtesy Ed Cullerton 20 LLRF Embedded Development 9/27/2018

Arm Product Line https://www.silabs.com/documents/public/white- papers/Which-ARM-Cortex-Core-Is-Right-for-Your- Application.pdf 21 LLRF Embedded Development 9/27/2018

Microsemi SmartFusion2 Single Event Upset Immune design 22 LLRF Embedded Development 9/27/2018

NXP Kinetis K60 LPC17xx i.MX 23 LLRF Embedded Development 9/27/2018

Why not Pis? • The question is really: what does real-time mean? – Bucket-level resolution and repeatability? – TCLK decoding accuracy? Event decoding accuracy? • What runs at each layer? – Firmware/embedded RTOS – Software/application layer • Lifetime and long-term support – Open Hardware • Interconnect standards – SPI + I2C support? (1 SPI, 1 I2C, 1 UART) – Standard interconnects? – Bus/backplane protocols? 24 LLRF Embedded Development 9/27/2018

DSP • LLRF requires higher-performance DSP and DAQ • Newer FPGAs are offering dedicated floating-point blocks – Require use of special tools or overloading standard operators • Integrated simulation and verification environment being more important • Collaboration requires the ability to transfer IP easily 25 LLRF Embedded Development 9/27/2018

SoM Benefits https://www.intel.com/content/www/us/en/programmable/products/soc/ecosystem/system-on-modules.html 26 LLRF Embedded Development 9/27/2018

Linux (and variants) • Linux w/ preemptRT • RTAI (Linux as task) • NI Real-Time Linux (dual-schedulers) 27 LLRF Embedded Development 9/27/2018

RTOS • FreeRTOS -- https://www.freertos.org/ • Contiki (IoT) -- http://www.contiki-os.org/ • MicroC/OS-IIx • Mbed (IoT) • RTEMS • vxWorks – Now a Linux variant • MQX 28 LLRF Embedded Development 9/27/2018

MPC603e 29 LLRF Embedded Development 9/27/2018

MPC750 30 LLRF Embedded Development 9/27/2018

MPC7457 31 LLRF Embedded Development 9/27/2018

Processing Models • IOC+Epics – “Dumb” front -end with managing IOC – Subscriber model • Crate processors – Slot 0 manages all cards in crate. Advanced data processing happens ??? • NADs – Multiple models available • Acnet • Requirements on who does each portion is not defined – Controls communication (MOOC(AN), libacnet, erlang, PC) – Debug communication (Labview, Python) – Front-end intelligence 32 LLRF Embedded Development 9/27/2018