Slide 1 Slide 1 P i Paired Redundant IOCs Paired Redundant IOCs d R d d t IOC with Redundant Hardware with Redundant Hardware ith R d d t H d S A Baily and Eric Bjorklund S. A. Baily and Eric Bjorklund S. A. Baily and Eric Bjorklund UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 2 Slide 2 Why Does LANCSE Use Redundant IOCs Why Does LANCSE Use Redundant IOCs Wh y D LANCSE U R d d t IOC For Its Timing System? F For Its Timing System? It Ti i g S y t ? • • Since the 1990s we’ve had redundant hardware for the master timer Since the 1990s we ve had redundant hardware for the master timer system. system. t • If the master timer stops putting out timing gates, all of our RF If the master timer stops putting out timing gates, all of our RF If th t ti t tti t ti i t ll f RF stands trip off. stands trip off p – It s not difficult to recover a single stand, but recovering all stands takes It’s not difficult to recover a single stand but recovering all stands takes a lot of time. a lot of time. l t f ti – Historically, dropping all RF power abruptly at 120 Hz caused city-wide Hi t i ll y, d pp g i ll RF p b p y tl t 120 H d it y id power outages power outages. – Power dispatch calls when our electricity usage drops quickly (at lower Power dispatch calls when our electricity usage drops quickly (at lower ( repetition rates). repetition rates) • • Without redundancy maintenance opportunities on the timing Without redundancy, maintenance opportunities on the timing system would be very limited. system would be very limited. t ld b li it d UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 3 Slide 3 Redundant IOC software Redundant IOC software Redundant IOC software • Developed by • Developed by D l p d b y DESY in DESY in DESY in collaboration collaboration collaboration with SLAC with SLAC ith SLAC • Maintained by Maintained b Maintained by DESY DESY DESY. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 4 Slide 4 How Redundant IOC Software Works How Redundant IOC Software Works How Redundant IOC Software Works • • Redundancy Monitoring Task Redundancy Monitoring Task – Monitors state health of network Monitors state health of network and drivers and drivers • • Continuous Control Executive Continuous Control Executive – Synchronizes EPICS databases Synchronizes EPICS databases • • RMT Driver RMT Driver – Plugs in to RMT. Plugs in to RMT g – Reports state of health Reports state of health p – Reports synchronization status Reports synchronization status p y – Accepts commands to enter Accepts commands to enter p master or slave states master or slave states. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 5 Slide 5 Why a Redundant IOC Pair? Why a Redundant IOC Pair? Why a Redundant IOC Pair? • • Master Timer Master Timer – Timing Pattern Generator (TPG) g ( ) • • VME-64 IOC with solution and zero crossing detector VME-64 IOC with solution and zero crossing detector. • Generates sched led timing patterns and timing e ent link Generates scheduled timing patterns and timing event link. • Includes RF gates and triggerable beam sequences. g gg q – MRF event generators. MRF event generators. – Gate Enable Inhibit and Countdown Controller (GEICCO) Gate Enable, Inhibit, and Countdown Controller (GEICCO) • cRIO IOC determines which beam events get sent. g • Triggers actual beam gates (slave event generators in the TPG) in response to operator Triggers actual beam gates (slave event generators in the TPG) in response to operator switches, and counters. switches and counters – cRIO with TTL Binary I/O cRIO with TTL Binary I/O. • A second pair provides redundancy A A second pair provides redundancy d i id d d • A fib A fiber optic switch (MRF-fanout concentrator) selects which system is p ti it h (MRF f ( t t t ) ) l t hi h y t i used used. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 6 Slide 6 G i Geicco-1 1 Hardware Setup Hardware Setup Hardware Setup TPG-1 TPG 1 • Two IOC pairs • Two IOC pairs. Two IOC pairs. • Two single points of • Two single points of Two single points of RF Dist RF Dist. f il failure failure. RF Input RF Input RF Dist RF Dist. – RF input (may not matter) RF input (may not matter) RF i t ( t tt ) – Event link (unavoidable) Event link (unavoidable) ( ) TPG 2 TPG-2 • Systems monitor each • Systems monitor each S y i h other as part of other as part of other as part of GEICCO-2 GEICCO-2 synchronization synchronization. y h i ti E Event-link switch Event link switch t li k it h UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 7 Slide 7 LANSCE Timing System IOC Functions LANSCE Timing System IOC Functions LANSCE Timing System IOC Functions • Timing Pattern Generator (TPG) Timing Pattern Generator (TPG) Ti i P tt G t (TPG) – Generates scheduled timing patterns Generates scheduled timing patterns. – Includes RF gates and triggerable beam sequences Includes RF gates and triggerable beam sequences. • MRF event generators (VME-64). MRF event generators (VME 64). • G t Gate Enable, Inhibit, and Countdown Controller (GEICCO) Gate Enable Inhibit and Countdown Controller (GEICCO) E bl I hibit d C td C t ll (GEICCO) – Triggers actual beam gates (slave event generators in the TPG) Triggers actual beam gates (slave event generators in the TPG) in response to operator switches and counters in response to operator switches, and counters. p p , • cRIO with TTL Binary I/O • cRIO with TTL Binary I/O. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 8 Slide 8 Differs From Traditional Redundant IOC Differs From Traditional Redundant IOC Differs From Traditional Redundant IOC • Not two identical IOCs on the same bus • Not two identical IOCs on the same bus. N t t id ti l IOC th b • Has redundant hardware • Has redundant hardware. H d d t h d • IOCs should run continuously IOCs should run continuously. IOC h ld ti l y • IOCs should fail over in pairs IOC IOCs should fail over in pairs. h ld f il i p i UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 9 Slide 9 Broadly Applicable Improvements Broadly Applicable Improvements Broadly Applicable Improvements • Added support for syncing fields with • Added support for syncing fields with pp y g SPC_DBADDR (waveform records) SPC DBADDR (waveform records) SPC DBADDR (waveform records) – Modified the Continous Control Executive (CCEXEC) – Modified the Continous Control Executive (CCEXEC) to support larger fields to support larger fields. • Made it possible to specify which records should Made it possible to specify which records should Made it possible to specify which records should be synced b be synced d – Configurable via info nodes. Configurable via info nodes g – Modified e2db so that CapFast can be used to Modified e2db so that CapFast can be used to M difi d 2db th t C F t b d t configure info nodes. configure info nodes. fi i f d UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 10 Slide 10 Why Not Sync All Records? Why Not Sync All Records? Why Not Sync All Records? • IOC stats should be individual. • IOC stats should be individual IOC t t h ld b i di id l • Conserve bandwidth • Conserve bandwidth C b d idth • Redundant hardware read-back channels R d Redundant hardware read-back channels d t h d d b k h l should come from the actual hardware should come from the actual hardware. h ld f th t l h d UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Slide 11 Slide 11 R d Redundant Hardware Specific Redundant Hardware Specific d t H d S p ifi Improvements Improvements I p t • Added an option to pause only the channel access • Added an option to pause only the channel access p p y server instead of the entire IOC server instead of the entire IOC. server instead of the entire IOC. • Call post event on the slave when the database • Call post event on the slave when the database p _ beco becomes synced with the master becomes synced with the master es sy ced e as e (CCEXEC SLV INSYNC state) so that passive (CCEXEC SLV INSYNC state) so that passive (CCEXEC_SLV_INSYNC state) so that passive records can be made to process on the slave records can be made to process on the slave. d b d t th l – Uses event-driven fanout records. Uses event driven fanout records – Can control the processing order C Can control the processing order. t l th i d – Particularly useful for commands. Particularly useful for commands y UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA
Recommend
More recommend
