Anomalies Detection for HEP Experiments Maxim Borisyak Denis - PowerPoint PPT Presentation

Anomalies Detection for HEP Experiments Maxim Borisyak Denis Derkach, Fedor Ratnikov, Andrey Ustyuzhanin HEP/ML Group, Yandex School of Data Analysis, National Research University High School of Economics with incredible help from CMS colleagues

Content ◊ Yandex ◊ Supervised anomalies detection ◊ Decomposition of anomalies by source ◊ Rare Anomalies Fedor.Ratnikov@cern.ch 2 Anomalies in High Energy Physics

Yandex in Wikipedia IT resources: ∼ 10% × Fedor.Ratnikov@cern.ch 3 Anomalies in High Energy Physics

Yandex in HEP ◊ Member of CERN OpenLab ◊ Member of LHCb Collaboration ◊ trigger ◊ B-tagging ◊ monitoring ◊ anomalies detection ◊ computing resources ◊ Member of SHIP Collaboration ◊ detector optimisation ◊ computing resources ◊ Cooperating with CMS Collaboration ◊ data certification ◊ Cooperating with ATLAS Collaboration ◊ GRID optimisation ◊ Contributing to other Particle Physics experiments beyond CERN Fedor.Ratnikov@cern.ch 4 Anomalies in High Energy Physics

Levels of Data Quality Monitoring ◊ Detector Level ◊ hit maps, occupancies… ◊ Routine Physics Level ◊ basic physics objects: hadrons, leptons, photons… ◊ Physics Candles ◊ J/ ψ , Z, W, top, … Fedor.Ratnikov@cern.ch 5 Anomalies in High Energy Physics

Formalising the Problem Use Routine Physics operation level ◊ Continuously supervised learning approach ◊ we have historical data processed by experts ◊ expert classified data as “good” or “bad” ◊ the system learns typical patterns ◊ establishes procedure to split data samples into “black” (definitely bad), “white” (definitely good), and “grey” (expert intervention needed) zones ◊ “definitely bad” ≡ FalsePositive < cut_bad ◊ “definitely good” ≡ FalseNegative < cut_good ◊ let system classify “black” and “white” domains, pass “grey” domain for expert decision As new data is coming, supervisor continue making complicated labelling Ultimate goal: take burden of routine classification from experts, let experts deal with non-trivial cases Fedor.Ratnikov@cern.ch 6 Anomalies in High Energy Physics

Practical Approach ◊ CMS 2010B run open data ◊ http://opendata.cern.ch/record/8 ◊ Streams: MinimalBias, Muons, Photons ◊ LumiSections (minimal chunk of data defined in metadata) are labelled as “good” or “bad” by the experiment ◊ Objects: Particle Flow Jets, Calorimeter Jets, Photons, Muons ◊ (p T , 𝜃 , 𝜒 , V xyz , mass) for 5 particles in quantiles in p T ◊ 7 features for every variable: ◊ quantiles: 0, 0.25, 0.5, 0.75, 1. + mean + variance ◊ over objects of all events in given LumiSection ◊ ∼ 2500 features describing every LumiSection Fedor.Ratnikov@cern.ch 7 Anomalies in High Energy Physics

Reference Performance ◊ Use training part of all available data to train classifier ◊ ultimate best case scenario ◊ Analyse test part of data ◊ get probability for the given LS to be “good” ◊ select two probability thresholds: Cut _bad , Cut _good ◊ define three zones ◊ “black zone” - LS is definitely bad ◊ “white zone” - LS is definitely good ◊ “grey zone” - classifier is in doubt, expert decision is needed expert decision automatic decision black zone grey zone white zone 0 Cut “bad” Cut “good” 1 Fedor.Ratnikov@cern.ch 8 Anomalies in High Energy Physics

Performance black zone grey zone white zone 0 Cut “bad” Cut “good” 1 ◊ Loss Rate ◊ “good” LS is classified as “definitely bad” and thus is lost for physics ◊ LR = FN(“black”) / (TP(“white”) +FN (“black”)) ◊ Pollution Rate ◊ “bad” LS is classified as “definitely good” and thus pollutes certified data ◊ PR = FP(“white”) / (TP(“black”) + FP(“white”)) ◊ Rejection Rate ◊ fraction of all LS which are not automatically classified as “definitely bad” or “definitely good” ◊ RR = (“grey”) / (“black” + “grey” + “white”) ◊ ManualWork = RejectionRate ∼ 80% saving on manual work is feasible for PR and LR at 5‰ Fedor.Ratnikov@cern.ch 9 Anomalies in High Energy Physics

Decomposing Anomalies ◊ Study effect of anomalies on individual channels ◊ what channels are responsible for anomalies? ◊ if only photons are affected, may muon data still be used? ◊ which plots should receive more attention from Data Quality experts? ◊ Decomposition of Channels ◊ build separate NN for every channel ◊ corresponding NN scores each channel ◊ connect networks by ◊ “min” operator with dropout ◊ exp ( (f i subnetwork - 1)) a kind of “fuzzy AND” ◊ train network to approximate a global score ◊ individual NN has high predictive power against anomalies within corresponding ◊ this may be mathematically proven in some reasonable for our case assumptions Fedor.Ratnikov@cern.ch 10 Anomalies in High Energy Physics

NN Design Cal Particle Flow Photon Muon ◊ Use 3 - layer NN ◊ Each subnetwork returns score ◊ close to 1 for good lumisections ◊ close to 1 for anomalies “invisible” from subnetwork’s channel data ◊ close to 0 for anomalies “visible” from subnetwork’s channel data ◊ Thus NN decomposes anomalies by channels Fedor.Ratnikov@cern.ch 11 Anomalies in High Energy Physics

Decomposition Results globally good lumisections globally anomalous lumisections ◊ Different channels contribute differently Fedor.Ratnikov@cern.ch 12 Anomalies in High Energy Physics

Correlations T r a c k s m u o n s E G a m m a J e t M e t ◊ Reverse test ◊ trying to predict output of the network for subsystems. ROC AUCs: ◊ muon: 0.89 ◊ photons: 0.95 ◊ particle flow: 0.86 ◊ calo: 0.94 ◊ 𝒪ℬ : expect ==1 in our assumptions Fedor.Ratnikov@cern.ch 13 Anomalies in High Energy Physics

Rare Anomalies ◊ 2010 Open Data contains significant fraction of bad data ◊ 1:2 bad-to-good lumisections ◊ Better data quality in Run 2 ◊ 1:100 bad-to-good lumisections ◊ lack of anomaly data for supervised learning ◊ also the case for LHCb ◊ Need other approaches Fedor.Ratnikov@cern.ch 14 Anomalies in High Energy Physics

Rare Anomalies ◊ Assumptions 1. good samples are embedded in small region of low-dimensional subspace 2. every point outside this region is an anomaly ◊ Technically, two-class problem ◊ suffers from class disbalance ◊ very few anomalous data ◊ assumptions allow using one-class methods ◊ but then still available information about anomalies would not be used ◊ Need to merge one-class and two-class approaches Fedor.Ratnikov@cern.ch 15 Anomalies in High Energy Physics

Mixed Objective ◊ Consider classification of objects of class 𝓓 ◊ can use “one-class on 𝓓 ”, e.g. one-class SVM ◊ Add artificial noise data 𝓞 to fill initial phase space ◊ then classifier 𝓓 against 𝓞 effectively separates 𝓓 from the rest of the phase space ◊ “one-class on 𝓓 ” = “ 𝓓 against everything” ◊ Now anomalous data may be added to the noise ◊ Loss function: 𝓜 = 𝓜 + + (1- 𝛽 ) 𝓜 - + 𝛽𝓜 noise ◊ 𝓜 +, 𝓜 - , 𝓜 noise - losses on normal, anomalous, and noise examples ◊ 𝛽 - trade-off parameter Fedor.Ratnikov@cern.ch 16 Anomalies in High Energy Physics

Illustration ◊ If negative samples are nearby positive region, produce solution as in classification problem ◊ Otherwise produce one-class bordering ◊ Toy example: 2D Gaussian normal (green), random anomalous (red) ◊ mixed objective produces more accurate separation Fedor.Ratnikov@cern.ch 17 Anomalies in High Energy Physics

Noise Injection ◊ Each layer of the deep network acts like a dimensionality reduction ◊ can inject noise more consistently into the middle of the net ◊ 𝓜 noise imposes a bias proportional to the phase volume of the positive class ◊ positive class volume tends to collapse to a single point ◊ add embedded Auto Encoder to penalties positive class volume shrinking ◊ 𝓜 = 𝓜 + + (1- 𝛽 ) 𝓜 - + 𝛽𝓜 noise + 𝛾𝓜 AE Fedor.Ratnikov@cern.ch 18 Anomalies in High Energy Physics

Tests on the Same Problem Noise Noise Noise Noise Injection Injection Injection Injection ◊ Data from the decomposition studies ◊ train/test: 10K/10K - positive, 64/6.4K - negative ◊ 800 features (reduced) ◊ ROC AUC (32 experiments) - 0.85±0.02 ◊ 0.80±0.05 without noise injection and autoencoder Fedor.Ratnikov@cern.ch 19 Anomalies in High Energy Physics

Access to Actual Data ◊ Actual studies need access to actual data ◊ historical (open) data may not represent the current status ◊ Both DS and detector operation expertise are necessary to implement advanced approaches into the detector operation chain ◊ cooperation between Yandex and CMS via CERN Open Lab is established ◊ conditional access to real time data is granted Fedor.Ratnikov@cern.ch 20 Anomalies in High Energy Physics

Conclusions ◊ Yandex group develops procedures to detect anomalies in detector data ◊ different approaches may work for different run conditions ◊ Current data access policies allow technical access to data in real time (break through since last year) ◊ CERN Open Lab in action ◊ Started moving from academic studies to practical solutions Fedor.Ratnikov@cern.ch 21 Anomalies in High Energy Physics