anomalies detection for hep experiments
play

Anomalies Detection for HEP Experiments Maxim Borisyak Denis - PowerPoint PPT Presentation

Anomalies Detection for HEP Experiments Maxim Borisyak Denis Derkach, Fedor Ratnikov, Andrey Ustyuzhanin HEP/ML Group, Yandex School of Data Analysis, National Research University High School of Economics with incredible help from CMS


  1. Anomalies Detection for HEP Experiments Maxim Borisyak Denis Derkach, Fedor Ratnikov, Andrey Ustyuzhanin HEP/ML Group, Yandex School of Data Analysis, National Research University High School of Economics with incredible help from CMS colleagues

  2. Content ◊ Yandex ◊ Supervised anomalies detection ◊ Decomposition of anomalies by source ◊ Rare Anomalies Fedor.Ratnikov@cern.ch 2 Anomalies in High Energy Physics

  3. Yandex in Wikipedia IT resources: ∼ 10% × Fedor.Ratnikov@cern.ch 3 Anomalies in High Energy Physics

  4. Yandex in HEP ◊ Member of CERN OpenLab ◊ Member of LHCb Collaboration ◊ trigger ◊ B-tagging ◊ monitoring ◊ anomalies detection ◊ computing resources ◊ Member of SHIP Collaboration ◊ detector optimisation ◊ computing resources ◊ Cooperating with CMS Collaboration ◊ data certification ◊ Cooperating with ATLAS Collaboration ◊ GRID optimisation ◊ Contributing to other Particle Physics experiments beyond CERN Fedor.Ratnikov@cern.ch 4 Anomalies in High Energy Physics

  5. Levels of Data Quality Monitoring ◊ Detector Level ◊ hit maps, occupancies… ◊ Routine Physics Level ◊ basic physics objects: hadrons, leptons, photons… ◊ Physics Candles ◊ J/ ψ , Z, W, top, … Fedor.Ratnikov@cern.ch 5 Anomalies in High Energy Physics

  6. Formalising the Problem Use Routine Physics operation level ◊ Continuously supervised learning approach ◊ we have historical data processed by experts ◊ expert classified data as “good” or “bad” ◊ the system learns typical patterns ◊ establishes procedure to split data samples into “black” (definitely bad), “white” (definitely good), and “grey” (expert intervention needed) zones ◊ “definitely bad” ≡ FalsePositive < cut_bad ◊ “definitely good” ≡ FalseNegative < cut_good ◊ let system classify “black” and “white” domains, pass “grey” domain for expert decision As new data is coming, supervisor continue making complicated labelling Ultimate goal: take burden of routine classification from experts, let experts deal with non-trivial cases Fedor.Ratnikov@cern.ch 6 Anomalies in High Energy Physics

  7. Practical Approach ◊ CMS 2010B run open data ◊ http://opendata.cern.ch/record/8 ◊ Streams: MinimalBias, Muons, Photons ◊ LumiSections (minimal chunk of data defined in metadata) are labelled as “good” or “bad” by the experiment ◊ Objects: Particle Flow Jets, Calorimeter Jets, Photons, Muons ◊ (p T , 𝜃 , 𝜒 , V xyz , mass) for 5 particles in quantiles in p T ◊ 7 features for every variable: ◊ quantiles: 0, 0.25, 0.5, 0.75, 1. + mean + variance ◊ over objects of all events in given LumiSection ◊ ∼ 2500 features describing every LumiSection Fedor.Ratnikov@cern.ch 7 Anomalies in High Energy Physics

  8. Reference Performance ◊ Use training part of all available data to train classifier ◊ ultimate best case scenario ◊ Analyse test part of data ◊ get probability for the given LS to be “good” ◊ select two probability thresholds: Cut _bad , Cut _good ◊ define three zones ◊ “black zone” - LS is definitely bad ◊ “white zone” - LS is definitely good ◊ “grey zone” - classifier is in doubt, expert decision is needed expert decision automatic decision black zone grey zone white zone 0 Cut “bad” Cut “good” 1 Fedor.Ratnikov@cern.ch 8 Anomalies in High Energy Physics

  9. Performance black zone grey zone white zone 0 Cut “bad” Cut “good” 1 ◊ Loss Rate ◊ “good” LS is classified as “definitely bad” and thus is lost for physics ◊ LR = FN(“black”) / (TP(“white”) +FN (“black”)) ◊ Pollution Rate ◊ “bad” LS is classified as “definitely good” and thus pollutes certified data ◊ PR = FP(“white”) / (TP(“black”) + FP(“white”)) ◊ Rejection Rate ◊ fraction of all LS which are not automatically classified as “definitely bad” or “definitely good” ◊ RR = (“grey”) / (“black” + “grey” + “white”) ◊ ManualWork = RejectionRate ∼ 80% saving on manual work is feasible for PR and LR at 5‰ Fedor.Ratnikov@cern.ch 9 Anomalies in High Energy Physics

  10. Decomposing Anomalies ◊ Study effect of anomalies on individual channels ◊ what channels are responsible for anomalies? ◊ if only photons are affected, may muon data still be used? ◊ which plots should receive more attention from Data Quality experts? ◊ Decomposition of Channels ◊ build separate NN for every channel ◊ corresponding NN scores each channel ◊ connect networks by ◊ “min” operator with dropout ◊ exp ( (f i subnetwork - 1)) a kind of “fuzzy AND” ◊ train network to approximate a global score ◊ individual NN has high predictive power against anomalies within corresponding ◊ this may be mathematically proven in some reasonable for our case assumptions Fedor.Ratnikov@cern.ch 10 Anomalies in High Energy Physics

  11. NN Design Cal Particle Flow Photon Muon ◊ Use 3 - layer NN ◊ Each subnetwork returns score ◊ close to 1 for good lumisections ◊ close to 1 for anomalies “invisible” from subnetwork’s channel data ◊ close to 0 for anomalies “visible” from subnetwork’s channel data ◊ Thus NN decomposes anomalies by channels Fedor.Ratnikov@cern.ch 11 Anomalies in High Energy Physics

  12. Decomposition Results globally good lumisections globally anomalous lumisections ◊ Different channels contribute differently Fedor.Ratnikov@cern.ch 12 Anomalies in High Energy Physics

  13. Correlations T r a c k s m u o n s E G a m m a J e t M e t ◊ Reverse test ◊ trying to predict output of the network for subsystems. ROC AUCs: ◊ muon: 0.89 ◊ photons: 0.95 ◊ particle flow: 0.86 ◊ calo: 0.94 ◊ 𝒪ℬ : expect ==1 in our assumptions Fedor.Ratnikov@cern.ch 13 Anomalies in High Energy Physics

  14. Rare Anomalies ◊ 2010 Open Data contains significant fraction of bad data ◊ 1:2 bad-to-good lumisections ◊ Better data quality in Run 2 ◊ 1:100 bad-to-good lumisections ◊ lack of anomaly data for supervised learning ◊ also the case for LHCb ◊ Need other approaches Fedor.Ratnikov@cern.ch 14 Anomalies in High Energy Physics

  15. Rare Anomalies ◊ Assumptions 1. good samples are embedded in small region of low-dimensional subspace 2. every point outside this region is an anomaly ◊ Technically, two-class problem ◊ suffers from class disbalance ◊ very few anomalous data ◊ assumptions allow using one-class methods ◊ but then still available information about anomalies would not be used ◊ Need to merge one-class and two-class approaches Fedor.Ratnikov@cern.ch 15 Anomalies in High Energy Physics

  16. Mixed Objective ◊ Consider classification of objects of class 𝓓 ◊ can use “one-class on 𝓓 ”, e.g. one-class SVM ◊ Add artificial noise data 𝓞 to fill initial phase space ◊ then classifier 𝓓 against 𝓞 effectively separates 𝓓 from the rest of the phase space ◊ “one-class on 𝓓 ” = “ 𝓓 against everything” ◊ Now anomalous data may be added to the noise ◊ Loss function: 𝓜 = 𝓜 + + (1- 𝛽 ) 𝓜 - + 𝛽𝓜 noise ◊ 𝓜 +, 𝓜 - , 𝓜 noise - losses on normal, anomalous, and noise examples ◊ 𝛽 - trade-off parameter Fedor.Ratnikov@cern.ch 16 Anomalies in High Energy Physics

  17. Illustration ◊ If negative samples are nearby positive region, produce solution as in classification problem ◊ Otherwise produce one-class bordering ◊ Toy example: 2D Gaussian normal (green), random anomalous (red) ◊ mixed objective produces more accurate separation Fedor.Ratnikov@cern.ch 17 Anomalies in High Energy Physics

  18. Noise Injection ◊ Each layer of the deep network acts like a dimensionality reduction ◊ can inject noise more consistently into the middle of the net ◊ 𝓜 noise imposes a bias proportional to the phase volume of the positive class ◊ positive class volume tends to collapse to a single point ◊ add embedded Auto Encoder to penalties positive class volume shrinking ◊ 𝓜 = 𝓜 + + (1- 𝛽 ) 𝓜 - + 𝛽𝓜 noise + 𝛾𝓜 AE Fedor.Ratnikov@cern.ch 18 Anomalies in High Energy Physics

  19. Tests on the Same Problem Noise Noise Noise Noise Injection Injection Injection Injection ◊ Data from the decomposition studies ◊ train/test: 10K/10K - positive, 64/6.4K - negative ◊ 800 features (reduced) ◊ ROC AUC (32 experiments) - 0.85±0.02 ◊ 0.80±0.05 without noise injection and autoencoder Fedor.Ratnikov@cern.ch 19 Anomalies in High Energy Physics

  20. Access to Actual Data ◊ Actual studies need access to actual data ◊ historical (open) data may not represent the current status ◊ Both DS and detector operation expertise are necessary to implement advanced approaches into the detector operation chain ◊ cooperation between Yandex and CMS via CERN Open Lab is established ◊ conditional access to real time data is granted Fedor.Ratnikov@cern.ch 20 Anomalies in High Energy Physics

  21. Conclusions ◊ Yandex group develops procedures to detect anomalies in detector data ◊ different approaches may work for different run conditions ◊ Current data access policies allow technical access to data in real time (break through since last year) ◊ CERN Open Lab in action ◊ Started moving from academic studies to practical solutions Fedor.Ratnikov@cern.ch 21 Anomalies in High Energy Physics

Recommend


More recommend