L2RPN Challenge - Learning to Run a Power Network through AI Di Shi - PowerPoint PPT Presentation

L2RPN Challenge - Learning to Run a Power Network through AI Di Shi Team: Tu Lan, Jiajun Duan, Bei Zhang, Zhiwei Wang, Xiaohu Zhang, Ruisheng Diao, Yan Zan AI & System Analytics GEIRI North America (GEIRINA) @PSERC Summer Workshop July 16, 2019

Intro of Artificial Intelligence Meaning of Different Terms: AI, ML, DL Source: Nvidia A process where a computer A subset of ML and refers to A subset of AI and refers to solves a task in a way that mimics artificial neural networks algorithms that parse data, human behavior. composed of many layers. learn from them, and then apply what they’ve learnt to Generalized AI vs. Applied AI. make intelligent decisions. Credit: Nvidia 2

Intro of Artificial Intelligence Milestones of AI Development Self-driving cars 2016 - present AlphaStar defeated top human players in Star Craft II Biometrics recognition 3 Source: https://www.pinterest.com/pin/786792997375069862/?lp=true

Intro of Artificial Intelligence AI Categories and Applications 4 *Source: https://towardsdatascience.com/machine-learning-for-biginners-d247a9420dab

Summary of Key AI Technologies Deep Learning is an extension Supervised Semi-supervised Reinforcement Unsupervised Learning Learning Learning Learning of supervised, unsupervised target many unlabeled & reward environment unlabeled error labeled & state few labeled data data and semi-supervised learning data In Out In Out In Out In Out using many layers Application Application Application Application  Classification  Clustering  Google Photos  DeepMind’s AlphaGo  Predict a target  Visualization  Webpage classification  Fire-extinguish robots  Dimensionality  Grid Mind numeric value reduction  Anomaly detection Common Algorithms Common Algorithms Common Algorithms Common Algorithms o k -Nearest Neighbors o k -Means o Combination of o Dynamic programming o Linear Regression o Hierarchical Cluster o Monte Carlo unsupervised and o Temporal Difference o Decision Trees DRL= DL + RL Analysis supervised learning o Principal Component o Naïve Bayes (TD)  Q-Learning o SVM Analysis  SARSA o Neural Networks 5

Applications of AI in Power Systems • Model validation and calibration • Excitation and damping control • Generation Maintenance Scheduling • Renewable Forecasting • Intelligent monitoring & early warning • Intelligent diagnosis of equipment Transmission • Image recognition of power lines • Situational awareness • Knowledge map & intelligent reasoning • Distribution Fault detection and location • Intelligent analysis and self-healing ctrl • Demand forecasting End user • Load clustering and par. identification Trend of AI in Power Grids Potential Applications • Power system operation and control Reasoning/planning Monitoring • Power system asset management GEIRINA’s R&D Focus! Diagnosis Decision making • Power system mid/long term Forecasting Autonomous control planning • • • • • LSTM (D)DQN PPO Power system economics and market RNN • • • • GAN CNN DDPG SAC 6 • SVM… • • • TRPO… GNN A3C

Outline • L2RPN Challenge • Early Stage Attempts • Proposed Methodology – the winning algorithm • Imitation Learning and DRL • Training Methods and Adaptive Adjustments • Results 7

About the Competition Timeline for the Competition May 15th, 2019 : Beginning of the competition with the release of public RL environment. Participants can start submitting agent models on Codalab platform and obtaining immediate feedback in the leaderboard on validation scenarios. May 27th, 2019 : Potential release of a new baseline to foster competition if several participants are already doing better than this baseline. June 15th, 2019 : Start of the testing days on unseen test scenarios. June 19th, 2019 : End of the competition, beginning of the post-competition process Jul 1rst, 2019 : Announcement of the L2RPN Winners. Jul 14th, 2019 : Beginning of IJCNN 2019. This was later extended to Jun. 23 rd , 2019 • https://l2rpn.chalearn.org/ • https://competitions.codalab.org/competitions/22845. 8

Problem Statement • Run the power network through topology control 13 Why should we care? 12 14 • Rising complexity of 11 the power grid 10 • Integration of 9 6 renewables 7 8 • AC + DC loads … 1 4 5 • Costly to build new lines 3 2 Q: How to alleviate the burden through the topology control? 9

Problem Analysis - System System Summary • 14 Buses 100 A • 155.3 A 5 Generators • 353.7 A 175.1 A 11 Loads 390.5 A • 208.9 A 20 Lines (thermal 211.8 A 100 A limits as indicated) 123 A Slack Bus 161.6 A 150 A 315.5 A 241 A 399.9 A 301.9 A • The system is successively running at an interval of every 5 min 996.8 A 447.1 A 374.4 A 221 A • Training: thousands of scenarios considered; around 1-month time series data in each scenario 428.4 A • Official Test: 10 scenarios ; data of 1-3 days in each scenario

Problem Analysis - Objective • Analyze the problem in the framework of an optimization problem Optimization problem: Maximize the remaining power transfer capability over all time steps of all scenarios obj. Min/Max (Objective) *Note: Game Transfer Capability at a Time Step: s.t. Constraint_1 over when certain Constraint_2 constraint is violated Constraint_3 Transfer Capability at a Scenario: … * • Decision Variable Transfer Capability of All Scenarios: • Parameters 1 st day 2 nd day n th day Scenario 1: 1 st day 2 nd day n th day Scenario 2: … 1 st day 2 nd day n th day Scenario n : 11

Problem Analysis – Decision Variables • Decision Variables: what can we control to maximize the power transfer capability? - Topology of the network at all time steps in all scenarios Node Splitting ( 156 for 14 nodes) Line Switching ( 20 lines) Bus1-1 Bus2 + Bus2 Bus1 Bus2 Bus1 Bus1-2 e.g. *Note: A Maximum of 1 action at Bus1-1 Bus1-1 the node + 1 action at a line per timestep is allowed Bus2 Bus2 Bus1-2 Bus1-2 Totally 3120 (156*20) topologies! 1 st day 2 nd day n th day Scenario 1: 1 st day 2 nd day n th day Scenario 2: … 1 st day 2 nd day n th day Scenario n : 12

Problem Analysis – Hard Constraints • Game Over if any of the hard constraints is violated: • Load should be met over all time steps of all scenarios • No more than 1 power plants get disconnected over all time steps of all scenarios • The grid should not get split apart into isolated sub-grids over all time steps of all scenarios • AC power flow solution should converge over all time steps of all scenarios 1 st day 2 nd day n th day Scenario 1: 1 st day 2 nd day n th day Scenario 2: … 1 st day 2 nd day n th day Scenario n : 13

Problem Analysis – Soft Constraints • Violation on soft constraints may lead to certain consequences though not immediate “game over”: • Line overload should be controlled over all time steps of all scenarios : Scenario Consequence Time Steps to Recover Line Flow >= 150% Line immediately broken and disconnected 10 100% < Line Flow < Wait for 2 more timestep to see whether the 3 150% overflow is resolved; If not, line gets disconnected • Cooldown should be considered: 3 steps of cool down is required before a line or node can be reused, the violation on this will cause: 1) step score to be 0; 2) the action will not be taken, resulting in no action. 1 st day 2 nd day n th day Scenario 1: 1 st day 2 nd day n th day Scenario 2: … 1 st day 2 nd day n th day Scenario n : 14

Problem Analysis – Parameters Load Profile Maintenance Profile Fault Profile Gen Profile Not considered in the competition, for future extension Voltage Profile 2018-01-02 2018-01-01 … … Interval: Every 5 min! 15

Problem Analysis - Summary • Hard problem to solve within the conventional optimization framework: 1) hard to solve the mixed-integer nonlinear dynamic optimization (AC power flow); 2) so many hard and soft constraints; 3) hard to mathematically model those dynamic constraints; 4) huge scale due to the consideration of long continuous timesteps • We may refer to DRL-based method but still with difficulties: • Selection on the action space (huge action space leads to difficulties in convergence), which will be further explained later • Long continuous timestep: the desired DRL agent should be able to operate the system in hundreds and thousands of timesteps • Many hard and soft constraints: this complicates the problem, and greatly increases the difficulty in training the agent. 16

Problem Formulation - If Using Traditional Optimization Approaches • As illustrated by the figure at the bottom, a large number of binary variable should be introduced to represent the connection status of each component, e.g., generator, load, line, etc. • The objective is to maximize the system available transmission capacity, an  auxiliary variable is introduced: k    0, k k  is constrained by max 𝜇 𝑙 S k     2 k 1 ( ) , k k max 𝑙∈𝛻 𝑙 S k 17 Generalized model for network topology change