market simulation data model validation
play

Market Simulation Data & Model Validation Presentation to SEM - PowerPoint PPT Presentation

Oct 05, 2023 •358 likes •845 views

Market Simulation Data & Model Validation Presentation to SEM Market Participants Agenda Description of Project Calibration of PLEXOS Validation of PLEXOS Input Data Observations for the Future 1 Description of Project 2

  1. Market Simulation Data & Model Validation Presentation to SEM Market Participants

  2. Agenda � Description of Project � Calibration of PLEXOS � Validation of PLEXOS Input Data � Observations for the Future 1

  3. Description of Project 2

  4. Objectives � Examine the ability of PLEXOS to produce results that reflect the MSP Software that determines market prices by half hour � Develop validated PLEXOS database for the October 2008 to September 2009 period suitable for use by the Regulatory Authorities for various purposes including the development of Directed Contract quantities and prices 3

  5. PLEXOS Calibration � Actual market results are available from November 1, 2007 � These results provide an opportunity to view the ability of PLEXOS to simulate market operation � A PLEXOS database was developed using actual load, actual generator availability and actual commercial offers for all thermal units � PLEXOS results using these data produced SMPs consistent with actual SMPs - that is within 2% averaged over all intervals � Insight was gained as to impact of PLEXOS settings 4

  6. PLEXOS Database Development � NERA began with the validated PLEXOS database developed by KEMA in 2007 � Updated forecast of load, wind and embedded generation were provided by Eirgrid � Market participants were provided an opportunity to review and update data � NERA updated the 2007 representation of the availability of Moyle energy � Fuel and carbon price inputs were developed from forward fuel prices with transport, currency and tax adjustments 5

  7. Calibration 6

  8. Calibration of PLEXOS Objective � Calibrate PLEXOS vs. actual half-hourly prices from the SEM & recommend required adjustments to settings/ assumptions and/or results Process 1. Run PLEXOS with actual load and commercial offer data & compare results to actuals 2. Identify reasons for differences 3. Modify assumptions or settings to reduce those differences 4. Go back to Step 1 until results sufficiently calibrated 7

  9. PLEXOS is Significantly Different than the MSP Softw are Model � Examines a longer time frame � Typically used with availability not known � While it has a large scale optimization option for unit commitment (MIP) the option is not always feasible to utilize � Attempts to simulate a full scale optimal unit commitment with approximations (rounded relaxation) 8

  10. Calibration of PLEXOS � Poor initial results using 2007 settings � Correction to MSL filter � Use of cold, warm and hot starts the largest issue � Rounded relaxation settings & MIP analysed in detail � Alternative peaker MSLs investigated � Alternative treatments of Moyle investigated � Final results are acceptably well calibrated 9

  11. Calibration of PLEXOS SMP Comparison: November 2007 - February 2008 200.00 150.00 €/MWh 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 10

  12. Calibration of PLEXOS SMP Comparison: November 2007 200.00 150.00 €/MWh 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 11

  13. Calibration of PLEXOS SMP Comparison: December 2007 200.00 150.00 €/MWh 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 12

  14. Calibration of PLEXOS SMP Comparison: January 2008 200.00 150.00 €/MWh 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 13

  15. Calibration of PLEXOS SMP Comparison: February 2008 200.00 150.00 €/MWh 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 14

  16. Calibration of PLEXOS SMP Comparison: Week 7 € 350.00 € 300.00 € 250.00 € 200.00 €/MWh € 150.00 € 100.00 € 50.00 € 0.00 Actual SMP Plexos SMP 15

  17. Calibration of PLEXOS SMP & Shadow Price Comparison � Overall SMP calibration is reasonable and no systematic bias warranting an adjustment of results was found � PLEXOS shadow price is low � PLEXOS uplift is high � The effects are more-or-less offsetting even on a half hourly basis � The cause is over-commitment by PLEXOS � Over-commitment by PLEXOS appears to be systematic 16

  18. Over Commitment Requires Modelling Adjustments � MSL filter removes impact on uplift of many units scheduled in PLEXOS at minimum stable load which does not happen in MSP Software � Use of warm start cost only avoids significant overstatement of SMP as a result of very high uplift values � Higher rounded relaxation tolerance can improve backcast results by reducing over-commitment. However, – In forecast mode higher rounded relaxation tolerance led to high instances of unserved energy. – For this reason, the middle RR level of 5 is recommended in forecast modelling and was used in NERA’s final backcast calibrations. 17

  19. Calibration of PLEXOS SMP Comparison: Use of 3 start states € 90.00 € 80.00 € 70.00 € 60.00 € 50.00 €/mwH € 40.00 € 30.00 € 20.00 € 10.00 € 0.00 Actual Plexos Warm Only Plexos 3 State Shadow Price Uplift 18

  20. Calibration of PLEXOS SMP & Shadow Price: Nov 2007-Feb 2008 (Actual) 200.00 150.00 €/MWh 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual SMP Actual Shadow Price 19

  21. Calibration of PLEXOS SMP & Shadow Price: Nov 2007-Feb 2008 (PLEXOS) 200.00 150.00 €/MWh 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Plexos SMP Plexos Shadow Price 20

  22. Calibration of PLEXOS Base prices Nov-Feb Mid prices Nov-Feb Peak prices Nov-Feb Sh. Price Uplift SMP Sh. Price Uplift SMP Sh. Price Uplift SMP Actual € 63.08 € 5.28 € 68.36 Actual € 71.74 € 7.33 € 79.07 Actual € 99.86 € 20.57 € 120.42 PLEXOS € 54.68 € 15.06 € 69.74 PLEXOS € 58.98 € 20.21 € 79.19 PLEXOS € 67.66 € 55.40 € 123.05 Variance (€ 8.40) € 9.78 € 1.37 Variance (€ 12.77) € 12.88 € 0.12 Variance (€ 32.20) € 34.83 € 2.63 Base prices Nov Mid prices Nov Peak prices Nov Sh. Price Uplift SMP Sh. Price Uplift SMP Sh. Price Uplift SMP Actual € 58.50 € 5.21 € 63.72 Actual € 67.30 € 6.96 € 74.26 Actual € 95.13 € 19.22 € 114.35 PLEXOS € 49.46 € 15.70 € 65.16 PLEXOS € 54.38 € 21.94 € 76.33 PLEXOS € 62.53 € 61.38 € 123.91 Variance (€ 9.04) € 10.48 € 1.44 Variance (€ 12.92) € 14.99 € 2.07 Variance (€ 32.59) € 42.16 € 9.57 Base prices Dec Mid prices Dec Peak prices Dec Sh. Price Uplift SMP Sh. Price Uplift SMP Sh. Price Uplift SMP Actual € 55.85 € 5.24 € 61.09 Actual € 64.47 € 7.03 € 71.50 Actual € 99.44 € 18.59 € 118.02 PLEXOS € 47.46 € 21.38 € 68.84 PLEXOS € 51.89 € 26.81 € 78.70 PLEXOS € 58.88 € 63.32 € 122.20 Variance (€ 8.39) € 16.14 € 7.75 Variance (€ 12.58) € 19.78 € 7.20 Variance (€ 40.56) € 44.73 € 4.18 Base prices Jan Mid prices Jan Peak prices Jan Sh. Price Uplift SMP Sh. Price Uplift SMP Sh. Price Uplift SMP Actual € 70.52 € 5.96 € 76.48 Actual € 79.70 € 8.57 € 88.27 Actual € 104.18 € 25.53 € 129.70 PLEXOS € 62.04 € 10.76 € 72.80 PLEXOS € 66.24 € 15.68 € 81.93 PLEXOS € 79.53 € 53.12 € 132.65 Variance (€ 8.48) € 4.80 (€ 3.68) Variance (€ 13.46) € 7.12 (€ 6.34) Variance (€ 24.65) € 27.60 € 2.95 Base prices Feb Mid prices Feb Peak prices Feb Sh. Price Uplift SMP Sh. Price Uplift SMP Sh. Price Uplift SMP Actual € 67.51 € 4.65 € 72.16 Actual € 75.57 € 6.66 € 82.23 Actual € 100.44 € 18.67 € 119.10 PLEXOS € 59.82 € 12.19 € 72.01 PLEXOS € 63.52 € 16.12 € 79.65 PLEXOS € 69.53 € 42.94 € 112.47 Variance (€ 7.69) € 7.54 (€ 0.15) Variance (€ 12.05) € 9.46 (€ 2.59) Variance (€ 30.90) € 24.27 (€ 6.63) 21

  23. Moyle, Hydro and Pumped Storage Were Also Examined � KEMA GB model was updated and used in calibration as opposed to actual Moyle flows � Monthly actual hydro generation was used and PLEXOS shaped hydro � Pumped storage capacity and efficiency was put in model and PLEXOS determined pumped storage schedule � This provides a better test than using actual data for these values � Using actual values was examined and did not alleviate the over commitment issue 22

  24. Calibration of PLEXOS Moyle Comparison: November 2007 - February 2008 450 400 350 300 MW Flow into SEM 250 200 150 100 50 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 23

  25. Calibration of PLEXOS Hydro Comparison: November 2007 - February 2008 200.00 150.00 MW 100.00 50.00 0.00 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 24

  26. Calibration of PLEXOS Hydro Comparison: November 2007 200 150 MW 100 50 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Time of Day Actual Plexos 25

Recommend

AIP Plexos Market Simulation Model Validation Project Workshop 1 Proposed Model Review

AIP Plexos Market Simulation Model Validation Project Workshop 1 Proposed Model Review

AIP Plexos Market Simulation Model Validation Project Workshop 1 Proposed Model Review Process Mike Wilks, Principal Consultant, Markets & Regulation 18 January 2007, Davenport Hotel, Dublin Experience you can trust. Presentation

447 views • 18 slides
Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides
AIP (PLEXOS) Market Simulation Model Validation Project Workshop 3 Final Conclusions Mike

AIP (PLEXOS) Market Simulation Model Validation Project Workshop 3 Final Conclusions Mike

AIP (PLEXOS) Market Simulation Model Validation Project Workshop 3 Final Conclusions Mike Wilks, Principal Consultant Dave Lenton, Senior Consultant Adrian Palmer, Senior Consultant 30 March 2007, Davenport Hotel, Dublin Experience you

2.23k views • 66 slides
AIP (PLEXOS) Market Simulation Model Validation Project Workshop 2 Initial Findings Mike

AIP (PLEXOS) Market Simulation Model Validation Project Workshop 2 Initial Findings Mike

AIP (PLEXOS) Market Simulation Model Validation Project Workshop 2 Initial Findings Mike Wilks, Principal Consultant Dave Lenton, Senior Consultant Adrian Palmer, Senior Consultant 2 March 2007, Belfast Hilton, Belfast Experience you can

944 views • 56 slides
Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides
Development and Validation of a Thermal Simulation Model for Li-ion Batteries in HEVs/PEVs

Development and Validation of a Thermal Simulation Model for Li-ion Batteries in HEVs/PEVs

Development and Validation of a Thermal Simulation Model for Li-ion Batteries in HEVs/PEVs Project Partners Dr. Ing Porsche, Adam Opel AG, Daimler AG CD-adapco, Battery Design LLc Automotive Simulation Center e.V. 1 07.10.2011 20/03/2012

313 views • 16 slides
SEM PLEXOS Model Validation Information Seminar RAs Market Modelling Group 11:30am, 7 th June

SEM PLEXOS Model Validation Information Seminar RAs Market Modelling Group 11:30am, 7 th June

SEM PLEXOS Model Validation Information Seminar RAs Market Modelling Group 11:30am, 7 th June 2012, CER Office Agenda Background to PLEXOS Validation Project - Andrew Ebrill Calibration of Backcast Model - Kevin Hagan Validation

838 views • 34 slides
Chapter 10 Verification and Validation of Simulation Models Banks, Carson, Nelson & Nicol

Chapter 10 Verification and Validation of Simulation Models Banks, Carson, Nelson & Nicol

Chapter 10 Verification and Validation of Simulation Models Banks, Carson, Nelson & Nicol Discrete-Event System Simulation Purpose & Overview The goal of the validation process is: To produce a model that represents true

518 views • 28 slides
SEM PLEXOS Model Validation Public Workshop RAs Market Modelling Group 15:00, 13 th June 2011,

SEM PLEXOS Model Validation Public Workshop RAs Market Modelling Group 15:00, 13 th June 2011,

SEM PLEXOS Model Validation Public Workshop RAs Market Modelling Group 15:00, 13 th June 2011, CER Office Agenda Background to PLEXOS Validation Project - Andrew Ebrill Calibration of Backcast Model - Kevin Hagan & Thomas Quinn

798 views • 44 slides
Verification Verification and and Validation Validation 1 /41 1 /41 C2 C2 Overview

Verification Verification and and Validation Validation 1 /41 1 /41 C2 C2 Overview

C2 C2 Verification Verification and and Validation Validation 1 /41 1 /41 C2 C2 Overview Overview It is very simple to create a simulation ! It is very difficult difficult to model something to model something accurately .

736 views • 41 slides
Chapter 10 Verification and Validation of Simulation Models Banks, Carson, Nelson & Nicol

Chapter 10 Verification and Validation of Simulation Models Banks, Carson, Nelson & Nicol

Chapter 10 Verification and Validation of Simulation Models Banks, Carson, Nelson & Nicol Discrete-Event System Simulation The Black Box [Bank Example: Validate I-O Transformation] A model was developed in close consultation with bank

414 views • 19 slides
Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides
Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist

Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist

Introduction to hyperparameter tuning MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist Model parameters Parameters are: Learned or estimated from the data The result of tting a model Used when making future predictions Not

1.82k views • 35 slides
Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using

Cross-Validation Machine Learning 1 Model selection Very broadly: Choosing the best model using given data What makes a model Features Hyper-parameters that control the hypothesis space Example: depth of a decision tree, neural

328 views • 12 slides
The problems with holdout sets MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist

The problems with holdout sets MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist

The problems with holdout sets MODEL VALIDATION IN P YTH ON Kasey Jones Data Scientist Transition validation X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2) rf = RandomForestRegressor() rf.fit(X_train, y_train)

570 views • 26 slides
Model Validation, Sensitivity Analysis and Policy Analysis Jayendran Venkateswaran Model

Model Validation, Sensitivity Analysis and Policy Analysis Jayendran Venkateswaran Model

Model Validation, Sensitivity Analysis and Policy Analysis Jayendran Venkateswaran Model Validation Did we build the right model? Corresponds largely with testing if the model can replicate past behavior/ historical patterns

440 views • 13 slides
CS626 Data Analysis and Simulation Instructor: Peter Kemper R 104A, phone 221-3462,

CS626 Data Analysis and Simulation Instructor: Peter Kemper R 104A, phone 221-3462,

CS626 Data Analysis and Simulation Instructor: Peter Kemper R 104A, phone 221-3462, email:kemper@cs.wm.edu Today: Verification and Validation of Simulation Models Reference: Law/Kelton, Simulation Modeling and Analysis, Ch 5. Sargent,

361 views • 22 slides
Testing & Validation HIGH PERFORMANCE COMPUTING FRAMEWORK FOR CO-SIMULATION OF VEHICLE

Testing & Validation HIGH PERFORMANCE COMPUTING FRAMEWORK FOR CO-SIMULATION OF VEHICLE

Modeling & Simulation, Testing & Validation HIGH PERFORMANCE COMPUTING FRAMEWORK FOR CO-SIMULATION OF VEHICLE TERRAIN INTERACTION Radu Serban, Nicholas Olsen, Dan Negrut University of Wisconsin Madison 8/22/2018 Modeling &

663 views • 23 slides
Why Bayesian methods in Simulation? Simulation Simulation Model Inputs BAYESIAN IDEAS

Why Bayesian methods in Simulation? Simulation Simulation Model Inputs BAYESIAN IDEAS

Outline Why Bayesian methods in Simulation? Simulation Simulation Model Inputs BAYESIAN IDEAS Output Random Simulated Statistical, pr FOR DISCRETE EVENT SIMULATION: Y r Numbers Variates Control,

473 views • 20 slides
The Modelling and Simulation Process 1. History of Modelling and Simulation 2. Modelling and

The Modelling and Simulation Process 1. History of Modelling and Simulation 2. Modelling and

The Modelling and Simulation Process 1. History of Modelling and Simulation 2. Modelling and Simulation Concepts 3. Levels of Abstraction 4. Experimental Frame 5. Validation 6. Studying a mass-spring system 7. The Modelling and Simulation

692 views • 34 slides
Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A,

Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A,

Lecture #6: Model Selection & Cross Validation Data Science 1 CS 109A, STAT 121A, AC 209A, E-109A Pavlos Protopapas Kevin Rader Margo Levine Rahul Dave Lecture Outline Review Multiple Regression with Interaction Terms Model Selection:

522 views • 33 slides
Testing & Validation Co-Simulation of MBD Models with DEM Code to Predict Mobility on Soft

Testing & Validation Co-Simulation of MBD Models with DEM Code to Predict Mobility on Soft

Modeling & Simulation, Testing & Validation Co-Simulation of MBD Models with DEM Code to Predict Mobility on Soft Soil Brian Edwards Pratt & Miller Engineering Sophie Broad DEM Solutions 8/22/2018 Mobility of MBD Models on

550 views • 13 slides
Market Simulation Winter 2017 Release Christopher McIntosh Lead Market Simulation

Market Simulation Winter 2017 Release Christopher McIntosh Lead Market Simulation

Market Simulation Winter 2017 Release Christopher McIntosh Lead Market Simulation Coordinator February 15, 2018 ISO PUBLIC ISO PUBLIC Agenda Winter 2017 MAP Stage Availability Winter 2017 Market Simulation Initiatives and

366 views • 23 slides
Development and Validation of Data Assimilative East Sea Regional Ocean Model Kyung-Il Chang 1 ,

Development and Validation of Data Assimilative East Sea Regional Ocean Model Kyung-Il Chang 1 ,

Development and Validation of Data Assimilative East Sea Regional Ocean Model Kyung-Il Chang 1 , Young Ho Kim 2 , Gyun-Do Park 1 , Young-Gyu Kim 3 1 Research Institute of Oceanography/School of Earth and Environmental Sciences, Seoul National

662 views • 45 slides

More recommend