Skip to Content

Tag: assumptions

Opening the black box (Part 3 of 3)

Posted on by Angela Vumbaca

A beginner’s guide to wholesale market modelling

Context – where we left off

Last time we completed our description of the end-to-end modelling process. But we noted that there remained many different tricks and tips that were hidden from the layperson.

With that in mind, we promised that in our final instalment in the series we would examine the assumptions and methods that fall into the category of ‘the Dark Arts’ of modelling.

At the outset, we make clear that our intention is to shed light on these often overlooked parts of the modelling process. We want the reader to understand what models can and cannot do, and the ways, means, and devices that together are sometimes used that can make it hard to derive value from modelling exercises. We want to arm you with the basics of ‘Defence against the Dark Arts’.

In the remainder of this article, we explain our thoughts on the top four ‘Dark Arts’ of the modelling process, specifically:

  • Crude chronologies – playing with time itself.
  • Carbon budgets and renewable targets – hiding costs in shadow prices.
  • Market based new entry – a no longer necessary evil.
  • Post-processing of results – what is helpful and what is not.

We conclude by setting out how we think about the role of modelling, and the philosophy that underpins our own work in this area.

Crude chronologies – playing with time itself

For capacity expansion, it is necessary to simplify the number of periods (sometimes called time slices or load blocks) over which we model the system. For example, a single day might be represented by 3 time slices: one for each eight-hour period of the day. In turn, this day might represent 30 days, which have similar characteristics. In so doing, 30 * 24 =720 hours could be represented by a mere 3 time slices.

The benefit is that the problem is easier, and so faster, to solve. But the approach of using very few time slices has been limited by the advent of:

  • intermittent renewables, whose output shapes change greatly from one day to the next over the course of the year, and
  • storage devices that force us to use time slices that obey a time-sequential chronology (ie, a set of time slices where time constraints are preserved).

It is now the case that unless we use a very large number of time slices, the system that results from the capacity expansion problem is not resilient. The solution is to use more time slices – this is not really an option, unless we are looking at a very simple power system.

Despite this reality, some modellers often want to speed up the process, or do not have access to commercial solvers, and so they resort to use of a crude set of time slices. The outcome can often be a brittle or, on the other hand, massively overbuilt system. When working with a consultant using capacity expansion modelling, it is worth being aware about the assumptions they have made as to time slices – a Dark Art if ever there was one.

Carbon budgets and renewable targets – hiding costs in shadow prices

One of the powerful aspects of using an optimisation framework for solving models is the ability to impose constraints. It is trivial to add constraints to the model that do the following:

  • Place a limit on the amount of CO2 emissions that can be produced in a given timeframe.
  • Force in a particular level of renewable generation by some date.
  • Represent other limits or restrictions on the operation of the capacity expansion model.

In practice, these constraints are just like any other constraint, such as the supply-demand constraint, or a limit on the amount of generation that can connect in a region. These constraints have the potential to change the outcomes that would have otherwise occurred in their absence.

When these constraints are imposed, the model will do whatever it takes (such as over-building renewable or artificially suppressing thermal generation out-of-the-merit-order) to meet these targets. This will occur regardless of whether the new investment will recover its costs from the energy prices in a potentially over-supplied market. This is not a fault of the least cost expansion model per se – imposing these constraints is to assume they will be met regardless of the cost, and the model is faithfully satisfying the request of the modeller by identifying the cheapest way to meet these requirements.

However, a modeller should know to look for these ‘hidden costs’. Just like the supply-demand constraint, every constraint in the model gives rise to a ‘shadow price’. As explained in our first article, this is the marginal value of alleviating this constraint by one unit on the total system cost. In the case of a renewable target or carbon budget, there is a clear meaning to their associated shadow prices:

  • In the case of a carbon budget, it is the implied cost of abatement of (the last) tonne of CO2 – ie, the implicit cost of reducing CO2 emissions due to the additional clean technology investment required.
  • In the case of a renewable target, it is the implied subsidy per MWh that is required to achieve that level of renewable penetration. This could be seen as the renewable energy certificate value that is required to make renewables whole.

Of incredible importance is that, when we invoke these constraints, the prices that fall out of the capacity expansion model assume that there is also a carbon emissions price, and a renewable energy certificate price. And so if we look only at the price that falls out of the model and assume this will be the cost of electricity, we will fail to account for the effect of the assumed carbon and renewable energy certificate prices, or the ‘missing money’ that must be funded outside the wholesale electricity market. These ‘hidden’ prices are often overlooked or forgotten about, but they are critical to understanding how generation recovers its costs. Moreover:

  • without some sort of carbon price or emissions trading mechanism, it makes no sense to include a carbon budget constraint; and
  • without some sort of subsidy to renewables, it makes no sense to include a renewable energy target constraint.

It is critical to be aware of this – otherwise we are not going to factor in the whole cost of policies, and so we may not understand what is required to achieve them.

Market-based new entry – a no-longer necessary evil

One of the perennial questions in the sector is whether a least-cost capacity expansion model is the correct way to determine new entry. The alternative, which is used by many advisors, is called ‘market-based new entry’. 

The recent surge of market-based new entry

Recent years have seen a surge of market-based new entry in market modelling. We often hear the motivation of using this approach is that new entrant generation built by least-cost capacity expansion models does not seem to earn enough pool revenue to recover its costs. Market-based new entry appears to fix this problem by allowing the modeller to alter the investment path manually until the all new entrants recover their costs.

We will address the economic fallacy of market-based new entry shortly. However, it is first worth noting that in a market where renewables are built to meet ambitious government renewable constraints or carbon budgets, new entrants will not recover their cost from pool prices alone. This is because of the “missing money” phenomenon discussed in the last section. The least cost expansion model merely exposes this fact, which is either misunderstood or ignored by proponents of market based new entry when communicating their results.

The quasi-economic argument against least-cost capacity expansion

One argument against least-cost new entry is that the decision to build is based on cost rather than price which is what we see in the market. The argument is quasi-economics at its best. It fails to recognise that price is inherently linked to long run marginal cost, and so solving for cost gives us the outcome we seek.

The proponents of market-based new entry fail to recognise that the capacity expansion model solves for cost but also produces prices. These prices are equal to the long-run marginal cost of generation.  Taken to its logical conclusion, a market-based new entry proponent would suggest that dispatch also needs to be ‘based on price’. This would involve iterating between different combinations of plant to be dispatched in a given interval until eventually we stumbled upon the same outcome that is yielded by the dispatch engine. Why would we do such a thing, when we can get the answer directly by solving as a linear program?

Market based new entry as a low-quality capacity expansion model

In the absence of constraints, the prices yielded by the least-cost capacity expansion are exactly the level needed so that every technology recovers its costs. In effect this is equivalent to a world where investors build new plant right up until the point where any additional plant would be uneconomic. This is also the common goal shared with market-based new entry.

Put another way, we see the following outcomes:

  • Least-cost capacity expansion satisfies the requirement that every plant that gets built receives a price that at least satisfies the costs over the life of the plant. At its best, market based new entry should do the same, but this is not guaranteed by the iterative process.
  • Least-cost capacity expansion satisfies the requirement that whenever price is high enough to support a new entrant, it gets built. Again, at its best, market based new entry should do the same, but this is not guaranteed by the iterative process.

The key phrase here is ‘at its best’. In reality, the market-based new entry process is trying to solve a very complicated problem with very primitive mathematical machinery. We know that some consultants complete this process manually, with choices being made by an analyst on-the-fly. This is essentially ‘dealer’s choice’ – ie, the outcomes depend on the gut-instinct of the modeller.

Even when an algorithm, or heuristics, are codified in a program, there is no guarantee that the process will converge to an outcome that satisfies both (a) and (b) above. The chances of stumbling on the exact combination of build over the next 25 years that yields prices that walk the tightrope between (a) and (b) above is vanishingly small. This means that there will always be either plants that are not built that should have been, or plants that are built that should not have been.

The computational argument against least-cost capacity expansion

A decade ago, least-cost capacity expansion models took a very long time to solve, and so required very crude chronologies. These models therefore were internally consistent but gave a poor representation of the planning needs of the system. It was not uncommon for these models to lack resilience. Against this backdrop, market-based new entry was an alternative but sometimes necessary evil to provide a meaningful view of the future needs of the power system.

However, in the last decade the speed of solvers has improved dramatically. Gurobi can now solve a least-cost capacity expansion problem with thousands of time slices per year over a 25-year horizon in a matter of hours. It is no longer necessary to stumble around with iteration and primitive heuristics to find a profile of the needs of the system. What was once a helpful tool to plan out the system is now no longer necessary, and there is no basis for it.

Market-based new entry is now a Dark Art

As the system becomes increasingly complex, market-based new entry becomes less and less helpful for planning, resilience testing, and projections of the future needs of the system. For example, when we have been asked to reconcile market-based new entry models against the results of least-cost capacity expansion, we find that there are either large amounts of unserved energy, or many plants that are built that do not recover their costs.

More importantly, the ‘dealer’s choice’ phenomenon means that we cannot easily replicate or reproduce the analysis of other consultants who have used market-based new entry. Outcomes based on choices made in the early hours of the morning by a bleary-eyed analyst as to what plant should get built where, are not consistent with robust modelling practices.

The continued use of these models without recognition of their limitations is surely one of the most prevalent Dark Arts of the sector.

Post-processing of results – what is helpful and what is not

By now it has become clear that there are elements of the market and the power system that are fundamentally challenging to capture. And so not everything can be included in the capacity expansion and operational dispatch models. Many consultants (ourselves included) provide services that alter the outputs of the model in a post-processing step. For example, it is not unusual to add interval-to-interval volatility or ‘noise’ to prices to reflect the patterns we see in the market.

We do not see anything wrong with this, provided it is clear as to what is the effect of this post-processing. Advisors should decompose their results so that the contribution of these post-processing steps can be captured – sometimes they are highly material to their findings.

In addition, some post-processing seeks to account for high-impact system events (such as the Callide C explosion) that cannot be meaningfully included in a small number of deterministic model runs. This is not to say that models cannot simulate this type of price impact if we know exactly when and how they happen in the future. In fact, the revenue impact on other assets, which is what most commercial clients are interested in, can even be easily obtained via back-of-the-envelope calculations. The real challenge is to project the frequency with which such events will happen and how the system unfolds immediately following it, which is often highly event-specific and beyond the realm of market simulation models. Including events of this type in a projection of the future and pretending that there is a sophisticated methodology for ‘modelling’ them goes beyond the Dark Arts and verges on the Unforgivable Curses. Market models are not designed to simulate such events – be wary of anyone telling you they can do so.

Conclusion – what’s it all about

We have listed here some of the Dark Arts, but in truth there are too many to list. Modellers can always use sleights of hand, and tricks to deceive their audience – the Dark Arts. In my opinion, the problem here stems from our understanding of the purpose of modelling. From my perspective, the purpose of a model is to create a mental latticework on which we can build intuition. When constructed properly, that latticework will remain rigid and so will not simply yield to your gut instincts – it will require that you adhere to its assumptions and logic, and so build your own understanding of the problem. We can change the latticework (through changing assumptions) but that gives rise to a new set of constraints on our logic.

In contrast, when we force models to yield to our own intuition, and to give us the answers that we want, those models lose all meaning. This is why modelling is particularly unhelpful in adversarial processes, where the objective is to show that one model is ‘wrong’ and one is ‘right’. Models were never meant to be ‘wrong’ or ‘right’ – they were meant to inform our thinking.

Sometimes models give us helpful insights, and sometimes they do not. But once problems become sufficiently complex, without them we are left without any sound basis for decision making. The energy industry poses many such problems, where the stakes are high, and the risks are many. We are foolish to leave decision-making to gut instinct alone and not use models, but the more we can learn about those models and understand them, the better our decisions will be. A more informed world is the outcome. Our purpose in discussing this topic is to arm you with the basics of ‘Defence against the dark arts’. This is the information you need to understand both what models cannot do, but more importantly the many powerful questions that can indeed be answered by modelling if we are able to stretch our understanding.

Opening the black box (Part 2 of 3)

Posted on by Angela Vumbaca

A beginner’s guide to wholesale market modelling

Context – where we left off

Last time we spoke about the use of wholesale market models to answer a wide variety of questions across the energy sector. We described the concepts of linear programming, solvers, and even how we derive prices from these tools. We then explained the first step in the modelling process – capacity expansion modelling, which allows us to determine the least-cost combination of generation technologies to meet demand.

But there were some aspects missing from our capacity expansion model, because of the computational complexity of including them. We concluded by foreshadowing the next step in the standard modelling framework, ie, the simulation of real-time dispatch – the subject of this second article.

Part (2) Operational dispatch modelling – creating a more granular picture

Having built the arena, the next step is to watch the operation of the system play out. There are three specific factors that we wish to capture in more detail in this process:

  • Bids from participants.
  • Detailed operational plant limitations, such as ramp rates, minimum-stable levels and outages.
  • Variation in renewable energy and demand traces.

We note that there is no reason that a capacity expansion model could not have captured each of these factors, save for the computational burden of doing so. When we have limited time and resources, it makes sense to ‘lock the build’, and examine these other factors in more detail.

The process is to run a time-sequential simulation model, which takes as an input the technologies and capacities from the capacity expansion model. This is a far simpler problem, which can be run many times over with different inputs for demand, renewable energy traces, fuel costs, and any other parameters of interest.

In the remainder of this article, we will examine each of the three factors described above, and how they are handled in the dispatch modelling.  

Choose your poison – bidding assumptions

Let us start with the most important, most controversial, of all assumptions: bids from technologies. So often when we try and explain a strange phenomenon in the market, the answer comes back: bidding behaviour. An unexpected price spike to the market price cap – bidding behaviour; counter-price flows on an interconnector – bidding behaviour; high prices for sustained periods on a mild day – bidding behaviour. Indeed, if one is at a loss for explaining a phenomenon, the best bet to avoid embarrassment is to give a sagely shake of the head and appeal to the higher power: ‘bidding behaviour’.

Bids are of such great importance because they collectively give rise to the supply curve that, together with demand, is responsible for price formation. And because demand is highly inelastic – ie, it does not respond to price – it is the supply curve that is responsible for a many of the phenomena in the sector that are otherwise inexplicable.

But what bidding assumptions should we use? Bids can vary from day to day, hour to hour and, even in some cases, minute to minute because of rebidding. Although it is entirely possible to reconstruct outcomes given historical bids, making projections about future bidding behaviour is far more complex. This is particularly the case in an environment where the technology capacity mix is changing, eg, when new plants are rapidly entering the system, or older plants are retiring.

The Holy Grail of bidding assumptions is some mechanism for determining how plants will bid in their capacity in any future world, whether that world be defined by:

  • a high penetration of low-cost renewables,
  • an aging and less reliable thermal fleet,
  • an increasingly interconnected system, or
  • any combination of the above.

Despite many claims to the contrary, no such mechanism exists. Yes, it is possible to create bids based on rules, or game-theoretic frameworks, but in the end they all result in the same outcome – players will bid some proportion (potentially none) of their capacity at a level that exceeds their short-run marginal cost. Some typical assumptions are as follows.

Approach 1: Contract bidding:

Players will bid in their contracted level of output at SRMC, but will then bid all remaining output with some mark-ups. A problem with this outcome is that we must assume a contract level. How contracting changes with changes in market conditions and the change of asset ownership will be difficult to forecast for every plant into the future, and so requires us to make assumptions. In effect, we are still assuming a supply curve.

Approach 2: Game theoretic bidding:

Players are assumed to bid based on the assumption of maximising their profit, subject to the strategies of other players. The assumption is that by iterating between players and giving them opportunities to change their bids, we will converge to a Nash-equilibrium (ie, a world where nobody has a reason to change their bids unilaterally). This is not mathematically correct – there is no assurance of convergence to a single Nash equilibrium given the way the supply curve is represented – and it drastically increases the computational overhead of the exercise, slowing down run times and forcing the modeller to make simplifications in other parts of the model. Moreover, the assumption that generators seek outcomes that are Nash is elegant but unrealistic. As one of my old colleagues was fond of saying: ‘I’ve never seen a rebid reason that says ‘Seeking Nash Equilibrium’.

Approach 3: Using historical, or other assumed profile of, bids:

In this case, Players are assumed to bid their capacity in at levels based on recent outcomes. This approach suffers from the weakness that the bids are once again being driven by the world we know and understand, and may not align with future changes in contracting behaviour, portfolio changes, or other developments of the system.

There seem to be no good solutions – one must choose their poison. At one time or another, we have used each of the above methods depending on the task at hand. But in general we have found that the approaches that limit the computational complexity (ie, Approaches 1 and 3) are more favourable, because they allow us to investigate different sensitivities to the supply curve. In addition, using historical information tends to provide a helpful reference point for any such discussion. For example, we can ask the question of ‘what happens if more generation is bid in at the market price cap than historically’, or ‘what if batteries start to bid more generation at a lower bid band’.

Regardless, there is no way to avoid the challenge that at its core we are making assumptions about future behaviours and that as the power system changes, the current information set we have will become more out of date.

In addition, we recognise that all existing models are poor at capturing the type of volatility (ie, instances of super-high prices well in excess of the $300 – $500 range) that are so important to market outcomes. This is because volatility occurs in the actual market due to unexpected transient factors such as system constraints (ie, temporary local FCAS constraints due to the risk of islanding), occasional bidding behaviours that are often quickly outcompeted by competitor responses, and unexpected major events such as an explosion at a major unit or lightning strike on a transmission line. Put simply, the reason why super high prices are difficult to forecast is not because the market model is not “good enough” (eg, 30-minute vs 5-minute resolution) but because the modeller cannot systematically forecast transient market disturbances. Where these super high price volatilities are included, they typically occur through post-processing of results, such as adding some historical “noise” component – it is not an outworking of the model. This is a clear limitation of market modelling. Our advice is to be aware of this limitation and be suspicious of anyone who tells you they have a model that can forecast this type of volatility.

High fidelity modelling – capturing detailed operational plant limitations and outages

One of the benefits of ‘locking the build’ is that we can simultaneously unlock large amounts of computational power to capture other factors. This can come in the form of more runs of the model (see next section), or in the form of increased fidelity of representation of the system.

There are three, and potentially many more, ways that this computational windfall is spent. The first is the inclusion of operational plant limitations, such as ramp rate and minimum stable level constraints. Historically ramp rates were generally of limited importance because of the relatively small amounts of ramping required across the system. But with the advent of renewables – in particular solar – minimum stable levels in the middle of the day and ramping on either side of the morning and evening peaks have become more and more important. Dispatch models can easily capture the inter-temporal restrictions on generation caused by limited plant flexibility, and so the benefits of fast-ramping technologies are more evident.

The second change in the dispatch modelling is the use of outages. Now here we face a conundrum: how is one best placed to capture the effect of outages, given that they are a random variable. In some studies, such as reliability modelling, we are not just interested in one realisation of outages, but in the distribution of outcomes across many potential different outage traces. Such modelling often involves rerunning the same model hundreds or even thousands of times to build a picture of the distribution of unserved energy. Here again we see the benefits of the dispatch model being simpler and faster – we can spend the computational windfall on running many different simulations, rather than just one.

But what if we are restricted to just one simulation? It would seem that in this world, we need some concept of a ‘normal’ outage pattern. This is indeed the approach that is taken by most modellers. For example, some modellers derate all capacity uniformly over the course of the year. However, this averaging approach tends to crimp volatility further, because it does not capture the extreme events which occur when outages are greater than their long-term average. We have typically adopted the approach of examining many different outage profiles and selecting the median profile according to a metric of the frequency of extreme events. Regardless of the approach adopted, it is important to understand the degree to which outages are affecting outcomes, because a single sustained outage at the wrong time can lead to a massive impact on reliability.

These factors tend to provide more granular results, because they impose additional constraints. All else being equal:

  • Operational plant limitations tighten the ability of the plant to respond to system fluctuations, so they increase the daily price spread.
  • The inclusion of outages removes generation from the supply curve, so it also acts to lift price.
  • More generally, any factor that adds constraints to the system will tend to lift price, whether that constraint be in the form of interconnector losses, complex heat rate equations or even cycle limits on batteries.

When all is said and done, these many different factors can give a great deal more shape to prices, as well as leading to different marginal costs or prices being observed in the system.

Diving into distributions – variation in renewable energy and demand traces

We have described using the computational windfall from locking the build that comes out of the capacity expansion model to increase the complexity of dispatch. But another way to spend that windfall is by running our dispatch models many times. This is particularly important when we want to understand how random factors influence outcomes.

We can look at many different potential realisations of a random variable, to understand not just a single point estimate of outcomes, but an entire distribution. This can help us answer questions like:

  • Can our power system withstand extreme demand events?
  • How might different weather conditions (ie, temperature, wind and solar irradiance) lead to different outcomes, and how different are those outcomes from one another?

The key here is to create the inputs – ie, the weather and demand traces – that will feed into these simulations. In the National Electricity Market, the market operator publishes a range of traces for demand and weather going back 13 years. But we can go further using historical data sets such as the MERRA-2 data set to create a longer history. The challenge is always to ensure that the weather and demand conditions are correlated. For example, it would be a mistake to use temperature from 2011, but wind data from 2022. The two would be misaligned with the potential for outcomes like high output from wind farms occurring at the same time as high temperature outcomes in summer. In general, the solution here is to ensure that all the trace variables are aligned, and so it is not possible to ‘mix and match’ traces without compromising the value of the exercise.

In the event that a model requires even more data than is historically available, the solution is to create synthetic data, which preserves the relationships between the variables, but which is generated using probabilistic machine learning or some other suitable technique. For more information about this type of approach, we refer the reader to Probabilistic Deep Learning by Oliver Dürr and Beate Sick.

Once we have run the model across all the available data, we can look at the distribution of outcomes and see how much additional information has been revealed. In our opinion, this type of ‘stress testing’ is massively underapplied across the sector. And even when it is applied, for example in reliability studies, not enough analysis occurs of the distribution of outcomes. As more and more data sets become available, and the system gets more and more dependent on random factors, this type of approach will become increasingly powerful.

Where to from here?

So we now have an end-to-end modelling process. But even after two articles, we have barely scratched the surface of the process. The power system is a complex beast, and a model that seeks to simulate its operation will be similarly intricate.

This intricacy can sometimes lead modellers to avoid talking about the fine details of their modelling and, in some instances, to use modelling to justify poor decisions.

With this in mind, we think that the most helpful tool for someone trying to engage with, or commission, energy market modelling is a guide to some of these tricks. In our final instalment of this series, we therefore examine the assumptions and methods – the Dark Arts – that your modeller would rather not talk about. Our intention in doing so is to arm you with the basics of ‘Defence against the dark arts’. This is the information you need to understand both what models cannot do, but more importantly the many powerful questions that can indeed be answered by modelling if we are able to stretch our understanding.

ph. +61 2 9037 0370
e. info@endgameanalytics.com.au
a.
24.01, 9 Castlereagh St, Sydney NSW 2000

"*" indicates required fields

This field is for validation purposes and should be left unchanged.

Get in touch.