Last time we completed our description of the end-to-end modelling process. But we noted that there remained many different tricks and tips that were hidden from the layperson.
With that in mind, we promised that in our final instalment in the series we would examine the assumptions and methods that fall into the category of ‘the Dark Arts’ of modelling.
At the outset, we make clear that our intention is to shed light on these often overlooked parts of the modelling process. We want the reader to understand what models can and cannot do, and the ways, means, and devices that together are sometimes used that can make it hard to derive value from modelling exercises. We want to arm you with the basics of ‘Defence against the Dark Arts’.
In the remainder of this article, we explain our thoughts on the top four ‘Dark Arts’ of the modelling process, specifically:
We conclude by setting out how we think about the role of modelling, and the philosophy that underpins our own work in this area.
For capacity expansion, it is necessary to simplify the number of periods (sometimes called time slices or load blocks) over which we model the system. For example, a single day might be represented by 3 time slices: one for each eight-hour period of the day. In turn, this day might represent 30 days, which have similar characteristics. In so doing, 30 * 24 =720 hours could be represented by a mere 3 time slices.
The benefit is that the problem is easier, and so faster, to solve. But the approach of using very few time slices has been limited by the advent of:
It is now the case that unless we use a very large number of time slices, the system that results from the capacity expansion problem is not resilient. The solution is to use more time slices – this is not really an option, unless we are looking at a very simple power system.
Despite this reality, some modellers often want to speed up the process, or do not have access to commercial solvers, and so they resort to use of a crude set of time slices. The outcome can often be a brittle or, on the other hand, massively overbuilt system. When working with a consultant using capacity expansion modelling, it is worth being aware about the assumptions they have made as to time slices – a Dark Art if ever there was one.
One of the powerful aspects of using an optimisation framework for solving models is the ability to impose constraints. It is trivial to add constraints to the model that do the following:
In practice, these constraints are just like any other constraint, such as the supply-demand constraint, or a limit on the amount of generation that can connect in a region. These constraints have the potential to change the outcomes that would have otherwise occurred in their absence.
When these constraints are imposed, the model will do whatever it takes (such as over-building renewable or artificially suppressing thermal generation out-of-the-merit-order) to meet these targets. This will occur regardless of whether the new investment will recover its costs from the energy prices in a potentially over-supplied market. This is not a fault of the least cost expansion model per se – imposing these constraints is to assume they will be met regardless of the cost, and the model is faithfully satisfying the request of the modeller by identifying the cheapest way to meet these requirements.
However, a modeller should know to look for these ‘hidden costs’. Just like the supply-demand constraint, every constraint in the model gives rise to a ‘shadow price’. As explained in our first article, this is the marginal value of alleviating this constraint by one unit on the total system cost. In the case of a renewable target or carbon budget, there is a clear meaning to their associated shadow prices:
Of incredible importance is that, when we invoke these constraints, the prices that fall out of the capacity expansion model assume that there is also a carbon emissions price, and a renewable energy certificate price. And so if we look only at the price that falls out of the model and assume this will be the cost of electricity, we will fail to account for the effect of the assumed carbon and renewable energy certificate prices, or the ‘missing money’ that must be funded outside the wholesale electricity market. These ‘hidden’ prices are often overlooked or forgotten about, but they are critical to understanding how generation recovers its costs. Moreover:
It is critical to be aware of this – otherwise we are not going to factor in the whole cost of policies, and so we may not understand what is required to achieve them.
One of the perennial questions in the sector is whether a least-cost capacity expansion model is the correct way to determine new entry. The alternative, which is used by many advisors, is called ‘market-based new entry’.
The recent surge of market-based new entry
Recent years have seen a surge of market-based new entry in market modelling. We often hear the motivation of using this approach is that new entrant generation built by least-cost capacity expansion models does not seem to earn enough pool revenue to recover its costs. Market-based new entry appears to fix this problem by allowing the modeller to alter the investment path manually until the all new entrants recover their costs.
We will address the economic fallacy of market-based new entry shortly. However, it is first worth noting that in a market where renewables are built to meet ambitious government renewable constraints or carbon budgets, new entrants will not recover their cost from pool prices alone. This is because of the “missing money” phenomenon discussed in the last section. The least cost expansion model merely exposes this fact, which is either misunderstood or ignored by proponents of market based new entry when communicating their results.
The quasi-economic argument against least-cost capacity expansion
One argument against least-cost new entry is that the decision to build is based on cost rather than price which is what we see in the market. The argument is quasi-economics at its best. It fails to recognise that price is inherently linked to long run marginal cost, and so solving for cost gives us the outcome we seek.
The proponents of market-based new entry fail to recognise that the capacity expansion model solves for cost but also produces prices. These prices are equal to the long-run marginal cost of generation. Taken to its logical conclusion, a market-based new entry proponent would suggest that dispatch also needs to be ‘based on price’. This would involve iterating between different combinations of plant to be dispatched in a given interval until eventually we stumbled upon the same outcome that is yielded by the dispatch engine. Why would we do such a thing, when we can get the answer directly by solving as a linear program?
Market based new entry as a low-quality capacity expansion model
In the absence of constraints, the prices yielded by the least-cost capacity expansion are exactly the level needed so that every technology recovers its costs. In effect this is equivalent to a world where investors build new plant right up until the point where any additional plant would be uneconomic. This is also the common goal shared with market-based new entry.
Put another way, we see the following outcomes:
The key phrase here is ‘at its best’. In reality, the market-based new entry process is trying to solve a very complicated problem with very primitive mathematical machinery. We know that some consultants complete this process manually, with choices being made by an analyst on-the-fly. This is essentially ‘dealer’s choice’ – ie, the outcomes depend on the gut-instinct of the modeller.
Even when an algorithm, or heuristics, are codified in a program, there is no guarantee that the process will converge to an outcome that satisfies both (a) and (b) above. The chances of stumbling on the exact combination of build over the next 25 years that yields prices that walk the tightrope between (a) and (b) above is vanishingly small. This means that there will always be either plants that are not built that should have been, or plants that are built that should not have been.
The computational argument against least-cost capacity expansion
A decade ago, least-cost capacity expansion models took a very long time to solve, and so required very crude chronologies. These models therefore were internally consistent but gave a poor representation of the planning needs of the system. It was not uncommon for these models to lack resilience. Against this backdrop, market-based new entry was an alternative but sometimes necessary evil to provide a meaningful view of the future needs of the power system.
However, in the last decade the speed of solvers has improved dramatically. Gurobi can now solve a least-cost capacity expansion problem with thousands of time slices per year over a 25-year horizon in a matter of hours. It is no longer necessary to stumble around with iteration and primitive heuristics to find a profile of the needs of the system. What was once a helpful tool to plan out the system is now no longer necessary, and there is no basis for it.
Market-based new entry is now a Dark Art
As the system becomes increasingly complex, market-based new entry becomes less and less helpful for planning, resilience testing, and projections of the future needs of the system. For example, when we have been asked to reconcile market-based new entry models against the results of least-cost capacity expansion, we find that there are either large amounts of unserved energy, or many plants that are built that do not recover their costs.
More importantly, the ‘dealer’s choice’ phenomenon means that we cannot easily replicate or reproduce the analysis of other consultants who have used market-based new entry. Outcomes based on choices made in the early hours of the morning by a bleary-eyed analyst as to what plant should get built where, are not consistent with robust modelling practices.
The continued use of these models without recognition of their limitations is surely one of the most prevalent Dark Arts of the sector.
By now it has become clear that there are elements of the market and the power system that are fundamentally challenging to capture. And so not everything can be included in the capacity expansion and operational dispatch models. Many consultants (ourselves included) provide services that alter the outputs of the model in a post-processing step. For example, it is not unusual to add interval-to-interval volatility or ‘noise’ to prices to reflect the patterns we see in the market.
We do not see anything wrong with this, provided it is clear as to what is the effect of this post-processing. Advisors should decompose their results so that the contribution of these post-processing steps can be captured – sometimes they are highly material to their findings.
In addition, some post-processing seeks to account for high-impact system events (such as the Callide C explosion) that cannot be meaningfully included in a small number of deterministic model runs. This is not to say that models cannot simulate this type of price impact if we know exactly when and how they happen in the future. In fact, the revenue impact on other assets, which is what most commercial clients are interested in, can even be easily obtained via back-of-the-envelope calculations. The real challenge is to project the frequency with which such events will happen and how the system unfolds immediately following it, which is often highly event-specific and beyond the realm of market simulation models. Including events of this type in a projection of the future and pretending that there is a sophisticated methodology for ‘modelling’ them goes beyond the Dark Arts and verges on the Unforgivable Curses. Market models are not designed to simulate such events – be wary of anyone telling you they can do so.
We have listed here some of the Dark Arts, but in truth there are too many to list. Modellers can always use sleights of hand, and tricks to deceive their audience – the Dark Arts. In my opinion, the problem here stems from our understanding of the purpose of modelling. From my perspective, the purpose of a model is to create a mental latticework on which we can build intuition. When constructed properly, that latticework will remain rigid and so will not simply yield to your gut instincts – it will require that you adhere to its assumptions and logic, and so build your own understanding of the problem. We can change the latticework (through changing assumptions) but that gives rise to a new set of constraints on our logic.
In contrast, when we force models to yield to our own intuition, and to give us the answers that we want, those models lose all meaning. This is why modelling is particularly unhelpful in adversarial processes, where the objective is to show that one model is ‘wrong’ and one is ‘right’. Models were never meant to be ‘wrong’ or ‘right’ – they were meant to inform our thinking.
Sometimes models give us helpful insights, and sometimes they do not. But once problems become sufficiently complex, without them we are left without any sound basis for decision making. The energy industry poses many such problems, where the stakes are high, and the risks are many. We are foolish to leave decision-making to gut instinct alone and not use models, but the more we can learn about those models and understand them, the better our decisions will be. A more informed world is the outcome. Our purpose in discussing this topic is to arm you with the basics of ‘Defence against the dark arts’. This is the information you need to understand both what models cannot do, but more importantly the many powerful questions that can indeed be answered by modelling if we are able to stretch our understanding.
We often hear people talk about volatility in the NEM: ‘the market has been volatile lately’, ‘we’re expecting some volatility this afternoon’, and so on. Moreover, there is a direct connection that is often drawn between volatility and the business case for batteries. But there is no unified view as to what we mean by volatility, and how we can measure it.
In this article, we examine the historical outcomes for three different definitions and attendant measures of volatility. We then seek to answer the question: is the NEM becoming more volatile, and what does this mean for batteries?
Our first measure is an old favourite – the contribution of prices above $300 per MWh to average prices. Chart 1 shows this analysis for all mainland regions of the NEM on a calendar year basis for 2000 to 2024. We note the following:
Chart 1 – Scarcity prices have been rising in importance over the last 5 years
Average prices in bands by region on a calendar year basis, 2010 to 2024
In general, the last 5 years has seen far more value in scarcity pricing across the NEM than at virtually any other period in its history, with the possible exception of 2008 and 2009 in South Australia.
But high prices alone are not the only measure of volatility. People often refer to variability in prices as being a measure of volatility. For example, frequent periods of low prices followed by high prices are often perceived as being volatile periods, particularly because they give rise to the opportunity for energy arbitrage.
How do we measure this type of volatility and how it is changing over time? One way is to examine how a hypothetical price-taking battery would have performed over a given period. Chart 2 shows the calculated trading profit for a 1 MW, 2-hour battery operating with perfect foresight over the period from 2010 to 2024. The height of the bar shows how much a battery could have earned in revenue less the cost of charging.
Maximum energy arbitrage profit available for batteries has been rising steadily, and has seen record levels in NSW and Victoria in 2024. Similarly, the levels seen in Queensland and South Australia have been relatively high.
Chart 2 – Annual operational profit has been rising over the last 10 years
Annual operational profit for 1MW, 2-hour battery, 2010 to 2024
We have described two concepts of volatility, both of which are inherently linked to the concept of battery profit. But there is another more mathematical view of volatility – the concept of variation. Variation is defined as the cumulative interval-to-interval change in a time series. So if we consider the series (10, 20, 5, 40), the cumulative variation is given by:
abs(20-10) + abs(5-20) + abs(40-5) = 10 + 15 + 35 = 60.
We can then take an average of this, which yields 20 per step (ie, 60/3 steps). In the case of the NEM, we can look at the average variation in spot prices for a region, to get an indication of how much prices are rising and falling from one 5-minute interval to the next.
Chart 3 shows the average variation for each mainland region and for each calendar year from 2000 to 2024. To filter out the effect of scarcity pricing, we have capped prices at $500 per MWh, though we note that this does not affect the trend in the results.
Average variation has increased markedly over the last decade. Where previously prices were very stable, there are now regular rises and falls over the course of the day. On average, prices are changing by $15 per MWh every 5-minutes across all regions. Put simply, there is more noise in prices than ever before.
Chart 3 – Variation has been rising steadily over the last 10 years
Average absolute price variation by region on a calendar year basis, 2000 to 2024
This an astonishing result, which confirms what many NEM-watchers, who live and breathe the market have noticed for the last few years, ie, that prices are ‘going up and down’ more and more.
What are the consequences of this rise in variation of prices for batteries? All else being equal, adding noise to a price series increases the opportunity for arbitrage. That said, capturing the low and high points of a noisy price series is harder than capturing the low and high points of a more predictable price series. It follows that:
The third point is particularly relevant, as it speaks to how much of the profitability that we see through backwards looking analysis could have reasonably been captured. Nevertheless, all else being equal, more interval-to-interval volatility is going to improve the business case for batteries, even if they cannot capture the entirety of the available arbitrage revenue.
We have looked at three measures of volatility, and each of them tells a different story about the outlook for batteries. But, regardless of how we define it, there has indeed been an increase in volatility in the market, as demonstrated by the following:
Put simply, the odds have been shifting in favour of battery profitability. However, the immediate question is whether these trends will persist, reverse, or stabilise as the energy transition continues. We will consider this question in a subsequent article.