The energy transition presents unprecedented changes in electricity systems. Regulators and system operators are struggling to keep up. In such an environment, leadership must prioritize any proposed changes. Innovations that are simpler with less costly implementation yet promise significant benefits should be at the front of the list. The forward energy market is such an innovation since it offers substantial benefits at relatively modest implementation costs. We list some of the benefits.
Provides transparent and efficient forward prices to guide investment in and operation of supply and demand resources. The gains from efficient pricing are empirically significant; for example, see Gowrisankaran et al. (2023) for an empirical analysis of the distortions from mispricing in nonrestructured markets.
Allows fine time and space granularity in trades. This attracts responsive resources when and where they are needed most. The energy products include monthly forwards, 48 to 1 month ahead, by hour, load zone, and type of day (weekday and weekend), and hourly forwards, 30 to 1 day ahead by load zone. The complexity of the forward energy market is robust to finer granularity from a computation, liquidity, and behavioral perspective.
Allows efficient management of renewable energy certificates (RECs) and jurisdictional renewable requirements efficiently and encourages a shift to a consistent carbon pricing approach.
Replaces the contentious capacity auctions or capacity requirements, including the anticompetitive and discriminatory minimum offer price requirements, with a more straightforward and effective instrument.
Uses capacity values for resource adequacy assessments rather than in market rules that directly tie a resource’s payments to the administrative accreditation of capacity values. This limited role encourages resource innovation and avoids costly fights over accreditation rules. Estimating capacity values becomes a technical exercise like forecasting the weather.
Embraces rapid resource innovation through technology-neutral rules and payments. The system rewards each resource for the value it creates. The playing field is level and transparent.
Relies on few administrative parameters, the chief being the price ceiling. Regulators set the price ceiling to achieve the reliability and resiliency standard. A higher price ceiling induces more investment and achieves a higher standard. With time, this parameter becomes less critical as improved flexibility reduces the frequency and duration of shortages.
Provide regulators and system operators with detailed price information to better understand and manage resource adequacy.
With the complementary reform of an intraday rolling settlement, extends forward prices to intraday operations, improving incentives and efficiency.
Enables market participants to express preferences and trade consistent with their interests, much as they do in the day-ahead market. Flow trading lets market participants efficiently manage risk, create value, and avoid adverse price impacts.
Enables the independent market monitor, the system operator, and regulators to understand and manage market power concerns.
Empowers the system operator to optimize collateral to reduce counterparty risk and reduce participants’ costs of satisfying collateral requirements.
Improves trading opportunities across markets and provides valuable information for inter-market transmission planning.
But what about the costs? Changes that alter the day-ahead and real-time markets are costly in time and money. The forward energy market alters neither. The forward energy market is an additional module that straightforwardly interfaces with the settlement system. The essential system integration is passing the prices and quantities from the forward market to the settlement system, where some matrix multiplication and addition are required to perform the final settlement. The interface is easy to build, test, and deploy. A prototype forward energy market platform is being developed as part of this research. The open-source platform will be freely available in the first quarter of 2024. At the paper’s end, we present incremental implementation strategies that are low in cost and risk.
As with any change, there are winners and losers. Forward trade benefits the market through reduced risk and more efficient operation and investment. The gains to consumers are seen in lower prices and reduced risk. Suppliers and many load-serving entities benefit from reduced risk and improved opportunities from innovation. The one group not benefitted is legacy utilities—those paid for exiting owned or controlled generation or curtailable demand and ill-suited to innovate. Strengthened competition will push these laggard incumbents aside.
A state-of-the-art capacity market has excellent performance incentives (Cramton et al., 2013), making it like the forward energy market. A forward energy market always settles against the day-ahead market, providing perfect performance incentives. If a day-ahead position differs from a forward position, the difference settles at the day-ahead price. There are no exceptions. Likewise, the best capacity markets are a financial option to deliver energy during reserve shortages. Again, there are no excuses. The sale of capacity comes with an obligation to provide energy during reserve shortages. Real-time deviations from the obligation settle at the real-time price. Both markets provide a financial hedge to electricity demanders. Both markets have a similar mantra, “Buy enough in advance.”
A difference between the markets is the hedge’s form. The forward energy market provides complete price coverage. Demanders are hedged regardless of the day-ahead price, and the day-ahead market provides an efficient hedge of the real-time price. Energy options provide a hedge against price spikes when demand is apt to be higher than expected. The combination of forward energy for expected demand and energy options for unanticipated demand provides complete price coverage. By contrast, a capacity market defines the energy obligation as an option: only during a reserve shortage does the capacity supplier have an obligation to deliver energy at the shortage price. (There is an obligation to offer day-ahead, but the offer may be at the price cap and not lead to energy delivery.) The capacity market only provides price coverage at the shortage price. Variations of the market offer broader price coverage by triggering the obligation at a lower scarcity price, say $200/MWh rather than $5,000/MWh. A lower strike price makes sense in markets where the scarcity events tend to be of long duration, such as Colombia, where El Nino can cause low hydro production for months (Cramton & Stoft, 2007). In the Eastern US, where shortages are of short duration, typically 5-120 minutes, a high trigger price is used to minimize the role of the capacity market in energy contracting.
As described in the introduction, the market designer’s desire to minimize the capacity market’s role in energy contracting stems from a fundamental capacity market problem: market power. Capacity markets use big-event auctions to procure capacity. Typically, a single auction procures 100 percent of the required capacity three years ahead. Since some market participants are dominant, the scope for exercising market power is considerable. Thus, the capacity market must have market rules to mitigate the exercise of market power. These rules limit the behavior of participants in potentially undesirable ways. Market power mitigation is controversial and imperfect.
By contrast, the forward energy market mitigates market power by allowing market participants to trade forward energy slowly. Market power is effectively eliminated, and no controversial rules limit behavior. The equilibrium behavior is to trade slowly and not exercise market power.
The second essential difference between the markets is the role of accreditation, which defines how much generators can sell in a capacity market. The best capacity markets use state-of-the-art forecasting methods to determine a resource’s capacity value, which is its ability to deliver energy during reserve shortages. Electricity spot markets often have missing money—the profits in the spot market are insufficient to cover the long-run average cost. Missing money arises from spot market flaws. The most common distortions are 1) a too-low price cap, 2) treatment of nonconvex costs, and 3) unpriced operator decisions, such as the intraday commitment of additional units to address reliability concerns. The capacity market in equilibrium restores the missing money without fixing the spot market flaws, which implies that the money paid to generators from selling capacity exceeds the financial cost of the option. Every resource wants to sell as much capacity as possible. Accreditation has a direct impact on generator revenues.
The stakeholder process about capacity values becomes a debate about how market rules allocate money. In such discussions, money trumps efficiency. The market participants care more about getting more money than making the market more efficient. The result is an unproductive stakeholder process. Every system operator with a capacity market has experienced this problem.
A capacity market is like training wheels on a bike. The training wheels do not necessarily help the child learn the critical aspect of biking: balance. Yet, the stakeholder process results in a bike with training wheels and other bells and whistles (Figure A1, left). The child is better off with a far simpler balance bike that focuses on the art of balance (Figure A1, right). Aagaard and Kleit (2022a; Aagaard & Kleit, 2022b) discuss these challenges.
By contrast, the forward energy market has no role for accreditation. Suppliers and system operators estimate capacity values. Suppliers use these estimates in their bidding strategies; system operators use them in resource adequacy assessments.
A secondary role of capacity values is in setting collateral. A resource’s capacity value will determine the physical hedge the resource brings to the generator’s resource portfolio. The greater the physical hedge, the lower the collateral without introducing counterparty risk. This use of capacity values is second order. It does not directly lead to money transfers to the generator. It has a modest impact on the generator’s collateral requirement.
The forward energy market motivates the proper response to missing money. If the market is missing money, there will be too little generator investment. Resource adequacy assessments will identify this shortfall, and the regulator will restore the missing money by raising the price cap or shifting the operating reserve demand curve upward. The higher price cap and demand curve will increase spot market revenues to encourage efficient investment. Alternatively, the system operator will redouble efforts to reduce entry barriers if that limits investment.
We can think of investment dynamics as an optimal control problem. The control is the price cap. Raising the price cap increases revenues to generators—holding resources fixed. More revenues increase the incentives for investment, raising the reserve margin. The investment only occurs if the payments cover costs. There is no missing money. Thus, the long-run equilibrium is that forward prices equal the expectation of real-time prices plus a risk premium. There is no missing money. Further, long-run cost is minimized because the participants have the information and incentives to make efficient operating and investment decisions. Should things get out of equilibrium, the market monitor, the system operator, and the regulator have the forward prices to recognize potential trouble and make an adjustment.
Significant sources of missing money are capacity market flaws and unpriced operator decisions, primarily reliability unit commitments. The forward energy market eliminates the capacity market flaws. It mitigates missing money from reliability unit commitments with the rolling settlement reform of the two-settlement system. Rolling settlement sets prices for what is unpriced today. It also injects convex arbitrage bids into the optimization, which mitigates the nonconvexities, another source of missing money.
The primary role of accreditation is resource adequacy assessments, a technical modeling exercise all markets do. A further advantage of the forward energy market is that it provides transparent and reliable information about resource adequacy. The forward energy prices are an outstanding measure of resource adequacy, and far-in-advance price information gives market participants and regulators ample time to respond.
Finally, the role of scarcity pricing—the operating reserve demand curve—vanishes as demanders provide sufficient flexibility so that reserve shortages do not occur. The primary administrative device becomes irrelevant. Given the rapid adoption of electric vehicles, we should expect the day of negligible reserve shortages to arrive within a decade or two. Outages will be at the distribution level, not systemwide.
The forward energy market promises a gradual transition to a perfectly competitive market. The capacity market does not. If stakeholders are engaged in a fight over the allocation of money, there will be no tendency for the market to become more efficient. With the forward energy market, the stakeholders are not fighting over money and can focus on efficiency. The result is a bicycle ruthlessly optimized for the required task (Figure A2).
No single bicycle is best for all races. Likewise, no single market design is best for all settings. The road bike on the left is optimized for maximum efficiency on varied terrain while drafting a quarter meter behind another rider. The triathlon bike on the right is optimized for maximum efficiency on flat terrain without drafting, which is illegal in triathlons. The triathlon bike puts the rider in a much more aerodynamic position to minimize wind resistance but handles poorly and affords poor visibility.
The most significant differences in electricity markets stem from the different supply and demand characteristics. The best market designs tune the parameters of the market to the setting. However, all the best market designs are grounded in the same fundamental principles, like the bicycle designs above. There are more similarities than differences. Indeed, the best market designs reoptimize the market parameters dynamically as circumstances change. For example, operating reserve requirements may differ by season or as the share of intermittent resources grows. A cyclist has multiple bikes: an ultralight bike for the mountain stages, a triathlon bike for the time trials, and an aero road bike for the other stages. A market with dynamic parameters can motivate investments in resources that can profit from the changing setting. For example, the operator of a state-of-the-art combined cycle unit can configure it differently based on its best use in the market. In periods where the unit runs constantly, the operator tunes it for maximum efficiency to produce the most energy per fuel input. When the unit provides flexibility, it can be adjusted to operate like several gas peakers with rapid ramping and lower startup costs. The best markets encourage such resources to bid multiple configurations into the market and let the system operator optimize the unit’s configuration daily.
It is precisely this close tie to the spot market that makes the market work. It is reasonable for people to be concerned that investment in generation is challenging in a market that bounces from negative prices to shortage prices in a day. However, long periods of negative prices are not equilibrium predictions based on detailed investment analysis through the energy transition. (For equilibrium predictions, see Cramton et al. 2022.) Instead, they are snapshots of resource scenarios inconsistent with investment incentives.
The reality is that tying the forward energy market to the spot market is sound and consistent with our foundational understanding of markets. As part of this research, we will present an equilibrium-based analysis of the forward energy market at different points in the energy transition.
An issue with any longer-term market is the possibility that the term length limits change. Such lock-in is not a feature of the forward energy market. Since the forward energy market is a derivative of the day-ahead market, it automatically inherits any changes to the day-ahead and real-time markets. Market participants understand this when they engage in forward trade.
Changes to the system operator’s markets are common. Many changes occur each year. Today’s energy futures do not depend on market rules being held fixed. The same is true if the system operator conducts the forward market.
Indeed, by defining the forward product as energy at a delivery point, the system operator—on instruction from the regulator—could adopt finer demand-side location granularity without changing the forward market rules. The settlement would reflect less demand-side aggregation when the switch is made to the day-ahead energy product.
The system operator should focus on making the spot market as good as possible. The forward energy market helps in that endeavor. It is complementary to the spot market, leveraging the system operator’s crucial work to improve the spot market. As discussed below, extending the forward market to an intraday rolling settlement is a helpful fix. Such an extension is consistent with the forward energy market.
Traditional markets manage liquidity by limiting the number of products. Take wheat. Wheat trading involves many grades and classifications, which vary by country and the organization responsible for grading. The United States Department of Agriculture categorizes wheat into eight classes based on kernel hardness, color, and planting season. Within these classes, wheat is further graded on a scale from 1 through 5 based on additional attributes like test weight, defects, and moisture content. Thus, there are 40 wheat products traded in the US.
As the US nodal systems prove, modern markets, like electricity, can trade products with much richer granularity. Liquidity is managed by allowing near-perfect substitution among products that are near-perfect substitutes. The nodal spot markets and the forward energy market have this feature.
The forward energy market, like the nodal market, has a high level of transparency, robust pricing, and low transaction costs, which favors liquidity. The forward energy market has three further advantages. First, preferences are convex. Market participants enter piecewise linear net demand curves, which yield a quadratic objective in the clearing optimization. Second, because the forward market is conducted well before the day-ahead market, the market participants have time to adjust positions as uncertainty resolves. Third, the frequent batch auction approach allows participants to make thousands of minor adjustments over months. Slow trading enables participants to minimize adverse price impact, improving the market’s competitiveness and increasing liquidity.
Gradual trade and gradual resolution of uncertainty imply that forward prices typically move smoothly. Even discrete events, such as the entry of a new 1 GW combined cycle unit, can impact prices smoothly. First, the event is apt to be anticipated by the market. The probability of approval gradually increases to one. Second, the resource owner, the party most impacted by the approval, is motivated to minimize price impact by trading slowly.
The most disruptive event would be a significant and sustained shock in the natural gas price, as happened in Europe with Russia’s invasion of Ukraine in February 2022. Since gas units typically are on the margin, the spot electricity price is set by the marginal gas unit at its heat rate times the price of natural gas. The invasion is a systemic event affecting energy prices every hour of every day for an extended period. The implication for the forward energy market would be a gradual increase in forward prices as the probability of invasion rose. Then, with a discrete price jump in the hour, the likelihood of invasion jumped to one. The discrete change in forward energy prices would require market participants to adjust their target positions. Still, trade would occur smoothly and orderly since each participant is motivated to trade-to-target slowly to mitigate adverse price impact.
One often thinks of risk as a zero-sum game. Suppose the system operator conducts the forward energy market. Doesn’t this transfer the risk of forward trading from the private market to the ISO market, which ratepayers bear? The answer is that efficient and transparent forward trade reduces counterparty risk and lowers the cost to ratepayers. Vibrant forward trade puts market participants in more balanced positions, reducing risk, market power, and system cost.
One can look at the costly defaults in electricity markets since 2000 to confirm this. In the 2000-2001 California electricity crisis, the utilities entered a long scarcity period caused by drought (low hydro production) with a large short position (Borenstein, 2002). The utilities required rescue by the state, costing about $40 billion (California State Auditor, 2001). In the February 2021 Texas crisis, the market participants were in much more balanced positions, and defaults were rare despite a real-time energy value of over $50 billion in four days (Cramton, 2022). In Britain’s crisis of 2021-2022, poorly hedged suppliers defaulted, costing consumers more than $10 billion (Waddams, 2023).
In the forward energy market, imbalanced positions are known, and the associated risk is priced and mitigated through higher collateral. Total system risk is reduced.
Physical risk is something that generators should be concerned about. If your 1GW combined cycle unit goes down minutes before real-time, the real-time price will spike, and you will be a large buyer at this high price. To mitigate this risk, you will want to only sell some of your expected production in the day-ahead market and sell the remainder in real-time. Thus, you want to enter the real-time market as a net seller during normal times.
The forward market helps you manage your sales of expected production to maximize a combination of profit and risk avoidance. Not all risks can be eliminated. The best we can do with physical risk is leave it to those who can do something about it. That would be the generators. If the generators want to mitigate the risk of unplanned outages, they can:
Maintain their resources.
Buy better resources.
Buy a diversified portfolio of resources.
Set a lower target position.
The forward energy market reduces risk by enabling market participants to take positions at known prices consistent with their anticipated needs or capacities. Production uncertainty varies by resource type, location, and other factors. Suppliers account for this production uncertainty in their forward trading strategies. A wind resource may choose to sell ahead a quantity close to its capacity value during net peak hours. The capacity value is production conditional on a shortage event. Then, the wind resource will not have to buy energy at the shortage price. Since the mandatory obligation is on demanders, not suppliers, the wind resource can pursue this strategy to avoid downside risk.
Service providers typically have many thousands or millions of customers. Most of the customers are small. This allows service providers to understand and manage the risk around switching. For larger customers, longer-term contracts can help manage churn.
Another way to manage switching is by having less of it. A common problem with retail choice is companies exploiting behavioral biases to make money. The primary bias is “inattention.” Customers have learned over many decades to be inattentive to utility bills. Many service providers offer a below-market price for a short time, such as six months or a year. The consumer adopts, and then the provider doubles the price at the contract end, invoking the “automatic renewal.” Unfortunately, most regulators do not protect customers from this practice, which is common.
Some markets with retail choice and capacity markets have complex provisions for LSE obligations to move with the customer automatically. One can do that to reduce further switching risk, but we do not recommend it. The problem is poor rate plan regulation that induces switching among the more attentive customers.
A better practice is for the regulator to have a default contract for the consumer with fair terms that do not exploit inattention and other behavioral biases. Instead, the default contract is a simple time-of-use contract. Service providers would be required to offer the default contract, competing on price and reputation.
The best service providers would offer innovative contracts that use the customer’s low-carbon technologies, such as electric vehicles, batteries, and solar panels. The contract would track the real-time wholesale price but would employ hedging. The customer would see and feel the real-time energy price on the margin, but nearly all her electricity would be purchased at much more stable forward prices. The contract would automatically manage the customer’s low-carbon technologies to create maximum value for the customer, charging the EV when the price is low, discharging when the price is high, anticipating the customer’s needs, and the production of the customer’s solar panels. Such a contract would create substantial value for the consumer and system, motivating investment in low-carbon technologies. The forward energy market and retail choice are complements in this consumer engagement. Later, we explain how consumer engagement improves resiliency.
As the share of renewables grows, the system operator may need other services for operating reliability. For example, thermal generation retirements may create an inertia shortage. The system operator identifies and introduces the required service as a product in the day-ahead and real-time optimizations. It is natural for the set and quantity of ancillary services to change with the resource structure. We do not anticipate a need to include these ancillary services in forward products before day-ahead apart from reserves, discussed later.
Oversupply is best addressed with negative prices. There is no need for arbitrary rationing. Price will do the job. Price will also lead to the correct response. Show us a market with persistent negative prices, and someone will start mining Bitcoin there.
Financial markets suffer from the same limitations of the stakeholder process as electricity markets. The most dominant stakeholders lobby the regulators to adopt market rules that favor them. For example, high-frequency traders dominate the technical committee advising the Commodity Futures Trading Commission. It is no wonder that the CFTC is slow to adopt reforms that limit the profits enjoyed by traders with speed advantages, especially since the other influential stakeholders, the exchanges, make most of their money selling tools—low-latency data and collocation services—to high-frequency traders. There is little tendency for the market to adopt efficiency-enhancing reforms. Regulators are risk-averse and easily scared that a reform may have unintended consequences. Such is the tyranny of the status quo (Budish et al., 2015; Budish et al., 2021).
Indeed, the market design challenges in financial markets are worse than in electricity markets. Important financial stakeholders were entrenched when information technology made the reforms discussed here possible. Thus, these stakeholders provided immediate resistance to change. A norm of transparency did not exist and was prevented by legislation from the early 1900s that prevents the disclosure of bidding information in Treasury markets—even after 100 years. By contrast, electricity markets were new, and the importance of transparency was well-understood by early market designers.
Efficient forward prices reward those providing flexibility. Market participants can easily see and enjoy the value of flexibility. Storage technologies, such as batteries, are especially adept at delivering value. With excellent price information, storage devices are trivially optimized with a linear program (Crampes & Trochet, 2019). Similarly, demanders are encouraged to be price responsive. Transparent and efficient prices will motivate the demand-side innovation essential to consumer engagement. Consumer engagement breeds resiliency (Bobbio et al., 2023).
To manage risk, a wind resource owner should not bid more than its capacity value far in advance. The resource’s capacity value is its expected production conditional on a reserve shortage. Only as uncertainty is resolved should the resource gradually shift toward its expected production. Capacity value is far less than its anticipated production since wind will produce little during shortages. The resource does not want to have sold expected production at a low forward price and then be forced to buy at $5,000 in a day-ahead shortage. This means that wind resources will be sellers in the days leading up to day-ahead much of the time. Waiting to sell close to day-ahead is no problem. The resource has no obligation to sell earlier, and the system operator will expect their performance to be poor during shortages.
The forward energy market works well for variable renewables, even when renewables dominate. The renewables have the flexibility, price, and production expectations to optimize positions throughout the 48 months.
Current markets in the US and Europe lack liquid forward markets. The price information could be better. Poor pricing and limited liquidity create challenges for small market participants and opportunities for dominant market participants, perpetuating an undesirable market structure. Dominant market participants can better manage risk and optimize the scheduling of their resources. Self-scheduling is a viable option for participants with large resource portfolios, whereas small market participants need the system operator to optimize their resources centrally. Thus, a lack of price transparency harms small market participants. Dominant participants can benefit from opaque markets since they can do large bilaterals at more favorable terms. Transparent prices level the playing field.
The disclosure requirement is like the mandatory disclosure of resource plans in the day-ahead and real-time markets, which is essential for system operation. Forward position transparency helps system operators establish optimized collateral requirements and assess market power. The disclosure can be price and quantity or quantities only.
Translation is needed if the trade is inconsistent with the forward energy products. This translation is an essential issue for legacy contracts. Once the market is established, there should be no reason for parties to transact outside the forward energy market and no reason to trade products inconsistent with forward energy products.
The translation is not perfect for options, but it should be sufficient for the primary use, collateral optimization. For example, an option to buy a modest fraction of demand at $1000/MWh is equivalent to purchasing and selling the same quantity in forward energy since the collateral concern primarily arises from exceptionally high prices when the demander is largely hedged with forward energy.
It is common in Europe for governments to conduct renewable energy contract-for-difference (CfD) auctions. These auctions aim to encourage renewable energy entry at a low price. The procurement cost is lowered by shifting price risk to the government. The contract-for-difference structure pays the generator a fixed price for each MWh delivered over a long term. It is like a forward energy contract, except the quantity varies by the generator’s real-time production. The generator provides energy in the spot market. The spot price for energy is paid, and then a contract-for-difference transfer is made between the generator and the government so that the net price the generator receives is the fixed price from the renewable auction. Money is paid to the government when the spot price exceeds the fixed price and to the generator when the fixed price exceeds the spot price.
These CfDs have three problems (Schlecht et al., 2024): 1) they destroy investment and operating incentives to produce more during price spikes, 2) they distort incentives after the day-ahead because the CfD is based on the day-ahead price, not the real-time price, and 3) they do not address quantity risk. A solution is for governments to auction improved financial CfDs that address these problems (Schlecht et al., 2024). A financial CfD is a fixed-for-floating swap: each hour, the government pays the generator a fixed amount minus a variable payment. The variable payment approximates spot energy revenue and is based on an independent estimate of hourly production. Neither payment depends on price and quantity realizations in the physical market. Thus, financial CfDs do not harm real-time incentives.
Government CfDs, even financial CfDs, remain problematic. Since the CfD provides a hedge, it harms the generator’s incentive to participate in the forward market, undermining the service provider’s ability to purchase ahead. The solution is to remove renewable production contracted in this way from demand. Obligations are stated in terms of demand minus the expected renewable production from contract-for-difference power purchase agreements (CfDs).
A more serious problem is that the service providers, not the government, should purchase long-term power. The government should motivate procurement of renewables, not directly through big-event auctions, which are vulnerable to market power and other distortions, but instead guide the market to meet climate goals with either renewable requirements or a carbon price. A quantity or price instrument works well and is consistent with the forward energy market. An efficient instrument allows service providers to purchase an energy portfolio gradually that is consistent with their needs. Market power and climate goals are addressed. The forward energy market gives market participants the tools to optimize positions over time to manage risk. The government purchase of CfDs, even financial CfDs, introduces distorts because the government is ill-suited to optimize the timing and form of CfD procurement. Trade is better left to the market participants operating in transparent and efficient forward and spot markets.
States’ standards can be included by defining renewable electricity certificates (RECs) as a product. Qualifying generators produce both energy and RECs. The RECs are defined based on the location of the generator. Buyers can then purchase energy and RECs consistent with their state’s renewables portfolio and clean energy standards. The standards can vary by load zone and hour, providing states with a rich way to establish standards and giving service providers enormous flexibility in managing energy and climate obligations.
Reliability must-run contracts are a sign that something is wrong with the market. Entry needs to be faster to keep up with the system’s needs. There are three primary causes: 1) interconnection entry barriers, 2) poor price signals, and 3) inadequate transmission. The forward energy market mitigates 2. However, the system operator, market monitor, and regulator should investigate the root cause of the RMR need and attempt to address it. The changes required may be improvements in the interconnection process, an increase in the price cap, or better transmission planning with faster execution.
Although the forward energy market cannot directly mitigate entry barriers and transmission shortcomings, it does so indirectly. The improved price information helps the planning process determine where and when new resources—supply, demand, and transmission—are needed. Thus, better price information leads to a better planning process, which allows a more timely and effective response to system needs. In this way, the forward energy market makes RMR contracts less likely and of shorter duration.
Beginning obligations 48 months ahead is a suggestion. It is longer than we see with current capacity markets, which tend to be three years ahead. It is possible and desirable to have forwards start ten years or even twenty years ahead. Doing so may reduce capital costs for generators.
Even without far-forward products, the price information embedded in the forward energy market would give investors much greater revenue assurance than the existing capacity and spot energy markets.
As with locational granularity, it makes sense to start four years ahead and then increase the window over time as participants gain experience with the market. Nonetheless, we would have no problem if the market started with seven- or ten-year forwards to reduce capital costs.
Fostering demand-side innovation to improve resilience may require multiple steps. For example, consider a market like Germany that has no smart meters. Demand-side innovation is impossible.
The first step is to install meters systemwide that measure electricity use at each time and location. Suppose Germany takes this first step but does not improve its spot market. There is still a single “German” price for electricity at every time despite persistent congestion. Utilities could introduce time-of-use rates that reflect systemwide scarcity but not local scarcity.
Suppose Germany introduces a nodal market that correctly prices wholesale energy at every time and location, but the existing monopoly utilities still deliver electricity to consumers. It is now possible for a utility to offer dynamic rates that reflect marginal social cost, but there is no incentive for a monopoly utility to do so. Only the consumer benefits from dynamic rates, and they do not know they need it. How many consumers thought they needed a mobile phone with a camera in the early days of mobile phones? Consumers dislike change. The utility might conduct consumer polls to confirm to regulators that German customers do not want dynamic rates. Innovation would not occur. This was the experience in California. The 2000-2001 California energy crisis led to systemwide installation of smart meters. It was many years before the meters were used to improve consumer behavior. Even today, some 23 years after the crisis, the use of the meters is limited to time-of-use rates. The forward energy market encourages dynamic retail rates by providing service providers an easy means to let consumers capture value from being price responsive.
But what if German regulators take the third step and allow retail choice? Innovative service providers enter the market and offer rate plans tailored to the needs of consumers. The innovative service provider offers the EV owner a dynamic rate so the owner can capture the value of her battery resource through optimized charging and discharging. The gains to the EV owner are dramatic. The net cost of EV ownership falls, and EV adoption accelerates. Improved price information stimulates this virtuous cycle.
Retail choice and the forward energy market are complements. The forward price information lets consumers adopt dynamic rates that are optimally hedged based on each consumer’s risk tolerance, load shape, and portfolio of low-carbon technologies. Energy consumption is then managed by sophisticated AI algorithms that control household devices based on rich data. These innovations are best brought to consumers through vigorous retail competition. The best role for the regulator is to stimulate effective competition through plan transparency and bans on harmful marketing practices that exploit behavioral biases, such as inattention.
Although bid-in demand is not an essential feature of the forward energy market, we mention it because it is aligned with the forward energy market and supports resiliency.
Consumers like low electricity bills. They ask for dynamic rates when it lets them save money. Saving money is why consumers who have invested in low-carbon technologies want to be price-responsive. Electric vehicle (EV) owners top this list. An EV owner usually has her car parked in the garage. The owner is indifferent to when the car charges if she has enough battery for her daily trips.
California has low EV rates in the super-off-peak window from midnight until 6 am. Peter pays 15.4 cents/kWh at night versus 81.6 cents/kWh at peak. His Tesla understands this and starts charging after midnight. He will do even better in the future by discharging to the grid in the on-peak window. Even now, his nighttime consumption is massive relative to his on-peak consumption (his July on-peak was -72 kWh, including solar production versus 523 kWh super-off-peak). The July savings from shifting his consumption to nighttime was $346.23 (a monthly bill of $21.79 versus $368.02). The scope for consumers benefiting from being price-responsive will grow as the pricing and algorithms improve and the cost of low-carbon technologies decline. A minority of price-responsive consumers is enough to enhance market resiliency dramatically. The majority can keep flat rates as long as they like.
Consumer adoption of electric vehicles has reached a tipping point. Today, over one-quarter of California automobile purchases are electric vehicles; in Norway, 83.5 percent of new car purchases are EVs. The scope for bid-in demand is rapidly increasing.
Using household-level data in Britain, we have measured consumers’ willingness to respond to dynamic rates. A one-percent price increase reduces demand by 0.26 percent (Figure A3). Moreover, this response increases by a factor of three for those who own electric vehicles (Figure A4).
Well-designed dynamic rates expose the consumer to the wholesale price on the margin yet include a built-in hedge against real-time price volatility. The service provider buys the consumer’s expected consumption in the forward market, hedging price volatility. Even better, the service provider uses machine learning to optimize the hedge to individual customers. A battery or electric vehicle desires a smaller hedge than a consumer with electric heat, requiring extra energy during a cold snap (Brandkamp, 2023). The hedge transforms what would be a downside risk—paying a high price during scarcity events—to an opportunity—consuming less than expected during these events. The consumer then automatically sells her surplus back to the market at the high scarcity price. The behavioral optics of dynamic rates are turned upside down when the hedge is included. Consumers with less income who are more price-sensitive become the winners from the shift to dynamic rates.
One challenge with dynamic rates is that they benefit those with low-carbon technologies the most. Wealthier consumers are most able to invest in low-carbon technologies. Thus, wealthy early adopters initially enjoy the advantages of dynamic rates. Programs subsidizing low-income consumers’ adoption of low-carbon technologies can reverse this tendency. Such subsidies are especially desirable when carbon prices are too low and borrowing rates are regressive.
As a thought experiment, we took the British elasticity of 0.26 percent and asked what fraction of Texans would need to be on dynamic rates to eliminate the large gap in supply during winter storm Uri. The answer is 44 percent. Thus, Texas could have survived the extreme winter storm with an outage if only a significant minority of retail customers were on dynamic rates (Figure A5).
The calculation is crude and invokes extreme extrapolation. The point, however, is valid. A minority share of price-responsive consumers can significantly improve market resiliency.
Over time, electric vehicles and other low-carbon technologies will be adopted, increasing incentives for bid-in demand. Eventually, we will move to the ideal where consumer engagement is sufficient to eliminate all shortages, and scarcity pricing becomes irrelevant. However, before we get there (20 years?), the market provides near-ideal incentives and robust information to invest and operate resources like batteries.
Yes. The best way to enhance the resiliency of the electricity market is to adopt the same reforms in the natural gas market. It would be easy for regulators to adopt a similar transparent and efficient trading mechanism for natural gas using the same principles and methodology. Electricity markets depend on gas markets, as the 2021 Winter Storm Uri demonstrated in Texas. The failure of the gas market was a proximate cause of the Texas crisis. Generators could not get gas, which led to a massive loss of thermal units that lacked fuel. Even those with dual fuel ran out because the delivery of oil was made impossible by failures in the transportation network. The critical infrastructures are linked.
Unfortunately, the gas market has none of the reliability protections we have for electricity. The absence of protection is the result of poor regulation. Gas is essentially unregulated within Texas. Transparency of prices or volumes is poor. The gas regulator is the Railroad Commission, comprised of commissioners who advocate for the gas industry. There is little consideration of consumer interest.
During Winter Storm Uri, the most significant loss was from natural gas production, which declined by about 15 GWs. Coal and nuclear energy sources also had failures. Wind resources performed better than expected in almost all hours during the storm.
As we add renewables, we must rely even more on natural gas to provide energy during shortages, at least until there is an economic alternative for long-duration storage. Therefore, a significant concern is the poor performance of natural gas during a crisis. Preliminary evidence suggests that many gas units failed because of a lack of gas or inadequate gas pressure. Gas supply was a problem. Texas natural gas production dropped 45 percent when gas demand surged because of the extreme cold. Detailed forensic work will uncover to what extent the shortage was from lack of gas or gas pressure versus freezing at the generator. This distinction is essential. Texas can spend billions winterizing its gas-generation fleet. The grid will remain vulnerable to winter storms unless the gas supply is also winterized.
During the Texas crisis, the gas market had a significant shortage. There was a failure to deliver at any price. Many who bought gas forward and did not receive it got a force majeure excuse rather than compensation for the failed delivery.
Technically, reforming the gas market to require efficient and transparent trade is easy. The same flow trading methodology would provide an efficient and transparent gas market framework. All that is needed is a regulator motivated to maximize social welfare rather than one captured by special interests.
The forward energy market settles against the day-ahead price rather than the real-time price because the day-ahead market already provides the hedge between day-ahead and real-time. Thus, there is no need for the forward energy market to duplicate the real-time hedge; doing so would harm the day-ahead market.
However, as in any forward market, a short squeeze is possible. The squeeze would take place in the real-time market. Such a squeeze is possible today. A participant takes a significant imbalanced position in the day-ahead market, causing others to take imbalanced positions day-ahead and then squeeze them in real-time. A dominant supplier has a comparative advantage in executing the squeeze in electricity. A generator buys a large quantity in the day-ahead market, leaving others short, then offers supply at high prices in real-time and strategically withholds with an “unplanned” outage. During periods of scarcity, enhanced market power improves the effectiveness of such strategies.
The forward energy market mitigates this possibility through the transparency of positions. The system operator and market monitor would observe the imbalanced position, which would prompt regulatory action. Moreover, the single-price auction makes a squeeze prohibitively expensive.
Recall Salomon Brothers’ famous squeeze in the US Treasury markets in 1990-1991. To be successful, Salomon Brothers needed to hold a considerable position. They acquired majority shares in some Treasury auctions. Although illegal, winning a majority was possible because of the pay-as-bid pricing and large price-tick size at the time. Salomon Brothers could acquire most of the issue and squeeze the short dealers in the subsequent market by bidding one tick above the obvious clearing price. Developing such a significant stake would be prohibitively expensive with single pricing, which we have in electricity.
Hundreds of market participants exist in the forward energy and day-ahead markets. The participants include natural buyers, natural sellers, and arbitragers. Each natural buyer and seller also act as arbitragers—the arbitrage behavior results in price convergence. The day-ahead price equals the expected real-time price plus a small risk premium (Jha & Wolak, 2023).
The market is highly competitive. Therefore, the scope for strategic bidding is limited. Flow trading further mitigates incentives for strategic bidding by incentivizing participants to seek balanced positions to manage risk and limit collateral. With balanced positions, there is no incentive to distort bids.
Market power only arises in the day-ahead market and within the day. Then, market participants can take actions that may result in more significant and favorable price impacts because other participants will not have time to take corrective measures to mitigate this behavior. Intraday market power is best addressed with the rolling settlement discussed earlier.
Contract clearing will be consistent with the product. In markets without persistent congestion, having a single-zone forward energy market and letting the firm transmission rights (FTR) market handle locational hedging will suffice. In markets with persistent congestion, the products are readily split into zones. In a market with six load zones, there would be about 18,000 products rather than 3,000.
There is a tradeoff in deciding how many locational elements should be hedged in the forward energy market versus the FTR market. Many markets now have three-year-ahead FTRs. The FTR market can remain the primary method for congestion hedging, or the FTR market can be replaced with a forward energy market with fine location granularity.
Until the forward energy market has fine location granularity, the FTR market remains essential to price transport of generation from one node to a load zone. A generator in zone A, selling zone B forward energy, would need to hedge the congestion cost of delivery in the FTR market. The FTR market design could adopt a similar flow trading methodology with frequent batch auctions to improve the hedge. However, since the FTR clearing optimization is challenging, it makes sense to clear the FTR market daily rather than hourly. This change to smooth trading and frequent clearing would improve the FTR market’s liquidity and mitigate market power.
Strategic reserves, sometimes called emergency or reliability reserves, stand idle until a scarcity event, most commonly a reserve shortage. Only in shortage do they run to mitigate the shortage. The “market” price throughout the shortage is the price cap, say $5000/MWh. The strategic reserve is procured periodically via competitive bid. The winning resources receive the clearing price for the quantity procured. The payment is in $/MW of capacity. The product is a reliability option to deliver 1 MWh of energy for each MW of capacity for every hour a reserve shortage occurs. Performance deviations settle at the price cap. The resources cannot participate outside these shortage events in the energy or reserve markets.
The strategic reserves do not harm the market when structured in this way. They are decidedly outside the market. Further, they face the same strong performance incentives of resources inside the market. The strategic reserves aim to provide additional insurance to reduce the probability and duration of shortages. In this way, they can support reliability and resiliency.
There are, however, some problems. When energy prices are high for an extended period, it is tempting for regulators to step in and use the strategic reserves to reduce prices. This pre-shortage use harms the market because it becomes difficult for generators to guess under what circumstances the strategic reserves are used—a classic moral hazard problem. It is difficult for the regulators to commit to not interfering with the market. A recent example is Germany’s strategic reserve. Germany had rules about when the system operator would use the strategic reserve. However, when sustained high energy prices arrived, the regulator discarded the rules and instructed the system operator to use the strategic reserve to reduce market prices. Such behavior creates missing money.
Still, provided regulators can commit to only using the resources during shortages, strategic reserves can provide additional insurance during the transition. The forward energy market works perfectly with these strategic reserves. It is simply a matter of cost. Is buying more strategic reserves or having a higher reserve margin within the market more cost-effective? The answer depends on the setting and is primarily a question of price discrimination. Suppose the price of strategic reserves is low because there are ample near-retirement resources whose best use is to stay idle except in a shortage. In that case, it may be economical to buy strategic reserves. However, if these resources can create more value in the energy market than in emergency response, then it is likely that welfare will be greater without the strategic reserve.
A second problem with strategic reserves arises when they are adopted without sufficient lead time. Until the expansion of resources is consistent with the increase in the reserve requirement, the strategic reserves are simply a wealth transfer from load to generators. The reason is that strategic reserves only enhance reliability if they expand the set of resources; otherwise, they raise energy prices.
A good example is the ERCOT Contingency Reserve Service introduced in summer 2023. CRS is a version of strategic reserves, procuring reserves only used during reserve shortages. When procured on a short-term basis, this amounts to mandating the withholding of resources until the price is at the price cap. The near-term implication of this required withholding is higher prices for consumers with little improvement in resiliency or reliability. This implication was borne out in summer 2023, as reported by the Independent Market Monitor (Bivens, 2023). The additional cost to consumers for the three months of June-August was about $8.5 billion. The cost through November was about $12.5 billion.
Relying solely on strategic reserves for reliability would be astronomically expensive. The forward energy market is a better approach, which provides the forward prices to guide investors and operators to optimize welfare.
However, emergency reserves can provide a reliability backstop if the forward prices suggest an emergency. If everything has been done to encourage entry and shortage is still likely, a reasonable emergency response would be for the regulator to authorize the system operator to make a timely competitive procurement of the shortfall. The procured resources, which must be additional resources, would be used as emergency reserves to generate energy and operating reserves during reserve shortages when prices are at the price cap. This backstop does not distort the market since the resources are only used during shortages; no quantities or revenues are taken from market resources.
Reliability is resource adequacy 1.0 — the electricity system’s ability to satisfy 100 percent of demand. It measures the frequency, duration, and magnitude of shortage events and looks at the system’s average interruption duration and frequency. Outages are short and localized, caused by routine events that cause demand to spike and supply to drop. Events are triggered, for example, by the failure of large units on a windless hot summer day. A reliable system, according to the North American Electric Reliability Corporation (NERC), has two components: 1) operational reliability—the ability of the system to withstand disturbances, such as outage of the largest resource, and 2) adequacy—the ability of the system to always meet aggregate demand. (*)
Resilience is resource adequacy 2.0. It examines the system’s robustness to a wide range of environments. Events are rare and involve systemic failure of many elements. An example is extreme cold. The same event triggers a drop in supply and a spike in demand. The events are systemwide, long, and have implications for other critical infrastructure. They are front-page news.
There are four stages to resilience: 1) preparing for events before they happen, 2) alleviating problems during the event, 3) recovering quickly after the event, and 4) learning from the experience to improve for next time. The forward energy market is primarily about 1 and 2 but also helps with 3 and 4.
While reliability will become a private good with improved demand engagement, discussed in the Appendix, resiliency will remain primarily a public good. Regulators and system operators can improve resiliency through better markets and preparation for systemic events (Cramton, 2022).
Resiliency is essential in avoiding system outages. Historically, system outages are resiliency failures:
The 2000-2001 California energy crisis was caused by an arid year, resulting in low hydro production. The tight market, combined with a design flaw, caused market failure. The flaw was preventing the utilities from managing risk. The utilities entered the tight year without adequate hedging, purchasing their energy in the day-ahead and real-time markets. When the tight market caused sustained high prices, the utilities neared bankruptcy, interrupting trade from excessive counterparty risk.
The 2003 Northeast blackout’s proximate cause was a software bug in FirstEnergy’s alarm system that led to an inadequate response when foliage fell onto transmission lines. The cascading outage caused 55 million people to lose power for seven hours or more.
The 2021 Texas crisis was caused by extreme cold. Electricity demand spiked, driven by electric heat, and supply tanked, primarily from a lack of gas, creating a gap of about 20 Gigawatts in a system with a 66-Gigawatt winter peak. Millions of households lost power for multiple days in extreme cold. At least 246 people died, and damages were about $130 billion.
These three failures include all the major North American failures since 2000. Including Europe, we would add the 2022 European electricity crisis caused by the Russian invasion of Ukraine, resulting in sustained high natural gas prices and the failure of many inadequately hedged service providers.
Systemic events caused all four failures. Resource adequacy and the traditional reliability measures had nothing to do with these costly events. To focus on reliability—non-systemic events—rather than resiliency is to ignore the essential lessons of history. Systemic events from extreme weather and high renewable variability are increasing in frequency and magnitude as our dependence on electricity is increasing. The need for system operators and their regulators to focus on resiliency is immediate and critical.
Rather than a resource adequacy assessment, we need a resiliency assessment. The assessment finds the most effective ways to improve resiliency, reducing the system’s vulnerability to a broad set of systemic events: extreme weather, high variability in renewable production, cyber-attacks, fuel supply disruption, and critical system failure. The forward energy market improves resiliency to each of these vulnerabilities.
(*) See Macey et al. (2024) for an illuminating discussion of NERC’s failure to move beyond reliability and better address resiliency.
