Machine Learning for Probabilistic Short-Term Electricity Demand Forecasting in Great Britain

A personal research project. Code and full week-by-week working notes: PLAN.md. Every result below is reproducible from a numbered notebook in notebooks/.

1. Motivation

Great Britain's electricity system operator (NESO) forecasts national demand for every half-hourly settlement period, every day, to balance supply and demand in real time. Forecast error is not a purely academic inconvenience — it has a direct operational cost: more reliance on fast-reacting (often gas) balancing reserve, more expensive balancing actions, and in some conditions more curtailment of renewable generation that would otherwise have been usable. Better probabilistic forecasts — a calibrated range, not just a point estimate — let a system lean less on conservative fossil backup and more on variable renewables with confidence.

This project asks two linked questions: (1) how much of NESO's forecasting problem can be reproduced from scratch, using only free, public data and a fairly standard ML toolchain, and (2) does the answer actually matter, in £ and tCO2 terms, at the accuracy differences found. It is built as a genuine research exercise — including reporting negative results and correcting earlier mistakes in the open — not a leaderboard exercise.

Why this problem, not another

The cause-prioritization case for this project, made explicit rather than assumed: importance is real but moderate — grid balancing efficiency matters for decarbonization, but marginal grid-forecasting work isn't among the highest-importance technical cause areas by typical effective-altruism-style cause rankings (AI safety and biosecurity are usually ranked above marginal climate/energy technical work, partly because climate/energy already attracts large non-EA capital and talent). Neglectedness is low — GB demand forecasting is a well-resourced space (NESO's own in-house forecasting team, an established academic literature via GEFCom-style competitions, commercial vendors) — this project doesn't fill a gap nobody else is working on. Tractability, for the specific purpose this project was actually chosen for, is high: free public data (NESO, Open-Meteo, NASA POWER, Elexon, Carbon Intensity API) made an end-to-end, honestly-evaluated probabilistic forecasting pipeline buildable solo in 8 weeks, with a real, falsifiable headline claim (§7, §9) — exactly the property needed to demonstrate applied ML rigor credibly to a prospective employer or collaborator.

In short: this problem was chosen for career-capital-building and rigor-demonstration in a genuinely public, verifiable, decarbonization-adjacent domain — not on a claim that it is the most neglected or highest-importance technical problem an equivalent 8 weeks could have gone toward. (AI safety evaluations or epidemic/biosecurity forecasting would likely rank higher on importance and neglectedness, at the cost of far less available public data and a much steeper solo on-ramp — a plausible direction for where this skill set goes next, see §12.)

2. Data

Data decisions worth stating explicitly: wind/solar (embedded generation) are used as features, never subtracted from demand a second time — NESO's own FAQ confirms embedded generation is already invisibly netted into ND by physics before it's ever measured, so subtracting it again would double-count the effect. interconnector is the sum of all individual link flows (individual links are highly collinear, all driven by the same GB-vs-Europe price differential).

(A date-parsing bug corrupting part of the 2025 data was found and fixed during this project — see PLAN.md's Follow-up 7 for details; all results below use the corrected data.)

3. Methodology

Time-aware validation only. No random train/test splits — expanding-window / walk-forward splits throughout. The project's splits were fixed once (Week 1) as TRAIN=2020–2023, VALIDATION=2024, TEST=2025, and later extended forward by one year (TRAIN=2020–2024, VALIDATION=2025, TEST=2026 provisional) once 2025 had fully elapsed and sat unused — a genuinely idle year being promoted into service, not a reopening of anything already evaluated against. TEST under either definition has never been evaluated against for model selection; every number reported below is VALIDATION-only unless stated otherwise.

One comparable scale across every result. MAE and RMSE are reported alongside MASE (MAE scaled against the seasonal-naive baseline), so results from different weeks and different feature sets stay comparable.

Forecast horizon is an explicit, fixed parameter. An early result (a 30-minute-ahead persistence forecast trivially beating every seasonal baseline) turned out to be a horizon artifact, not a real finding — GB demand barely moves in 30 minutes. 1d (48 settlement periods) is used throughout the results below: it's the horizon genuinely comparable to NESO's own day-ahead forecast.

Model: LightGBM, direct per-horizon models (found to beat a single recursively-rolled model at this horizon), with a randomized walk-forward-CV hyperparameter search confined entirely to TRAIN. Features: deterministic time/calendar features (half-hour-of-day, day-of-week, month, Fourier terms at daily/weekly/annual periods, bank/school holidays, event tiers), lag and trailing-average demand features (filtered to only those valid at the forecast horizon — a lag shorter than the horizon would use data not yet known at issue time), and weather features (temperature, heating/cooling degree-days, wind speed, cloud cover, shortwave radiation, and — the most recent addition — a 1-day trailing cumulative degree-day feature). wind/solar/ interconnector outturn values are deliberately never used as model features, since they are same-period actuals with no free historical forecast archive — using them would misrepresent deployable accuracy.

4. Results: baseline → ML → weather → probabilistic

Significance-tested (§9): is any of this real, or noise? Re-run independently under a Diebold-Mariano test (notebooks/24_significance_testing.ipynb) — weather-on vs. weather-off, p = 6.9e-51; outturn wind/solar vs. none, p = 7.7e-16; self-forecast wind/solar vs. none, p = 0.0042 (a genuine update to the "0.09%, noise" framing above — see §9). All three survive Benjamini-Hochberg FDR correction alongside three other headline comparisons tested together. | + 1-day cumulative degree-days (this session) | modest overall (~0.2%), real bias fix in extreme buckets | see §6 |

Weather is the standout lever, and it's mostly a proxy effect, not a direct one. shortwave_radiation_wm2 and wind_speed_ms rank above temperature_c in feature importance — not what a temperature-only hypothesis predicted. The likely mechanism: embedded wind/solar generation is already netted into demand before it's ever measured, so weather variables that correlate with that suppression effect (wind speed with embedded wind, shortwave radiation with embedded solar) are legitimate forecast-based proxies for it, and evidently strong ones.

Probabilistic forecasting (LightGBM quantile objective, same feature set): the nominal-80% [P10, P90] interval achieved only 65.2% empirical coverage (PICP) — meaningfully overconfident, and not a tuning artifact (an untuned check showed essentially the same gap). The reliability diagram shows a textbook tail-compression pattern: the model knows the centre of the distribution well but systematically narrows both tails, a well-documented failure mode of independent per-quantile tree boosting. See §5.

5. Calibration

The overconfidence in §4 is not uniform — it concentrates exactly where it matters most.

• Bias and calibration failure are the same finding, not two separate ones. The point model systematically over-forecasts on every demand-suppressing extreme (is_hot: bias −560 MW, is_high_wind: −796 MW, is_christmas: −1,242 MW), and PICP collapses in exactly those buckets (is_hot PICP 0.558, is_christmas 0.426, both far below the 0.652 overall figure) — a systematic over-forecast pulls the whole quantile band upward, so actual demand falls below even P10 more often.
• A targeted fix (an is_christmas calendar feature + 3× sample-weighting on the three worst buckets) reduced point-model bias 10–18% in the targeted buckets with no cost to overall accuracy — but made quantile-model calibration worse in those same buckets, even as it improved the pinball-loss objective and the aggregate PICP number. Kept for the point model, explicitly not applied to the quantile model — a concrete demonstration of why pinball loss and PICP need to be evaluated separately, not assumed to move together.
• Renewable-output deciles show two different shapes. Point accuracy degrades smoothly and continuously across the whole wind+solar output range (MAE 558→1,417 MW from lowest to highest decile) — a gradual, physically coherent gradient. PICP, by contrast, is flat and stable across deciles 1–9 and then falls off a cliff at decile 10 (0.52) — calibration failure is a concentrated tail phenomenon, accuracy degradation is not.

Stretch: does a structurally different model fix the overconfidence, rather than just describing it? Two candidates compared head-to-head on 1d, on a fresh evaluation year (VALIDATION=2025): Conformalized Quantile Regression (CQR — Romano et al. 2019, a post-hoc correction fit on a held-out calibration year) and NGBoost (Duan et al. 2019 — one Normal distribution boosted per row, so quantiles come from a single coherent CDF and can't cross by construction). Full comparison in notebooks/23_calibration_robustness_comparison.ipynb.

CQR wins outright. A single scalar correction, fit on one held-out year and costing nothing to retrain, lifts PICP from 62.6% to 75.1% — most, not all, of the way to nominal 80% (the shortfall is plausibly because the calibration year and the evaluation year aren't perfectly exchangeable, the one assumption CQR's guarantee actually needs) — while improving lower-tail pinball loss, not just widening the interval defensively. NGBoost, a genuinely different model, lost on every metric here — for a specific, honest reason, not because the idea is wrong: its median forecast is 8.9% worse than LightGBM's because matching LightGBM's walk-forward-CV tuning budget was computationally infeasible in this project's time budget (one 600-tree NGBoost fit took 862s vs. ~313s for LightGBM's entire tuned quantile ensemble). Its one clean structural win — zero quantile crossing by construction, vs. LightGBM's 3.6% needing a rearrangement patch — wasn't enough to make up the gap. Decision: CQR is now applied to every horizon in the live demo (edf.models.registry, app.predict); NGBoost remains a legitimate idea for later, conditional on a real tuning budget. Significance-tested (§9): an independent DM test on the coverage-indicator series (baseline non-coverage vs. CQR non-coverage) gives p ≈ 4.7e-255 — the most significant of every comparison tested, expected given CQR's near-uniform per-row correction (see §9).

6. Extreme events and the newest feature finding

COVID-19 lockdown (2020), examined as a case study, not a held-out accuracy number (the model was trained on 2020–2021 data, so it had the chance to fit this exact historical pattern — a stated limitation of the case study itself). Isolating the correct comparison (same calendar window across years, not a naive within-2020 before/after) shows naive forecasting genuinely blindsided (~25–30% worse than the equivalent window in other years) while the point model was essentially unaffected — visually, the naive baseline stays pinned near pre-lockdown levels for about a week after the announcement, while the point model tracks the collapse almost immediately.

Two new feature ideas, sourced from a literature search rather than intuition, tested this session: substituting apparent ("feels-like") temperature for dry-bulb temperature in the degree-day formula was rejected (0.50% worse); a 1-day trailing cumulative heating/cooling degree-day feature was adopted — a small overall MAE gain (0.23%) but a real, broad bias reduction concentrated in exactly the buckets already known to be weak (is_hot: 14.5% smaller bias, is_high_wind: 5.6%, is_christmas: 12.6%), the first improvement this project has found in several rounds of feature research. Longer cumulative windows (2–7 days) made things worse, monotonically — the literature's "longer persistence helps" framing didn't transfer to a model that already carries explicit multi-week demand-lag features.

7. Does this beat NESO — and the forecast-combination result

No, not from scratch, and the honest comparison is worse than an early, coarser check suggested. Against NESO's own portal archive (~12 cardinal points/day), the from-scratch model lost overall (~6.3%) but won at "other" (non-peak, non-trough) cardinal points by 7.9%. Re-checked against Elexon's NDF — a denser, fresher benchmark — that edge disappears entirely: NDF beats the from-scratch model in every regime checked (trough 43.8%, peak 40.8%, other 46.9% — its largest margin). A residual-correction model (predict actual − NDF, add back) barely helped and made things slightly worse overall.

A plain linear forecast combination does better than NDF alone — the first improvement over NDF found by any mechanism. Error correlation between the from-scratch model and NDF is only 0.36, and the from-scratch model is individually more accurate on 33% of half-hours despite losing on every aggregate regime — textbook Bates-Granger combination territory. A combination weight (≈0.17 on the from-scratch model, fit via expanding-window walk-forward, zero overlap with the evaluation window) beat NDF alone on two independent, non-overlapping years:

(Updated 2026-09-11, PLAN.md Follow-up 10: re-run after Follow-ups 8/9 revised calendar_events.py for ~2% of rows in this window — was 572.11/3.55%, now 572.50/3.48%. Same finding, same order of magnitude; the DM-test/FDR numbers directly below were computed on the pre-Follow-up-10 feature table and were not individually re-run, a stated residual rather than a re-verified claim.)

Significance-tested (§9): yes, this is real. An independently re-run Diebold-Mariano test on this exact comparison — blend vs. NDF alone, half-hourly, VALIDATION=2025 — gives p = 1.55e-9 (mean loss differential −24.81 MW/row in the blend's favor), and survives Benjamini-Hochberg FDR correction alongside five other headline comparisons tested together (§9). This is the project's most important and most heavily-scrutinized claim, so it's also the one most worth a rigor check beyond a single point-estimate percentage.

A more flexible regime-aware stacking meta-model (LightGBM combining both forecasts plus calendar/bucket context) initially did worse than the simple blend (overfitting on a single-season, six-month fit window — its is_hot bias flipped to +352 MW). Refitting on a proper multi-year, multi-season out-of-sample set fixed most of that gap (515.28 vs. the blend's 513.07) but still didn't beat the simple weighted average — added complexity without measured benefit, not evidence regime-awareness can't work with more data.

This combination result is the project's final headline forecasting number. It survived being independently re-derived on a second year with a real, previously-undetected data bug found and fixed along the way (§2) — about as strong a robustness check as this project has produced.

8. Impact estimate

A rough, deliberately transparent back-of-envelope estimate of what §7's forecast-combination result — 21.03 MW average absolute-error reduction versus NDF alone, on the corrected 2025 VALIDATION — could plausibly be worth. Full arithmetic in notebooks/22_impact_estimate.ipynb. This is a directional estimate, not a causal claim — shown in full so a reader can disagree with an assumption rather than just the conclusion.

Illustrative potential impact, not realized impact. Nothing below describes an effect that has actually happened. NDF is NESO's own live production forecast; this project's model and its combination with NDF have never been deployed; no balancing decision has ever been made using either. The numbers answer a conditional question — if a system operator adopted this specific, free-data combination technique, what would it plausibly be worth per year, at current demand scale and current data quality — not "what has this project saved so far" (zero, precisely) and not "what it will save if adopted" (unknown: real deployment would add integration cost, ongoing recalibration of the combination weight — see Limitations — and its own implementation risk). The realized-impact figure for this project, today, is zero; the table below is the case for why that number is worth trying to move, which is the actual purpose of §12.

Method: treat the 21.03 MW average error reduction as if sustained across every settlement period of a year (184,223 MWh of "avoided forecast-uncertainty volume," ≈0.08% of GB's annual demand), then price it two ways. £: NESO's own half-hourly Balancing Services Use of System (BSUoS) cost data, FY2024-25, matched to this project's own demand data over the identical dates, gives two rates — a broad one (all six BSUoS cost categories ÷ total demand, £8.95/MWh, which generously treats the entire system-operation cost as forecast-sensitive) and a narrow one (only the Positive Reserve + Energy Imbalance categories — the two most mechanistically tied to "the system needed more than expected" — £0.40/MWh, the more defensible figure). tCO2: the Carbon Intensity API's standard generation-type factors for CCGT (394 gCO2/kWh, GB's typical marginal plant) and OCGT (651 gCO2/kWh, faster-reacting peaking plant, a plausible upper bound).

The wide range is the honest answer, not a flaw to resolve — public aggregate data can't determine how much of GB's balancing bill is actually sensitive to demand-forecast accuracy specifically. If a single number is wanted, the narrow-rate/CCGT low end (~£74k, ~72,600 tCO2/year) is the more mechanistically defensible one — it isolates the cost categories closest to demand imbalance and uses typical, not extreme, marginal generation.

What this number represents, precisely: not "this project's model saves the system this much" — NDF is NESO's own already-deployed forecast, not something this project replaces. It's "combining a free, from-scratch model with NESO's own published forecast — a technique not currently known to be in operational use — would be worth roughly this much per year if adopted." A smaller, more honest claim than "beating NESO," and a more useful one for judging whether any of this actually matters.

9. Statistical significance and false-discovery-rate correction

Every result above (§4–§8) reports a point-estimate delta — "27.65% improvement", "3.48% improvement" — with no test of whether it's distinguishable from noise, and no correction for the dozens of comparisons this project has run over its life. That gap is addressed directly here for the project's six most load-bearing, currently-live comparisons (notebooks/24_significance_testing.ipynb, src/edf/significance.py), all trained on TRAIN (2020–2024) only and evaluated on the full, untouched VALIDATION year (2025).

Method: the Diebold-Mariano test (1995) on the paired per-row loss differential between two forecasts, using a Newey-West/Bartlett HAC variance estimator with truncation lag h−1 (h=48, the 1d horizon in half-hourly periods) — the standard tool for this setting, chosen over a plain paired t-test because GB demand's strong day/week autocorrelation violates a t-test's i.i.d. assumption, understating the true variance and giving artificially small p-values. Then Benjamini-Hochberg FDR correction (1995) across all six p-values tested together, appropriate for a set of independent exploratory questions rather than one confirmatory claim.

All six survive FDR correction. With p-values this small — even the weakest, self-forecast wind/solar at p=0.0042, clears the 0.05/6≈0.0083 a naive Bonferroni correction would have required — there was nothing genuinely borderline in this particular set for BH to catch. That's a real result (these six claims are solid), not evidence the correction was pointless: a more marginal set of comparisons (e.g. the individual extreme-bucket findings in §6) would be a more realistic place to expect FDR correction to actually change a conclusion, and hasn't been tested this way yet (see §11).

One finding changed on re-test, reported honestly rather than smoothed over. Self-forecast wind/solar (§4) was originally found to give a 0.09% MAE change ("noise, not signal") under the project's older split and CV-tuned hyperparameters. Under the current split and untuned but matched-capacity models, the same comparison shows a small but statistically significant ~4.1% MAE reduction (p=0.0042) — a genuine update attributable to the different configuration, not a contradiction. It remains the weakest and smallest effect of the six by a wide margin, and doesn't overturn the practical recommendation not to deploy it (§4) — but "small, real, and not clearly worth the complexity" is a more accurate summary than "zero effect."

Caveat applying to all six: statistical significance answers "is this effect distinguishable from zero," not "is this effect large enough to matter" — with n=17,519 half-hourly rows, even modest effects are detectable, as the self-forecast wind/solar result illustrates directly. The practical-significance judgment calls made throughout §4–§8 (e.g. "not adopted", "kept for the point model") stand on their own merits and aren't overridden by a small p-value alone.

10. Limitations

• Weather is observed (ERA5 hindsight) for most of the project's date range, not forecast. The one exception — a genuine day-ahead weather forecast archive — only starts 2024-03-07; results using it are an honest lower bound on how much of the weather gain survives real-world lead time, not a claim the whole 2020–2025 window's weather results are deployable as reported.
• No free historical forecast archive exists for wind/solar/interconnector. A from-scratch forecast of wind/solar was built and tested; it didn't recover the leaky upper-bound gain. Interconnector flow (price-driven, not weather-driven) remains an unaddressed limitation.
• Embedded generation is invisibly netted into demand by physics, not by any subtraction NESO performs — this project's wind/solar features exploit that correlation but cannot fully reconstruct a true net-demand ("duck curve") series without transmission-scale generation data.
• No systematically-collected TV-audience dataset exists for this project's window; the event-tier table is a researched qualitative judgement, cross-checked against one hard NESO MW figure, not a continuously measured variable. A paid feed (BARB overnight ratings) would sharpen the tier boundaries retrospectively, but wouldn't change deployability: it publishes next-day audience actuals, not a forecast, so a live day-ahead model would still only have the fixture schedule available ahead of time — the same information the tier table already encodes. See PLAN.md's Week 1 limitations for the fuller argument.
• The post-2026-09-11 slice of the event-tier table was added by an AI agent doing live web search, not by hand-checking primary sources the way the 2020-2025 table was (see calendar_events.py's module docstring and PLAN.md's "keeping calendar_events.py current" follow-up) — a lower-confidence batch, not yet human-reviewed, kept only because the alternative (an already-year-stale table) was worse for a live app. The same pass also revised the tiering rule to be UK-audience-based rather than England-only (PLAN.md Follow-up 9) — e.g. the 2022/2026 World Cup finals now count as "large" despite no England team, and Scotland's 2026 World Cup games count on the same NESO MW evidence England's do. That's a real methodology change mid-project, not just new rows in an unchanged table, and it makes the table more exposed to a wrong published-audience number than the old rule was, not less — there's no fixture list to sanity-check a viewing figure against the way there was for "was England playing."
• Population-weighted weather across 5 conurbations is a coarse proxy, not a full gridded weighting — a stated, unfixed simplification.
• The COVID case study is not a held-out accuracy number — the point model was trained on data including the lockdown period itself, so its apparent robustness there is a different claim than genuine zero-warning forecasting skill.
• TEST (2026) has now been evaluated once, per the pre-registration below — see that subsection for the result. All results in §4–§9 above remain VALIDATION-only, computed before TEST was touched. The 2026 source's changed CSV filename and NESO-flagged "missing Scottish transfer data" issue (PLAN.md's Follow-up 6) turned out not to block this: the flagged column (SCOTTISH_TRANSFER) isn't one this project uses, and ND itself — checked directly — has zero NaNs and matches an independent live NESO feed exactly over their overlap window. See PLAN.md's Follow-up 11 for the full verification.
• Tested, not assumed: does the combination weight need periodic recalibration, and does the result hold up across more than two splits? (notebooks/25_rolling_combination_recalibration, edf.models.combination.rolling_combination_weights/apply_rolling_combination, edf.evaluate.rolling_folds.) Refitting the weight monthly (expanding window, 42 folds, mid-2022 to end-2025) instead of once shows the weight is stable, not drifting — it settles to ~0.17 within a few months of NDF-archive history and stays in a 0.143–0.175 band for the remaining three years, including all of VALIDATION. Consequently, monthly recalibration does not measurably beat the static, fit-once weight (pooled MAE 527.47 rolling vs. 527.20 static — rolling only wins in 16/42 months) — a genuine negative result, kept rather than discarded. The combination beats NDF alone in 35/42 individual months (83.3%), a finer-grained robustness check than the original two splits, though the 42 folds share nested expanding windows and autocorrelated demand, so they aren't 42 independent replications the way the two splits were. Practical upshot: no live recalibration pipeline is needed for this weight at this data scale — HEADLINE_COMBINATION_WEIGHT stays a static constant — but the tooling to detect future drift now exists and is tested, rather than the weight being frozen on faith.

Pre-registered TEST (2026) expectation — and the observed result

Written before TEST was evaluated against, so it couldn't be adjusted with hindsight once the 2026 data-quality issue above was resolved. This was a prediction to check the eventual result against — not a target to hit, and not a reason to re-derive anything (the combination weight, the feature set) if TEST disagreed with it. TEST has since been evaluated once (2026-09-11, PLAN.md Follow-up 11) — the observed result is reported inline with each prediction below, unedited from what was first written.

• Combination result (§7). The blend-vs-NDF-alone improvement was found twice, independently, on non-overlapping windows: 3.75% (2024 H2) and 3.55%, corrected to 3.48% by Follow-up 10's feature-table fix (2025) — a citation correction made before TEST was touched, not an adjustment informed by it; the stated range below is unchanged by it. Both land in a narrow band. Expectation: TEST=2026 lands in roughly the same 2–5% MAE improvement range. A result outside that range — including a negative one — is a genuine finding (the combination weight failed to generalize to a third year), to be reported as such, not explained away. Observed (2026-01-01 to 2026-09-11, n=12,159 half-hourly rows): MAE 1,042.12 MW (our model alone) vs. 648.29 MW (NDF alone) vs. 623.10 MW (blend) — a 3.89% improvement. Inside the predicted range, and the closest of the three independent measurements yet to the other two (3.75%, 3.48%). This is the confirmation the pre-registration was designed to either give or withhold — it gives it.
• From-scratch model alone vs. NDF (§7, §9). NDF beat the from-scratch model in every regime checked under the original evaluation (40–47% in the worst buckets), and the same ordering was independently reconfirmed on VALIDATION=2025 via the DM test (§9): a mean loss differential of +336.23 MW in NDF's favor, p=5.22e-53. Expectation: the same ordering — NDF wins outright — holds on TEST=2026. A result where the from-scratch model wins outright would be the most surprising finding in the project and would need independent replication before being trusted. Observed: NDF wins outright, as predicted — but by 60.75%, a substantially wider gap than the 40–47% seen before. Direction confirmed, magnitude is the genuine surprise here, reported as such rather than smoothed into the expected range. The from-scratch model's own point predictions are 8+ months past its training cutoff (2025-12-31) by the end of this window, so some real drift is a plausible contributor alongside ordinary between-year variance — not independently isolated, a stated open question rather than a settled explanation.
• Weather-driven error reduction (§4, §9). The original evaluation found a 27.65% MAE reduction from adding weather features (VALIDATION=2024); the same weather-on/weather-off comparison was independently reconfirmed as significant on VALIDATION=2025 (§9: −359.34 MW mean loss differential, p=6.92e-51), though the exact percentage hasn't been recomputed on the new split. Expectation: TEST=2026 continues to show a large, significant weather-driven error reduction — not necessarily exactly 27.65%, but clearly present — conditional on 2026 weather data being available at comparable quality. Not separately re-run on TEST=2026 — the evaluation performed (Follow-up 11) scored the already-deployed registry model end-to-end (weather included throughout, per the live app's feature set) rather than repeating the §4 weather-on/weather-off ablation a third time; that ablation on TEST remains open work (see §11).

What would count as a genuine surprise, not noise: any of the three landing meaningfully outside its stated range. With n≈17,500 half-hourly rows, small differences are easily statistically significant per §9's caveat, so "different from this expectation" isn't automatically "important" — but a large or directionally-reversed gap would be. Two of the three predictions were checked: the headline combination result confirmed cleanly; the from-scratch- model-alone comparison confirmed on direction but surprised on magnitude, reported as such rather than explained away, the same way §2's date bug and §9's self-forecast-wind/solar reversal were.

11. What you'd do with more time

• Give NGBoost (§5) a fair tuning budget. Its loss in the CQR/NGBoost comparison traces to an un-tuned, shallow default base learner, not to the Normal-distribution assumption being wrong — a proper walk-forward-CV search (deeper trees, more estimators, possibly a skewed Dist given this project's own bias findings) is expensive (~15min per fit at this project's data volume) but not yet ruled out as competitive with CQR.
• A literal GB "duck curve" / net-load analysis using transmission-scale wind/solar generation (Elexon BMRS fuel mix) — embedded generation alone can't reconstruct it.
• Verify the 2026 data source properly (the Scottish-transfer-data gap NESO itself has flagged, and the new CSV filename pattern) and extend the combination result to a third independent year.
• Regional decomposition (GB's 14 Grid Supply Point regions) rather than one national-aggregate model — published GB net-load work suggests region-specific weather effects (e.g. wind speed mattering more for Scotland, solar irradiance more for southern England) that a single national model averages away.
• A properly walk-forward-validated (not single-split) version of the forecast-combination weight, and a test of whether combination helps the probabilistic forecasts the same way it helped the point forecast.
• Economic/social signal features (GDP, unemployment, or news-derived signals) — a UK/Ireland academic study found ~6% improvement from this class of feature; not yet tried here.
• Extend §9's significance testing to the extreme-bucket findings (is_hot/is_high_wind/ is_christmas bias fixes, the renewable-decile PICP cliff) — smaller, more marginal effects than the six headline comparisons tested so far, and a more realistic place for BH-FDR correction to actually flip a conclusion rather than confirm an already-robust one.

12. Next step: seeking outside scrutiny

§8's number is illustrative, not realized (§8). The gap between the two is external validation — someone outside this project checking the methodology and the headline claim, and ideally someone positioned to act on it if it holds up. The next concrete step is to put this work in front of people who could do that, rather than treating the repo and this report as the end point:

• NESO / National Grid ESO's open-data and innovation channels. The most direct route, and the only party who could actually deploy the combination result (§7) if it survives their scrutiny.
• Climate Change AI and similar applied-ML-for-climate communities, where a from-scratch demand-forecasting write-up with a stated, ranged impact estimate is a natural fit for technical feedback from people already working in the field.
• A public technical write-up (a relevant subreddit, Hacker News, or similar) — lower-friction than a direct pitch to NESO, and a reasonable first test of whether the methodology survives contact with a technically adversarial audience before a more targeted approach.

The honest framing: this is a test of whether §8's number is worth anyone's attention, not an assumption that it will be. No interest, or scrutiny that surfaces a flaw, is itself a useful and reportable outcome — and will be recorded here, in this section, whichever way it goes.

Findings

A living log of concrete, checked findings from data exploration — updated whenever an analysis produces a real result, not every intermediate exploration step. See PLAN.md for the plan and notebooks/ for the underlying analysis behind each entry.

1. Demand shape & the "is the decline real?" question

• GB demand is winter-peaking with no material summer air-conditioning peak — structurally different from summer-peaking systems (e.g. most of the US). Cold, not heat, is the extreme worth caring about most (see §2).
• ND declined ~3.7% from 2020 to 2025, but a "true demand" proxy (ND + embedded wind + embedded solar, reconstructing what ND is hiding — see PLAN.md's embedded-generation note) declined only ~1.2% over the same period. Roughly two-thirds of the apparent ND decline is a growing-embedded -generation visibility artifact, not real demand reduction — though a genuine ~1.2% underlying decline remains, not yet explained (not weather-normalized). Verified the arithmetic holds exactly at the row level (true_demand_proxy − (ND + wind_gen + solar_gen) = 0 for every half-hour), not just approximately at the annual aggregate.
• This is almost entirely a solar story, not a wind one — worth being precise about, since "wind+solar capacity grew ~39%" (an earlier, blurrier framing) obscures that wind and solar moved in opposite directions. 2020→2025: wind capacity +1.4% (6,515→6,606 MW) while wind generation fell ~6% (2,077→1,951 MW, capacity factor 31.9%→29.5%) — most plausibly 2025 was simply a less windy year on average, a weather-variability effect, not a capacity one. Solar capacity grew +57.8% (13,188→20,813 MW) and solar generation grew almost exactly in proportion (+55.1%, 1,374→2,130 MW, capacity factor flat at ~10%) — a clean, capacity-driven trend. Capacity and generation are different scales (capacity factor ~10-32%, nowhere near 100%) and shouldn't be compared directly — a real correction made after a user question caught the first draft conflating them. Source: notebooks/01_explore_raw_data.ipynb.
• 2021 was the highest-demand year in both ND and the true-demand proxy — plausibly a colder winter rather than a COVID-rebound effect (lockdown 3 was still running through Q1 2021). Not yet checked against weather data.

2. Weather

• Open-Meteo (ERA5) and NASA POWER agree closely but not identically — 0.98 temperature correlation, ~1°C mean absolute difference. Genuinely independent sources, not redundant.
• Open-Meteo's "Historical Forecast API" is a short-lead (~0-3h) nowcast, not a day-ahead forecast — corrected after initially mischaracterizing it as "archived forecasts" (see PLAN.md).
• Unresolved: a genuine day-ahead forecast source (_previous_day1) shows lower MAE against ERA5 hindsight than the short-lead nowcast does — even after pinning both endpoints to the same explicit model (gfs_seamless), ruling out "different default model" as the explanation. The naive "more lead time = more forecast error" expectation doesn't hold here and the cause isn't understood yet. Flagged as a Week 4 pre-req, not resolved. Source: notebooks/02_explore_weather_data.ipynb.
• Cold vs. heat asymmetry, confirmed with real numbers: the 13 Dec 2022 cold snap (-17.3°C overnight, Braemar) added +5,319 MW average demand vs. a control day (~17% higher, peak +4,869 MW); the 19 Jul 2022 record heatwave (40.3°C, Coningsby, UK's hottest day on record) added only +712 MW (~2.8% higher, peak +888 MW). Cold has ~7x the demand impact heat does — consistent with GB's gas-heated, low-AC-penetration housing stock. Source: notebooks/03_interesting_days.ipynb.

3. COVID as a natural experiment

Not yet investigated in depth — a later thread. lockdown_level (none/partial/full) is already built in src/edf/data/calendar_events.py, sourced from the Institute for Government's timeline.

4. TV/event "pickup" — verified in our own data, not just cited from NESO

• Euro 2020 R16, England v Germany (29 Jun 2021): NESO's own article reports ~1GW pickup at half-time, ~1.6GW at full-time. At half-hourly resolution, the match evening's demand declines more slowly than a control evening's from kickoff through full-time — the same direction as NESO's reported effect, though the sharp minute-scale spike itself isn't resolvable at this granularity (a stated resolution limitation, not a failure to find the effect).
• Euro 2020 Final, England v Italy (11 Jul 2021): a clearer pattern — demand dips below a control evening during play, then crosses over and exceeds it from full-time through the penalty shootout (the match went to extra time + penalties, ending ~22:30 BST).
• Day-type "how unusual" ranking, confirmed with real December 2023 numbers: ordinary weekday (~32.2 GW avg) > ordinary weekend (~28.2 GW) > bank holiday/Boxing Day (~23.8 GW) > Christmas Day (~22.8 GW). The weekday-to-weekend gap is comparable in size to the weekend-to-holiday gap — worth remembering when weighing calendar features later.

5. Renewable growth & interconnectors

• Embedded wind+solar capacity grew ~39% 2020→2025 (see §1).
• Interconnector go-live dates are directly visible in the raw flow columns — a nice concrete verification that the data reflects real-world construction: NSL (Norway) near-zero before 1 Oct 2021, then ~997 MW average magnitude after; Viking (Denmark) near-zero before 29 Dec 2023, then ~763 MW after; Greenlink (Ireland) near-zero before 30 Jan 2025, then ~403 MW after.
• GB is a net interconnector importer on average in every year 2020-2025 except 2022 (net exporter) — plausibly tied to the 2022 European gas price crisis following Russia's invasion of Ukraine (Feb 2022), which may have made continental gas-fired generation relatively more expensive than GB's own mix that year. Not yet checked against actual price data — a hypothesis, not a confirmed cause.
• Weak but directionally sensible correlations: interconnector flow vs. wind generation ≈ -0.16 (some tendency to export more / import less when windy); vs. demand ≈ +0.21 (some tendency to import more at high demand). Consistent with interconnectors acting as a release valve in both directions, just not a dominant one.

Ideas for future data (renewable-growth context — not yet sourced, for later)

Factors plausibly driving the embedded wind/solar capacity growth seen in the data, worth sourcing if a fuller causal/contextual story is wanted:

• Smart Export Guarantee (SEG, started Jan 2020) — pays small-scale generators for exports, replacing the closed Feed-in Tariff scheme; overlaps our window almost exactly.
• The 2021-2023 European gas price crisis — sharply higher household/business energy bills, creating a strong private payback incentive for rooftop solar specifically in this period.
• Falling solar panel / battery storage costs (global LCOE trend).
• CfD (Contracts for Difference) auction rounds (AR4 2022, AR5 2023, AR6 2024) — mostly larger-scale; worth checking how much of this is embedded vs. transmission-connected capacity.
• UK Net Zero by 2050 legal commitment (June 2019) and associated policy support.
• DNO grid-connection queue reforms / permitted development rights changes for small-scale installations.
• Possible sources to pull from later: DESNZ (Dept for Energy Security & Net Zero) solar/wind deployment statistics; Ofgem's SEG/FiT installation registers; historical wholesale gas/electricity prices (to test the 2022 price-crisis interconnector hypothesis directly).

6. Data-quality "gotchas" found (methodology notes — all corrected, not left as bugs)

• Embedded wind/solar generation is already invisibly netted into ND (confirmed via NESO's own FAQ) — not a separate additive component. Corrected the original Week 7 plan, which had double- subtracted it.
• Grouping by UTC calendar date trivially always shows 48 settlement periods/day — the real clock-change anomalies (46/50-period days) only show up when grouping by the raw local SETTLEMENT_DATE.
• SETTLEMENT_DATE's text format is inconsistent across NESO's own yearly files (01-JAN-2020, 01-Jan-23, 2025-01-01).
• SCOTTISH_TRANSFER is genuinely absent pre-2023 (NESO added the column), not a data defect.
• Weather source #3 was originally mischaracterized as "archived forecasts" — corrected after re-reading Open-Meteo's own docs properly instead of trusting the first read.
• wind/solar/interconnector have no free historical forecast archive available anywhere — a genuine, permanent limitation (Week 4b proposes forecasting them ourselves instead of solving the data-availability problem directly, since it can't be).

Last updated: 2026-09-06.