Updating the electricity supply and trade mix

This workflow builds a database whose electricity supply mix (generation by technology) and trade mix (bilateral sourcing between regions) reflect recent statistics. Both updates rewrite the composition of the table while preserving every column total: within each region the weight only moves between the electricity labels, so every buyer keeps its total electricity input and the rest of the table is untouched.

The shares come from two open data providers:

• EMBER — generation-by-fuel, used for the supply mix. EMBER covers every country worldwide.

• ENTSO-E — scheduled cross-border commercial exchanges, used for the trade mix. Bilateral flows only exist for the European synchronous area, which is exactly where a trade mix is meaningful.

The workflow applies to any database that exposes a disaggregated power sector (separate generation technologies rather than a single electricity sector): EXIOBASE (hybrid and monetary, IOT and SUT) and EMERGING-E (IOT). The examples below parse EXIOBASE Hybrid v3.3.18 as a SUT and go end to end: parse → supply mix → pool the electricity trade → trade mix → export.

Core methods

• Database.update_supply_mix — rewrite the generation mix in place (from EMBER, or from explicit shares);

• Database.pool_trade — prepare a SUT so the technology mix and the trade mix become two independent market-share columns;

• Database.update_trade_mix — rewrite the regional sourcing of a traded item per destination market (from ENTSO-E, or from explicit shares);

• Database.get_mix — read back a region-by-sector production mix (IOT).

Database.update_supply_mix_iot and Database.update_mix_iot are kept as backward-compatible IOT-only aliases of update_supply_mix.

Configuring the data-provider API keys

Passing the special string "electricity" to update_supply_mix / update_trade_mix makes MARIO derive the shares from the provider data: EMBER generation for the supply mix, the ENTSO-E import mix for the trade mix.

MARIO ships a reduced snapshot of each provider inside the package, so both methods run offline with no key — the reproducible fallback, frozen at the snapshot vintage. To fetch a specific or more recent year live, register a free key and let MARIO query the provider API:

• EMBER — register at https://ember-energy.org/data/api/

• ENTSO-E — request the API token from https://transparency.entsoe.eu/

Keys can be provided four ways; the first one found wins, so a secret never has to live in code:

1. inline at the call site, api_key={"ember": "<key>"};

2. mario.set_api_keys(ember="...", entsoe="..."), set once per session;

3. the EMBER_API_KEY / ENTSOE_API_KEY environment variable;

4. the git-ignored mario/settings/api_keys.yaml file (copy api_keys.example.yaml; never commit real keys).

In a shared notebook, read the keys from the environment rather than typing them in a cell:

import os
import mario

# Configure the provider keys once for the session. Reading them from the
# environment keeps the actual secrets out of the notebook.
mario.set_api_keys(
    ember=os.environ.get("EMBER_API_KEY"),
    entsoe=os.environ.get("ENTSOE_API_KEY"),
)

Once the keys are configured you refer to a provider by name at the call site (api_key="ember" / api_key="entsoe"). Omit api_key entirely to use the packaged snapshot instead of a live fetch.

Parse a database

Download the EXIOBASE Hybrid v3.3.18 files from Zenodo and parse the SUT. The hybrid table ships the disaggregated electricity generation activities and commodities that the mix updates need.

info = mario.download_hybrid_exiobase(path="/path/to/3.3.18", table="SUT")

db = mario.parse_exiobase(
    path="/path/to/3.3.18",
    table="SUT",
    unit="Hybrid",
    extensions="all",
)

Update the supply mix from EMBER

update_supply_mix("electricity") rewrites the generation mix of one scenario in place. The EMBER taxonomy is coarser than the disaggregated electricity labels, so MARIO:

1. aggregates the database electricity bundle to the compatible EMBER fuel groups (coal, gas, nuclear, hydro, wind, solar, bioenergy, other renewables, other fossil);

2. reads the EMBER generation shares for the requested year;

3. redistributes each group total back to the original database labels using the current internal composition of each group.

On a SUT, MARIO first aggregates the disaggregated electricity commodities into one shared electricity commodity — using the packaged profile, no mapping required for EXIOBASE-style layouts — and rewrites the market shares of the generation activities in the supply block s. Transmission and distribution labels are left unchanged. On an IOT the generation sectors are redistributed across the z and Y blocks instead.

# Live EMBER fetch for a specific year (uses the key configured above);
# omit api_key to read the packaged snapshot instead.
db.update_supply_mix("electricity", scenario="baseline", year=2023, api_key="ember")

Regions that EMBER does not cover, or that report no positive generation, are left unchanged and reported in the logs. When a covered region has no observation for the requested year, MARIO falls back to the nearest available year for that region.

Inspecting and customising the mix. A few common variations:

• Explicit shares — instead of "electricity", pass an explicit region -> {label: share} mapping to move weight between named labels (rescale=True normalises arbitrary positive totals). On an IOT database get_mix("electricity") reads the current mix back as region -> {sector: share}.

• Aggregated regions — EXIOBASE Rest-of-World regions and user aggregations are expanded to their member countries automatically; pass region_aggregation=... to supply the concordance explicitly.

• ``aggregate_as_ember=True`` — collapse the disaggregated electricity labels to the EMBER fuel groups after the update.

• Column selectors — column_regions, column_sectors and column_categories restrict the rewrite to a subset of buyer/demand columns.

See Database.update_supply_mix in the API reference for the full argument list. The diagrams below show where the weight lands for each table layout.

Chenery-Moses tables: update the mix on every destination market. On tables converted with to_chenery_moses, the supply block entangles technology and trade: one region’s technology mix is replicated — scaled by its trade shares — inside the s columns of every destination it exports to. The default column_regions=None correctly rewrites the region’s rows in all destination markets, preserving each destination’s trade share. Restricting column_regions to the domestic market would leave the exported production on the old mix (one fleet, two mixes — physically inconsistent). Isard tables and pooled tables built with pool_trade are immune by construction, because the technology mix is stored once.

Pool the electricity trade

On a raw Isard SUT the bilateral sourcing lives in the use-side rows, so every buyer of a region can have its own origin split and the technology mix and the trade mix are entangled. update_supply_mix("electricity") above already aggregated the disaggregated generation commodities into one commodity named electricity; pool_trade pools that commodity, adding per region one "electricity - supply" pass-through activity and one "electricity - need" market commodity:

1. the pass-through activity consumes the whole domestic output of electricity;

2. every buyer moves onto the domestic need commodity, keeping its total input;

3. the supply block routes each destination market to the origin pass-through activities with the bilateral flows observed on the Isard use side.

Destination-level trade totals are preserved exactly, and the initial market shares equal the observed origin shares. Afterwards the technology mix (on the electricity commodity column) and the trade mix (on the electricity - need column) live in two separate market-share columns of s and can be updated independently.

db.pool_trade("electricity")

Update the trade mix from ENTSO-E

update_trade_mix("electricity") reads the ENTSO-E import mix — a first-order net commercial decomposition of scheduled exchanges (ENTSO-E A09), not physical cross-border flows — and rewrites each destination’s electricity sourcing. On the pooled table the traded rows live in the supply block s, so this is an Activity-level mix: pass level="Activity" and name the pooled pair through items (the - supply pass-through activity) and commodities (the - need market column). Read the exact labels back from db.meta.pooled_trade_map: update_supply_mix aggregated the disaggregated generation commodities into one commodity named electricity (lowercase), so the pooled pair is electricity - supply / electricity - need. ENTSO-E supplies only the shares.

fill_uncovered_domestic=True sets the regions ENTSO-E does not cover (non-European: US, CN, JP, … and the RoW aggregates, which are electrically near-isolated) to domestic-only. Leave it False to keep them on their original sourcing.

# Live ENTSO-E fetch into a new scenario cloned from the supply-mix baseline.
# Read the exact pooled labels back from the map pool_trade recorded, rather
# than hard-coding them.
pooled = db.meta.pooled_trade_map["electricity"]

db.update_trade_mix(
    "electricity",
    items=pooled["supply"],
    commodities=pooled["need"],
    level="Activity",
    scenario="electricity_mix",
    clone_from="baseline",
    year=2023,
    api_key="entsoe",
    fill_uncovered_domestic=True,
)

The trade mix covers the ENTSO-E (European) area only; non-EU zones with poor coverage are kept as-is. The basis was validated against Electricity Maps, whose data is proprietary and cannot be redistributed.

The update_trade_mix API reference documents the explicit destination -> {origin: share} form (for arbitrary items and non-electricity trade), the column_sectors / column_categories selectors, and the block each table layout rewrites.

Export

Write the updated scenario to MARIO’s text format (or to_excel / to_parquet / export).

db.to_txt(path="/path/to/output", scenario="electricity_mix")

Excel route

Both updates can also be authored in the shock workbook, without writing any code: the coefficient sheets accept the Supply mix N and Trade mix N types, and shock_calc(...) applies them on top of a new scenario. See Shock analyses for the sheet layouts and worked examples.

Governed data sources: nxbase and nxsut

The live provider APIs (EMBER, ENTSO-E) are one way to feed the mix. A second is to read the same inputs from a governed data backbone that snapshots and versions them, which keeps a pipeline reproducible offline:

• nxbase, the eNextGen data backbone, exposes EMBER generation and ENTSO-E trades through a read-only query API. nxbase is currently in beta and public API tokens are not available yet, so this path is not open to external users at the moment. nxbase

• nxsut, the reference supply-use database, is generated with exactly this pipeline. Its gen_v3.ipynb generator parses EXIOBASE Hybrid, updates the supply mix from EMBER and the trade mix from ENTSO-E (adding sector-specific detail on top), and is a complete, working end-to-end example. nxsut

Download this notebook