Compute matrices#
This notebook shows how MARIO calculates matrices that are not stored yet in the packaged test tables, first on the IOT fixture and then on the SUT fixture.
[1]:
import mario
mario.set_log_verbosity("info") # you can set the log verbosity as "debug", "info", "warning", "error", or "critical"
[2]:
db = mario.load_test("IOT")
INFO Parser: excel reading IOT flows from /Users/lorenzorinaldi/Documents/GitHub/MARIO/mario/test/tables/test_IOT_standard.xlsx.
INFO Parser: state payload ready with 6 canonical blocks.
INFO Parser: excel state ready for IOT.
INFO Metadata: initialized.
See what is already available#
[3]:
db.matrices["baseline"].keys()
[3]:
dict_keys(['E', 'EY', 'V', 'VY', 'Y', 'Z'])
Calculate selected matrices#
Use calc_all(...) to calculate a list of matrices. MARIO resolves whatever dependencies are needed internally, but by default it materializes only the requested target matrices.
[4]:
db.calc_all(["X", "z", "f"])
db.matrices["baseline"].keys()
INFO Resolver: resolving X for baseline.
INFO Resolver: trying X via formula build_iot_X_from_Z_Y.
INFO Resolver: resolved X via formula build_iot_X_from_Z_Y.
INFO Resolver: resolving z for baseline.
INFO Resolver: trying z via formula build_iot_z_from_Z_X.
INFO Resolver: resolved z via formula build_iot_z_from_Z_X.
INFO Resolver: resolving f for baseline.
INFO Resolver: trying f via formula build_iot_f_from_e_w (compute_method=auto, runtime=inverse).
INFO Resolver: resolved w via formula build_iot_w_from_z (compute_method=auto, runtime=inverse).
INFO Resolver: resolved f via formula build_iot_f_from_e_w (compute_method=auto, runtime=inverse).
[4]:
dict_keys(['E', 'EY', 'V', 'VY', 'Y', 'Z', 'X', 'z', 'f'])
Inspect the calculated results#
[5]:
db.X
[5]:
| production | |||
|---|---|---|---|
| Region | Level | Item | |
| Reg1 | Sector | Agriculture | 9.972169e+06 |
| Industry | 5.311550e+07 | ||
| Services | 1.044952e+08 | ||
| Reg2 | Sector | Agriculture | 7.984966e+04 |
| Industry | 1.195683e+06 | ||
| Services | 2.418662e+06 |
[6]:
db.z
[6]:
| Region | Reg1 | Reg2 | ||||||
|---|---|---|---|---|---|---|---|---|
| Level | Sector | Sector | ||||||
| Item | Agriculture | Industry | Services | Agriculture | Industry | Services | ||
| Region | Level | Item | ||||||
| Reg1 | Sector | Agriculture | 0.093346 | 0.100871 | 0.010290 | 0.029593 | 0.042325 | 0.004276 |
| Industry | 0.112553 | 0.431001 | 0.101928 | 0.017704 | 0.112831 | 0.012555 | ||
| Services | 0.189887 | 0.183327 | 0.292816 | 0.016302 | 0.023262 | 0.024197 | ||
| Reg2 | Sector | Agriculture | 0.000042 | 0.000049 | 0.000008 | 0.037102 | 0.025621 | 0.003292 |
| Industry | 0.000691 | 0.002514 | 0.000627 | 0.121941 | 0.240282 | 0.066618 | ||
| Services | 0.000159 | 0.000293 | 0.000368 | 0.208897 | 0.261236 | 0.298549 | ||
[7]:
db.f
[7]:
| Region | Reg1 | Reg2 | ||||
|---|---|---|---|---|---|---|
| Level | Sector | Sector | ||||
| Item | Agriculture | Industry | Services | Agriculture | Industry | Services |
| Item | ||||||
| Employment | 0.205834 | 0.071561 | 0.041096 | 0.026342 | 0.034874 | 0.018308 |
| CO2 | 508912.512077 | 492572.973155 | 332919.806550 | 392045.986026 | 228374.179654 | 145915.355204 |
Calculate selected matrices in other scenarios#
By default, the matrices will calculated in the “baseline” scenario
[8]:
db.clone_scenario("baseline", "new_scenario")
db
[8]:
name = IOT test (standard)
table = IOT
scenarios = ['baseline', 'new_scenario']
Factor of production = 3
Satellite account = 2
Consumption category = 1
Region = 2
Sector = 3
[9]:
db.query("p","new_scenario")
INFO Resolver: resolving p for new_scenario.
INFO Resolver: trying p via formula build_iot_p_from_v_w (compute_method=auto, runtime=inverse).
INFO Resolver: resolved w via formula build_iot_w_from_z (compute_method=auto, runtime=inverse).
INFO Resolver: resolved p via formula build_iot_p_from_v_w (compute_method=auto, runtime=inverse).
[9]:
| price index | |||
|---|---|---|---|
| Region | Level | Item | |
| Reg1 | Sector | Agriculture | 1.0 |
| Industry | 1.0 | ||
| Services | 1.0 | ||
| Reg2 | Sector | Agriculture | 1.0 |
| Industry | 1.0 | ||
| Services | 1.0 |
[10]:
db.matrices['baseline'].keys()
[10]:
dict_keys(['E', 'EY', 'V', 'VY', 'Y', 'Z', 'X', 'z', 'f'])
[11]:
db.matrices['new_scenario'].keys()
[11]:
dict_keys(['E', 'EY', 'V', 'VY', 'Y', 'Z', 'X', 'z', 'f', 'p'])
Compute selected matrices on a SUT#
SUT workflows use the same calc_all(...) interface, but the resolved matrix names include split and unified activity/commodity blocks.
[12]:
sut = mario.load_test("SUT")
INFO Parser: excel reading SUT flows from /Users/lorenzorinaldi/Documents/GitHub/MARIO/mario/test/tables/test_SUT_standard.xlsx.
INFO Parser: state payload ready with 10 canonical blocks.
INFO Parser: excel state ready for SUT.
INFO Metadata: initialized.
[13]:
sut.matrices["baseline"].keys()
[13]:
dict_keys(['EY', 'Ea', 'Ec', 'S', 'U', 'VY', 'Va', 'Vc', 'Ya', 'Yc'])
[14]:
sut.calc_all(["Xc", "u", "gac"])
sut.matrices["baseline"].keys()
INFO Resolver: resolving Xc for baseline.
INFO Resolver: trying Xc via formula build_sut_Xc_from_U_Yc.
INFO Resolver: resolved Xc via formula build_sut_Xc_from_U_Yc.
INFO Resolver: resolving u for baseline.
INFO Resolver: trying u via extract.
INFO Resolver: trying u via formula build_sut_u_from_U_Xa.
INFO Resolver: resolved u via formula build_sut_u_from_U_Xa.
INFO Resolver: resolving gac for baseline.
INFO Resolver: trying gac via extract.
INFO Resolver: trying gac via formula build_sut_gac_from_gaa_bs.
INFO Resolver: resolved gac via formula build_sut_gac_from_gaa_bs.
[14]:
dict_keys(['EY', 'Ea', 'Ec', 'S', 'U', 'VY', 'Va', 'Vc', 'Ya', 'Yc', 'Xc', 'u', 'gac'])
Inspect the calculated SUT results#
[15]:
sut.Xc
[15]:
| Item | production | ||
|---|---|---|---|
| Region | Level | Item | |
| Region 1 | Commodity | Goods | 1.311430e+12 |
| Services | 2.382765e+12 | ||
| Region 2 | Commodity | Goods | 6.302777e+13 |
| Services | 1.045551e+14 |
[16]:
sut.u
[16]:
| Region | Region 1 | Region 2 | ||||
|---|---|---|---|---|---|---|
| Level | Activity | Activity | ||||
| Item | Manufacturing | Services | Manufacturing | Services | ||
| Region | Level | Item | ||||
| Region 1 | Commodity | Goods | 0.261452 | 0.068540 | 0.002359 | 0.000660 |
| Services | 0.255721 | 0.299919 | 0.000187 | 0.000343 | ||
| Region 2 | Commodity | Goods | 0.150731 | 0.016951 | 0.481652 | 0.110734 |
| Services | 0.020500 | 0.024077 | 0.183058 | 0.294300 | ||
[17]:
sut.gac
[17]:
| Region | Region 1 | Region 2 | ||||
|---|---|---|---|---|---|---|
| Level | Commodity | Commodity | ||||
| Item | Goods | Services | Goods | Services | ||
| Region | Level | Item | ||||
| Region 1 | Activity | Manufacturing | 1.355147 | 0.290279 | 0.337316 | 0.198263 |
| Services | 0.313276 | 1.443459 | 0.098479 | 0.080553 | ||
| Region 2 | Activity | Manufacturing | 0.009015 | 0.004155 | 2.015638 | 0.550454 |
| Services | 0.002194 | 0.001910 | 0.329334 | 1.495274 | ||
Control the calculation runtime#
By default, calc_all(...), resolve(...), resolve_many(...), and dotted access such as db.f use the automatic runtime.
Conceptually, MARIO can compute many derived targets in two different ways:
inverse: build the explicit inverse/intermediate path first, then reuse it downstream.solve: avoid materializing the full inverse and compute the requested target directly through linear solves.auto: the default choice. MARIO decides between the two depending on the requested target, the size of the system, and the estimated memory cost of materializing the inverse.
In the public API this behavior is controlled through compute_options, rather than through a dedicated compute_method= argument. Most users can simply keep the default automatic behavior.
[18]:
iot = mario.load_test("IOT")
# automatic choice (default)
iot.resolve("f")
# prefer the explicit inverse-based path
iot.resolve(
"f",
force_rewrite=True,
compute_options=mario.ComputeOptions(
backend_override="dense_inverse",
),
)
# prefer direct target solves
iot.resolve(
"f",
force_rewrite=True,
compute_options=mario.ComputeOptions(
planning_override="prefer_direct_targets",
),
)
INFO Parser: excel reading IOT flows from /Users/lorenzorinaldi/Documents/GitHub/MARIO/mario/test/tables/test_IOT_standard.xlsx.
INFO Parser: state payload ready with 6 canonical blocks.
INFO Parser: excel state ready for IOT.
INFO Metadata: initialized.
INFO Resolver: resolving f for baseline.
INFO Resolver: trying f via formula build_iot_f_from_e_w (compute_method=auto, runtime=inverse).
INFO Resolver: resolved w via formula build_iot_w_from_z (compute_method=auto, runtime=inverse).
INFO Resolver: resolved f via formula build_iot_f_from_e_w (compute_method=auto, runtime=inverse).
INFO Resolver: resolving f for baseline.
INFO Resolver: trying f via formula build_iot_f_from_e_w (compute_method=inverse, runtime=inverse).
INFO Resolver: resolved w via formula build_iot_w_from_z (compute_method=inverse, runtime=inverse).
INFO Resolver: resolved f via formula build_iot_f_from_e_w (compute_method=inverse, runtime=inverse).
INFO Resolver: resolving f for baseline.
INFO Resolver: trying f via formula build_iot_f_from_e_z (compute_method=solve, runtime=solve).
INFO Resolver: resolved f via formula build_iot_f_from_e_z (compute_method=solve, runtime=solve).
[18]:
| Region | Reg1 | Reg2 | ||||
|---|---|---|---|---|---|---|
| Level | Sector | Sector | ||||
| Item | Agriculture | Industry | Services | Agriculture | Industry | Services |
| Item | ||||||
| Employment | 0.205834 | 0.071561 | 0.041096 | 0.026342 | 0.034874 | 0.018308 |
| CO2 | 508912.512077 | 492572.973155 | 332919.806550 | 392045.986026 | 228374.179654 | 145915.355204 |
The same pattern works for SUT targets such as pc, fc, Xc, wcc, and the other split activity/commodity blocks. If you need lower-level control of the solve backend, advanced options such as backend_override="sparse_direct" and backend_override="sparse_iterative" are also available, but in most workflows auto is the recommended choice.
Compare auto, inverse, and solve with large database#
The example below mirrors the same benchmarking pattern you can use on larger real-world databases. On the small packaged test fixture, absolute timings are not especially meaningful so an example with GLORIA is performed.
[19]:
import time
import numpy as np
mario.set_log_verbosity("critical")
target = "pc" # target matrix to resolve (e.g: pc = price index vector by commodity)
repeats = 3 # number of times to repeat each pipeline for comparing median timing
benchmark_db = mario.parse_gloria(
path = '/Users/lorenzorinaldi/Library/CloudStorage/OneDrive-SharedLibraries-PolitecnicodiMilano/DENG-SESAM - Documenti/c-Research/a-Datasets/_Input Output Databases/GLORIA',
year = 2023,
)
auto_options = mario.ComputeOptions()
inverse_options = mario.ComputeOptions(backend_override="dense_inverse")
solve_options = mario.ComputeOptions(planning_override="prefer_direct_targets")
pipelines = [
("auto", auto_options),
("inverse-only", inverse_options),
("solve-only", solve_options),
]
results = {}
for label, options in pipelines:
times = []
resolved = None
for _ in range(repeats):
candidate = benchmark_db.copy()
t0 = time.perf_counter()
resolved = candidate.resolve(target, compute_options=options)
times.append(time.perf_counter() - t0)
results[label] = {
"times": times,
"median": float(np.median(times)),
"value": resolved,
}
print(label, "times:", times)
print(label, "median:", results[label]["median"])
auto times: [105.73984454094898, 106.24737650004681, 127.93017966602929]
auto median: 106.24737650004681
inverse-only times: [454.232233250048, 432.189996042056, 405.2793996250257]
inverse-only median: 432.189996042056
solve-only times: [99.94249587494414, 95.362556417007, 106.08363883302081]
solve-only median: 99.94249587494414
[20]:
inverse_values = results["inverse-only"]["value"].to_numpy(dtype=float)
auto_values = results["auto"]["value"].to_numpy(dtype=float)
solve_values = results["solve-only"]["value"].to_numpy(dtype=float)
auto_relative_error = np.linalg.norm(inverse_values - auto_values) / np.linalg.norm(inverse_values)
solve_relative_error = np.linalg.norm(inverse_values - solve_values) / np.linalg.norm(inverse_values)
print("relative error auto vs inverse:", auto_relative_error)
print("relative error solve vs inverse:", solve_relative_error)
print("inverse/auto speed ratio:", results["inverse-only"]["median"] / results["auto"]["median"])
print("inverse/solve speed ratio:", results["inverse-only"]["median"] / results["solve-only"]["median"])
relative error auto vs inverse: 1.0673892003248946e-08
relative error solve vs inverse: 1.0673892003248946e-08
inverse/auto speed ratio: 4.067770991426462
inverse/solve speed ratio: 4.324386661134078