Data Utilities

sindae.data_utils

Containers for multi-trajectory solution data and helpers for extracting and generating training data from solved Pyomo models.

• TrajectoryData holds the per-trajectory arrays (sampling_times, nn_input, nn_output, obs, aux_vars).

• InstanceData is a list-like container of TrajectoryData that also computes the input/output normalization statistics used throughout training.

• extract_instance_data pulls an InstanceData out of any solved model.

• generate_data solves the true model to synthesize (optionally noisy) observations for examples and benchmarks.

Usage¶

import numpy as np
from sindae import generate_data, extract_instance_data

# Synthesize noisy observations from a problem whose true output is known
# (requires the problem to implement add_true_output_constraints):
true_data = generate_data(problem, noise_std=np.array([0.05]), obs_every=2, seed=0)

# After any solve, pull the arrays out of the Pyomo model:
data = extract_instance_data(problem, solved_model)

traj0 = data[0]                    # a TrajectoryData
traj0.sampling_times               # (num_t,)
traj0.nn_input, traj0.nn_output    # (num_t, input_dim) / (num_t, output_dim)

data.input_mean, data.input_std    # normalization stats across all trajectories
data.output_mean, data.output_std

generate_data also populates problem.obs_times / problem.obs_values in place, so the same problem is ready to hand to the smoother and training routines.

API reference¶

TrajectoryData¶

class TrajectoryData(
    sampling_times: np.ndarray,
    nn_input: np.ndarray,
    nn_output: np.ndarray,
    obs: np.ndarray,
    aux_vars: Optional[np.ndarray] = None,
)

Fields

• sampling_times (np.ndarray)

• nn_input (np.ndarray)

• nn_output (np.ndarray)

• obs (np.ndarray)

• aux_vars (Optional[np.ndarray], default None)

InstanceData¶

class InstanceData(trajectories: List[TrajectoryData])

Container for multi-trajectory solution data extracted from a solved Pyomo model.

Per-trajectory data lives in TrajectoryData objects (accessible via indexing). Normalization statistics (input_mean/std, output_mean/std) are computed on the fly from the stored nn_input / nn_output arrays across all trajectories.

Note: obs is stored in original (un-normalised) space. Obs normalisation statistics are available via problem.obs_mean / problem.obs_std.

Parameters

• trajectories (List[TrajectoryData])

Properties

• num_trajectories

• sampling_times

• nn_input

• nn_output

• obs

• input_mean

• input_std

• output_mean

• output_std

Methods

append_trajectory¶

append_trajectory(traj_data: TrajectoryData) -> None

Parameters

• traj_data (TrajectoryData)

save_to_file¶

save_to_file(filename: str) -> None

Parameters

• filename (str)

load_from_file¶

load_from_file(filename: str) -> InstanceData

Parameters

• filename (str)

Returns

• InstanceData

NormStats¶

class NormStats(
    input_mean: np.ndarray,
    input_std: np.ndarray,
    output_mean: np.ndarray,
    output_std: np.ndarray,
)

The four normalization vectors solve_inference consumes.

An :class:InstanceData exposes the same four attributes as properties; NormStats is the lightweight stand-in restored by HybridDAE.load, which persists the scaler but not the full training trajectories. Anywhere an InstanceData is used only for normalization statistics (e.g. make_inference_model), a NormStats is a drop-in replacement.

Fields

• input_mean (np.ndarray)

• input_std (np.ndarray)

• output_mean (np.ndarray)

• output_std (np.ndarray)

extract_instance_data¶

extract_instance_data(problem, model: pyo.ConcreteModel) -> InstanceData

Extract trajectory data from a solved Pyomo model.

Iterates over model.traj_set, calling problem.get_input_vars, get_output_vars, get_obs_vars, and (if non-empty) get_aux_vars at each time point.

Parameters

• problem

• model (pyo.ConcreteModel)

Returns

• (InstanceData)

generate_data¶

generate_data(
    problem: ProblemDefinition,
    obs_every: int = 1,
    seed: int = 0,
    noise_std: Optional[np.ndarray] = None,
    solver_options: Optional[dict] = None,
    nlp_solver: str = 'pounce',
    tee: bool = False,
) -> InstanceData

Solve the true model for all trajectories and populate problem with data.

Builds a Pyomo model using problem.build_trajectory + problem.add_true_output_constraints + problem.discretize, solves with POUNCE, then:

• Sets problem.obs_times (subsampled collocation times)

• Sets problem.obs_values (noisy observations of get_obs_vars)

• Returns the true trajectories as an InstanceData

Parameters

• problem (ProblemDefinition) — Must implement add_true_output_constraints.

• obs_every (int, default 1) — Keep every N-th collocation time point as an observation. 1 = observe at all collocation points (default).

• seed (int, default 0) — RNG seed for reproducible noise.

• noise_std (Optional[np.ndarray], default None) — Std of Gaussian noise added to observations (0 = noiseless).

• solver_options (Optional[dict], default None) — Extra NLP solver options, e.g. {‘tol’: 1e-9}.

• nlp_solver (str, default 'pounce') — NLP solver backend used for the true-model solve.

• tee (bool, default False) — Pass through to the NLP solver (print output if True).

Returns

• (InstanceData) — True trajectories at all collocation points (nn_input, nn_output, obs, and aux_vars per trajectory). Returns None if the POUNCE solve fails or does not reach optimality.