Batch Data Analysis #

predict_online(batch, upto_k, *, initial_conditions=None, method='tsr', ridge=0.0)[source]#

Project a partially-complete batch, treating the future as missing data.

During a running batch, the trajectory data for the future (time samples upto_k and later) are not yet known. This method marks those unfolded columns as missing and delegates to the shared missing-data projection (process_improve.multivariate.PCA.project()), which estimates the score vector from the observed columns only. Initial conditions, known from the batch start, are always part of the observed set, so they sharpen the projection from the first sample. The default estimator is trimmed score regression, the method recommended for exactly this batch-so-far problem by Garcia-Munoz, Kourti and MacGregor (2004).

The batch is expected to be aligned to the training length; upto_k selects how many leading time samples are treated as observed. To compare the returned statistics against control limits, use process_improve.batch.BatchMonitor, which builds the time-varying limits from good batches with the same estimator.

Parameters:

batch (pd.DataFrame) – A single aligned batch (n_timesteps rows, the training tags as columns).
upto_k (int) – Number of leading time samples to treat as observed, in 1 .. n_timesteps_. At upto_k == n_timesteps_ every trajectory column is observed and the result matches diagnose() for that batch.
initial_conditions (pd.Series or pd.DataFrame, optional) – The Z block for this batch (required if the model was fitted with one). A Series of the initial-condition values, or a single-row DataFrame.
method ({"tsr", "scp", "pmp"}, default="tsr") – The missing-data score estimator; see process_improve.multivariate.PCA.project().
ridge (float, default=0.0) – Regularisation for the "tsr" / "pmp" estimators.

Returns:

result – With keys scores (Series, one entry per component), hotellings_t2 (float, cumulative over all components), spe (float, the length of the residual over the observed columns), spe_instantaneous (float, the length of the residual over the newest observed sample only), condition_number (float, the estimator’s conditioning diagnostic at this pattern), residuals (Series over feature_columns_, NaN where unobserved) and forecast (DataFrame, n_timesteps_ rows by the training tags, in engineering units: the batch’s own values up to upto_k and the model’s imputation of the remainder, Eq. 4 of Wold et al., 2009).

Return type:

predict_online_trace(batch, *, initial_conditions=None, method='tsr', ridge=0.0)[source]#

Project a batch at every time sample in one vectorized call.

Equivalent to calling predict_online() for upto_k in 1 .. n_timesteps_, but the batch is unfolded and scaled once and all the per-sample patterns are projected together, which is what an online monitor needs (process_improve.batch.BatchMonitor builds its per-sample limits this way).

Parameters:

batch (pd.DataFrame) – A single aligned batch (n_timesteps rows, the training tags as columns).
initial_conditions (pd.Series or pd.DataFrame, optional) – The Z block for this batch; required if the model was fitted with one.
method ({"tsr", "scp", "pmp"}, default="tsr") – The missing-data score estimator.
ridge (float, default=0.0) – Regularisation for the "tsr" / "pmp" estimators.

Returns:

result – With keys time (1-based sample indices), scores (DataFrame, n_timesteps x n_components; row k-1 is the score estimate using samples up to k), hotellings_t2, spe, spe_instantaneous (the residual over the newest observed sample only) and condition_number (np.ndarray of length n_timesteps).

Return type:

unfold_and_scale(X, *, initial_conditions=None)[source]#

Unfold batches batchwise and apply the training centring and scaling.

Parameters:

X (dict[Hashable, pd.DataFrame]) – Standard batch-data dictionary of aligned batches with the same tags and number of samples as the training data.
initial_conditions (pd.DataFrame, optional) – The Z block for the batches; required if (and only if) the model was fitted with one.

Returns:

The one-row-per-batch [Z | X] matrix in the model’s scaled space, indexed by batch identifier, with the 2-level unfolded column index. This is the X argument that score_contributions(), spe_contributions() and t2_contributions() expect; passing the training batches reproduces the fitted scores.

Return type:

pd.DataFrame of shape (n_batches, n_unfolded_features)

hotellings_t2_limit(conf_level=0.95)[source]#

Hotelling’s T2 limit at the given confidence level.

Parameters:: conf_level (float)
Return type:: float

ellipse_coordinates(score_horiz, score_vert, conf_level=0.95, n_points=100)[source]#

Coordinates of the T2 confidence ellipse for a score plot.

Parameters:

score_horiz (int)
score_vert (int)
conf_level (float)
n_points (int)

Return type:

tuple[ndarray, ndarray]

score_plot(pc_horiz=1, pc_vert=2, pc_depth=-1, items_to_highlight=None, settings=None, fig=None, *, sizes=None, size_name='')#

Generate a 2D or 3D score plot for the given latent variable model.

A 2D scatter on (pc_horiz, pc_vert) is produced by default. Supplying pc_depth >= 1 adds a third score axis and switches the underlying trace to Scatter3d.

Parameters:

model (MVmodel object (PCA, or PLS)) – A latent variable model generated by this library.
pc_horiz (int, optional) – Which component to plot on the horizontal axis, by default 1 (the first component)
pc_vert (int, optional) – Which component to plot on the vertical axis, by default 2 (the second component)
pc_depth (int, optional) – If pc_depth >= 1, then a 3D score plot is generated, with this component on the 3rd axis
items_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier; values are lists of index names to highlight. For example:
```
items_to_highlight = {'{"color": "red", "symbol": "cross"}': items_in_red}
```
will highlight the items in items_in_red with the given colour and shape.
sizes (pd.Series, optional) – One non-negative value per observation, indexed as the scores are. The marker area is made proportional to it, so that a marker of twice the area stands for twice the value, and the largest value is drawn settings["size_max"] pixels across. The plain and the highlighted traces share one scale, and a highlighted point keeps its own area rather than being enlarged, because two meanings on one channel cannot both be read. Give the reader that scale as well: an area cannot be read off a plot on its own.
size_name (str, optional) – What sizes measures, for example "SPE"; it names the value in the hover text.

settings (dict) –

Default settings:

{
    "show_ellipse": True,          # bool: show the Hotelling's T2 ellipse
    "ellipse_conf_level": 0.95,    # float: ellipse confidence level (< 1.00)
    "title": "",                   # str: overall plot title. The
                                   # default is the empty string on
                                   # the 2D path (pc_depth <= 0) and
                                   # a "Score plot of component ..."
                                   # sentence on the 3D path
                                   # (pc_depth > 0).
    "show_labels": False,          # bool: add a label for each observation
    "show_legend": True,           # bool: show clickable legend
    "size_max": 26,                # float: diameter in pixels of the
                                   # largest marker, when `sizes` is given
    "html_image_height": 500,      # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",        # str: registered Plotly theme name
}

fig (Figure | None)

Return type:

Figure

Examples

>>> pca = PCA(n_components=3).fit(X_scaled)
>>> pca.score_plot()                          # PC1 vs PC2
>>> pca.score_plot(pc_horiz=1, pc_vert=3)     # PC1 vs PC3
>>> pca.score_plot(pc_horiz=1, pc_vert=2, pc_depth=3)  # 3D

spe_plot(with_a=-1, items_to_highlight=None, settings=None, fig=None)#

Generate a squared-prediction error (SPE) plot for the given latent variable model using with_a number of latent variables. The default will use the total number of latent variables which have already been fitted.

Parameters:

model (MVmodel object (PCA, or PLS)) – A latent variable model generated by this library.
with_a (int, optional) – Uses this many number of latent variables, and therefore shows the SPE after this number of model components. By default the total number of components fitted will be used.
items_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier; values are lists of index names to highlight. For example:
```
items_to_highlight = {'{"color": "red", "symbol": "cross"}': items_in_red}
```
will highlight the items in items_in_red with the given colour and shape.

settings (dict) –

Default settings:

{
    "show_limit": True,            # bool: show the SPE confidence limit line
    "conf_level": 0.95,            # float: confidence level for limit (< 1.00)
    "title": "SPE plot ...",        # str: overall plot title
    "default_marker": {...},        # dict: e.g. dict(symbol="circle", size=7)
    "show_labels": False,           # bool: add a label for each observation
    "show_legend": False,           # bool: show clickable legend
    "html_image_height": 500,       # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",         # str: registered Plotly theme name
}

fig (Figure | None)

Return type:

Figure

Examples

>>> pca.spe_plot()
>>> pca.spe_plot(settings={"conf_level": 0.99, "show_labels": True})

t2_plot(with_a=-1, items_to_highlight=None, settings=None, fig=None)#

Generate a Hotelling’s T2 (T^2) plot for the given latent variable model using with_a number of latent variables. The default will use the total number of latent variables which have already been fitted.

Parameters:

model (MVmodel object (PCA, or PLS)) – A latent variable model generated by this library.
with_a (int, optional) – Uses this many number of latent variables, and therefore shows the Hotelling’s T2 after this number of model components. By default the total number of components fitted will be used.
items_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier; values are lists of index names to highlight. For example:
```
items_to_highlight = {'{"color": "red", "symbol": "cross"}': items_in_red}
```
will highlight the items in items_in_red with the given colour and shape.

settings (dict) –

Default settings. The default title interpolates the class-level conf_level default (0.95), so a user-supplied conf_level correctly changes the limit line but the auto-generated title text still reads 95.0% unless a title override is passed too:

{
    "show_limit": True,            # bool: show the T2 confidence limit line
    "conf_level": 0.95,            # float: confidence level for limit (< 1.00)
    "title": "T2 plot ...",         # str: overall plot title
    "default_marker": {...},        # dict: e.g. dict(symbol="circle", size=7)
    "show_labels": False,           # bool: add a label for each observation
    "show_legend": False,           # bool: show clickable legend
    "html_image_height": 500,       # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",         # str: registered Plotly theme name
}

fig (Figure | None)

Return type:

Figure

Examples

>>> pca.t2_plot()
>>> pca.t2_plot(settings={"conf_level": 0.99, "show_labels": True})

loading_plot(loadings_type='p', pc_horiz=1, pc_vert=2, settings=None, fig=None)#

Generate a 2-dimensional loadings for the given latent variable model.

Parameters:

model (MVmodel object (PCA, or PLS)) – A latent variable model generated by this library.
loadings_type (str, optional) –

A choice of the following:
’p’ : (default for PCA) : the P (projection) loadings: only option possible for PCA ‘w’ : the W loadings: Suitable for PLS ‘w*’ : (default for PLS) the W* (or R) loadings: Suitable for PLS ‘w*c’ : the W* (from X-space) with C loadings from the Y-space: Suitable for PLS ‘c’ : the C loadings from the Y-space: Suitable for PLS

For PCA model any other choice besides ‘p’ will be ignored.
pc_horiz (int, optional) – Which component to plot on the horizontal axis, by default 1 (the first component)
pc_vert (int, optional) – Which component to plot on the vertical axis, by default 2 (the second component)

settings (dict) –

Default settings:

{
    "title": "Loadings plot ...",  # str: overall plot title
    "show_labels": True,           # bool: add a label for each variable
    "html_image_height": 500,      # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",        # str: registered Plotly theme name
}

fig (Figure | None)

Return type:

Figure

Examples

>>> pca.loading_plot()                                 # P loadings, PC1 vs PC2
>>> pls.loading_plot(loadings_type="w*c")              # W* and C loadings
>>> pls.loading_plot(loadings_type="w", pc_vert=3)     # W loadings, PC1 vs PC3

explained_variance_plot(settings=None, fig=None)#

Generate an explained-variance plot for a fitted latent variable model.

Shows the variance explained by each component as bars, with the cumulative variance explained overlaid as a line. For PCA the variance refers to the X-block; for PLS it refers to the Y-block.

Parameters:

model (MVmodel object (PCA, or PLS)) – A fitted latent variable model generated by this library.

settings (dict) –

Default settings:

{
    "as_percentage": True,         # bool: y-axis as a percentage, else a fraction
    "title": "Variance explained ...",   # str: overall plot title
    "bar_color": None,              # str|None: bar colour; None uses the theme
    "line_color": None,             # str|None: line colour; None uses the theme
    "show_legend": True,            # bool: show clickable legend
    "html_image_height": 500,       # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",         # str: registered Plotly theme name
}

fig (go.Figure, optional) – An existing figure to draw onto. A new figure is created if omitted.

Return type:

Figure

Examples

>>> pca.explained_variance_plot()
>>> pls.explained_variance_plot(settings={"as_percentage": False})

spe_limit(conf_level=0.95)#

Return the squared prediction error limit at the given level of confidence.

Parameters:

model (BaseEstimator) – A fitted multivariate model exposing a spe_ attribute and an n_components_ attribute (e.g. a fitted PCA or PLS instance). The fitted n_components_ (with the trailing underscore) is used, not the constructor’s n_components parameter, so a model whose component count was clamped at min(n_samples, n_features) at fit time is read with the value it actually holds.
conf_level (float, optional) – Fractional confidence limit, less that 1.00; by default 0.95

Returns:

The squared prediction error limit at the given level of confidence.

Return type:

float

score_limit(conf_level=0.95)#

Return two-sided confidence limits for each score component.

The scores of component a have mean zero, and their standard deviation is estimated from the same N observations, so the symmetric limit at the requested confidence level is t_{N - 1} * std(score_a) with t_{N - 1} the Student-t quantile on N - 1 degrees of freedom. A score outside [-limit, +limit] is unusual at that confidence level.

The Student-t quantile is used rather than the standard-normal z because s_a is an estimate: z is only its large-N limit and is too narrow otherwise (1.96 against 2.26 at N = 10, a limit 15 percent too tight; the gap falls below 3 percent by N = 50).

Parameters:

model (BaseEstimator) – A fitted PCA or PLS model exposing a scores_ attribute.
conf_level (float, optional) – Fractional confidence level in (0, 1); by default 0.95.

Returns:

Array of length n_components with the positive score limit for each component.

Return type:

np.ndarray

References

Score limits: the limit is t_{(1 + conf_level) / 2, N - 1} * s_a. Equivalently (t_a / s_a) ** 2 follows an F(1, N - 1) distribution, since sqrt(F(1, N - 1)) is exactly the two-sided t_{N - 1} quantile.

t2_contributions(X, components=None, *, method='scp', ridge=0.0)#

Per-variable contributions to Hotelling’s $T^2$.

Works with fitted PCA and PLS models. Decomposes each observation’s $T^2$ onto the original variables. The contribution of variable $k$ for observation $i$ is

\[c^{T^2}_{ik} = x_{ik} \sum_{a} \frac{t_{ia}}{s_a^2}\, R_{ka},\]

where $t_{ia}$ are the scores, $s_a^2$ is the score variance of component $a$ (scaling_factor_for_scores_ squared) and $R$ is the score-generating matrix (loadings for PCA, direct_weights_ for PLS, so that $T = XR$). Summed over the variables this telescopes to $\sum_a t_{ia}^2 / s_a^2$, i.e. the observation’s $T^2$. The values are signed; a large magnitude flags a variable that drives the observation away from the model centre. This is the standard MSPC diagnostic (Westerhuis, Gurden and Smilde, 2000).

A row with missing cells (NaN) gets its scores from its observed cells alone, with the estimator named by method, the missing-data operators of PCA.project() and PLS.project(). Its contributions are then defined at every observed cell and NaN at the missing ones, so the row sums (sum(axis=1), which skips NaN) keep their meaning. The default "scp" is the single-component projection the NIPALS fit uses, so on a PCA model fitted with missing data the training rows reproduce the stored scores_, hotellings_t2_ and spe_ to within the NIPALS convergence tolerance, as the complete rows do.

Parameters:

model (PCA or PLS) – A fitted PCA or PLS model.
X (array-like of shape (n_samples, n_features)) – Preprocessed data, scaled the same way as the training data (for example with MCUVScaler). Passing the training data reproduces the model’s stored hotellings_t2_.
components (list of int, optional) – 1-based component indices to decompose over, matching the model’s column convention. None (default) uses all fitted components, so the row sums equal the cumulative $T^2$.
method ({"scp", "tsr", "pmp"}, default="scp") – Score estimator for the rows with missing cells; ignored when X is complete. See PCA.project().
ridge (float, default=0.0) – Regularisation for the "tsr" and "pmp" estimators, as in PCA.project().

Returns:

Signed contributions of shape (n_samples, n_features), NaN at the missing cells. Each row sums to the observation’s $T^2$ over the selected components.

Return type:

pd.DataFrame

Examples

>>> pca = PCA(n_components=3).fit(X_scaled)
>>> contrib = pca.t2_contributions(X_scaled)
>>> contrib.sum(axis=1)  # equals pca.hotellings_t2_.iloc[:, -1]

See also

spe_contributions: The residual-space counterpart.
PCA.score_contributions: Decomposes a single score-space movement.

spe_contributions(X, *, method='scp', ridge=0.0)#

Per-variable squared-prediction-error (SPE / DModX) contributions.

Works with fitted PCA and PLS models. Returns the signed residual of each variable after reconstructing the X-block from the full model:

\[e_{ik} = x_{ik} - \hat{x}_{ik}, \qquad \hat{X} = T P^\top,\]

where $P$ is the reconstruction loadings (loadings_ for PCA, x_loadings_ for PLS). The squared residuals sum across variables to the observation’s SPE; equivalently (spe_contributions(X) ** 2).sum(axis=1) equals the stored spe_ (final column) squared. The signs show whether a variable sits above or below its reconstruction, which is the standard SPE contribution plot used to diagnose why an observation has a high residual.

Parameters:

model (PCA or PLS) – A fitted PCA or PLS model.
X (array-like of shape (n_samples, n_features)) – Preprocessed data, scaled the same way as the training data. Passing the training data reproduces the model’s stored spe_.
method ({"scp", "tsr", "pmp"}, default="scp") – Score estimator for the rows with missing cells; ignored when X is complete. See PCA.project().
ridge (float, default=0.0) – Regularisation for the "tsr" and "pmp" estimators, as in PCA.project().

Returns:

Signed per-variable residuals of shape (n_samples, n_features), NaN at the missing cells. The squared row sums equal the observation’s SPE squared, over its observed cells.

Return type:

pd.DataFrame

Examples

>>> pca = PCA(n_components=2).fit(X_scaled)
>>> resid = pca.spe_contributions(X_scaled)
>>> (resid ** 2).sum(axis=1)  # equals pca.spe_.iloc[:, -1] ** 2

See also

t2_contributions: The $T^2$ (score-space) counterpart.

score_contributions(X, component=1, scaling='none', *, method='scp', ridge=0.0, **deprecated)#

Per-variable contributions to a single score, $t_a$.

Works with fitted PCA and PLS models. A score is a weighted sum of the (preprocessed) variables, so it splits exactly into one term per variable. The contribution of variable $k$ to the score of observation $i$ on component $a$ is

\[c_{ik}^{(a)} = x_{ik}\, R_{ka}, \qquad \sum_{k=1}^{K} c_{ik}^{(a)} = t_{ia},\]

where $R$ is the score-generating matrix (loadings_ for PCA, direct_weights_ for PLS, so that $T = XR$). This is the contribution of Miller, Swanson and Heckler (1994); the generalisation from PCA loadings to any latent-variable model’s score-generating weights follows Westerhuis, Gurden and Smilde (2000).

The distinction from a loading plot is the point of the diagnostic. A loading $R_{ka}$ describes the whole data set; a contribution $x_{ik} R_{ka}$ describes one observation, and a variable with a large loading contributes nothing when that observation sits at its mean. Ranking variables by loading can therefore point at a different cause than ranking them by contribution.

Parameters:

model (PCA or PLS) – A fitted PCA or PLS model.
X (array-like of shape (n_samples, n_features)) – Preprocessed data, scaled the same way as the training data (for example with MCUVScaler). Passing the training data reproduces the model’s stored scores_.
component (int, default=1) – 1-based component index whose score is decomposed, matching the model’s column convention.
scaling ({"none", "maximum", "within"}, default="none") – Presentation scaling from Miller, Swanson and Heckler (1994). "none" returns the raw contributions, which sum to the score. "maximum" divides by the largest absolute contribution anywhere in X, so a bar of $\pm 1$ marks the most extreme variable-observation pair in the data set. "within" divides each row by the sum of its absolute contributions, so each row is on a common footing. Both scalings leave the pattern of bars within a row unchanged; neither preserves the sum to the score.
method ({"scp", "tsr", "pmp"}, default="scp") – Score estimator for the rows with missing cells; ignored when X is complete. See PCA.project().
ridge (float, default=0.0) – Regularisation for the "tsr" and "pmp" estimators, as in PCA.project().
**deprecated – Rejected. Captures t_end, components and weighted so that a call passing a score vector rather than X raises a TypeError explaining the correct usage. Passing a 1-D X raises the same error.

Returns:

Signed contributions of shape (n_samples, n_features), NaN at the missing cells. With the default scaling="none", each row sums to that observation’s score on the selected component.

Return type:

pd.DataFrame

Examples

>>> pca = PCA(n_components=2).fit(X_scaled)
>>> contrib = pca.score_contributions(X_scaled, component=1)
>>> contrib.sum(axis=1)  # equals pca.scores_[1]
>>> contrib.loc["33"].abs().sort_values()  # what makes observation 33 extreme

References

Miller, P., Swanson, R.E. and Heckler, C.E. (1994). “Contribution plots: a missing link in multivariate quality control.” Applied Mathematics and Computer Science, 8(4), 775-792.

Westerhuis, J.A., Gurden, S.P. and Smilde, A.K. (2000). “Generalized contribution plots in multivariate statistical process monitoring.” Chemometrics and Intelligent Laboratory Systems, 51(1), 95-114.

See also

group_contributions: The same decomposition for a group of observations, or for the difference between two groups.
t2_contributions: Decomposes Hotelling’s $T^2$, which pools all components rather than reading one at a time.
spe_contributions: The residual-space counterpart.

set_fit_request(*, initial_conditions='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

initial_conditions (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for initial_conditions parameter in fit.
self (BatchPCA)

Returns:

self – The updated object.

Return type:

set_transform_request(*, initial_conditions='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the transform method.

The options for each parameter are:

True: metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to transform.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

initial_conditions (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for initial_conditions parameter in transform.
self (BatchPCA)

Returns:

self – The updated object.

Return type:

class process_improve.batch.BatchPLS(n_components, *, scale=True, group_by_batch=False)[source]#

Bases: RegressorMixin, BaseEstimator

Batchwise-unfolded PLS from batch trajectories (and Z) to final quality Y.

Unfolds an aligned batch-data dictionary batchwise (one row per batch), optionally joins an initial-conditions (Z) block onto that row, centres and scales every column with its own MCUVScaler, and fits a process_improve.multivariate.PLS model against a batch-indexed quality block Y. The fitted model relates the initial conditions and the time-varying trajectory deviations to the final quality, and predicts the quality of a completed batch.

The batches must be aligned before fitting (every batch the same number of samples; see process_improve.batch.resample_to_reference() and process_improve.batch.batch_dtw()) with no missing values.

Parameters:

n_components (int) – Number of PLS components.
scale (pd.Series of length n_unfolded_features) – Scale each unfolded [Z | X] column to unit variance after centring. Centring always happens (it removes the average trajectory); set this to False to keep the columns in their centred, unscaled units. The quality block Y is always centred and scaled to unit variance internally, and every reported prediction is mapped back to the original quality units.
group_by_batch (bool, default=False) – Ordering of the unfolded column index, passed to process_improve.batch.dict_to_wide().
fitting) (Attributes (after)
--------------------------
x_weights (pd.DataFrame of shape (n_unfolded_features, n_components)) – X-block weights (w), indexed by the 2-level unfolded column index so the trajectory part reshapes to a (tag, time) grid.
loadings (pd.DataFrame) – Alias of x_weights_, so process_improve.batch.time_varying_loading_plot() can plot the time-varying weights.
x_loadings (pd.DataFrame of shape (n_unfolded_features, n_components)) – X-block loadings (p), on the same 2-level index.
direct_weights (pd.DataFrame of shape (n_unfolded_features, n_components)) – Direct weights R = W (P'W)^{-1}, so scores are T = X_scaled R.
y_loadings (pd.DataFrame of shape (n_targets, n_components)) – Y-block loadings (c), mapping scores to the scaled quality space.
explained_variance (np.ndarray of shape (n_components,)) – Variance of each training score.
beta_coefficients (pd.DataFrame) – Regression coefficients from the unfolded X to Y, in the original (engineering) units of both blocks.
r2_cumulative (pd.Series) – Cumulative R2 of the quality block after each component.
rmse (pd.DataFrame of shape (n_targets, n_components)) – Root-mean-square error of the fit, on the original quality units.
predictions (pd.DataFrame of shape (n_batches, n_targets)) – Fitted quality of every training batch, on the original quality units, indexed by batch identifier.
r2_per_variable (pd.DataFrame of shape (n_unfolded_features, n_components)) – Cumulative R2 of each unfolded [Z | X] column after each component, on the 2-level unfolded index. With scale=True every column has unit variance, so the column mean is the R2 of the whole block; with scale=False it is an unweighted average.
scores (pd.DataFrame) – Batch-level scores and diagnostics; one row per batch.
spe (pd.DataFrame) – Batch-level scores and diagnostics; one row per batch.
hotellings_t2 (pd.DataFrame) – Batch-level scores and diagnostics; one row per batch.
center (pd.Series of length n_unfolded_features) – The per-column centring and scaling of the [Z | X] block.
scale – The per-column centring and scaling of the [Z | X] block.
y_center (pd.Series of length n_targets) – The centring and scaling of the quality block.
y_scale (pd.Series of length n_targets) – The centring and scaling of the quality block.
n_batches (int) – Problem dimensions.
n_tags (int) – Problem dimensions.
n_timesteps (int) – Problem dimensions.
n_initial_conditions (int) – Problem dimensions.
batch_ids (list) – Labels for the batches, tags, initial conditions, targets, and time samples.
tag_names (list) – Labels for the batches, tags, initial conditions, targets, and time samples.
initial_condition_names (list) – Labels for the batches, tags, initial conditions, targets, and time samples.
target_names (list) – Labels for the batches, tags, initial conditions, targets, and time samples.
time_index (list) – Labels for the batches, tags, initial conditions, targets, and time samples.

Examples

>>> from process_improve.batch import BatchPLS, load_dryer, resample_to_reference
>>> import pandas as pd
>>> batches = load_dryer()
>>> tags = [c for c in next(iter(batches.values())).columns if c != "ClockTime"]
>>> trimmed = {k: v[tags] for k, v in batches.items()}
>>> aligned = resample_to_reference(trimmed, columns_to_align=tags, reference_batch=1)
>>> quality = pd.DataFrame({"final": [float(b["DryerTemp"].iloc[-1]) for b in aligned.values()]},
...                        index=list(aligned.keys()))
>>> model = BatchPLS(n_components=2).fit(aligned, quality)
>>> model.predict(aligned).y_hat.shape
(71, 1)

See also

process_improve.batch.BatchPCA: the unsupervised (monitoring) counterpart.
process_improve.multivariate.PLS: the underlying estimator.

fit(X, Y, *, initial_conditions=None)[source]#

Fit the batchwise-unfolded PLS model against the quality block Y.

Parameters:

X (dict[Hashable, pd.DataFrame]) – Standard batch-data dictionary of aligned batches.
Y (pd.DataFrame) – Final-quality block: one row per batch (indexed by the same batch identifiers as X), one column per quality variable.
initial_conditions (pd.DataFrame, optional) – The Z block: one row per batch, joined onto the unfolded row.

Returns:

self

Return type:

BatchPLS

predict(X, *, initial_conditions=None)[source]#

Predict the final quality of completed batches.

Parameters:

X (dict[Hashable, pd.DataFrame]) – Standard batch-data dictionary of aligned batches with the same tags and length as the training data.
initial_conditions (pd.DataFrame, optional) – The Z block for the new batches; required if the model was fitted with one.

Returns:

result – With keys y_hat (predicted quality in the original units, one row per batch), scores, hotellings_t2 and spe (batch diagnostics).

Return type:

prediction_interval(X, *, conf_level=0.95, initial_conditions=None)[source]#

Prediction interval for the final quality of completed batches.

Forwards to process_improve.multivariate.PLS.prediction_interval() on the scaled row and maps the bounds back to the original quality units.

Parameters:

X (dict[Hashable, pd.DataFrame]) – Standard batch-data dictionary of aligned batches.
conf_level (float, default=0.95) – Confidence level for the interval, in (0.5, 1.0).
initial_conditions (pd.DataFrame, optional) – The Z block for the new batches; required if the model was fitted with one.

Returns:

result – With keys y_hat, lower and upper (DataFrames on the original quality units) and conf_level.

Return type:

transform(X, *, initial_conditions=None)[source]#

Return the batch-level PLS scores for X (in the input batch order).

Parameters:

X (dict[Hashable, pd.DataFrame])
initial_conditions (pd.DataFrame | None)

Return type:

pd.DataFrame

predict_online(batch, upto_k, *, initial_conditions=None, method='tsr', ridge=0.0)[source]#

Predict the final quality of a running batch from its data so far.

The unfolded row of a batch that has run for upto_k samples is complete up to that sample and missing after it. This method scales the observed part, estimates the batch’s scores from those cells alone with the shared missing-data projection (process_improve.multivariate.PLS.project()), and maps the scores to a quality prediction through the model’s Y loadings. It is the “batch so far” primitive of Garcia-Munoz, Kourti and MacGregor (2004), and Eqs. 2 and 5 of Wold, Kettaneh-Wold, MacGregor and Dunn (2009) when method="pmp". The default estimator is trimmed score regression.

The batch may be truncated to the samples observed so far (at least upto_k rows) or be a complete aligned batch; only its first upto_k rows are used, the tags are matched by name, and a NaN cell among them counts as one more missing cell. The first few samples constrain the scores weakly, so read the early predictions together with condition_number (and consider ridge); the estimate settles as samples accumulate. To compare the returned statistics against control limits use process_improve.batch.BatchMonitor, which builds per-sample limits from reference batches with the same estimator; the hotellings_t2 returned here uses the end-of-batch score scaling and is not a per-sample statistic.

Parameters:

batch (pd.DataFrame) – The batch’s trajectories (the training tags as columns), at least upto_k rows.
upto_k (int) – Number of leading time samples to treat as observed, in 1 .. n_timesteps_. At upto_k == n_timesteps_ the row is complete and y_hat equals predict() for that batch.
initial_conditions (pd.Series or pd.DataFrame, optional) – The Z block for this batch (required if the model was fitted with one).
method ({"tsr", "scp", "pmp"}, default="tsr") – The missing-data score estimator; see process_improve.multivariate.PLS.project().
ridge (float, default=0.0) – Regularisation for the "tsr" / "pmp" estimators.

Returns:

result – With keys scores (Series, one entry per component), y_hat (Series in the original quality units, one entry per target), hotellings_t2 (float, end-of-batch scaling), spe (float, the length of the residual over the observed cells), spe_instantaneous (float, the length of the residual over the newest observed sample only), condition_number (float), residuals (Series over feature_columns_, NaN where unobserved) and forecast (DataFrame, n_timesteps_ rows by the training tags, in engineering units: the batch’s own values up to upto_k and the model’s imputation of the remainder, Eq. 4 of Wold et al., 2009).

Return type:

predict_online_trace(batch, *, initial_conditions=None, method='tsr', ridge=0.0)[source]#

Predict the final quality at every time sample of a complete batch, in one call.

Equivalent to predict_online() for upto_k in 1 .. n_timesteps_: the batch is unfolded and scaled once and all the per-sample missingness patterns are projected together. This is the evolving prediction of a batch as it would have looked in real time, and what process_improve.batch.BatchMonitor uses to build its per-sample limits.

Parameters:

batch (pd.DataFrame) – A single complete batch, aligned to the training length, the training tags as columns.
initial_conditions (pd.Series or pd.DataFrame, optional) – The Z block for this batch; required if the model was fitted with one.
method ({"tsr", "scp", "pmp"}, default="tsr") – The missing-data score estimator.
ridge (float, default=0.0) – Regularisation for the "tsr" / "pmp" estimators.

Returns:

result – With keys time (1-based number of samples observed), scores (DataFrame, n_timesteps x n_components; row k - 1 uses samples up to k), y_hat (DataFrame, n_timesteps x n_targets, original quality units, index named upto_k), hotellings_t2, spe, spe_instantaneous and condition_number (arrays of length n_timesteps). For a training batch the last row of y_hat equals its entry in predictions_.

Return type:

online_rmse(X, Y, *, initial_conditions=None, method='tsr', ridge=0.0)[source]#

Root-mean-square error of the evolving quality prediction, per sample and target.

Each batch in X is traced with predict_online_trace() and its prediction after k samples compared with its measured quality in Y; the errors are squared, averaged over the batches and rooted, giving one curve per target over the batch. On the training batches this is the estimation error (RMSEE) as a function of how much of the batch has been observed. On batches the model was not fitted on (for example one held-out batch at a time) it is the prediction error (RMSEP).

Parameters:

X (dict[Hashable, pd.DataFrame]) – Standard batch-data dictionary of complete, aligned batches.
Y (pd.DataFrame) – Measured final quality, one row per batch in X (indexed by batch identifier), the training targets as columns.
initial_conditions (pd.DataFrame, optional) – The Z block for the batches; required if the model was fitted with one.
method ({"tsr", "scp", "pmp"}, default="tsr") – The missing-data score estimator.
ridge (float, default=0.0) – Regularisation for the "tsr" / "pmp" estimators.

Returns:

Indexed by upto_k (1-based number of samples observed), in the original quality units.

Return type:

pd.DataFrame of shape (n_timesteps, n_targets)

projection_matrix(observed, *, method='tsr', ridge=0.0)[source]#

Build the fixed operator mapping observed unfolded columns to score estimates.

Forwards to process_improve.multivariate.PLS.projection_matrix() on the inner model. The operator acts on the scaled space of the unfolded [Z | X] row; use the public center_ and scale_ attributes to move engineering-unit values into that space. This is the primitive the mid-course corrector precomputes once per decision point: for a fixed pattern of observed columns, the score estimate is an affine function of any subset of those columns.

Parameters:

observed (array-like) – Boolean mask over feature_columns_ (True = observed) or a list of unfolded column labels, e.g. [("temperature", 4), ...].
method ({"tsr", "scp", "pmp"}, default="tsr")
ridge (float, default=0.0)

Returns:

result – With keys matrix (DataFrame, n_components x n_observed), condition_number (float) and method.

Return type:

unfold_and_scale(X, *, initial_conditions=None)[source]#

Unfold batches batchwise and apply the training centring and scaling.

Parameters:

X (dict[Hashable, pd.DataFrame]) – Standard batch-data dictionary of aligned batches with the same tags and length as the training data.
initial_conditions (pd.DataFrame, optional) – The Z block for the batches; required if the model was fitted with one.

Returns:

Return type:

pd.DataFrame of shape (n_batches, n_unfolded_features)

hotellings_t2_limit(conf_level=0.95)[source]#

Hotelling’s T2 limit at the given confidence level.

Parameters:: conf_level (float)
Return type:: float

ellipse_coordinates(score_horiz, score_vert, conf_level=0.95, n_points=100)[source]#

Coordinates of the T2 confidence ellipse for a score plot.

Parameters:

score_horiz (int)
score_vert (int)
conf_level (float)
n_points (int)

Return type:

tuple[ndarray, ndarray]

predictions_vs_observed_plot(y_observed, variable=None, settings=None, fig=None)[source]#

Observed-versus-predicted (parity) plot of the training batches.

Both axes are on the original quality units: the fitted values come from predictions_ and the observed values from y_observed, matched by batch identifier.

Parameters:

y_observed (pd.DataFrame) – The quality block passed to fit(), indexed by batch identifier. Rows are aligned to predictions_ by label, so the row order does not matter.
variable (str, optional) – Which quality variable to plot. Defaults to the first one.
settings (dict, optional) – Plot settings, as for process_improve.multivariate.plots.predictions_vs_observed_plot().
fig (go.Figure, optional) – An existing figure to draw onto.

Return type:

go.Figure

score_plot(pc_horiz=1, pc_vert=2, pc_depth=-1, items_to_highlight=None, settings=None, fig=None, *, sizes=None, size_name='')#

Generate a 2D or 3D score plot for the given latent variable model.

A 2D scatter on (pc_horiz, pc_vert) is produced by default. Supplying pc_depth >= 1 adds a third score axis and switches the underlying trace to Scatter3d.

Parameters:

model (MVmodel object (PCA, or PLS)) – A latent variable model generated by this library.
pc_horiz (int, optional) – Which component to plot on the horizontal axis, by default 1 (the first component)
pc_vert (int, optional) – Which component to plot on the vertical axis, by default 2 (the second component)
pc_depth (int, optional) – If pc_depth >= 1, then a 3D score plot is generated, with this component on the 3rd axis
items_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier; values are lists of index names to highlight. For example:
```
items_to_highlight = {'{"color": "red", "symbol": "cross"}': items_in_red}
```
will highlight the items in items_in_red with the given colour and shape.
sizes (pd.Series, optional) – One non-negative value per observation, indexed as the scores are. The marker area is made proportional to it, so that a marker of twice the area stands for twice the value, and the largest value is drawn settings["size_max"] pixels across. The plain and the highlighted traces share one scale, and a highlighted point keeps its own area rather than being enlarged, because two meanings on one channel cannot both be read. Give the reader that scale as well: an area cannot be read off a plot on its own.
size_name (str, optional) – What sizes measures, for example "SPE"; it names the value in the hover text.

settings (dict) –

Default settings:

{
    "show_ellipse": True,          # bool: show the Hotelling's T2 ellipse
    "ellipse_conf_level": 0.95,    # float: ellipse confidence level (< 1.00)
    "title": "",                   # str: overall plot title. The
                                   # default is the empty string on
                                   # the 2D path (pc_depth <= 0) and
                                   # a "Score plot of component ..."
                                   # sentence on the 3D path
                                   # (pc_depth > 0).
    "show_labels": False,          # bool: add a label for each observation
    "show_legend": True,           # bool: show clickable legend
    "size_max": 26,                # float: diameter in pixels of the
                                   # largest marker, when `sizes` is given
    "html_image_height": 500,      # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",        # str: registered Plotly theme name
}

fig (Figure | None)

Return type:

Figure

Examples

>>> pca = PCA(n_components=3).fit(X_scaled)
>>> pca.score_plot()                          # PC1 vs PC2
>>> pca.score_plot(pc_horiz=1, pc_vert=3)     # PC1 vs PC3
>>> pca.score_plot(pc_horiz=1, pc_vert=2, pc_depth=3)  # 3D

spe_plot(with_a=-1, items_to_highlight=None, settings=None, fig=None)#

Parameters:

model (MVmodel object (PCA, or PLS)) – A latent variable model generated by this library.
with_a (int, optional) – Uses this many number of latent variables, and therefore shows the SPE after this number of model components. By default the total number of components fitted will be used.
items_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier; values are lists of index names to highlight. For example:
```
items_to_highlight = {'{"color": "red", "symbol": "cross"}': items_in_red}
```
will highlight the items in items_in_red with the given colour and shape.

settings (dict) –

Default settings:

{
    "show_limit": True,            # bool: show the SPE confidence limit line
    "conf_level": 0.95,            # float: confidence level for limit (< 1.00)
    "title": "SPE plot ...",        # str: overall plot title
    "default_marker": {...},        # dict: e.g. dict(symbol="circle", size=7)
    "show_labels": False,           # bool: add a label for each observation
    "show_legend": False,           # bool: show clickable legend
    "html_image_height": 500,       # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",         # str: registered Plotly theme name
}

fig (Figure | None)

Return type:

Figure

Examples

>>> pca.spe_plot()
>>> pca.spe_plot(settings={"conf_level": 0.99, "show_labels": True})

t2_plot(with_a=-1, items_to_highlight=None, settings=None, fig=None)#

Parameters:

model (MVmodel object (PCA, or PLS)) – A latent variable model generated by this library.
with_a (int, optional) – Uses this many number of latent variables, and therefore shows the Hotelling’s T2 after this number of model components. By default the total number of components fitted will be used.
items_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier; values are lists of index names to highlight. For example:
```
items_to_highlight = {'{"color": "red", "symbol": "cross"}': items_in_red}
```
will highlight the items in items_in_red with the given colour and shape.

settings (dict) –

{
    "show_limit": True,            # bool: show the T2 confidence limit line
    "conf_level": 0.95,            # float: confidence level for limit (< 1.00)
    "title": "T2 plot ...",         # str: overall plot title
    "default_marker": {...},        # dict: e.g. dict(symbol="circle", size=7)
    "show_labels": False,           # bool: add a label for each observation
    "show_legend": False,           # bool: show clickable legend
    "html_image_height": 500,       # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,  # float: width as ratio of height
    "template": "pi_journal",         # str: registered Plotly theme name
}

fig (Figure | None)

Return type:

Figure

Examples

>>> pca.t2_plot()
>>> pca.t2_plot(settings={"conf_level": 0.99, "show_labels": True})

spe_limit(conf_level=0.95)#

Return the squared prediction error limit at the given level of confidence.

Parameters:

model (BaseEstimator) – A fitted multivariate model exposing a spe_ attribute and an n_components_ attribute (e.g. a fitted PCA or PLS instance). The fitted n_components_ (with the trailing underscore) is used, not the constructor’s n_components parameter, so a model whose component count was clamped at min(n_samples, n_features) at fit time is read with the value it actually holds.
conf_level (float, optional) – Fractional confidence limit, less that 1.00; by default 0.95

Returns:

The squared prediction error limit at the given level of confidence.

Return type:

float

score_limit(conf_level=0.95)#

Return two-sided confidence limits for each score component.

Parameters:

model (BaseEstimator) – A fitted PCA or PLS model exposing a scores_ attribute.
conf_level (float, optional) – Fractional confidence level in (0, 1); by default 0.95.

Returns:

Array of length n_components with the positive score limit for each component.

Return type:

np.ndarray

References

score_contributions(X, component=1, scaling='none', *, method='scp', ridge=0.0, **deprecated)#

Per-variable contributions to a single score, $t_a$.

\[c_{ik}^{(a)} = x_{ik}\, R_{ka}, \qquad \sum_{k=1}^{K} c_{ik}^{(a)} = t_{ia},\]

Parameters:

model (PCA or PLS) – A fitted PCA or PLS model.
X (array-like of shape (n_samples, n_features)) – Preprocessed data, scaled the same way as the training data (for example with MCUVScaler). Passing the training data reproduces the model’s stored scores_.
component (int, default=1) – 1-based component index whose score is decomposed, matching the model’s column convention.
scaling ({"none", "maximum", "within"}, default="none") – Presentation scaling from Miller, Swanson and Heckler (1994). "none" returns the raw contributions, which sum to the score. "maximum" divides by the largest absolute contribution anywhere in X, so a bar of $\pm 1$ marks the most extreme variable-observation pair in the data set. "within" divides each row by the sum of its absolute contributions, so each row is on a common footing. Both scalings leave the pattern of bars within a row unchanged; neither preserves the sum to the score.
method ({"scp", "tsr", "pmp"}, default="scp") – Score estimator for the rows with missing cells; ignored when X is complete. See PCA.project().
ridge (float, default=0.0) – Regularisation for the "tsr" and "pmp" estimators, as in PCA.project().
**deprecated – Rejected. Captures t_end, components and weighted so that a call passing a score vector rather than X raises a TypeError explaining the correct usage. Passing a 1-D X raises the same error.

Returns:

Signed contributions of shape (n_samples, n_features), NaN at the missing cells. With the default scaling="none", each row sums to that observation’s score on the selected component.

Return type:

pd.DataFrame

Examples

>>> pca = PCA(n_components=2).fit(X_scaled)
>>> contrib = pca.score_contributions(X_scaled, component=1)
>>> contrib.sum(axis=1)  # equals pca.scores_[1]
>>> contrib.loc["33"].abs().sort_values()  # what makes observation 33 extreme

References

Miller, P., Swanson, R.E. and Heckler, C.E. (1994). “Contribution plots: a missing link in multivariate quality control.” Applied Mathematics and Computer Science, 8(4), 775-792.

See also

group_contributions: The same decomposition for a group of observations, or for the difference between two groups.
t2_contributions: Decomposes Hotelling’s $T^2$, which pools all components rather than reading one at a time.
spe_contributions: The residual-space counterpart.

spe_contributions(X, *, method='scp', ridge=0.0)#

Per-variable squared-prediction-error (SPE / DModX) contributions.

Works with fitted PCA and PLS models. Returns the signed residual of each variable after reconstructing the X-block from the full model:

\[e_{ik} = x_{ik} - \hat{x}_{ik}, \qquad \hat{X} = T P^\top,\]

Parameters:

model (PCA or PLS) – A fitted PCA or PLS model.
X (array-like of shape (n_samples, n_features)) – Preprocessed data, scaled the same way as the training data. Passing the training data reproduces the model’s stored spe_.
method ({"scp", "tsr", "pmp"}, default="scp") – Score estimator for the rows with missing cells; ignored when X is complete. See PCA.project().
ridge (float, default=0.0) – Regularisation for the "tsr" and "pmp" estimators, as in PCA.project().

Returns:

Signed per-variable residuals of shape (n_samples, n_features), NaN at the missing cells. The squared row sums equal the observation’s SPE squared, over its observed cells.

Return type:

pd.DataFrame

Examples

>>> pca = PCA(n_components=2).fit(X_scaled)
>>> resid = pca.spe_contributions(X_scaled)
>>> (resid ** 2).sum(axis=1)  # equals pca.spe_.iloc[:, -1] ** 2

See also

t2_contributions: The $T^2$ (score-space) counterpart.

t2_contributions(X, components=None, *, method='scp', ridge=0.0)#

Per-variable contributions to Hotelling’s $T^2$.

Works with fitted PCA and PLS models. Decomposes each observation’s $T^2$ onto the original variables. The contribution of variable $k$ for observation $i$ is

\[c^{T^2}_{ik} = x_{ik} \sum_{a} \frac{t_{ia}}{s_a^2}\, R_{ka},\]

Parameters:

model (PCA or PLS) – A fitted PCA or PLS model.
X (array-like of shape (n_samples, n_features)) – Preprocessed data, scaled the same way as the training data (for example with MCUVScaler). Passing the training data reproduces the model’s stored hotellings_t2_.
components (list of int, optional) – 1-based component indices to decompose over, matching the model’s column convention. None (default) uses all fitted components, so the row sums equal the cumulative $T^2$.
method ({"scp", "tsr", "pmp"}, default="scp") – Score estimator for the rows with missing cells; ignored when X is complete. See PCA.project().
ridge (float, default=0.0) – Regularisation for the "tsr" and "pmp" estimators, as in PCA.project().

Returns:

Signed contributions of shape (n_samples, n_features), NaN at the missing cells. Each row sums to the observation’s $T^2$ over the selected components.

Return type:

pd.DataFrame

Examples

>>> pca = PCA(n_components=3).fit(X_scaled)
>>> contrib = pca.t2_contributions(X_scaled)
>>> contrib.sum(axis=1)  # equals pca.hotellings_t2_.iloc[:, -1]

See also

spe_contributions: The residual-space counterpart.
PCA.score_contributions: Decomposes a single score-space movement.

set_fit_request(*, initial_conditions='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the fit method.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

initial_conditions (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for initial_conditions parameter in fit.
self (BatchPLS)

Returns:

self – The updated object.

Return type:

set_predict_request(*, initial_conditions='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the predict method.

The options for each parameter are:

True: metadata is requested, and passed to predict if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to predict.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

initial_conditions (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for initial_conditions parameter in predict.
self (BatchPLS)

Returns:

self – The updated object.

Return type:

set_score_request(*, sample_weight='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the score method.

The options for each parameter are:

True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to score.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.
self (BatchPLS)

Returns:

self – The updated object.

Return type:

set_transform_request(*, initial_conditions='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the transform method.

The options for each parameter are:

True: metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to transform.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

initial_conditions (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for initial_conditions parameter in transform.
self (BatchPLS)

Returns:

self – The updated object.

Return type:

class process_improve.batch.BatchMonitor(model, *, conf_level=0.99, method='tsr', spe_statistic='cumulative', spe_window=0, ridge=0.0)[source]#

Bases: BaseEstimator

Per-sample (online) control limits for a fitted BatchPCA or BatchPLS model.

Builds a Hotelling’s T2 and an SPE control limit at every time sample by passing each reference batch through the model’s missing-data score estimate at that sample and summarising the reference-batch spread. A new batch can then be tracked in real time: at each sample its projected T2 and SPE are compared against the limit for that sample, flagging abnormal behaviour while the batch is still running.

The T2 at sample k is t_k' S_k^-1 t_k with S_k the scatter of the reference batches’ score estimates at that sample about zero, the centre of the training scores (so the reference batches’ mean T2 is A (N - 1) / N at every sample), and its limit is the F-distribution limit for the number of reference batches and components. The reference batches are normally the batches the model was fitted on, as in Nomikos and MacGregor; a different reference set is centred on the model’s training batches, not on itself. The SPE is either the length of the residual over every cell observed so far ("cumulative") or over the newest sample only ("instantaneous", the per-interval SPE of Nomikos and MacGregor, which reacts in the sample a fault begins); its limit at each sample is the moment-matched chi-squared limit on the reference batches’ values at that sample, or pooled over a window of neighbouring samples (spe_window).

Parameters:

model (BatchPCA or BatchPLS) – A fitted batch model, ideally built from good (common-cause) batches only.
conf_level (float, default=0.99) – Confidence level for the control limits.
method ({"tsr", "scp", "pmp"}, default="tsr") – The missing-data score estimator used for every projection, passed to the model’s predict_online_trace. The limits and the monitored traces always use the same estimator, so the statistic at each sample is compared against the reference-batch spread computed the same way.
spe_statistic ({"cumulative", "instantaneous"}, default="cumulative") – Which SPE to chart and to build limits for (see above).
spe_window (int, default=0) – Half-width of the window of neighbouring samples whose reference SPE values are pooled with those of sample k before its limit is fitted: 0 fits each limit to the reference values of that sample alone, 2 to the values of samples k - 2 to k + 2 (fewer at the two ends of the batch). Pooling steadies the limits when few reference batches are available, at the cost of blurring a limit across samples where the reference SPE changes level. The mean trace spe_mean_over_time_ is never pooled.
ridge (float, default=0.0) – Regularisation for the "tsr" / "pmp" estimators, passed to predict_online_trace.
fitting) (Attributes (after)
--------------------------
spe_limit_over_time (np.ndarray of shape (n_timesteps,)) – The SPE limit at each sample.
t2_limit_over_time (np.ndarray of shape (n_timesteps,)) – The T2 limit at each sample (the same value at every sample, since the T2 is standardised by the per-sample score covariance).
spe_mean_over_time (np.ndarray of shape (n_timesteps,)) – The mean reference-batch statistic at each sample.
t2_mean_over_time (np.ndarray of shape (n_timesteps,)) – The mean reference-batch statistic at each sample.
score_covariance_over_time (np.ndarray of shape (n_timesteps, n_components, n_components)) – The scatter matrix (about zero, divided by N - 1) of the reference batches’ score estimates at each sample.
n_reference_batches (int) – Number of reference batches the limits were built from.
n_timesteps (int) – Number of time samples in an aligned batch.

fit(good_batches, y=None, *, initial_conditions=None)[source]#

Learn the per-sample control limits from reference batches.

Parameters:

good_batches (dict[Hashable, pd.DataFrame]) – Standard batch-data dictionary of aligned good (common-cause) batches, the same tags and length as the model’s training data.
y (ignored) – Present for sklearn Pipeline compatibility.
initial_conditions (pd.DataFrame, optional) – The Z block for the good batches; required if (and only if) the model was fitted with one.

Returns:

self

Return type:

BatchMonitor

monitor(batch, upto_k=None, *, initial_conditions=None)[source]#

Track a batch in real time against the per-sample limits.

This replays a complete, aligned batch and reports the statistics up to upto_k, which is how limits are checked on historical batches. A batch that is genuinely still running, with only its first samples in hand, is scored with the model’s predict_online and compared with spe_limit_over_time_ and t2_limit_over_time_ at that sample.

Parameters:

batch (pd.DataFrame) – A single complete, aligned batch to monitor (the training tags as columns).
upto_k (int, optional) – Report only up to this time sample (simulating a still-running batch). Defaults to the full batch length.
initial_conditions (pd.Series or pd.DataFrame, optional) – The Z block for this batch; required if the model was fitted with one.

Returns:

result – With keys time (1-based number of samples observed), scores (DataFrame, the score estimates at each sample), hotellings_t2 and spe (the batch’s statistic traces), t2_limit and spe_limit (the limits over the same samples), and t2_alarm / spe_alarm (boolean arrays where the statistic exceeds its limit).

Return type:

set_fit_request(*, good_batches='$UNCHANGED$', initial_conditions='$UNCHANGED$')#

Configure whether metadata should be requested to be passed to the fit method.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

good_batches (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for good_batches parameter in fit.
initial_conditions (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for initial_conditions parameter in fit.
self (BatchMonitor)

Returns:

self – The updated object.

Return type:

Mid-course correction of a running batch with a latent-variable model.

At a decision point during a batch, the initial conditions and the trajectories observed so far are known, the future responses are not, and the future manipulated-variable (MV) columns are ours to choose. This module solves that choice as a quadratic program in the scaled space of a fitted process_improve.batch.BatchPLS model:

the score vector of the candidate row is an affine function of the future MV columns, t = b + A_F u, built from the fixed-missingness-pattern projection operator (projection_matrix(), trimmed score regression by default), with the future response columns treated as missing;
the objective trades off quality tracking (or maximisation) against movement from the nominal remaining schedule, plus soft SPE and Hotelling’s T2 penalties that keep the correction where the model has data;
box bounds, rate-of-change limits between consecutive samples (including the seam to the last implemented sample) and optional hard SPE / T2 caps complete the program.

With the quadratic caps the problem is a convex quadratically-constrained QP. The workhorse is the penalty-form pure QP solved with osqp (the control extra); the hard-cap mode wraps the same QP in an outer scalar iteration on the two penalty multipliers, which is exact for this convex problem and converges in a handful of inner solves at this size (a few dozen decision variables).

The formulation follows the latent-variable batch control literature. Flores-Cerrillo and MacGregor (2004) give the quality-tracking objective, the movement-suppression term, the soft T2 term, and the SPE check on the measurements so far that gates whether a correction is computed at all; Yabuki and MacGregor (1997) give the no-correction dead band, their “no-control region”; Garcia-Munoz, Kourti and MacGregor (2004) give the per-decision-point score covariance and limits; Arteaga and Ferrer (2002) give the trimmed score regression used to estimate the scores of a partially observed row, which Golshan et al. (2010) also apply in the LV-MPC setting.

Two departures from Flores-Cerrillo and MacGregor are deliberate. Their optimisation is over an adjustment to the scores, with the remaining trajectories recovered by inverting the PLS model, which is what keeps those trajectories consistent with past operation. Here the decision variables are the future MV columns themselves, so actuator bounds and rate limits apply exactly in engineering units, and the terms that hold the answer inside the model’s region are stated explicitly instead. Second, the SPE of the candidate row is penalised and capped, not only checked on the measurements so far, because a candidate written directly in the MV columns can leave the model plane in a way a score adjustment cannot.

A practical caveat: models of this kind are identified on recorded (noisy, realised) trajectories, while the corrector outputs setpoints. That is the standard identification practice, and it attenuates the apparent gain slightly (the regression sees the control error as input noise); the executed-policy evaluation in evaluate_control_policies() (process_improve.simulation) measures the realised effect rather than trusting the model’s own prediction.

References

Flores-Cerrillo, J. and MacGregor, J.F., “Control of batch product quality by trajectory manipulation using latent variable models”, Journal of Process Control, 14, 539-553, 2004.

Arteaga, F. and Ferrer, A., “Dealing with missing data in MSPC: several methods, different interpretations, some examples”, Journal of Chemometrics, 16, 408-418, 2002.

Garcia-Munoz, S., Kourti, T. and MacGregor, J.F., “Model Predictive Monitoring for Batch Processes”, Industrial & Engineering Chemistry Research, 43, 5929-5941, 2004.

Yabuki, Y. and MacGregor, J.F., “Product quality control in semibatch reactors using midcourse correction policies”, Industrial & Engineering Chemistry Research, 36, 1268-1275, 1997.

Golshan, M., MacGregor, J.F., Bruwer, M.-J. and Mhaskar, P., “Latent Variable Model Predictive Control (LV-MPC) for trajectory tracking in batch processes”, Journal of Process Control, 20, 538-550, 2010.

process_improve.batch.control.midcourse_correction(model, *, observed, free_columns, mode='target', y_target=None, weights=None, bounds=None, rate_limits=None, seam=None, nominal_remaining=None, spe_cap=None, t2_cap=None, score_covariance=None, method='tsr', ridge=0.0, n_knots=None)[source]#

Optimise the remaining manipulated-variable columns of one batch.

This is the pure optimisation: everything is explicit and nothing is gated (no dead band, no validity check; use MidCourseCorrector for the full decision-point workflow). The unfolded row of the model splits three ways: observed columns carry known values; free_columns are the decision variables (the future MV columns); every other column is a missing future response, imputed by the projection operator.

Parameters:

model (BatchPLS) – A fitted process_improve.batch.BatchPLS model.
observed (pd.Series) – Known values in engineering units, indexed by unfolded column labels: the initial conditions (name, "") and the past trajectory columns (tag, sample).
free_columns (list) – Unfolded column labels of the decision variables, e.g. [("temperature", 12), ("temperature", 13), ...]. Must be disjoint from observed.
mode ({"target", "maximize"}, default="target") – "target" tracks y_target with a quadratic penalty (the Yabuki-MacGregor use case). "maximize" pushes the predicted quality up with a linear term; the quadratic movement penalty keeps the program bounded, which is the correct form for quality maximisation (an unreachable setpoint inside a quadratic is deliberately not used).
y_target (Series, dict, or float, optional) – The quality target in original units; required for mode="target" (a bare float is accepted for a single-target model).
weights (dict, optional) – Keys (all optional): "target" (scalar or per-target array; tracking weight, or the linear reward in maximize mode), "movement" (scalar or per-free-column array; penalty on the scaled deviation from nominal_remaining; must be positive in maximize mode), "spe" and "t2" (soft penalties on the candidate row’s SPE and Hotelling’s T2; the manufacturing-vs-development exploration dial). Defaults: {"target": 1.0, "movement": 0.1, "spe": 0.0, "t2": 0.0}.
bounds (dict, optional) – Per-tag box bounds in engineering units, {tag: (low, high)}, applied to every free column of that tag. Tighten the box inward by roughly two control-error standard deviations, so optimised setpoints do not sit on the actuator rails where clipping biases the realised mean.
rate_limits (dict, optional) – Per-tag limit on the change between consecutive samples, in engineering units, {tag: max_step}. Applied between consecutive free samples of the tag and, when seam provides the last implemented value, across the seam as well.
seam (dict, optional) – {tag: last_implemented_value} in engineering units, for the seam rate constraint.
nominal_remaining (pd.Series, optional) – The nominal remaining schedule in engineering units, indexed by free_columns; the movement penalty is measured from it. Default: the training average (the model’s centring) of those columns.
spe_cap (float, optional) – Hard cap on the candidate row’s SPE (on the square-root scale used throughout the package, so the quadratic constraint bounds SPE**2). Enforced by the outer multiplier iteration.
t2_cap (float, optional) – Hard cap on the candidate row’s Hotelling’s T2.
score_covariance (array-like of shape (A, A), optional) – Covariance used in the T2 quadratic. Default: the diagonal of the training score variances. Pass the per-decision-point covariance of the score estimates (Garcia-Munoz et al., 2004) for a reference that matches the pattern; MidCourseCorrector.limits_at() builds it.
method ({"tsr", "scp", "pmp"}, default="tsr") – Score-estimation method for the projection operator.
ridge (float, default=0.0) – Regularisation for the operator; see projection_matrix().
n_knots (int, optional) – Parameterise each tag’s free samples by n_knots linearly interpolated knot values. Shrinks the decision space (useful early in the batch) and smooths the schedule; the problem stays a QP.

Returns:

result – With keys mv (Series, the optimised free columns in engineering units, in model-feature order), y_hat and y_hat_no_change (Series, original quality units), scores (Series), spe and t2 (floats for the candidate row over the observed-plus-free pattern), active_constraints (dict with keys bounds, rate, spe_cap, t2_cap), solver (Bunch with status, n_solves, spe_multiplier, t2_multiplier, spe_unconstrained and t2_unconstrained), operator_condition_number and spe_offset (the constant term in the SPE quadratic, c_obs @ c_obs + c_free @ c_free).

Return type:

class process_improve.batch.control.MidCourseCorrector(model, nominal_schedule, *, mv_tags, mode='target', y_target=None, weights=None, bounds=None, rate_limits=None, spe_cap='limit', t2_cap='limit', conf_level=0.95, dead_band=1.0, target_side='both', method='tsr', ridge=0.0, n_knots=None)[source]#

Bases: object

Decision-point workflow around midcourse_correction().

Holds the model, the nominal schedule and the tuning, and at each decision point: checks the batch-so-far against the model (the SPE validity gate of Flores-Cerrillo and MacGregor, 2004), applies the no-correction dead band (Yabuki and MacGregor, 1997) in target mode, builds the per-decision-point reference limits (Garcia-Munoz et al., 2004), solves the QP, and returns the full corrected schedule ready to implement (or to hand to process_improve.simulation.BioreactorSimulator.simulate_batch()).

Parameters:

model (BatchPLS) – Fitted model whose X block unfolds recorded tag trajectories (and optionally initial conditions). Must be fitted with the default column layout (group_by_batch=False).
nominal_schedule (pd.DataFrame) – The nominal setpoint schedule: n_timesteps_ rows (positionally aligned with the tag samples), one column per manipulated tag.
mv_tags (list) – The manipulated tags (a subset of the model’s tag names); every other tag is a response, treated as missing after the decision point.
mode ({"target", "maximize"}, default="target")
y_target (Series, dict, or float, optional) – Required for mode="target".
weights (dict | None) – Passed through to midcourse_correction().
bounds (dict | None) – Passed through to midcourse_correction().
rate_limits (dict | None) – Passed through to midcourse_correction().
method (str) – Passed through to midcourse_correction().
ridge (float) – Passed through to midcourse_correction().
n_knots (int | None) – Passed through to midcourse_correction().
spe_cap (float, "limit", or None, default="limit") – Hard caps for the QP. "limit" resolves, per decision point, to the training-based limit for the same missingness pattern at conf_level (see limits_at()); a float is used as given; None disables the cap.
t2_cap (float, "limit", or None, default="limit") – Hard caps for the QP. "limit" resolves, per decision point, to the training-based limit for the same missingness pattern at conf_level (see limits_at()); a float is used as given; None disables the cap.
conf_level (float, default=0.95) – Confidence level for the per-decision-point limits and the dead-band prediction interval.
dead_band (float, default=1.0) – Multiplier on the prediction-interval half-width: in target mode the correction is skipped while the no-change prediction lies within dead_band half-widths of the target for every quality variable. Set to 0.0 to correct at every decision point. Ignored in maximize mode.
target_side ({"both", "below", "above"}, default="both") – Which deviations from the target warrant a correction. "below" treats the target as a floor (a more-is-better quality): batches predicted at or above it are left alone, whatever the dead band says. "above" is the mirror (a ceiling); "both" corrects deviations in either direction (an on-target specification).

limits_at(k)[source]#

Per-decision-point reference limits from the training batches.

The training rows are re-projected under decision point k’s two patterns: the monitoring pattern (initial conditions plus every tag up to k; the future entirely missing) for the SPE validity gate, and the candidate pattern (monitoring plus the future MV columns, which the optimiser treats as observed) for the QP’s SPE cap and the score covariance behind its T2 term. Limits: the g-chi-squared SPE limit of Nomikos and MacGregor on each pattern’s training SPE values, and the F-distribution T2 limit on the candidate-pattern score estimates with their own covariance (Garcia-Munoz et al., 2004).

Results are cached per k.

Parameters:: k (int)
Return type:: Bunch

correct(batch_so_far, *, initial_conditions=None, implemented_schedule=None, k=None)[source]#

Decide and (when warranted) compute the correction at one decision point.

Parameters:

batch_so_far (pd.DataFrame) – The recorded tag trajectories up to the decision point: the first k samples, columns = the model’s tags.
initial_conditions (pd.Series or pd.DataFrame, optional) – The batch’s Z values; required if the model was fitted with a Z block.
implemented_schedule (pd.DataFrame, optional) – The setpoint schedule actually implemented so far (same layout as nominal_schedule); its first k rows are carried into the returned schedule verbatim and its row k - 1 anchors the seam rate constraint. Defaults to the nominal schedule.
k (int, optional) – The decision point (number of completed samples). Defaults to len(batch_so_far).

Returns:

result – With keys schedule (the full setpoint DataFrame: implemented past plus the decided remainder), corrected (bool), reason ("corrected", "spe_gate", "dead_band" or "batch_complete"), k, spe_so_far and spe_limit_monitor (the validity gate), y_hat_no_change and, in target mode, dead_band_margin (Series; deviation of the no-change prediction from the target in units of the prediction-interval half-width), plus correction (the full midcourse_correction() Bunch) when a correction was computed.

Return type:

process_improve.batch.control.evaluate_control_policies(simulator, *, y_target, n_train=200, n_test=40, mv_variation=2.5, n_components=4, decision_points=(8,), target_side='below', dead_band=2.5, weights=None, bounds=None, rate_limits=None, spe_cap='limit', t2_cap='limit', n_knots=4, per_class=True, method='tsr', ridge=0.0, include_adapted=True, adapted_n_knots=4, adapted_n_starts=5, oracle='corrected', random_state=None)[source]#

Compare batch operating policies on the bioreactor simulator, executed.

Runs the full mid-course-correction workflow end to end and reports realised (executed) quality, not model predictions: every corrected schedule is fed back into the simulator with the identical seed, so the with- and without-correction titers are true same-batch counterfactuals. Four policies are compared:

replay: every batch runs the nominal schedule (the floor; what “replicate the golden batch” automation does).
mid-course: batches are corrected at the decision_points by MidCourseCorrector models fitted on a deliberately varied historical campaign; batches the dead band or the validity gate skips run the nominal schedule.
oracle-from-k: for every batch the mid-course policy corrected, the remaining schedule is instead optimised against the simulator itself at the same decision point (_oracle_remaining()). This is the ceiling for any mid-course scheme at that decision point; the gap to the mid-course row is the price of using an empirical model with limited historical excitation.
adapted: every batch runs the true optimal schedule for its own initial conditions from time zero (simulator.optimal_trajectory), the perfect-feedforward ceiling.

Parameters:

simulator (BioreactorSimulator) – The simulator (duck-typed: needs simulate_campaign, simulate_batch, nominal_trajectory, optimal_trajectory and config).
y_target (float) – The quality target handed to the corrector (original units).
n_train (int) – Sizes of the historical (training) campaign and the fresh test campaign.
n_test (int) – Sizes of the historical (training) campaign and the fresh test campaign.
mv_variation (float, default=2.5) – Deliberate setpoint variation of the historical campaign; the identification requirement is discussed in simulate_campaign().
n_components (int, default=4) – Components for each BatchPLS model.
decision_points (tuple of int, default=(8,)) – Sample indices at which the corrector is consulted, in order; later points see the schedule implemented by earlier ones.
per_class (bool, default=True) – Fit one model per feed class (labels from the training campaign; test batches are assigned to the nearest class centroid in standardised Z). With False, or when class labels are unavailable, a single global model is used; the executed experiments behind this module found the global linear model averages the class-dependent gain direction away, so per-class models are the default.
target_side (str) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
dead_band (float) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
weights (dict | None) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
bounds (dict | None) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
rate_limits (dict | None) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
spe_cap (float | str | None) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
t2_cap (float | str | None) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
n_knots (int | None) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
method (str) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
ridge (float) – Corrector settings, passed to MidCourseCorrector. bounds defaults to the simulator’s operating bounds tightened inward by about two control-error standard deviations (0.3 degC, 0.04 pH); rate_limits defaults to 3.0 degC and 0.5 pH per sample; weights defaults to {"target": 1.0, "movement": 0.1}.
include_adapted (bool, default=True) – Compute the adapted (perfect-feedforward) row; it costs one optimal_trajectory call per test batch (roughly 10 s each on the default configuration).
adapted_n_knots (int) – Passed to optimal_trajectory for the adapted policy.
adapted_n_starts (int) – Passed to optimal_trajectory for the adapted policy.
oracle ({"corrected", "none"}, default="corrected") – Compute the oracle-from-k row for the batches the mid-course policy corrected (a few direct-search optimisations against the simulator), or skip it.
random_state (int, np.random.Generator, or None) – Seed for the campaigns and the per-batch execution seeds; the whole comparison is reproducible end to end.

Returns:

result – With keys summary (DataFrame: one row per policy with mean, sd, min and max titer), batches (DataFrame: per-batch replay / mid-course / adapted / oracle titers, the assigned and true feed class, whether and why each batch was or was not corrected, the decision point used, and the corrector’s predicted quality), n_corrected, n_harmed (corrected batches whose executed titer fell more than 0.01 below replay), and models (per-class fit R2).

Return type:

Model-level plots for batchwise-unfolded (multiway) PCA and PLS of batch data.

These complement the batch score / SPE / T2 plots, which are inherited from the multivariate package (they operate on the internal PCA or PLS model). Three plots are specific to the batch (unfolded) structure:

time_varying_loading_plot(): the loadings (or PLS weights) of one component drawn as a function of time, one trace per tag, so the reader sees how each variable contributes to a component over the batch evolution.
unfolded_contribution_plot(): for one batch, its whole contribution vector over the unfolded (tag, time) axis, grouped and coloured by tag, or summed into one bar per tag. This is the classic batch contribution plot that answers “which variables, and when”.
contribution_at_time_plot(): for one batch, the per-tag contribution to SPE or Hotelling’s T2 at a chosen time sample, drawn as a bar chart to diagnose which variable drives an abnormal event.

process_improve.batch._batch_plots.time_varying_loading_plot(model, component=1, fig=None, show_initial_conditions=True)[source]#

Plot one component’s loadings (or weights) as a function of time, one trace per tag.

The batchwise-unfolded model has a separate loading for every (tag, time) cell, so a component’s loadings can be read as a set of time-varying weight profiles: how strongly each variable loads on the component at each point in the batch. Initial-condition (Z) loadings, which have no time axis, are drawn as a marker group to the left of time zero.

Parameters:

model (BatchPCA, BatchPLS or a fitted multivariate model) – A fitted process_improve.batch.BatchPCA (loadings p), a fitted process_improve.batch.BatchPLS (weights w), or any process_improve.multivariate model whose loadings_ or x_weights_ carry the 2-level (tag, sequence) index of a process_improve.batch.dict_to_wide() matrix.
component (int, default=1) – 1-based component index whose loadings to plot.
fig (plotly.graph_objects.Figure, optional) – Figure to draw into; a new one is created when omitted.
show_initial_conditions (bool, default=True) – Draw the initial-condition loadings (if the model has any) as a marker group before time zero.

Return type:

plotly.graph_objects.Figure

process_improve.batch._batch_plots.unfolded_contribution_plot(contributions, batch_id=None, *, by_tag=False, fig=None)[source]#

Bar chart of one batch’s contributions over the whole unfolded (tag, time) axis.

Takes a contribution matrix (one row per batch, the 2-level (tag, sequence) column index of the unfolded data) and draws one batch’s row as bars in unfolded column order, one trace per tag so the legend toggles tags and the colour identifies them. The tag names are written under the centre of each tag’s block of samples. Reading the plot left to right answers “which variables, and at which time” for the score, SPE or T2 of that batch.

With by_tag=True the bars are summed over time, one bar per tag, which is the compact summary used to rank the variables. The sum is signed: for score contributions it is the tag’s contribution to the score; for SPE contributions (signed residuals) pass contributions ** 2 to get each tag’s share of the SPE.

Parameters:

contributions (pd.DataFrame) – Output of score_contributions, spe_contributions or t2_contributions on process_improve.batch.BatchPCA or process_improve.batch.BatchPLS, or of the standalone process_improve.multivariate functions on a model fitted to a process_improve.batch.dict_to_wide() matrix whose column index was re-attached after scaling.
batch_id (Hashable, optional) – Which batch (row) to plot. Defaults to the first row.
by_tag (bool, default=False) – Sum the contributions over time and draw one bar per tag.
fig (plotly.graph_objects.Figure, optional) – Figure to draw into; a new one is created when omitted.

Return type:

plotly.graph_objects.Figure

process_improve.batch._batch_plots.contribution_at_time_plot(contributions, k, batch_id=None, fig=None)[source]#

Bar chart of per-tag contributions at one time sample, for one batch.

Takes the output of process_improve.batch.BatchPCA.spe_contributions() or t2_contributions() (one row per batch, columns indexed by the 2-level (tag, sequence) unfolded index) and shows, for a single batch and a single time sample k, how much each tag contributes. This localizes an abnormal event to the responsible variable(s).

Parameters:

contributions (pd.DataFrame) – Contribution matrix from BatchPCA.spe_contributions / t2_contributions: one row per batch, a 2-level (tag, sequence) column index.
k (int) – The time sample (sequence value) at which to show the contributions.
batch_id (Hashable, optional) – Which batch (row) to plot. Defaults to the first row; required to be a valid row label when the matrix has more than one batch.
fig (plotly.graph_objects.Figure, optional) – Figure to draw into; a new one is created when omitted.

Return type:

plotly.graph_objects.Figure

process_improve.batch._batch_plots.online_monitoring_plot(monitor, batch, statistic='spe', *, initial_conditions=None, fig=None)[source]#

Plot a batch’s online SPE or T2 trace against the per-sample limit.

Tracks the batch through the fitted process_improve.batch.BatchMonitor (built on a process_improve.batch.BatchPCA or process_improve.batch.BatchPLS model) and draws its statistic over time overlaid on the control limit and the mean reference-batch trace, with the alarm samples marked. This is the online (real-time) monitoring chart of Nomikos and MacGregor. The SPE drawn is the one the monitor was fitted with (cumulative over the observed cells, or the newest sample only).

Parameters:

monitor (BatchMonitor) – A fitted process_improve.batch.BatchMonitor.
batch (pd.DataFrame) – A single aligned batch to monitor.
statistic ({"spe", "t2"}, default="spe") – Which statistic to plot.
initial_conditions (pd.Series or pd.DataFrame, optional) – The Z block for this batch; required if the model was fitted with one.
fig (plotly.graph_objects.Figure, optional) – Figure to draw into; a new one is created when omitted.

Return type:

plotly.graph_objects.Figure

Loader functions for the batch datasets bundled with, or hosted for, the package.

Each trajectory loader returns the standard batch-data dictionary used throughout process_improve.batch: keys are batch identifiers, values are per-batch dataframes with identical, all-numeric columns (one column per tag). See process_improve.batch.data_input for the format definitions and converters to the melted and wide representations.

Three datasets are bundled (load_nylon(), load_dryer(), load_batch_fake_data()). Three larger case-study datasets are hosted on openmv.net and downloaded on demand (load_dupont(), load_fmc(), load_sbr()); the download is bounded by settings.dataset_fetch_timeout and every failure surfaces as a RuntimeError naming the URL (see process_improve._remote_data).

process_improve.batch.datasets.DUPONT_URL = 'https://openmv.net/file/polymerization.csv'#: Hosted copy of the DuPont batch polymerization data (load_dupont()).

process_improve.batch.datasets.FMC_URL = 'https://openmv.net/file/batch-dryer.xlsx'#: Hosted copy of the aligned, four-block FMC batch dryer data (load_fmc()).

process_improve.batch.datasets.SBR_URL = 'https://openmv.net/file/sbr-batch-reactor.xlsx'#: Hosted copy of the simulated SBR batch reactor data (load_sbr()).

process_improve.batch.datasets.load_nylon()[source]#

Return the nylon autoclave reactor batch dataset.

Trajectory data from an industrial nylon polymerization autoclave, used widely in the batch analysis and monitoring literature. Variables Tag01 to Tag10 are temperatures, pressures, and flows recorded during each batch. Batch durations vary slightly (113 to 135 samples), so resample or align the batches to a common length before unfolding (see process_improve.batch.resample_to_reference()).

Returns:

dict[Hashable, pd.DataFrame] – Standard batch-data dictionary: 57 batches, each a dataframe of 10 numeric tag columns.
Source
——
Kassidas, A., “Fault Detection and Diagnosis in Dynamic Multivariable
Chemical Processes Using Speech Recognition Methods”, PhD thesis,
McMaster University, 1997. Also analyzed in Wold, Kettaneh-Wold,
MacGregor and Dunn, “Batch Process Modeling and MSPC”, Comprehensive
Chemometrics, Elsevier, 2009.

Return type:

dict[Hashable, pd.DataFrame]

Examples

>>> from process_improve.batch.datasets import load_nylon
>>> batches = load_nylon()
>>> len(batches)
57

process_improve.batch.datasets.load_dryer()[source]#

Return the batch dryer dataset.

Trajectory data from an industrial batch drying process (a critical step in the manufacture of an agricultural chemical). Each batch records ten process tags plus ClockTime, the wall-time sample counter:

CollectorTankLevel: level of the solvent collector tank
DifferentialPressure: differential pressure in the dryer
DryerPressure: pressure in the dryer
AgitatorPower: power to the agitator
AgitatorTorque: torque resistance for the agitator
AgitatorSpeed: agitator speed
JacketTemperatureSP: set point for the jacket heating medium
JacketTemperature: temperature of the jacket heating medium
DryerTemperatureSP: set point for the temperature inside the dryer
DryerTemp: temperature inside the dryer
ClockTime: sample counter (samples assumed evenly spaced)

The batches have varying durations, so this dataset is a realistic candidate for alignment (see process_improve.batch.batch_dtw() and process_improve.batch.resample_to_reference()).

Returns:

dict[Hashable, pd.DataFrame] – Standard batch-data dictionary: 71 batches, each a dataframe of 11 numeric columns (10 tags plus ClockTime).
Source
——
Garcia-Munoz, S., “Batch process improvement using latent variable
methods”, PhD thesis, McMaster University, 2004. Also analyzed in Wold,
Kettaneh-Wold, MacGregor and Dunn, “Batch Process Modeling and MSPC”,
Comprehensive Chemometrics, Elsevier, 2009.

Return type:

dict[Hashable, pd.DataFrame]

Examples

>>> from process_improve.batch.datasets import load_dryer
>>> batches = load_dryer()
>>> "DryerTemp" in next(iter(batches.values())).columns
True

process_improve.batch.datasets.load_batch_fake_data()[source]#

Return a small synthetic batch dataset.

Simulated trajectory data for quick examples and tests: two temperature tags and one pressure tag per batch, plus UCI_minutes (minutes since the start of the batch). The wall-clock timestamp column in the raw CSV is dropped, so all returned columns are numeric.

Returns:: Standard batch-data dictionary of synthetic batches, each a dataframe with columns UCI_minutes, Temp1, Temp2, and Pressure1.
Return type:: dict[Hashable, pd.DataFrame]

Examples

>>> from process_improve.batch.datasets import load_batch_fake_data
>>> batches = load_batch_fake_data()
>>> sorted(next(iter(batches.values())).columns)
['Pressure1', 'Temp1', 'Temp2', 'UCI_minutes']

process_improve.batch.datasets.load_dupont(*, url=None, timeout=None)[source]#

Return the DuPont industrial batch polymerization dataset (downloaded).

The worked example of Nomikos and MacGregor (1995): 55 batches from an industrial batch polymerization reactor, each already aligned to 100 equal time intervals, with ten process measurements per interval. Values are scaled for confidentiality and there are no missing values. The ten tags, in file order, are TempR-1, TempR-2, TempR-3 (reactor temperatures), Press-1 (a pressure), Flow-1 (a feed flow), TempH-1 and TempC-1 (heating- and cooling-medium temperatures), Press-2 and Press-3 (pressures) and Flow-2 (a feed flow).

The final quality of each batch is not part of the dataset. The paper reports that batches 40, 41, 42, 50, 51, 53, 54 and 55 had a quality measurement well outside the acceptable limit, that batches 38, 45, 46, 49 and 52 were above or very close to it, and that batch 49 was barely acceptable. Batches 50 to 55 stand out in the score plot and batch 49 in the SPE.

Parameters:

url (str, optional) – Where to download from. Defaults to DUPONT_URL; pass a mirror or a file:// URL to read a local copy.
timeout (float, optional) – Download budget in seconds. Defaults to settings.dataset_fetch_timeout.

Returns:

Standard batch-data dictionary: batch identifiers 1 to 55, each a dataframe of 100 rows and the 10 numeric tag columns. The time column of the hosted file is dropped, since it is identical in every batch.

Return type:

dict[Hashable, pd.DataFrame]

Raises:

RuntimeError – When the download fails or times out.
Source –
------ –
Nomikos, P. and MacGregor, J.F., "Multivariate SPC Charts for Monitoring –
Batch Processes", Technometrics, 37(1), 41-59, 1995. Hosted at –
https://openmv.net/info/polymerization. –

Examples

>>> from process_improve.batch import load_dupont
>>> batches = load_dupont()
>>> len(batches), batches[1].shape
(55, (100, 10))

process_improve.batch.datasets.load_fmc(*, url=None, timeout=None)[source]#

Return the aligned, four-block FMC batch dryer dataset (downloaded).

An industrial batch drying step in the manufacture of an agricultural chemical: wet cake (solid plus embedded solvent) is charged, dried through three recipe phases (solvent collection, temperature ramp, cool-down), and the solvent is collected in a side tank. This is the multiblock case study of Garcia-Munoz et al. (2003), with the trajectories already aligned within each phase to 325 samples per batch. Compare load_dryer(), the raw, unaligned trajectories of the same process.

The four blocks are:

X: the batch trajectories, ten tags plus ClockTime, the wall time at each aligned sample, which after alignment is itself a trajectory that carries the time-warping information.
Zchem: eleven initial-condition chemistry measurements of the cake, Z1 to Z11.
Zop: nine initial operating conditions (levels, temperatures, the durations of the recipe steps, the temperature slope, the cake weight).
Y: eight final quality attributes, Y1 to Y11 (not all numbers are used) and SolventConc.

The data contain genuine missing values, kept as NaN: 1410 cells in X, 134 in Zchem and 21 in Y. Thirteen batches have no chemistry measurements at all; the original study excluded them, and their identifiers are returned as missing_chemistry so the exclusion can be reproduced.

Parameters:

url (str, optional) – Where to download from. Defaults to FMC_URL; pass a mirror or a file:// URL to read a local copy. Reading the workbook needs openpyxl (the batch extra).
timeout (float, optional) – Download budget in seconds. Defaults to settings.dataset_fetch_timeout.

Returns:

With fields X (standard batch-data dictionary of 59 batches, each 325 rows by 11 columns), Y (59 x 8), Zop (59 x 9) and Zchem (59 x 11), the last three indexed by batch identifier in the same order as the keys of X, plus batch_ids (the 59 non-consecutive identifiers) and missing_chemistry (the 13 identifiers without chemistry data).

Return type:

Raises:

RuntimeError – When the download fails or times out.
Source –
------ –
Garcia-Munoz, S., Kourti, T., MacGregor, J.F., Mateos, A.G. and Murphy, –
G., "Troubleshooting of an Industrial Batch Process Using Multivariate –
Methods", Industrial and Engineering Chemistry Research, 42, 3592-3601, –
2003. Hosted at https://openmv.net/info/batch-dryer. –

Examples

>>> from process_improve.batch import load_fmc
>>> fmc = load_fmc()
>>> len(fmc.X), fmc.Y.shape, fmc.Zop.shape, fmc.Zchem.shape
(59, (59, 8), (59, 9), (59, 11))

process_improve.batch.datasets.load_sbr(*, url=None, timeout=None)[source]#

Return the simulated SBR batch reactor dataset (downloaded).

Styrene-butadiene rubber (SBR) emulsion polymerization, simulated from a first-principles model for the batch monitoring work of Nomikos (1995): 53 batches of 200 samples with nine trajectories, and five final quality attributes per batch. Because the data are simulated, the fault is known (Nomikos and MacGregor, 1994): batch 37 received 30% more organic impurity in the butadiene feed than the normal batches from its start, and batch 34 50% more from midway through, at sample 100. The two feed flows and the feed temperature carry only the noise the simulation adds to them, under 2% and 0.1% of their values.

Parameters:

url (str, optional) – Where to download from. Defaults to SBR_URL; pass a mirror or a file:// URL to read a local copy. Reading the workbook needs openpyxl (the batch extra).
timeout (float, optional) – Download budget in seconds. Defaults to settings.dataset_fetch_timeout.

Returns:

With fields X (standard batch-data dictionary: batch identifiers 1 to 53, each 200 rows by the 9 tags StyreneFlow, ButadieneFlow, FeedTemp, ReactorTemp, CoolingTemp, JacketTemp, LatexDensity, Conversion and EnergyReleased), Y (53 x 5: Composition, ParticleSize, Branching, CrossLinking and Polydispersity, indexed by batch identifier), trajectory_tags (the six reactor tags this case study models, without the feed tags) and fault_batches ([34, 37]).

Return type:

Raises:

RuntimeError – When the download fails or times out.
Source –
------ –
Nomikos, P., "Statistical process control of batch processes", PhD –
thesis, McMaster University, 1995, and Nomikos, P. and MacGregor, J.F., –
"Monitoring batch processes using multiway principal component –
analysis", AIChE Journal, 40, 1361-1375, 1994, which describes the –
simulation and the two faulty batches. Hosted at –
https://openmv.net/info/sbr-batch-reactor. –

Examples

>>> from process_improve.batch import load_sbr
>>> sbr = load_sbr()
>>> len(sbr.X), sbr.X[1].shape, sbr.Y.shape
(53, (200, 9), (53, 5))

process_improve.batch.features.f_mean(data, tags=None, batch_col=None, phase_col=None)[source]#

Feature: mean.

The arithmetic mean for the given tags in tags, for each unique batch in the batch_col indicator column, and within each unique phase, per batch, of the phase_col column.

Parameters:

data (DataFrame)
tags (list[str] | None)
batch_col (str | None)
phase_col (str | None)

Return type:

DataFrame

process_improve.batch.features.f_median(data, tags=None, batch_col=None, phase_col=None)[source]#

Feature: median.

The median for the given tags in tags, for each unique batch in the batch_col indicator column, and within each unique phase, per batch, of the phase_col column.

Parameters:

data (DataFrame)
tags (list[str] | None)
batch_col (str | None)
phase_col (str | None)

Return type:

DataFrame

process_improve.batch.features.f_std(data, tags=None, batch_col=None, phase_col=None)[source]#

Feature: std.

The standard deviation for the given tags in tags, for each unique batch in the batch_col indicator column, and within each unique phase, per batch, of the phase_col column.

See also: f_sum, f_last

Parameters:

data (DataFrame)
tags (list[str] | None)
batch_col (str | None)
phase_col (str | None)

Return type:

DataFrame

process_improve.batch.features.f_slope(data, x_axis_tag, tags=None, batch_col=None, phase_col=None, age_col=None)[source]#

Feature: slope.

The slope of the given tags for each unique batch in the batch_col indicator column, of the phase_col column.

The slope is calculated against whichever variable is given by x_axis_tag. If this is the age_col of the batch (i.e. time duration), ensure that age_col is also specified.

Parameters:

data (DataFrame)
x_axis_tag (str)
tags (list[str] | None)
batch_col (str | None)
phase_col (str | None)
age_col (str | None)

Return type:

DataFrame

process_improve.batch.features.cross(series, threshold=0, direction='cross', only_index=False, first_point_only=False)[source]#

Given a Series returns all the index values where the data values equal the ‘threshold’ value. Will first drop all missing values from the series.

direction` can be ‘rising’ (for rising edge), ‘falling’ (for only falling edge), or ‘cross’ for both edges.

If only_index is True (default False), then it will return the 0-based index where crossing occur just after. E.g. if the returned index is 135, then the crossing takes place at, or after, index 135, but before index 136.

If the setting first_point_only is set to True, only the first point where the crossing occurs is reported. The rest are ignored. Default = all crossings are report (i.e. first_point_only=False).

https://stackoverflow.com/questions/10475488/calculating-crossing-intercept- points-of-a-series-or-dataframe

Parameters:

series (Series)
threshold (int | None)
direction (str | None)
only_index (bool | None)
first_point_only (bool | None)

Return type:

ndarray | list

process_improve.batch.features.f_crossing(data, tag, time_tag, threshold=0, direction='cross', only_index=False, batch_col=None, phase_col=None, suffix=None)[source]#

Feature: cross.

The time (time_tag) value at which tag crosses a certain numeric threshold`, either direction=’rising’` (for rising edge), or direction=’falling’’ (for falling edge), or ‘cross’ for both edges.

The time when the crossing occurs is found by linear interpolation between the indices. If you prefer the index itself, use only_index=True, but the default for that setting is False.

Does this for each unique batch in the batch_col indicator column. The phase_col argument is accepted for signature-compatibility with the other f_* features but is not consumed here: the crossing is looked up over the whole batch, not per phase.

suffix: what to add to the data tag, to name to this feature.

Note: NaN is returned for a given batch, if the crossing is not found.

Parameters:

data (DataFrame)
tag (str)
time_tag (str)
threshold (int)
direction (str)
only_index (bool)
batch_col (str | None)
phase_col (str | None)
suffix (str | None)

Return type:

DataFrame

process_improve.batch.features.f_elbow(data, x_axis_tag, tags=None, only_index=False, batch_col=None, phase_col=None)[source]#

Feature: elbow.

The “elbow” of the given tags for each unique batch in the batch_col indicator column, of the phase_col column.

The elbow is calculated against whichever variable is given by x_axis_tag (usually a time- based tag).

The function returns the value on the x-axis where the elbox occurs. Sometimes you might want the index of the value, so you can also find the corresponding y-axis value. Use only_index=True for such cases.

Parameters:

data (DataFrame)
x_axis_tag (str)
tags (list[str] | None)
only_index (bool | None)
batch_col (str | None)
phase_col (str | None)

Return type:

DataFrame

process_improve.batch.preprocessing.determine_scaling(batches, columns_to_align=None, settings=None)[source]#

Scales the batch data according to the variable ranges.

Parameters:

batches (dict[str, pd.DataFrame]) – Batch data, in the standard format (keyed by batch identifier).
columns_to_align (list, optional) – The column names (tags) to be scaled. If None, the columns of the first batch are used.
settings (dict, optional) –
Optional overrides:

"robust" (bool, default True)
Switches between a robust per-batch range and the raw max - min.

"robust_range" (str, default "q98-q02")
Which robust range to use when robust is True, either "q98-q02" or "iqr" (q75 - q25, the interquartile range that f_iqr() computes). Ignored when robust is False.

The two are not interchangeable, which is the answer to the question the old TODO here posed. The IQR spans the middle half of a batch; q98 - q02 spans nearly all of it. On a Gaussian tag the second is about 3.05 times the first, but a batch trajectory is not Gaussian, so the ratio varies by tag: measured per tag on the bundled data it runs from 1.02 to 4.23 (dryer) and 1.21 to 2.95 (nylon). Switching therefore re-weights the tags against each other rather than rescaling them together. The IQR also collapses to zero more often, because a tag that holds one value for more than half of a batch has no interquartile spread at all: DifferentialPressure collapses in 21 of the 71 dryer batches under the IQR against 16 under q98 - q02. Prefer the IQR when the tags carry excursions you want the scaling to ignore; the default otherwise.

Returns:

range_scalers – J rows, 2 columns: column 1 = the per-tag range (approximately q98 - q02 when settings["robust"] is True, else raw max - min); column 2 = the per-tag minimum. Both columns are aggregated across batches with the median when robust=True and the mean otherwise, but the per-batch minimum itself is always the raw batch.min(axis=0), not a quantile.

Return type:

DataFrame

process_improve.batch.preprocessing.apply_scaling(batches, scale_df, columns_to_align=None)[source]#

Scales the batches according to the information in the scaling dataframe.

Parameters:

batches (dict[str, pd.DataFrame]) – The batches, in standard format.
scale_df (pd.DataFrame) – The scaling dataframe, from determine_scaling.
columns_to_align (list, pd.Index, or None, optional) – Which columns of each batch to scale. Columns outside this list are dropped from the output. When None (the default) the columns of the first batch are used, matching determine_scaling().

Returns:

The scaled batch data. Each value carries only the columns_to_align columns, in that order.

Return type:

process_improve.batch.preprocessing.reverse_scaling(batches, scale_df, columns_to_align=None)[source]#

Reverse the scaling applied by apply_scaling.

Parameters:

batches (dict[str, pd.DataFrame]) – The scaled batches, in standard format.
scale_df (pd.DataFrame) – The scaling dataframe, from determine_scaling().
columns_to_align (list, pd.Index, or None, optional) – Which columns of each batch to un-scale. Columns outside this list are dropped from the output. When None (the default) the columns of the first batch are used, matching apply_scaling().

Returns:

The un-scaled batch data.

Return type:

class process_improve.batch.preprocessing.DTWresult(synced, penalty_matrix, md_path, warping_path, distance, normalized_distance)[source]#

Bases: object

Result class.

Parameters:

synced (np.ndarray | pd.DataFrame)
penalty_matrix (np.ndarray)
md_path (np.ndarray)
warping_path (np.ndarray)
distance (float)
normalized_distance (float)

class process_improve.batch.preprocessing.BatchScaler(columns_to_align=None, batch_col=None, robust=True, robust_range='q98-q02')[source]#

Bases: TransformerMixin, BaseEstimator

Range-scale batch trajectories, as a fit / transform estimator.

Wraps the three functions determine_scaling(), apply_scaling() and reverse_scaling() in the estimator shape the rest of this package uses, so batch preprocessing composes with Pipeline and with clone / get_params / set_params the way MCUVScaler already does (#199). The three functions remain public and unchanged; this adds a way to carry the fitted scaling around as one object instead of threading a scale_df through every call.

Each tag is mapped to roughly [0, 1] by subtracting a per-tag minimum and dividing by a per-tag range, both aggregated across the batches seen in fit(). That is a different normalisation from mean-centring to unit variance: it preserves the shape of a trajectory within its own operating range, which is what the alignment distance needs.

Parameters:

columns_to_align (list or None, optional) – The tags to scale. None (the default) takes the columns of the first batch, matching determine_scaling().
batch_col (str or None, optional) – When set, fit() and transform() also accept a single melted DataFrame holding every batch, and split it on this column. This is the DataFrame input case #199 asked for: the functions reject a DataFrame outright and tell the caller to split it themselves.
robust (bool, optional) – Use a robust per-batch range (the default) rather than max - min.
robust_range (str, optional) – Which robust range: "q98-q02" (the default) or "iqr". See determine_scaling() for what separates them.

scale_df_#

The fitted scaling, exactly as determine_scaling() returns it: a Range and a Minimum per tag.

Type:: pd.DataFrame

columns_to_align_#

The tags actually scaled, resolved during fit().

Type:: list

n_features_in_#

Number of tags scaled, for the sklearn contract.

Type:: int

Examples

>>> scaler = BatchScaler(columns_to_align=["Temperature"])
>>> scaled = scaler.fit_transform(batches)
>>> original = scaler.inverse_transform(scaled)

A melted frame works when the batch column is named:

>>> scaler = BatchScaler(batch_col="batch_id")
>>> scaled = scaler.fit_transform(melted_frame)

fit(X, y=None)[source]#

Determine the per-tag range and minimum from these batches.

y is accepted and ignored, per the sklearn transformer contract.

Parameters:

X (dict | DataFrame)
y (object)

Return type:

BatchScaler

transform(X)[source]#

Scale these batches with the fitted range and minimum.

Parameters:: X (dict | DataFrame)
Return type:: dict

inverse_transform(X)[source]#

Undo transform(), returning the batches to their original units.

Parameters:: X (dict | DataFrame)
Return type:: dict

process_improve.batch.preprocessing.align_with_path(md_path, batch)[source]#

Align a batch to the reference using the DTW path.

Where several samples of batch map to the same reference index (a compression in the warping path), the synced value for that index is the average of those batch samples. The running temp accumulator is therefore seeded with the first batch sample for the current index - the same value assigned to synced row 0 just below - not with a reference row. A former initial_row argument seeded it from the reference row (in one caller) or from an out-of-space batch index (in the other), which mixed an unrelated row into the row-0 average (#197).

Non-numeric columns are carried through rather than averaged. A batch frame from melted_to_dict() still holds its identifier column, and averaging a label is meaningless even when it happens to be a number: with string identifiers it raised TypeError: unsupported operand type(s) for /, and with integer ones it silently wrote the mean of the identifier into the aligned frame. Such a column is constant within a batch, so the first value is taken (#197).

Parameters:

md_path (ndarray)
batch (DataFrame)

Return type:

DataFrame

process_improve.batch.preprocessing.dtw_core(test, ref, weight_matrix, band=None)[source]#

Compute DTW alignment of test batch against reference batch.

band is an optional constraint on the warping path, resolved by resolve_band(). The default None places no constraint.

Parameters:

test (DataFrame)
ref (DataFrame)
weight_matrix (ndarray)
band (object)

Return type:

DTWresult

process_improve.batch.preprocessing.one_iteration_dtw(batches_scaled, refbatch_sc, weight_matrix, settings=None)[source]#

Perform one iteration of the DTW alignment algorithm.

Parameters:

batches_scaled (dict)
refbatch_sc (DataFrame)
weight_matrix (ndarray)
settings (dict | None)

Return type:

tuple[dict, DataFrame]

process_improve.batch.preprocessing.batch_dtw(batches, columns_to_align, reference_batch, settings=None)[source]#

Synchronize, via iterative DTW, with weighting.

Algorithm: Kassidas et al. (2004): https://doi.org/10.1002/aic.690440412

Parameters:

batches (dict[str, pd.DataFrame]) – Batch data, in the standard format.
columns_to_align (list) – Which columns to use during the alignment process. The others are aligned, but get no weight, and therefore do not influence the objective function.
reference_batch (str) – Which key in the batches is the reference batch to use.
settings (dict) –
Default settings are:
```
{
    "maximum_iterations": 25,  # stops here, even if not converged
    "tolerance": 0.1,          # convergence tolerance
    "robust": True,            # use robust scaling
    "show_progress": True,     # show progress
    "subsample": 1,            # use every sample
    "weighting": "quadratic",  # "quadratic" or "absolute"; see below
    "batch_weighting": "equal",  # "equal" or "huber"; see below
    "band": None,              # warping-path constraint; see below
    "interpolate_time_axis_maximum": 100,  # resample time axis to this scale
    "interpolate_time_axis_delta": 1,      # resolution of the resampled axis
    "interpolate_method": "cubic",         # any scipy.interpolate.interp1d method
}
```
The default settings resample the time axis to 100 points, starting at 0 and ending at 99, so each point is one percent of the batch’s duration however long the batch actually ran. Lower the delta for a finer axis (0.5 gives 200 points, 0.25 gives 400) or change the maximum for a different scale. The delta no longer has to divide the maximum exactly: values such as 0.3 or 7 used to fail an assertion.

weighting selects how a variable’s deviation from the average trajectory is accumulated before the weight is taken as its reciprocal:

"quadratic" (the default)
The sum of squared deviations, as in Kassidas et al. The reciprocal is then an inverse-variance (precision) weight, which is what the weighted distance in distance_matrix() expects: that distance is a Mahalanobis form, quadratic in the deviations.

"absolute"
The sum of absolute deviations. Less sensitive to a single badly aligned batch, but the reciprocal is no longer a precision, so the weighted distance loses its Mahalanobis reading. It does not simply flatten the weighting: on the bundled dryer data the ratio of largest to smallest weight rose from 2.8 to 6.0 and the iteration count from 2 to 3, so both the fixed point and the path to it differ. Offered for comparison; it is not the published method, and the effect on your own data should be measured rather than assumed.

batch_weighting decides how much each batch contributes to the variable weights. Under "equal" (the default) every batch counts the same, so one badly aligned batch inflates the summed deviation of whichever variables it misfits and depresses their weights for every other batch. "huber" weights each batch by Huber’s function applied to the robust z-score of its normalized_distance, against the median and MAD of the batch set: weight 1 inside a cutoff of 1.345, falling off as 1 / |z| beyond it, then rescaled to average 1.0.

Huber rather than a redescending function because it never reaches zero. A downweighted batch pulls the average trajectory away from itself, so it looks worse on the next iteration; a weight that could reach zero would make that a one-way door. The weights are recomputed from scratch each iteration and floored, so a batch that recovers is counted again.

band constrains the warping path: the reference rows each test sample may map to. None (the default) places no constraint. Pass an (n_test, 2) array of half-open row bounds, or a callable of (n_test, n_ref) returning one, since the two lengths differ from batch to batch and are not known until each pair is aligned:
```
from process_improve.batch.alignment_helpers import sakoe_chiba, itakura

settings = {"band": sakoe_chiba(window=0.1)}   # 10% of the batch duration
settings = {"band": itakura(max_slope=2.0)}
```
A constraint speeds up the dynamic programme, from O(n_ref * n_test) to O(window * n_test), which matters because every batch is re-aligned on every iteration. It also changes the result: a corridor that excludes the true warp changes the aligned trajectories, so the iterated average converges to a different fixed point. Widen it until the alignment stops changing.

Returns:

dict – Various outputs relevant to the alignment, keyed by scale_df, aligned_batch_objects, aligned_batch_dfdict, last_average_batch, weight_history and distances.

distances is a DataFrame indexed by batch identifier, with the Distance and Normalized distance of each batch to the reference on the final iteration. Normalized distance divides by the summed path length, so it is comparable across batches of unequal duration. Use it to see which batches aligned poorly, for instance outputs["distances"]["Normalized distance"] .nlargest(5).
Notation
——–
I = number of batches (index = i)
i = index for the batches
J = number of tags (columns in each batch)
j = index for the tags
k = index into the rows of each batch, the samples (0 … k … K_i)

Return type:

process_improve.batch.preprocessing.resample_to_reference(batches, columns_to_align, reference_batch, settings=None)[source]#

Resamples all batches (only the columns_to_align) to the duration of batch with identifier reference.

Parameters:

batches (dict[str, pd.DataFrame]) – Batch data, in the standard format.
columns_to_align (list) – Which columns to use. Others are ignored.
reference_batch (str) – Which key in the batches is the reference batch.
settings (dict, optional) – [description], by default None

Returns:

Batch data, in the standard format.

Return type:

process_improve.batch.preprocessing.find_average_length(batches, settings=None)[source]#

Find the batch in batches with the average length.

Parameters:

batches (dict[str, pd.DataFrame]) – Batch data, in the standard format.

settings (dict) –

Default settings are:

{"robust": True}  # use robust (median) average batch length

Return type:

One of the dictionary keys from batches.

process_improve.batch.preprocessing.find_reference_batch(batches, columns_to_align, settings=None)[source]#

Find a reference batch. Assumes NO missing data.

Starts with the average duration batch; resamples (simple interpolation) of all batches to that duration. Unfolds that resampled data. Does PCA on the wide, unfolded data. Fits, by default, 4 components. Excludes all batches with Hotelling’s T2 > 90% limit. Refits PCA with 4 components. Finds the batch which has the multivariate combination of scores which are the smallest (i.e. closest to the model center) and ensures this batch has SPE < 50% of the model limit.

Parameters:

batches (dict[str, pd.DataFrame]) – Batch data, in the standard format.
columns_to_align (list) – Which columns to use. Others are ignored.

settings (dict, optional) –

Default settings are:

{
    "robust": True,                    # use robust scaling
    "subsample": 1,                    # use every sample
    "method": "pca_most_average",      # most average batch from a crude PCA
    "n_components": 4,
    "number_of_reference_batches": 1,  # only a single batch returned
}

Returns:

When settings["number_of_reference_batches"] == 1 (the default), a single dictionary key from batches is returned. When more than one reference batch is requested, a list of that many keys is returned, ordered from most to least central in the PCA model.

Return type:

str or list[str]

process_improve.batch.preprocessing.unfold_blocks(blocks, *, initial_conditions=None, group_by_batch=False)[source]#

Unfold several aligned batch blocks batchwise, ready for a multi-block model (#193).

dict_to_wide() already unfolds one block of aligned batches into a one-row-per-batch matrix. What it cannot do is keep several blocks side by side and separate, which is what fit() and fit() want: they take a dict[str, pd.DataFrame] and preprocess each block on its own.

BatchPCA and BatchPLS unfold too, but they concatenate the initial-conditions block onto the trajectories to make a single wide frame, because the model underneath them is single-block. Here the blocks stay apart, so a block’s own variance decides its weight in the fit rather than its column count deciding it by accident.

Parameters:

blocks (dict[str, dict]) – One entry per block. Keys are block names, carried through to the result. Values are standard batch-data dictionaries: keys are batch identifiers, values are per-batch dataframes with identical numeric columns and the same number of rows within a block. Blocks may have different numbers of columns and different trajectory lengths from one another; they only have to describe the same batches.
initial_conditions (pd.DataFrame, optional) – One row per batch, indexed by batch identifier: measurements taken before the batch ran, which have no time axis. Added as its own block under the name "initial_conditions", not glued onto a trajectory block.
group_by_batch (bool, optional) – Passed to dict_to_wide() for every block. False (default) orders each block’s columns (tag, sequence); True swaps them to (sequence, tag).

Returns:

One wide dataframe per block, every one sharing the same row index in the same order, so row i is the same batch in every block.

Return type:

dict[str, pd.DataFrame]

Raises:

ValueError – If blocks is empty, if a block name collides with the initial_conditions block, or if the blocks do not all cover exactly the same batches.

Examples

>>> wide = unfold_blocks({"spectra": spectra, "process": process})
>>> MBPCA(n_components=2).fit(wide)

class process_improve.batch.plotting.MultiTagPlotSettings(*, nrows=1, ncols=0, x_axis_label='Time, grouped per tag', title='', show_legend=True, mode='lines', html_image_height=900, html_aspect_ratio_w_over_h=1.7777777777777777, default_line_width=2, colour_map=<function husl_palette>, animate=False, animate_batches_to_highlight=<factory>, animate_show_slider=True, animate_show_pause=True, animate_slider_prefix='Index: ', animate_slider_vertical_offset=-0.3, animate_line_width=4, animate_n_frames=None, animate_framerate_milliseconds=0)[source]#

Bases: BaseModel

Settings for plot_multitags().

All fields have sensible defaults; pass a plain dict of overrides to plot_multitags(..., settings=...).

Parameters:

nrows (int)
ncols (int)
x_axis_label (str)
title (str)
show_legend (bool)
mode (str)
html_image_height (int)
html_aspect_ratio_w_over_h (float)
default_line_width (float)
colour_map (Callable)
animate (bool)
animate_batches_to_highlight (list)
animate_show_slider (bool)
animate_show_pause (bool)
animate_slider_prefix (str)
animate_slider_vertical_offset (float)
animate_line_width (float)
animate_n_frames (int | None)
animate_framerate_milliseconds (int)

model_config = {'arbitrary_types_allowed': True}#: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

nrows: int#

ncols: int#

x_axis_label: str#

title: str#

show_legend: bool#

mode: str#

html_image_height: int#

html_aspect_ratio_w_over_h: float#

default_line_width: float#

colour_map: Callable#

animate: bool#

animate_batches_to_highlight: list#

animate_show_slider: bool#

animate_show_pause: bool#

animate_slider_prefix: str#

animate_slider_vertical_offset: float#

animate_line_width: float#

animate_n_frames: int | None#

animate_framerate_milliseconds: int#

process_improve.batch.plotting.get_rgba_from_triplet(incolour, alpha=1, as_string=False)[source]#

Convert the input colour triplet (list) to a Plotly rgba(r,g,b,a) string if as_string is True. If False it will return the list of 3 integer RGB values.

E.g. [0.9677975592919913, 0.44127456009157356, 0.5358103155058701] -> ‘rgba(246,112,136,1)’

Parameters:

incolour (Sequence[float])
alpha (float)
as_string (bool)

Return type:

str | list

process_improve.batch.plotting.plot_to_HTML(filename, fig)[source]#

Export a Plotly figure to an HTML file.

Parameters:

filename (str)
fig (dict)

Return type:

str

process_improve.batch.plotting.plot_all_batches_per_tag(df_dict, tag, tag_y2=None, time_column=None, extra_info='', batches_to_highlight=None, x_axis_label='Time [sequence order]', highlight_width=5, html_image_height=900, html_aspect_ratio_w_over_h=1.7777777777777777, y1_limits=(None, None), y2_limits=(None, None), mode='lines')[source]#

Plot a particular tag over all batches in the given dataframe df.

Parameters:

df_dict (dict) – Standard data format for batches.
tag (str) – Which tag to plot? [on the y1 (left) axis]
tag_y2 (str, optional) – Which tag to plot? [on the y2 (right) axis] Tag will be plotted with different scaling on the secondary axis, to allow time-series comparisons to be easier.
time_column (str, optional) – Which tag on the x-axis. If not specified, creates sequential integers, starting from 0 if left as the default, None.
extra_info (str, optional) – Used in the plot title to add any extra details, by default “”
batches_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier. For example:
```
batches_to_highlight = {'{"width": 2, "color": "rgba(255,0,0,0.5)"}': redlist}
```
will plot the batch identifiers in redlist with that colour and linewidth.
x_axis_label (str, optional) – String label for the x-axis, by default “Time [sequence order]”
highlight_width (int, optional) – The width of the highlighted lines; default = 5.
html_image_height (int, optional) – HTML image output height, by default 900
html_aspect_ratio_w_over_h (float, optional) – HTML image aspect ratio: 16/9 (therefore the default width will be 1600 px)
y1_limits (tuple, optional) – Axis limits enforced on the y1 (left) axis. Default is (None, None) which means the data themselves are used to determine the limits. Specify BOTH limits. Plotly requires (at the moment plotly/plotly.js#400) that you specify both. Order: (low limit, high limit)
y2_limits (tuple, optional) – Axis limits enforced on the y2 (right) axis. Default is (None, None) which means the data themselves are used to determine the limits. Specify BOTH limits. Plotly requires (at the moment plotly/plotly.js#400) that you specify both.
mode (str, optional) – Plotly trace draw mode, by default “lines”. Use “lines+markers” to also show a marker at each data point, or “markers” for markers only.

Returns:

Standard Plotly fig object (dictionary-like).

Return type:

go.Figure

process_improve.batch.plotting.colours_per_batch_id(batch_ids, batches_to_highlight, default_line_width, use_default_colour=False, colour_map=None)[source]#

Return a colour to use for each trace in the plot. A dictionary: keys are batch ids, and the value is a colour and line width setting for Plotly.

use_default_colour: bool: If True, then the default colour is used (grey: 0.5, 0.5, 0.5)

Parameters:

batch_ids (list)
batches_to_highlight (dict)
default_line_width (float)
use_default_colour (bool)
colour_map (Callable | None)

Return type:

dict[Any, dict]

process_improve.batch.plotting.plot_multitags(df_dict, batch_list=None, tag_list=None, time_column=None, batches_to_highlight=None, settings=None, fig=None)[source]#

Plot all the tags for a batch; or a subset of tags, if specified in tag_list.

Parameters:

df_dict (dict) – Standard data format for batches.
batch_list (list [default: None, will plot all batches in df_dict]) – Which batches to plot; if provided, must be a list of valid keys into df_dict.
tag_list (list [default: None, will plot all tags in the dataframes]) – Which tags to plot; tags will also be plotted in this order, or in the order of the first dataframe if not specified.
time_column (str, optional) – Which tag on the x-axis. If not specified, creates sequential integers, starting from 0 if left as the default, None.
batches_to_highlight (dict, optional) –
Keys are JSON strings parseable by json.loads into a Plotly line specifier. For example:
```
batches_to_highlight = {'{"width": 2, "color": "rgba(255,0,0,0.5)"}': redlist}
```
will plot the batch identifiers in redlist with that colour and linewidth.

settings (dict) –

Default settings:

{
    "nrows": 1,                             # int: number of subplot rows
    "ncols": None,                          # int or None: columns (None = auto)
    "x_axis_label": "Time, grouped per tag",# str: x-axis label
    "title": "",                            # str: overall plot title
    "show_legend": True,                    # bool: show legend
    "mode": "lines",                        # str: Plotly trace mode
                                            #      e.g. "lines+markers"
    "html_image_height": 900,               # int: image height in pixels
    "html_aspect_ratio_w_over_h": 16/9,     # float: width as ratio of height
}

fig (go.Figure) – If supplied, uses the existing Plotly figure to draw in.

Return type:

Figure

process_improve.batch.plotting.generate_one_frame(df_dict, tag_list, fig, up_to_index, time_column, batch_ids_to_animate, animation_colour_assignment, show_legend=False, hovertemplate='', max_columns=0, mode='lines')[source]#

Return a list of dictionaries.

Each entry in the list is for each subplot; in the order of the subplots. Since each subplot is a tag, we need the tag_list as input.

Parameters:

df_dict (dict)
tag_list (list)
fig (Figure)
up_to_index (int)
time_column (str | None)
batch_ids_to_animate (list)
animation_colour_assignment (dict)
show_legend (bool)
hovertemplate (str)
max_columns (int)
mode (str)

Return type:

list[Scatter]

Getting data into the required format for use with this library.

There are 3 useful ways to represent batch data.

dict: as a Python dictionary. Example:

data = {
    "batch 1": data frame with varying number of rows, but same number of columns,
    "batch 2": etc,
}

The keys are unique identifiers for each batch, such as integers or strings.

melt: as a single Pandas data frame:

data = pd.DataFrame(...)

Characteristics:

very large number of rows, for all batches stacked vertically on top of each other
some number of columns, one column per tag
one column, usually called batch_id, indicates what the batch number is for that row
another column, usually called time, indicates what the time is within that batch
typically sorted, but does not have to be

wide: as a single Pandas data frame, as for the “melted” version, but pivoted instead. These wide dataframes always have a multilevel column index to distinguish the tags from the time. This representation is only valid for aligned data. Example:

data = pd.DataFrame(...)

Characteristics:

each row is a unique batch number
the multilevel column index has level 0 = column name, level 1 = aligned time
only makes sense if the data are aligned (same number of elements in each level-1 index)

process_improve.batch.data_input.check_valid_batch_dict(in_dict, no_nan=False)[source]#

Check if the incoming dictionary of batch data is a valid dictionary of data.

Checks: 1. All batches in the dictionary have the same number of columns. 2. All columns are numeric. 3. If no_nan is True, also checks that there are no NaNs.

Parameters:

in_dict (dict) – A dictionary of batch data.
no_nan (bool) – If True, will also check that no missing values are present.

Returns:

True, if it passes the checks.

Return type:

bool

process_improve.batch.data_input.dict_to_melted(in_df, insert_batch_id_column=True, insert_sequence_column=False)[source]#

Reverse of melted_to_dict.

Parameters:

in_df (dict)
insert_batch_id_column (bool)
insert_sequence_column (bool)

Return type:

DataFrame

process_improve.batch.data_input.dict_to_wide(in_df, group_by_batch=False)[source]#

Convert aligned batch data from a dict to wide format.

Each row of the output is one batch; the columns are a 2-level ("tag", "sequence") index, so the data are only meaningful for aligned batches (every batch has the same number of samples).

Parameters:

in_df (dict) – Standard batch-data dictionary: keys are batch identifiers, values are per-batch dataframes with identical columns.
group_by_batch (bool, optional) –
Controls the ordering of the hierarchical column index.
- False (default): columns are ordered (tag, sequence), so all time samples for a tag are grouped together, side-by-side.
- True: the levels are swapped to (sequence, tag), so all tags for a given time sample are grouped together.

Returns:

Wide-format dataframe, one row per batch, with a 2-level column index.

Return type:

pd.DataFrame

process_improve.batch.data_input.melted_to_dict(in_df, batch_id_col)[source]#

Load a “melted” data set, where one of the columns is the batch_id_col. The data are grouped along the unique values of batch_id_col, and each group is stored in a dictionary. The dictionary keys are the batch identifier, and the corresponding value is a Pandas dataframe of the batch data for that batch.

Parameters:

in_df (DataFrame)
batch_id_col (str)

Return type: