Workbench#

The dataset workbench: refine a table step by step, and keep the Python that repeats it.

A Recipe is a source file and a list of steps (drop, keep or rename columns, drop or filter rows, add a computed column, transform a column, read text as numbers, identify the rows). It applies its steps, writes a script that repeats them with pandas alone, and round-trips through JSON.

A Variant is a model (PCA or PLS) over a recipe’s table. It leaves its excluded rows out of the fit and projects them through it, and its Fit explains why a group of rows differs from the rest (Fit.explain_group()). Variant.cross_validate() scores every number of components on values the model did not see (CrossValidation), for choosing how many to keep.

A Project keeps the source file, its recipe, the variants and the reader’s saved selections in one zip file, and refits and checks every variant on opening.

Recipes#

A recipe: the steps that refine a source table, and the Python that repeats them.

A Recipe is a source (a file and how to read it) and a list of steps. It is the source of truth for the table an analysis uses: Recipe.apply() runs the steps, Recipe.to_python() writes a script that does the same with pandas and numpy alone, and Recipe.to_dict() / Recipe.from_dict() round-trip it through JSON. Each step’s apply and code perform the same operations in the same order, so the script rebuilds the table bit for bit.

A recipe can arrive in someone else’s project file, so Recipe.from_dict() checks every field, an expression is parsed and never evaluated as Python, and every name, row id and reason reaches the script only as a quoted literal or a one-line comment.

process_improve.workbench.recipe.READ_OPTIONS: dict[str, frozenset[str]] = {'csv': frozenset({'decimal', 'encoding', 'header', 'index_col', 'na_values', 'sep', 'skiprows'}), 'excel': frozenset({'header', 'index_col', 'na_values', 'sheet_name', 'skiprows'})}#

The read options a source may carry, for CSV and Excel files.

process_improve.workbench.recipe.TRANSFORMS: dict[str, tuple[str, float | None]] = {'log': ('log({x} + {c})', 0.0), 'log10': ('log10({x} + {c})', 0.0), 'logit': ('log({x} / (1 - {x}))', None), 'neglog': ('sign({x}) * log1p(abs({x}))', None), 'power': ('sign({x}) * abs({x}) ** {c}', 0.5), 'reciprocal': ('1 / {x}', None), 'sqrt': ('sqrt({x})', None)}#

The transforms Transform offers, as expressions of the column x and the constant c.

class process_improve.workbench.recipe.Step[source]#

Bases: object

One step of a recipe. Subclasses set kind and implement the four methods.

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

to_dict()[source]#

Return the step as JSON-ready data, with its kind.

Return type:

dict[str, Any]

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

Step

class process_improve.workbench.recipe.DropColumns(columns)[source]#

Bases: Step

Remove columns.

Parameters:

columns (tuple[str, ...])

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

DropColumns

class process_improve.workbench.recipe.KeepColumns(columns)[source]#

Bases: Step

Keep only these columns, in this order.

Parameters:

columns (tuple[str, ...])

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

KeepColumns

class process_improve.workbench.recipe.RenameColumns(pairs)[source]#

Bases: Step

Rename columns, given as (old, new) pairs.

Parameters:

pairs (tuple[tuple[str, str], ...])

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

to_dict()[source]#

Return the step as JSON-ready data, with its kind.

Return type:

dict[str, Any]

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

RenameColumns

class process_improve.workbench.recipe.DropRows(rows, reason='')[source]#

Bases: Step

Remove rows by their id (index label), with the reason.

Parameters:
apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

DropRows

class process_improve.workbench.recipe.FilterRows(expression)[source]#

Bases: Step

Keep the rows where an expression is true, for example Tmax1 > 290.

Parameters:

expression (str)

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

FilterRows

class process_improve.workbench.recipe.AddColumn(name, expression)[source]#

Bases: Step

Add (or replace) a column computed from an expression, for example Tout1 - Tin.

Parameters:
apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

AddColumn

class process_improve.workbench.recipe.Transform(column, function, constant=None, new_name=None)[source]#

Bases: Step

Transform a column, in place or into a new column: one of TRANSFORMS.

constant is the offset of log and log10 (log(x + c), default 0) and the exponent of power (default 0.5, a signed square root).

Parameters:
  • column (str)

  • function (str)

  • constant (float | None)

  • new_name (str | None)

property expression: str#

The transform as an expression of the column, which is what it applies and writes.

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

Transform

class process_improve.workbench.recipe.ToNumeric(columns)[source]#

Bases: Step

Read text columns as numbers; a value that is not a number becomes missing.

Parameters:

columns (tuple[str, ...])

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

ToNumeric

class process_improve.workbench.recipe.SetIndex(column)[source]#

Bases: Step

Make a column the row ids.

Parameters:

column (str)

apply(df)[source]#

Return the table after this step; df is never modified.

Parameters:

df (DataFrame)

Return type:

DataFrame

code()[source]#

Return the lines of Python that perform this step on the table df.

Return type:

list[str]

describe()[source]#

Return the step in a sentence, for the recipe’s history.

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a step from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

SetIndex

process_improve.workbench.recipe.step_from_dict(data)[source]#

Rebuild any step from its Step.to_dict() output.

Parameters:

data (Mapping[str, Any])

Return type:

Step

class process_improve.workbench.recipe.SourceFile(name, format='csv', options=<factory>, sha256=None)[source]#

Bases: object

Where a recipe’s table comes from: a file’s name, format, read options and digest.

Parameters:
  • name (str) – The file name, as the script will look for it next to itself.

  • format ({"csv", "excel"})

  • options (dict) – Options for pandas.read_csv() or pandas.read_excel(), from READ_OPTIONS.

  • sha256 (str, optional) – The file’s SHA-256 digest. The script checks it before reading, so it refuses a different file of the same name.

classmethod of(name, content, format='csv', **options)[source]#

Describe a file from its name and content, recording its digest.

Parameters:
Return type:

SourceFile

read(content)[source]#

Read the table from the file’s content (bytes) or from a path.

Parameters:

content (bytes | str | Path)

Return type:

DataFrame

code()[source]#

Return the lines that check and read the file into df.

Return type:

list[str]

to_dict()[source]#

Return the source as JSON-ready data.

Return type:

dict[str, Any]

classmethod from_dict(data)[source]#

Rebuild a source from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

SourceFile

class process_improve.workbench.recipe.Recipe(source, steps=())[source]#

Bases: object

A source and the steps that refine it into the table an analysis uses.

Recipes are immutable: then() returns a longer recipe, so the previous one is the undo.

Examples

>>> recipe = Recipe(SourceFile("LDPE.csv", options={"index_col": 0}))
>>> recipe = recipe.then(DropColumns(("Tcin1", "Tcin2")), AddColumn("dT", "Tout1 - Tin"))
>>> table = recipe.apply(pd.read_csv("LDPE.csv", index_col=0))
>>> print(recipe.to_python())
Parameters:
then(*steps)[source]#

Return this recipe with steps added at the end.

Parameters:

steps (Step)

Return type:

Recipe

apply(df)[source]#

Run every step on df (the source’s table), which is not modified.

Raises:

ValueError – Naming the step that failed and why.

Parameters:

df (DataFrame)

Return type:

DataFrame

run(content)[source]#

Read the source from its content or path, then run every step.

Parameters:

content (bytes | str | Path)

Return type:

DataFrame

describe()[source]#

Return each step in a sentence, in order.

Return type:

list[str]

to_python()[source]#

Return a script that rebuilds the table with pandas and numpy alone, as df.

Return type:

str

to_dict()[source]#

Return the recipe as JSON-ready data.

Return type:

dict[str, Any]

classmethod from_dict(data)[source]#

Rebuild a recipe from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

Recipe

Model variants#

A model variant over a recipe’s table: what is modelled, and which rows it leaves out.

A Variant names the model (PCA or PLS), its number of components, the X and Y columns, and the rows it excludes. Each block is mean-centred and scaled to unit variance (MCUVScaler) on the fitted rows. Excluding is not dropping: an excluded row is left out of the fit (its centring and scaling too), then projected through the fitted model, so its scores, Hotelling’s T² and SPE are still reported and it can still be drawn. That is how a reader sees where a suspect batch would fall in a model that never saw it.

Variant.fit() returns a Fit, and Fit.explain_group() answers the question a cluster on a score plot asks: what do these rows have in common that the others do not? Variant.cross_validate() answers the one before it: how many components does the data support?

process_improve.workbench.variant.MODELS: dict[str, type[PCA | PLS]] = {'PCA': <class 'process_improve.multivariate._pca.PCA'>, 'PLS': <class 'process_improve.multivariate._pls.PLS'>}#

The models a variant can be, by the name a project file stores.

class process_improve.workbench.variant.Variant(kind='PCA', n_components=2, x_columns=(), y_columns=(), excluded=(), name='Model 1')[source]#

Bases: object

A model over a table: its kind, size, blocks, scaling, and excluded rows.

Parameters:
  • kind ({"PCA", "PLS"})

  • n_components (int)

  • x_columns (tuple of str) – The X block. Empty takes every numeric column that is not in y_columns.

  • y_columns (tuple of str) – The Y block; PLS needs at least one column, PCA none.

  • excluded (tuple of row ids) – Rows left out of the fit (and of its centring and scaling), then projected.

  • name (str) – What the reader calls this variant.

blocks(table)[source]#

Return the X block and the Y block (None for PCA) of table, every row kept.

Parameters:

table (DataFrame)

Return type:

tuple[DataFrame, DataFrame | None]

included(table)[source]#

Return the rows of table the model is fitted to: every row the variant does not exclude.

Raises:

ValueError – If an excluded row is not in the table, or too few rows remain to fit n_components components.

Parameters:

table (DataFrame)

Return type:

Index

fit(table)[source]#

Fit the variant to table, then project its excluded rows through the model.

Raises:

ValueError – If an excluded row is not in the table, or too few rows remain to fit n_components components.

Parameters:

table (DataFrame)

Return type:

Fit

cross_validate(table, *, random_state=0)[source]#

Cross-validate every number of components on the fitted rows, to choose n_components.

The library’s own selectors do the work, with their defaults: PCA holds out single cells and predicts each from a model that never saw it (PCA.select_n_components(), element-wise k-fold), and PLS holds out whole rows, in shuffled folds repeated ten times (PLS.select_n_components()). Both fit the centring and scaling inside every fold, so they are given the raw blocks. The answer does not depend on this variant’s n_components.

Parameters:
  • table (pd.DataFrame) – The recipe’s table; the variant’s excluded rows are left out, as in fit().

  • random_state (int, default 0) – Seed of the fold assignment, so that the same table gives the same answer.

Return type:

CrossValidation

to_python(frame='df')[source]#

Return the code that fits this variant to the table named frame, as a recipe’s script leaves it.

Parameters:

frame (str)

Return type:

str

to_dict()[source]#

Return the variant as JSON-ready data.

Return type:

dict[str, Any]

classmethod from_dict(data)[source]#

Rebuild a variant from to_dict()’s output, checking every field.

Parameters:

data (Mapping[str, Any])

Return type:

Variant

class process_improve.workbench.variant.Fit(variant, table, model, x_scaled, y_scaled, scalers)[source]#

Bases: object

A fitted variant: the model, and every row’s scores, T² and SPE, the excluded rows projected.

Parameters:
property excluded: Index#

The rows left out of the fit, in table order.

rows()[source]#

Return one row per table row: its scores, Hotelling’s T² and SPE at all components, and excluded.

A fitted row carries the values the fit computed; an excluded row the values of projecting it through the model, its missing cells estimated from the observed ones (PCA.project()).

Return type:

DataFrame

r2x()[source]#

Return the fraction of the fitted X block’s variance each component explains, in order.

Return type:

list[float]

limits(conf_level=0.95)[source]#

Return the T² and SPE limits at conf_level.

"T2" judges the rows the model was fitted to, by the beta law, and "T2 new rows" the rows it left out or never saw, by the F law (#978).

Parameters:

conf_level (float)

Return type:

dict[str, float]

explain_group(group, reference=None, components=None, top=5)[source]#

Explain why group sits where it does: which variables move its average score away from the reference’s.

Parameters:
  • group (sequence of row ids) – The rows of interest, for example a lasso on a score plot. Excluded rows may be part of it.

  • reference (sequence of row ids, optional) – The rows to compare against. Default: every fitted row outside the group.

  • components (sequence of int, optional) – 1-based components to explain; default all.

  • top (int, default 5) – How many variables GroupExplanation.top names.

Return type:

GroupExplanation

class process_improve.workbench.variant.CrossValidation(variant, r2, q2, q2_se, n_components, rule, held_out, folds, random_state)[source]#

Bases: object

How well each number of components predicts values the model was not fitted to.

Parameters:
variant#

The variant that was cross-validated; its n_components plays no part.

Type:

Variant

r2#

Cumulative R² of the fitted rows per number of components, indexed 1..A: R²X for PCA, R²Y for PLS. It only rises as components are added.

Type:

pd.Series

q2#

The same quantity measured on held-out values, Q², on the same index. It falls once a component fits noise, which is what separates it from r2.

Type:

pd.Series

q2_se#

One standard error of q2, from the spread over the folds.

Type:

pd.Series

n_components#

The number of components the selector’s rule recommends.

Type:

int

rule#

That rule: "min" (the lowest prediction error) or "1se" (the fewest components within one standard error of it).

Type:

str

held_out#

What a fold holds out: single cells (PCA) or whole rows (PLS).

Type:

{“cells”, “rows”}

folds#

The number of fold fits behind each number of components, over all repeats.

Type:

int

random_state#

The seed of the fold assignment.

Type:

int

to_python()[source]#

Return the code that repeats this cross-validation, after the variant’s code defines X and Y.

Return type:

str

class process_improve.workbench.variant.GroupExplanation(group, reference, contributions, top, summary, evidence)[source]#

Bases: object

Why a group of rows differs from a reference, variable by variable.

Parameters:
group, reference
Type:

list of row ids

contributions#

One row per variable, one column per component: each variable’s share of the difference between the group’s average score and the reference’s. A column sums to that difference. Rows are ordered by their largest absolute share.

Type:

pd.DataFrame

top#

The variables with the largest shares, the ones to look at first.

Type:

list of str

summary#

Per variable, in the same order, the group’s and the reference’s means in the table’s own units, their difference, and the difference in units of the reference’s standard deviation.

Type:

pd.DataFrame

evidence#

The raw values of the top variables for every row, in table order, with in_group and in_reference flags: what to draw beside the bars.

Type:

pd.DataFrame

score_difference()[source]#

Return, per component, the difference between the group’s average score and the reference’s.

Return type:

Series

Project files#

The project file: a source file, its recipe, its model variants and saved selections, in one zip.

Member

Holds

manifest.json

the format and library versions, the recipe, the variants, the saved selections

source/<file name>

the source file’s own bytes, exactly as read

recipe.py

the recipe as a script, to run next to the source file

fits/<k>/<array>.npy

variant k’s fitted scores, loadings, T² and SPE

Opening a project executes nothing from it: the manifest is JSON, checked field by field; the source is read as the recipe says (CSV or Excel); the arrays are read with numpy.load(allow_pickle=False). recipe.py is written for the reader and never read back. Project.open() refits every variant from the source and the recipe and compares the result with the stored arrays, so a project that no longer reproduces, because the library or the file changed, says so instead of being trusted.

process_improve.workbench.project.FORMAT = 'process-improve-project'#

What manifest.json calls this format, and the version of it this module writes.

process_improve.workbench.project.FORMAT_VERSION = 1#

What manifest.json calls this format, and the version of it this module writes.

process_improve.workbench.project.MAX_UNPACKED_BYTES = 536870912#

The largest project this module opens, unpacked, and the most members it reads.

process_improve.workbench.project.MAX_MEMBERS = 2000#

The largest project this module opens, unpacked, and the most members it reads.

process_improve.workbench.project.ARRAYS = ('scores', 'loadings', 'T2', 'SPE')#

The arrays stored for each variant, by the name of the member that holds them.

class process_improve.workbench.project.Check(variant, array, max_difference, ok)[source]#

Bases: object

How one stored array compares with the same array refitted on opening.

Parameters:
class process_improve.workbench.project.Opened(project, table, fits, checks, library)[source]#

Bases: object

A project opened: its table, every variant fitted, and how each fit compares with the stored one.

Parameters:
property reproduces: bool#

Whether every refitted array matches the one stored.

class process_improve.workbench.project.Project(recipe, source, variants=(), selections=<factory>)[source]#

Bases: object

A source file’s content, its recipe, the model variants over it, and the reader’s saved selections.

Parameters:
  • recipe (Recipe)

  • source (bytes) – The source file’s content. Its SHA-256 digest must match the recipe’s, when the recipe records one.

  • variants (tuple of Variant)

  • selections (dict) – Named sets of row ids, as the page saves them (a lasso kept under a name).

table()[source]#

Return the recipe’s table, read from the source and refined by every step.

Return type:

pd.DataFrame

to_bytes()[source]#

Return the project as a zip file, each variant fitted so its arrays are stored with it.

Return type:

bytes

save(path)[source]#

Write the project to path.

Parameters:

path (str | Path)

Return type:

None

classmethod from_bytes(content)[source]#

Read a project from its zip file’s content, checking every field; nothing in it is executed.

Parameters:

content (bytes)

Return type:

Project

classmethod load(path)[source]#

Read a project from path.

Parameters:

path (str | Path)

Return type:

Project

static open(content, *, rtol=1e-09)[source]#

Read a project, refit each variant, and compare every refitted array with the stored one.

Parameters:
  • content (bytes) – The project file’s content.

  • rtol (float, default 1e-9) – The largest difference, relative to the array’s largest magnitude, that still counts as the same.

Return type:

Opened

Expressions#

The expression language of a recipe’s computed columns and row filters.

An expression is parsed with ast and walked, never evaluated as Python, so an expression read from someone else’s project file is data. The same walk gives the value (evaluate()) and the pandas code that computes it (to_code()), so the code a recipe writes does exactly what the recipe did, operation for operation.

Accepted: numbers, column names, + - * / **, one comparison (< <= > >= == !=) against a number or a quoted text, &, | and ~ to combine conditions, and the functions in FUNCTIONS. and, or and not combine conditions as a reader expects (Tmax1 > 290 and Mn < 30000) and are written as pandas’ &, | and ~. A column name that is not a Python identifier is written in back-ticks, as in pandas.DataFrame.query(): ` `Real coffee` * 2 `.

process_improve.workbench._expressions.MAX_EXPRESSION_LENGTH = 500#

Longest expression accepted, which also bounds the depth of the parse.

process_improve.workbench._expressions.FUNCTIONS: dict[str, tuple[Callable[[Any], Any], str]] = {'abs': (<ufunc 'absolute'>, 'np.abs'), 'exp': (<ufunc 'exp'>, 'np.exp'), 'log': (<ufunc 'log'>, 'np.log'), 'log10': (<ufunc 'log10'>, 'np.log10'), 'log1p': (<ufunc 'log1p'>, 'np.log1p'), 'sign': (<ufunc 'sign'>, 'np.sign'), 'sqrt': (<ufunc 'sqrt'>, 'np.sqrt')}#

The functions an expression may call, each with one argument, and the code that calls it.

class process_improve.workbench._expressions.Expression(text, columns)[source]#

Bases: object

A parsed, checked expression over the columns of a table.

Parameters:
  • text (str) – The expression as typed.

  • columns (Iterable[str] or None) – The columns it may name. None takes every name in the text as a column, as typed: what writing the code of an expression already checked against its table needs.

Raises:

ValueError – If the text is too long, does not parse, uses syntax outside the language, or names a column that is not in columns.

evaluate(frame)[source]#

Return the expression’s value over frame: a Series, one value per row.

Parameters:

frame (pd.DataFrame)

Return type:

Any

to_code(frame='df')[source]#

Return the pandas code that computes the expression over the table named frame.

Every operation is parenthesised, so the code groups exactly as the parse did.

Parameters:

frame (str)

Return type:

str

process_improve.workbench._expressions.python_literal(value)[source]#

Return value as Python source: text in double quotes where it can be, numbers exactly.

Parameters:

value (object)

Return type:

str