Workbench#
The dataset workbench: refine a table step by step, and keep the Python that repeats it.
A Recipe is a source file and a list of steps (drop, keep or rename columns,
drop or filter rows, add a computed column, transform a column, read text as numbers,
identify the rows). It applies its steps, writes a script that repeats them with pandas
alone, and round-trips through JSON.
A Variant is a model (PCA or PLS) over a recipe’s table. It leaves its
excluded rows out of the fit and projects them through it, and its Fit
explains why a group of rows differs from the rest (Fit.explain_group()).
Variant.cross_validate() scores every number of components on values the model
did not see (CrossValidation), for choosing how many to keep.
A Project keeps the source file, its recipe, the variants and the reader’s
saved selections in one zip file, and refits and checks every variant on opening.
Recipes#
A recipe: the steps that refine a source table, and the Python that repeats them.
A Recipe is a source (a file and how to read it) and a list of steps. It is
the source of truth for the table an analysis uses: Recipe.apply() runs the
steps, Recipe.to_python() writes a script that does the same with pandas and
numpy alone, and Recipe.to_dict() / Recipe.from_dict() round-trip it
through JSON. Each step’s apply and code perform the same operations in the
same order, so the script rebuilds the table bit for bit.
A recipe can arrive in someone else’s project file, so Recipe.from_dict() checks
every field, an expression is parsed and never evaluated as Python, and every name,
row id and reason reaches the script only as a quoted literal or a one-line comment.
- process_improve.workbench.recipe.READ_OPTIONS: dict[str, frozenset[str]] = {'csv': frozenset({'decimal', 'encoding', 'header', 'index_col', 'na_values', 'sep', 'skiprows'}), 'excel': frozenset({'header', 'index_col', 'na_values', 'sheet_name', 'skiprows'})}#
The read options a source may carry, for CSV and Excel files.
- process_improve.workbench.recipe.TRANSFORMS: dict[str, tuple[str, float | None]] = {'log': ('log({x} + {c})', 0.0), 'log10': ('log10({x} + {c})', 0.0), 'logit': ('log({x} / (1 - {x}))', None), 'neglog': ('sign({x}) * log1p(abs({x}))', None), 'power': ('sign({x}) * abs({x}) ** {c}', 0.5), 'reciprocal': ('1 / {x}', None), 'sqrt': ('sqrt({x})', None)}#
The transforms
Transformoffers, as expressions of the columnxand the constantc.
- class process_improve.workbench.recipe.Step[source]#
Bases:
objectOne step of a recipe. Subclasses set
kindand implement the four methods.
- class process_improve.workbench.recipe.DropColumns(columns)[source]#
Bases:
StepRemove columns.
- class process_improve.workbench.recipe.KeepColumns(columns)[source]#
Bases:
StepKeep only these columns, in this order.
- class process_improve.workbench.recipe.RenameColumns(pairs)[source]#
Bases:
StepRename columns, given as
(old, new)pairs.
- class process_improve.workbench.recipe.DropRows(rows, reason='')[source]#
Bases:
StepRemove rows by their id (index label), with the reason.
- class process_improve.workbench.recipe.FilterRows(expression)[source]#
Bases:
StepKeep the rows where an expression is true, for example
Tmax1 > 290.- Parameters:
expression (str)
- class process_improve.workbench.recipe.AddColumn(name, expression)[source]#
Bases:
StepAdd (or replace) a column computed from an expression, for example
Tout1 - Tin.
- class process_improve.workbench.recipe.Transform(column, function, constant=None, new_name=None)[source]#
Bases:
StepTransform a column, in place or into a new column: one of
TRANSFORMS.constantis the offset oflogandlog10(log(x + c), default 0) and the exponent ofpower(default 0.5, a signed square root).- property expression: str#
The transform as an expression of the column, which is what it applies and writes.
- class process_improve.workbench.recipe.ToNumeric(columns)[source]#
Bases:
StepRead text columns as numbers; a value that is not a number becomes missing.
- class process_improve.workbench.recipe.SetIndex(column)[source]#
Bases:
StepMake a column the row ids.
- Parameters:
column (str)
- process_improve.workbench.recipe.step_from_dict(data)[source]#
Rebuild any step from its
Step.to_dict()output.
- class process_improve.workbench.recipe.SourceFile(name, format='csv', options=<factory>, sha256=None)[source]#
Bases:
objectWhere a recipe’s table comes from: a file’s name, format, read options and digest.
- Parameters:
name (str) – The file name, as the script will look for it next to itself.
format ({"csv", "excel"})
options (dict) – Options for
pandas.read_csv()orpandas.read_excel(), fromREAD_OPTIONS.sha256 (str, optional) – The file’s SHA-256 digest. The script checks it before reading, so it refuses a different file of the same name.
- classmethod of(name, content, format='csv', **options)[source]#
Describe a file from its name and content, recording its digest.
- Parameters:
- Return type:
- class process_improve.workbench.recipe.Recipe(source, steps=())[source]#
Bases:
objectA source and the steps that refine it into the table an analysis uses.
Recipes are immutable:
then()returns a longer recipe, so the previous one is the undo.Examples
>>> recipe = Recipe(SourceFile("LDPE.csv", options={"index_col": 0})) >>> recipe = recipe.then(DropColumns(("Tcin1", "Tcin2")), AddColumn("dT", "Tout1 - Tin")) >>> table = recipe.apply(pd.read_csv("LDPE.csv", index_col=0)) >>> print(recipe.to_python())
- Parameters:
source (SourceFile)
- apply(df)[source]#
Run every step on
df(the source’s table), which is not modified.- Raises:
ValueError – Naming the step that failed and why.
- Parameters:
df (DataFrame)
- Return type:
Model variants#
A model variant over a recipe’s table: what is modelled, and which rows it leaves out.
A Variant names the model (PCA or PLS), its number of components, the X and
Y columns, and the rows it excludes. Each block is mean-centred and scaled to unit
variance (MCUVScaler) on the fitted rows.
Excluding is not dropping: an excluded row is left out of the fit (its centring and
scaling too), then projected through the fitted model, so its scores, Hotelling’s T²
and SPE are still reported and it can still be drawn. That is how a reader sees where
a suspect batch would fall in a model that never saw it.
Variant.fit() returns a Fit, and Fit.explain_group() answers the
question a cluster on a score plot asks: what do these rows have in common that the
others do not? Variant.cross_validate() answers the one before it: how many
components does the data support?
- process_improve.workbench.variant.MODELS: dict[str, type[PCA | PLS]] = {'PCA': <class 'process_improve.multivariate._pca.PCA'>, 'PLS': <class 'process_improve.multivariate._pls.PLS'>}#
The models a variant can be, by the name a project file stores.
- class process_improve.workbench.variant.Variant(kind='PCA', n_components=2, x_columns=(), y_columns=(), excluded=(), name='Model 1')[source]#
Bases:
objectA model over a table: its kind, size, blocks, scaling, and excluded rows.
- Parameters:
kind ({"PCA", "PLS"})
n_components (int)
x_columns (tuple of str) – The X block. Empty takes every numeric column that is not in
y_columns.y_columns (tuple of str) – The Y block; PLS needs at least one column, PCA none.
excluded (tuple of row ids) – Rows left out of the fit (and of its centring and scaling), then projected.
name (str) – What the reader calls this variant.
- included(table)[source]#
Return the rows of
tablethe model is fitted to: every row the variant does not exclude.- Raises:
ValueError – If an excluded row is not in the table, or too few rows remain to fit
n_componentscomponents.- Parameters:
table (DataFrame)
- Return type:
- fit(table)[source]#
Fit the variant to
table, then project its excluded rows through the model.- Raises:
ValueError – If an excluded row is not in the table, or too few rows remain to fit
n_componentscomponents.- Parameters:
table (DataFrame)
- Return type:
- cross_validate(table, *, random_state=0)[source]#
Cross-validate every number of components on the fitted rows, to choose
n_components.The library’s own selectors do the work, with their defaults: PCA holds out single cells and predicts each from a model that never saw it (
PCA.select_n_components(), element-wise k-fold), and PLS holds out whole rows, in shuffled folds repeated ten times (PLS.select_n_components()). Both fit the centring and scaling inside every fold, so they are given the raw blocks. The answer does not depend on this variant’sn_components.- Parameters:
- Return type:
- class process_improve.workbench.variant.Fit(variant, table, model, x_scaled, y_scaled, scalers)[source]#
Bases:
objectA fitted variant: the model, and every row’s scores, T² and SPE, the excluded rows projected.
- Parameters:
variant (Variant)
table (DataFrame)
x_scaled (DataFrame)
y_scaled (DataFrame | None)
scalers (tuple[MCUVScaler, MCUVScaler | None])
- rows()[source]#
Return one row per table row: its scores, Hotelling’s T² and SPE at all components, and
excluded.A fitted row carries the values the fit computed; an excluded row the values of projecting it through the model, its missing cells estimated from the observed ones (
PCA.project()).- Return type:
- r2x()[source]#
Return the fraction of the fitted X block’s variance each component explains, in order.
- limits(conf_level=0.95)[source]#
Return the T² and SPE limits at
conf_level."T2"judges the rows the model was fitted to, by the beta law, and"T2 new rows"the rows it left out or never saw, by the F law (#978).
- explain_group(group, reference=None, components=None, top=5)[source]#
Explain why
groupsits where it does: which variables move its average score away from the reference’s.- Parameters:
group (sequence of row ids) – The rows of interest, for example a lasso on a score plot. Excluded rows may be part of it.
reference (sequence of row ids, optional) – The rows to compare against. Default: every fitted row outside the group.
components (sequence of int, optional) – 1-based components to explain; default all.
top (int, default 5) – How many variables
GroupExplanation.topnames.
- Return type:
- class process_improve.workbench.variant.CrossValidation(variant, r2, q2, q2_se, n_components, rule, held_out, folds, random_state)[source]#
Bases:
objectHow well each number of components predicts values the model was not fitted to.
- Parameters:
- r2#
Cumulative R² of the fitted rows per number of components, indexed
1..A: R²X for PCA, R²Y for PLS. It only rises as components are added.- Type:
pd.Series
- q2#
The same quantity measured on held-out values, Q², on the same index. It falls once a component fits noise, which is what separates it from
r2.- Type:
pd.Series
- q2_se#
One standard error of
q2, from the spread over the folds.- Type:
pd.Series
- rule#
That rule:
"min"(the lowest prediction error) or"1se"(the fewest components within one standard error of it).- Type:
- held_out#
What a fold holds out: single cells (PCA) or whole rows (PLS).
- Type:
{“cells”, “rows”}
- class process_improve.workbench.variant.GroupExplanation(group, reference, contributions, top, summary, evidence)[source]#
Bases:
objectWhy a group of rows differs from a reference, variable by variable.
- Parameters:
- group, reference
- Type:
list of row ids
- contributions#
One row per variable, one column per component: each variable’s share of the difference between the group’s average score and the reference’s. A column sums to that difference. Rows are ordered by their largest absolute share.
- Type:
pd.DataFrame
- summary#
Per variable, in the same order, the group’s and the reference’s means in the table’s own units, their difference, and the difference in units of the reference’s standard deviation.
- Type:
pd.DataFrame
- evidence#
The raw values of the top variables for every row, in table order, with
in_groupandin_referenceflags: what to draw beside the bars.- Type:
pd.DataFrame
Project files#
The project file: a source file, its recipe, its model variants and saved selections, in one zip.
Member |
Holds |
|---|---|
|
the format and library versions, the recipe, the variants, the saved selections |
|
the source file’s own bytes, exactly as read |
|
the recipe as a script, to run next to the source file |
|
variant |
Opening a project executes nothing from it: the manifest is JSON, checked field by
field; the source is read as the recipe says (CSV or Excel); the arrays are read with
numpy.load(allow_pickle=False). recipe.py is written for the reader and never
read back. Project.open() refits every variant from the source and the recipe and
compares the result with the stored arrays, so a project that no longer reproduces,
because the library or the file changed, says so instead of being trusted.
- process_improve.workbench.project.FORMAT = 'process-improve-project'#
What
manifest.jsoncalls this format, and the version of it this module writes.
- process_improve.workbench.project.FORMAT_VERSION = 1#
What
manifest.jsoncalls this format, and the version of it this module writes.
- process_improve.workbench.project.MAX_UNPACKED_BYTES = 536870912#
The largest project this module opens, unpacked, and the most members it reads.
- process_improve.workbench.project.MAX_MEMBERS = 2000#
The largest project this module opens, unpacked, and the most members it reads.
- process_improve.workbench.project.ARRAYS = ('scores', 'loadings', 'T2', 'SPE')#
The arrays stored for each variant, by the name of the member that holds them.
- class process_improve.workbench.project.Check(variant, array, max_difference, ok)[source]#
Bases:
objectHow one stored array compares with the same array refitted on opening.
- class process_improve.workbench.project.Opened(project, table, fits, checks, library)[source]#
Bases:
objectA project opened: its table, every variant fitted, and how each fit compares with the stored one.
- Parameters:
- class process_improve.workbench.project.Project(recipe, source, variants=(), selections=<factory>)[source]#
Bases:
objectA source file’s content, its recipe, the model variants over it, and the reader’s saved selections.
- Parameters:
- table()[source]#
Return the recipe’s table, read from the source and refined by every step.
- Return type:
pd.DataFrame
- to_bytes()[source]#
Return the project as a zip file, each variant fitted so its arrays are stored with it.
- Return type:
- classmethod from_bytes(content)[source]#
Read a project from its zip file’s content, checking every field; nothing in it is executed.
Expressions#
The expression language of a recipe’s computed columns and row filters.
An expression is parsed with ast and walked, never evaluated as Python, so an
expression read from someone else’s project file is data. The same walk gives the
value (evaluate()) and the pandas code that computes it (to_code()), so the
code a recipe writes does exactly what the recipe did, operation for operation.
Accepted: numbers, column names, + - * / **, one comparison (< <= > >= == !=)
against a number or a quoted text, &, | and ~ to combine conditions, and
the functions in FUNCTIONS. and, or and not combine conditions as a
reader expects (Tmax1 > 290 and Mn < 30000) and are written as pandas’ &, |
and ~. A column name that is not a Python identifier is
written in back-ticks, as in pandas.DataFrame.query(): ` `Real coffee` * 2 `.
- process_improve.workbench._expressions.MAX_EXPRESSION_LENGTH = 500#
Longest expression accepted, which also bounds the depth of the parse.
- process_improve.workbench._expressions.FUNCTIONS: dict[str, tuple[Callable[[Any], Any], str]] = {'abs': (<ufunc 'absolute'>, 'np.abs'), 'exp': (<ufunc 'exp'>, 'np.exp'), 'log': (<ufunc 'log'>, 'np.log'), 'log10': (<ufunc 'log10'>, 'np.log10'), 'log1p': (<ufunc 'log1p'>, 'np.log1p'), 'sign': (<ufunc 'sign'>, 'np.sign'), 'sqrt': (<ufunc 'sqrt'>, 'np.sqrt')}#
The functions an expression may call, each with one argument, and the code that calls it.
- class process_improve.workbench._expressions.Expression(text, columns)[source]#
Bases:
objectA parsed, checked expression over the columns of a table.
- Parameters:
- Raises:
ValueError – If the text is too long, does not parse, uses syntax outside the language, or names a column that is not in
columns.