Sensory Panel Analysis#

Descriptive panel-data analysis: validate the data, identify and optionally correct panel anomalies, then relate the panel attributes back to the product. The narrative walkthrough is in Descriptive panel data: validate, check the panel, relate to the product.

The subpackage __init__ re-exports every name below, so from process_improve.sensory import validate_descriptive works. Each object is documented once, under the module that defines it.

Kevin Dunn, 2010-2026. MIT License.

Descriptive panel-data analysis.

This subpackage provides a small, generic pipeline for descriptive panel data: validate the data, identify and (optionally) correct panel anomalies, and relate the panel attributes back to the product. For now the product is described by observational descriptors (measured covariates of products whose formulation is unknown), and the relationship is analysed by PLS as association rather than causation. A designed mode (controlled experimental runs, analysed as factor effects) is stubbed and planned for a later release.

The public entry points are validate_descriptive() and analyze_descriptive(). Agent-callable wrappers live in process_improve.sensory.tools.

Validation and ingest#

Kevin Dunn, 2010-2026. MIT License.

Schema validation for descriptive panel data.

Every analysis in this subpackage enters through validate_descriptive(), which coerces a caller-supplied table into the canonical descriptive_long schema and validates a product-covariate table alongside it. For now only the observational mode is supported: the covariate columns are measured descriptors of products whose formulation is unknown. The designed mode (covariate columns being controlled factor levels, analysed as effects) is stubbed and raises NotImplementedError; it is planned for a later release. The mode is recorded on the result and decides how process_improve.sensory.analysis.analyze_descriptive() relates the attributes back to the product.

The validated result carries a content hash and is cached, so downstream tools can refuse to run on data that has not passed validation.

process_improve.sensory.validation.DESCRIPTIVE_LONG_COLUMNS: tuple[str, ...] = ('panelist_id', 'session', 'product', 'attribute', 'replicate', 'score')#: Required columns of the descriptive_long schema, in canonical order.

class process_improve.sensory.validation.ValidationResult(ok, mode, normalized_df, covariates, warnings=<factory>, errors=<factory>, content_hash=None, stats=<factory>)[source]#

Bases: object

Outcome of validate_descriptive().

Parameters:

ok (bool)
mode (Literal['designed', 'observational'])
normalized_df (DataFrame | None)
covariates (DataFrame | None)
warnings (list[str])
errors (list[str])
content_hash (str | None)
stats (dict)

ok#

True when no blocking errors were found. Downstream analysis refuses to run when this is False.

Type:: bool

mode#

"designed" or "observational"; how the covariate table is interpreted.

Type:: str

normalized_df#

The panel data coerced to the descriptive_long schema.

Type:: pandas.DataFrame or None

covariates#

The product-covariate table, indexed by product.

Type:: pandas.DataFrame or None

warnings#

Non-blocking issues the caller should see.

Type:: list of str

errors#

Blocking issues; non-empty implies ok is False.

Type:: list of str

content_hash#

Stable hash of the normalized inputs and mode.

Type:: str or None

stats#

Summary counts (number of panelists, products, attributes, etc.).

Type:: dict

process_improve.sensory.validation.validate_descriptive(panel, covariates, mode, *, score_min=None, score_max=None, balance_warn=0.05, balance_error=0.2)[source]#

Validate panel data against the descriptive_long schema.

Parameters:

panel (pandas.DataFrame) – Long-format panel data; must contain the columns listed in DESCRIPTIVE_LONG_COLUMNS.
covariates (pandas.DataFrame) – Product-covariate table. Either has a product column or is indexed by product. In observational mode the remaining columns are measured numeric descriptors. designed mode (the columns being controlled factor levels) is not implemented yet.
mode ({"observational", "designed"}) – How the covariate table is interpreted. Only "observational" is supported for now; "designed" passes the up-front value check but then raises NotImplementedError, and is planned for a later release. Any other value raises ValueError.
score_min (float or None) – Optional inclusive bounds for the score column; out-of-range values are reported as a warning.
score_max (float or None) – Optional inclusive bounds for the score column; out-of-range values are reported as a warning.
balance_warn (float) – Missing-cell fractions (of the full panelist x product x attribute x replicate grid) above which an unbalanced-panel warning is raised. Imbalance is reported but never blocks the observational relate, which aggregates to product means; balance_error only controls the wording (badly-unbalanced vs unbalanced).
balance_error (float) – Missing-cell fractions (of the full panelist x product x attribute x replicate grid) above which an unbalanced-panel warning is raised. Imbalance is reported but never blocks the observational relate, which aggregates to product means; balance_error only controls the wording (badly-unbalanced vs unbalanced).

Returns:

See the class docstring. When ok is True the result is also stored in an in-process cache keyed by content_hash.

Return type:

ValidationResult

Examples

>>> result = validate_descriptive(panel_df, descriptors_df, mode="observational")
>>> result.ok
True

process_improve.sensory.validation.is_validated(content_hash)[source]#

Return True if content_hash refers to a cached validated result.

Parameters:: content_hash (str)
Return type:: bool

Kevin Dunn, 2010-2026. MIT License.

Deterministic reshaping of panel data into the descriptive_long schema.

Raw panel data usually arrives wide (one column per attribute) or already long. The front end (or a code sandbox) parses the spreadsheet into rows; this module performs the reshape deterministically and self-checks it, so the bulk transform never goes through an LLM’s tokens (which would risk silent mis-mapping on real-sized panels).

reshape_to_long() takes an explicit column mapping (the caller, possibly an LLM, decides which column is which) and a layout flag, melts to long when needed, and verifies a set of round-trip invariants: the grand mean, the mean per attribute, the mean per panelist, and the count of non-missing cells must be identical before and after the reshape. A mismatch (for example product and attribute columns swapped) raises rather than silently corrupting every downstream statistic.

process_improve.sensory.ingest.reshape_to_long(data, *, layout, mapping)[source]#

Reshape panel data into the descriptive_long schema, with round-trip checks.

Parameters:

data (pandas.DataFrame) – The parsed panel table (rows already read from the spreadsheet).
layout ({"long", "wide_by_attribute", "wide_by_product"}) – "long" passes through (renaming columns and canonicalising); "wide_by_attribute" melts attribute columns into rows (rows carry the product); "wide_by_product" melts product columns into rows (rows carry the attribute, i.e. one panelist-by-product matrix per attribute, stacked with an attribute label column).
mapping (dict) –
Explicit column roles:
- panelist_id (required): column name.
- session and replicate (optional): column names; default to a constant 1 when absent.
- For wide_by_attribute: product (required) and attributes (the list of attribute columns; if omitted, all non-id columns).
- For wide_by_product: attribute (required, the block-label column) and products (the list of product columns; if omitted, all non-id columns).
- For long: product, attribute and score column names.
- ignore (optional): columns to drop before reshaping (nuisance columns such as a site or batch code). When the attribute/product list is omitted, “all remaining columns” excludes these.

Returns:

long_df (pandas.DataFrame) – Data in the canonical descriptive_long schema, sample-major sorted.
checks (dict) – The round-trip invariants (grand mean, per-attribute and per-panelist max differences, cell counts) and ok.

Raises:

ValueError – If required mapping columns are missing, the data is means-only (no panelist column), or a round-trip invariant fails (which signals a wrong column mapping).

Return type:

tuple[DataFrame, dict[str, Any]]

Panel diagnostics#

Kevin Dunn, 2010-2026. MIT License.

Panel-anomaly scorecard for descriptive panel data.

Before any product conclusion is drawn, each panelist is scored on four axes:

discrimination - does the panelist separate the products (mean eta-squared of the product effect across attributes; higher is better);
agreement - does the panelist rank the products like the rest of the panel (mean correlation of the panelist’s product means with the panel’s, across attributes; higher is better);
scale use - a location shift and a spread ratio relative to the panel;
drift - association between session order and the panelist’s mean score (only when more than one session is present).

Panelists that discriminate poorly, disagree with the panel, or use the scale atypically are flagged so the caller can keep or drop them before relating the attributes to the product.

class process_improve.sensory.panel.PanelScorecard(table, flagged, reasons=<factory>)[source]#

Bases: object

Outcome of panel_scorecard().

Parameters:

table (DataFrame)
flagged (list[str])
reasons (dict[str, list[str]])

table#

One row per panelist, indexed by panelist_id, with the columns discrimination, agreement, scale_shift, scale_spread, and drift.

Type:: pandas.DataFrame

flagged#

Panelist ids flagged as anomalous.

Type:: list of str

reasons#

Maps each flagged panelist id to the list of reasons it was flagged.

Type:: dict

process_improve.sensory.panel.panel_scorecard(panel)[source]#

Score each panelist and flag anomalies.

Parameters:: panel (pandas.DataFrame) – Validated descriptive_long panel data (the normalized_df of a ValidationResult).
Returns:: See the class docstring.
Return type:: PanelScorecard

Examples

>>> card = panel_scorecard(validated.normalized_df)
>>> card.flagged
['P7']

process_improve.sensory.panel.apply_correction(panel, drop)[source]#

Return panel with the listed panelists removed.

Parameters:

panel (pandas.DataFrame) – Validated descriptive_long panel data.
drop (list of str) – Panelist ids to remove.

Returns:

The panel without the dropped panelists.

Return type:

pandas.DataFrame

Kevin Dunn, 2010-2026. MIT License.

Preconditions for panel analysis: can this attribute be modelled at all?

Every model in this subpackage assumes the attribute behaves like an intensity that assessors read off a linear scale, and that what separates assessors is how much of that scale they use. Two things break the assumption, and both of them break it quietly:

The attribute is pinned against a bound. In a region where everyone records the same value, no scaling difference is expressible, so the Mixed Assessor Model has nothing to estimate and reports the residue as disagreement. boundary_occupancy() measures how much of an attribute lives against the floor or the ceiling; detection_rate() gives the response that is appropriate instead, a probability of detection rather than an intensity.
Assessors differ in how noisy they are, not in how they scale. assessor_variance_equality() tests that directly. Grossmann et al. (2023) show that the Mixed Assessor Model reads unequal assessor variance as a scaling effect, which shifts its F-test so that real disagreement is understated. A small p-value here means the model’s scaling coefficients are measuring partly that, and the F-test should be read with the finding in mind.

All three take long-format panel data with panelist_id, product, attribute and score columns; the canonical descriptive_long schema from validate_descriptive() satisfies that.

References

Grossmann, Ellis, Hopfer and others, “The effect of unequal assessor variance on the Mixed Assessor Model”, Food Quality and Preference, 105, 104792, 2023, doi:10.1016/j.foodqual.2022.104792.

process_improve.sensory.diagnostics.boundary_occupancy(panel, lo=0.0, hi=10.0, band=0.1)[source]

Measure how much of each attribute sits against the ends of the scale.

An attribute pinned against a scale bound violates the Mixed Assessor Model’s premise that assessors compress or expand a linear scale: no scaling difference is expressible in a region where everyone records the same value. Use this before modelling to decide whether an attribute can be treated as an intensity at all, and detection_rate() when it cannot.

Floor, ceiling and exact-zero occupancy are reported separately because they are different questions. In particular, a panel whose convention is to record “not perceived” as a small positive number rather than an exact zero will look floor-pinned when it is not, and the exact_zero column is what distinguishes the two.

Parameters:

panel (pandas.DataFrame) – Long-format panel data with panelist_id, product, attribute and score columns.
lo (float, default 0.0) – Lower bound of the rating scale.
hi (float, default 10.0) – Upper bound of the rating scale.
band (float, default 0.10) – Width of the floor and ceiling bands, as a fraction of the scale range. The default counts a score as “at the floor” when it is within the bottom 10% of the scale. Must lie in [0, 0.5).

Returns:

One row per attribute, sorted by attribute, with columns:

attribute: The attribute name.
n: Number of non-missing scores.
at_floor, at_ceiling: Counts of scores inside the floor and ceiling bands.
exact_zero: Count of scores exactly equal to lo.
frac_floor, frac_ceiling, frac_exact_zero: The same three as fractions of n, which is what a keep/drop decision is actually made on.

Return type:

pandas.DataFrame

Raises:

ValueError – If a required column is missing, the panel has no rows, or the scale bounds and band are not a usable combination.

Examples

>>> occupancy = boundary_occupancy(validated.normalized_df)
>>> occupancy.query("frac_floor > 0.5")  # candidates for detection_rate instead

process_improve.sensory.diagnostics.detection_rate(panel, lo=0.0, band=0.1, hi=10.0)[source]

Report, per product and attribute, the fraction of assessments that detected it.

This is the appropriate response for an attribute that boundary_occupancy() shows is pinned against the floor: the question “how intense is it” has no answer when most assessors record nothing, but “how often is it perceived at all” does.

Warning

A detection rate is not comparable with an intensity score. It is a probability on [0, 1], it does not share the attribute’s units, and it must not be dropped into the same table, correlation matrix or PLS block as intensity-scored attributes without saying what it is. Two attributes with the same mean intensity can have very different detection rates, and vice versa.

Parameters:

panel (pandas.DataFrame) – Long-format panel data with panelist_id, product, attribute and score columns.
lo (float, default 0.0) – Lower bound of the rating scale.
band (float, default 0.10) – Width of the floor band, as a fraction of the scale range. A score strictly above lo + band * (hi - lo) counts as detected.
hi (float, default 10.0) – Upper bound of the rating scale.

Returns:

Products (rows) by attributes (columns) of detection probabilities. A product-attribute pair that nobody assessed is NaN rather than 0: “never detected” and “never asked” are different answers.

Return type:

pandas.DataFrame

Raises:

ValueError – If a required column is missing, the panel has no rows, or the scale bounds and band are not a usable combination.

Examples

>>> rates = detection_rate(validated.normalized_df)
>>> rates["burnt"].sort_values(ascending=False)

process_improve.sensory.diagnostics.assessor_variance_equality(panel)[source]

Test whether assessors are equally variable, per attribute.

The Mixed Assessor Model splits the assessor-by-product interaction into a scaling part and a disagreement part, and reads the scaling part as “this assessor uses a wider or narrower range of the scale”. Grossmann et al. (2023) show that an assessor who is simply noisier than the others loads onto that same scaling term, which shifts the MAM F-test so that real disagreement is understated. This function tests the precondition directly, so a caller can tell which of the two they are looking at.

Method: take residuals as score minus the product mean, within each attribute, which removes the genuine product effects that would otherwise dominate the spread. Then apply Levene’s test (median-centred, i.e. the Brown-Forsythe variant, for robustness against non-normal residuals) across assessors.

A small p_equal_variance means the assessors genuinely differ in spread, and the MAM scaling coefficients for that attribute are measuring partly that rather than scale use alone.

Parameters:

panel (pandas.DataFrame) – Long-format panel data with panelist_id, product, attribute and score columns. Replicates are used as-is; more replicates give the test more to work with, but it runs on unreplicated data too by drawing the spread from across products.

Returns:

One row per attribute, sorted by attribute, with columns:

attribute: The attribute name.
levene_stat: Levene’s test statistic. NaN when fewer than two assessors have enough observations to have a spread.
p_equal_variance: The p-value for the null “all assessors have the same residual spread”. Small means they do not.
spread_ratio_max_min: The largest assessor’s residual standard deviation divided by the smallest, an effect size to read alongside the p-value. inf where some assessor has no residual spread at all.
n_assessors: Number of assessors contributing to the attribute.

Return type:

pandas.DataFrame

Raises:

ValueError – If a required column is missing or the panel has no rows.

Examples

>>> equality = assessor_variance_equality(validated.normalized_df)
>>> equality.query("p_equal_variance < 0.05")["attribute"].tolist()

Kevin Dunn, 2010-2026. MIT License.

Mixed Assessor Model (MAM): per-assessor scaling and scale alignment.

The classical assessor-by-product interaction lumps together two different things: a panelist who simply uses a wider or narrower part of the scale (a multiplicative scaling difference), and a panelist who genuinely ranks the products differently (real disagreement). The MAM separates them.

For each attribute, regress every panelist’s product means on the panel consensus product means. The slope is the panelist’s scaling coefficient beta:

beta near 1: uses the scale like the panel;
beta < 1: compresses (narrow range);
beta > 1: expands (wide range).

What is left after removing the scaling part is the disagreement. Using the disagreement (rather than the inflated raw interaction) as the error term gives a more powerful product-effect F-test, and the beta coefficients let you align the panel: rescale each panelist onto a common scale instead of dropping them (align_scores()).

This is a pure-Python MAM. A later release may add the SensMixed / lmerTest random-effects F-test via an R bridge; see the tracking issue.

References

Brockhoff, Schlich and Skovgaard, “Taking individual scaling differences into account by analyzing profile data with the Mixed Assessor Model”, Food Quality and Preference, 39, 156-166, 2015.

class process_improve.sensory.mam.MAMResult(scaling, ftests)[source]#

Bases: object

Outcome of mixed_assessor_model().

Parameters:

scaling (DataFrame)
ftests (DataFrame)

scaling#

One row per (attribute, panelist) with the scaling coefficient beta, the panelist offset from the attribute grand mean, and the panelist mean.

Type:: pandas.DataFrame

ftests#

One row per attribute with the MAM and classical product-effect F-tests: f_product_mam / p_product_mam (disagreement as error) and f_product_classical / p_product_classical (raw interaction as error), plus the degrees of freedom.

Type:: pandas.DataFrame

process_improve.sensory.mam.mixed_assessor_model(panel)[source]#

Fit the Mixed Assessor Model per attribute.

Parameters:: panel (pandas.DataFrame) – Validated descriptive_long panel data.
Returns:: Per-panelist scaling coefficients and per-attribute F-tests; see the class docstring.
Return type:: MAMResult
Raises:: ValueError – If a required column is missing, or the panel has no rows. An empty panel is reachable whenever an upstream filter removes every attribute; the frames built from it would have no columns at all, so the caller would meet the problem as a KeyError on ftests["f_product_mam"] rather than here.

Examples

>>> mam = mixed_assessor_model(validated.normalized_df)
>>> mam.scaling.query("attribute == 'saltiness'").sort_values("beta").head()

process_improve.sensory.mam.align_scores(panel, *, method='both', robust=False)[source]#

Harmonize every panelist’s scores onto the common panel scale.

For each attribute and panelist, the location lever removes the panelist’s mean offset (so “rates everything high/low” goes away) and the scale lever divides by the panelist’s scaling coefficient beta (so a compressor’s narrow range is stretched toward the panel’s). This rescales the whole panel (standard MAM practice), keeping panelists rather than dropping them.

When a panelist is flat or anti-correlated with the panel (beta is not finite or below _MIN_SLOPE), the scale lever is skipped for that panelist and the fallback depends on method:

"both": the panelist is left location-corrected only.
"scale": the panelist is left as-is (no correction applied).
"location": the slope is not consulted, so the location correction is always applied.

Parameters:

panel (pandas.DataFrame) – Validated descriptive_long panel data.
method ({"both", "location", "scale"}) – "location" recentres each panelist to the grand mean; "scale" rescales the spread around the panelist’s own mean; "both" (default) does both, the full MAM alignment.
robust (bool) – Use the repeated-median slope for beta instead of least squares.

Returns:

A corrected copy of panel with the score column aligned.

Return type:

pandas.DataFrame

Relating the panel to the product#

Kevin Dunn, 2010-2026. MIT License.

Relate descriptive panel attributes to the product.

analyze_descriptive() runs the proof-of-concept pipeline on a validated dataset: score and (optionally) correct the panel, then relate each sensory attribute to the product. The relate step dispatches on the validation mode:

observational (supported) - the product has measured descriptors but unknown formulation, so the attribute block is related to the descriptors with PLS (process_improve.multivariate.PLS plus VIP) and per-descriptor correlations, reported as association rather than causation.
designed (stub, not implemented yet) - the product is a controlled experimental run; the plan is to regress each attribute on the design factors via process_improve.experiments.analyze_experiment() for factor effects. See relate_designed(); it raises NotImplementedError for now.

The observational relate corrects across the family of tests with Benjamini-Hochberg FDR and returns supporting product means with confidence intervals and a PCA sensory map. Both the marginal associations and the per-attribute predictive-descriptor search are additionally gated on a leave-one-out jackknife, so an association or predictive coefficient that rests on a single high-leverage observation (a predictor that is non-zero on only one product, common in sparse, wide descriptor blocks) is demoted rather than reported. The jackknife adds no threshold of its own: it reuses the same alpha and the number of observations, so a genuine multi-observation driver is unaffected.

class process_improve.sensory.analysis.AnalysisResult(mode, panel, dropped, mam, correction, relate, product_means, pca, config=<factory>)[source]#

Bases: object

Outcome of analyze_descriptive().

Parameters:

mode (str)
panel (PanelScorecard)
dropped (list[str])
mam (MAMResult)
correction (str)
relate (dict[str, Any])
product_means (DataFrame)
pca (dict[str, Any])
config (dict[str, Any])

mode#

"designed" or "observational".

Type:: str

panel#

The per-panelist scorecard and flags.

Type:: PanelScorecard

dropped#

Panelists removed before the relate step.

Type:: list of str

mam#

Mixed Assessor Model: per-panelist scaling coefficients and the MAM vs classical product-effect F-tests.

Type:: MAMResult

correction#

The panel correction applied before relating: "none", "align", or "drop".

Type:: str

relate#

Mode-specific relate results; see analyze_descriptive().

Type:: dict

product_means#

Per product-by-attribute mean with a confidence interval.

Type:: pandas.DataFrame

pca#

Product sensory map: scores, loadings, explained_variance.

Type:: dict

config#

The options the analysis ran with.

Type:: dict

process_improve.sensory.analysis.aggregate_to_product(panel)[source]#

Return a product-by-attribute table of mean scores.

Parameters:: panel (pandas.DataFrame) – Validated descriptive_long panel data.
Returns:: Index product, one column per attribute, values the mean score over panelists and replicates.
Return type:: pandas.DataFrame

process_improve.sensory.analysis.find_predictive_descriptors(agg, covariates, *, n_components=2, alpha=0.05, n_permutations=199, random_state=0, cluster_threshold=0.95, max_components_cv=4)[source]#

Find which descriptors carry predictive signal, per attribute.

The marginal associations (relate_observational()) flag every descriptor that correlates with an attribute in-sample, genuine drivers and proxies alike. This step adds out-of-sample evidence:

a per-attribute cross-validated Q-squared gate (is the attribute predictable from the descriptor block at all),
a selectivity ratio per descriptor on the target-projected predictive direction, with a permutation p-value corrected for multiplicity by the max-statistic (Westfall-Young) permutation, so a descriptor that merely correlates by chance but does not enter the predictive direction is demoted, and
a collinear-cluster id per descriptor.

What it cannot do is rank descriptors within a collinear cluster: two descriptors that carry the same information predict equally well out of sample, so they share a cluster id and both stay significant. Separating them needs an external dataset or a designed experiment.

Important

The multiplicity correction is within an attribute, not across them. The max-statistic null is rebuilt for each attribute over its own descriptors, so p_value_fwer controls the family-wise error rate for that attribute’s descriptor family only. Nothing corrects across attributes: on a panel of A attributes at alpha, roughly alpha * A attributes are expected to produce a spurious family by chance alone. Read a single flagged descriptor on a many-attribute panel with that in mind.

Parameters:

agg (pandas.DataFrame) – Product-by-attribute mean table (index product).
covariates (pandas.DataFrame) – One row per product with the measured descriptors (plus a product column, which is dropped here).
n_components (int) – Latent components for the in-sample selectivity-ratio fit.
alpha (float) – Target false-discovery rate for the permutation family.
n_permutations (int) – Number of label permutations for the selectivity-ratio null.
random_state (int) – Seed for the permutations and the cross-validation folds.
cluster_threshold (float) – Absolute-correlation threshold for the collinear clustering.
max_components_cv (int) – Cap on the component count the Q-squared gate may select.

Returns:

per_attribute (the Q-squared gate per attribute), descriptors (the per attribute-descriptor selectivity ratio, raw permutation p_value, family-wise-error-adjusted p_value_fwer, jackknife_significant flag from the leave-one-out beta confidence interval, is_predictive flag and cluster_id), clusters (the descriptor-to-cluster map), and the settings used. A descriptor is is_predictive only when it also survives the jackknife, so a coefficient carried by a single product is demoted.

Return type:

dict

See also

find_predictive_descriptors: The per-attribute counterpart, and the one to prefer when the answer has to name an attribute. It reports one record per (attribute, descriptor) pair and controls the family-wise error rate within each attribute by a max-statistic permutation. This function instead fits a single multi-response PLS over the whole attribute block and returns one record per descriptor, so it answers “which descriptors matter for the panel as a whole”; its knockoff quantile band is a calibrated screen rather than formal error control. Use it to triage a wide descriptor block before committing to the per-attribute work.

process_improve.sensory.analysis.product_means(panel, conf_level=0.95)[source]#

Return per product-by-attribute mean with a confidence interval.

Parameters:

panel (DataFrame)
conf_level (float)

Return type:

DataFrame

process_improve.sensory.analysis.analyze_descriptive(validated, *, drop_panelists=None, correction='none', align_method='both', model='main_effects', n_components=2, conf_level=0.95, alpha=0.05, find_predictive=True, n_permutations=199, random_state=0, influence_deletions=1, discriminator=None)[source]#

Run the descriptive pipeline: panel check, correction, and relate.

Parameters:

validated (ValidationResult) – A passing result from process_improve.sensory.validate_descriptive().
drop_panelists ({"auto", None} or list of str) – "auto" drops every flagged panelist; a list drops exactly those ids; None keeps all panelists.
correction ({"none", "align", "drop"}) – Panel correction before relating. "none" (default) leaves scores as is; "align" applies the Mixed Assessor Model scale alignment to all panelists (process_improve.sensory.mam.align_scores()); "drop" is a synonym for using drop_panelists. Alignment and dropping compose: panelists are aligned first, then any dropped.
align_method ({"both", "location", "scale"}) – Which MAM lever to apply when correction="align".
model (str) – Design model for the designed relate step (default "main_effects").
n_components (int) – Components for the PLS relate step and the PCA map.
conf_level (float) – Confidence level for the product-mean intervals.
alpha (float) – Target false-discovery rate for the relate step.
find_predictive (bool) – Whether to run the per-attribute predictive-descriptor search (find_predictive_descriptors()) in the observational relate step.
n_permutations (int) – Permutations for the selectivity-ratio null.
random_state (int) – Seed for the permutations and cross-validation folds.
influence_deletions (int) – How many observations the marginal-association jackknife removes together (default 1, ordinary leave-one-out). Raising it to 2 also demotes a correlation carried by a single pair of high-leverage observations, which leave-one-out cannot detect.
discriminator (bool | None)

Returns:

See the class docstring.

Return type:

AnalysisResult

Raises:

ValueError – If validated did not pass validation.

process_improve.sensory.analysis.discriminate_observational(agg, covariates, *, n_components=2, alpha=0.05, n_permutations=199, random_state=0, cluster_threshold=0.95, max_components_cv=4)[source]#

Forward to find_predictive_descriptors(); emits a DeprecationWarning.

Deprecated since version 1.77.0: Use find_predictive_descriptors() instead. Will be removed in 2.0.0.

Parameters:

agg (DataFrame)
covariates (DataFrame)
n_components (int)
alpha (float)
n_permutations (int)
random_state (int)
cluster_threshold (float)
max_components_cv (int)

Return type:

dict[str, Any]

Kevin Dunn, 2010-2026. MIT License.

Designed-mode comparison of descriptive panel data: factorial ANOVA of the product (formulation) effect, with all-pairwise and against-a-control post-hoc multiple comparisons.

The observational half of this subpackage relates attributes to measured covariates of products whose formulation is unknown. This module covers the complementary designed question: the products are controlled treatments (formulations, aging conditions, …), the same panelists score every treatment (a randomized complete block design, with panelist as the block), and we want to know which treatments differ, and by how much, on each attribute.

Per attribute the workhorse is a fixed-effects ANOVA

score ~ C(factor_1) * C(factor_2) * … + C(block)

fitted by ordinary least squares with Type III sums of squares, so an unbalanced grid (missing cells, a panelist who skipped a sample) is handled correctly and the interaction terms test whether one factor’s effect depends on another (e.g. does aging change some formulations more than others). When the omnibus factor effect is real, two post-hoc procedures answer the follow-up question:

tukey_hsd() - all-pairwise Tukey HSD, using the blocked-model error mean square and the studentized-range distribution, so the block (panelist) variance is removed from the yardstick. Answers “which treatments differ from which”. A compact-letter display groups treatments that are not separable.
dunnett_vs_control() - Dunnett’s two-sided test of every treatment against a single named control, controlling the family-wise error for the “many treatments, one control” comparison only (tighter than Tukey).

compare_products() runs the whole sequence and returns a ComparisonResult. Everything is generic in the factor column names, so the same code serves a one-factor formulation screen or a formulation-by-aging-condition stability study.

References

Tukey, J. W. (1949). Comparing individual means in the analysis of variance. Biometrics, 5(2), 99-114.

Dunnett, C. W. (1955). A multiple comparison procedure for comparing several treatments with a control. JASA, 50(272), 1096-1121.

Piepho, H.-P. (2004). An algorithm for a letter-based representation of all-pairwise comparisons. Journal of Computational and Graphical Statistics, 13(2), 456-466.

Naes, T., Brockhoff, P. B. & Tomic, O. (2010). Statistics for Sensory and Consumer Science. Wiley.

class process_improve.sensory.designed.ComparisonResult(anova, tukey, dunnett, letters, means, config)[source]#

Bases: object

Outcome of compare_products().

Parameters:

anova (DataFrame)
tukey (DataFrame)
dunnett (DataFrame)
letters (DataFrame)
means (DataFrame)
config (dict[str, Any])

anova#

Type III ANOVA table, one row per (attribute, source) with df, sum_sq, mean_sq, F and p_value. The Residual row carries the error mean square used by the post-hoc tests.

Type:: pandas.DataFrame

tukey#

All-pairwise Tukey HSD contrasts (see tukey_hsd()), prefixed with the stratum column when the comparison was stratified.

Type:: pandas.DataFrame

dunnett#

Dunnett-vs-control contrasts (see dunnett_vs_control()); empty when no control was given.

Type:: pandas.DataFrame

letters#

Compact-letter display: one row per (stratum, attribute, level) with a letters string. Treatments that share a letter are not separable at the chosen alpha.

Type:: pandas.DataFrame

means#

Per (stratum, attribute, level) mean, confidence interval and n.

Type:: pandas.DataFrame

config#

The resolved call arguments, for provenance.

Type:: dict

process_improve.sensory.designed.factorial_anova(panel, *, factors, block='panelist_id', interactions=True)[source]#

Type III factorial ANOVA of the panel scores, one model per attribute.

Parameters:

panel (pandas.DataFrame) – Descriptive-long panel data (columns attribute and score plus the factor and block columns named below).
factors (list of str) – Column names of the fixed factors of interest (e.g. ["formulation", "condition"]). With more than one factor and interactions=True their full crossed model is fitted.
block (str or None) – Column treated as a blocking factor (default "panelist_id"). Pass None for no block.
interactions (bool) – Include the factor-by-factor interaction terms (default True).

Returns:

One row per (attribute, source) with df, sum_sq, mean_sq, F and p_value. The Residual row gives the error term. An attribute whose model cannot be fitted (too few observations, a singular design) yields a single source="(model failed)" row rather than aborting the sweep.

Return type:

pandas.DataFrame

Examples

>>> factorial_anova(panel, factors=["formulation", "condition"]).head()

process_improve.sensory.designed.tukey_hsd(panel, *, factor, block='panelist_id', alpha=0.05)[source]#

All-pairwise Tukey HSD of factor levels, one comparison per attribute.

The critical difference uses the error mean square of the blocked model score ~ C(factor) + C(block) and the studentized-range distribution, so the block (panelist) variance is removed. Unequal group sizes use the Tukey-Kramer standard error.

Parameters:

panel (pandas.DataFrame) – Descriptive-long panel data.
factor (str) – Column whose levels are compared all-pairwise.
block (str or None) – Blocking column (default "panelist_id"); None for no block.
alpha (float) – Family-wise significance level (default 0.05).

Returns:

One row per (attribute, pair) with group1, group2, meandiff (group1 minus group2), se, q_stat, p_value, the (ci_low, ci_high) simultaneous interval and reject.

Return type:

pandas.DataFrame

process_improve.sensory.designed.dunnett_vs_control(panel, *, factor, control, alpha=0.05)[source]#

Dunnett’s two-sided test of every factor level against control.

Uses scipy.stats.dunnett(), which pools the within-level variance and controls the family-wise error for the many-treatments-vs-one-control family (tighter than all-pairwise Tukey when the control is the only reference of interest). Unlike tukey_hsd() it does not remove a block term.

Parameters:

panel (pandas.DataFrame) – Descriptive-long panel data.
factor (str) – Column whose levels are compared to the control.
control (str) – The level of factor used as the reference.
alpha (float) – Significance level (default 0.05).

Returns:

One row per (attribute, level) for every non-control level, with meandiff (level minus control), statistic, p_value and reject.

Return type:

pandas.DataFrame

process_improve.sensory.designed.compare_products(panel, *, factors, block='panelist_id', primary=None, within=None, control=None, interactions=True, alpha=0.05, conf_level=0.95)[source]#

Compare product treatments per attribute: ANOVA plus post-hoc contrasts.

Fits the factorial ANOVA over factors (with a blocking factor), then runs all-pairwise Tukey HSD and, when control is given, Dunnett vs the control, on the primary factor. When within is set the post-hoc tests are run as simple effects separately within each level of that factor (e.g. compare formulations within each aging condition), which is the right follow-up once the primary x within interaction is significant.

Parameters:

panel (pandas.DataFrame) – Descriptive-long panel data.
factors (list of str) – Fixed factors for the ANOVA (e.g. ["formulation", "condition"]).
block (str or None) – Blocking column (default "panelist_id").
primary (str or None) – Factor whose levels the post-hoc tests compare. Defaults to factors[0].
within (str or None) – If given, run the post-hoc tests separately within each level of this factor (simple effects). If None, they pool over the other factors.
control (str or None) – Level of primary used as the Dunnett reference. If None, the Dunnett table is empty.
interactions (bool) – Include interaction terms in the ANOVA (default True).
alpha (float) – Post-hoc significance level (default 0.05).
conf_level (float) – Confidence level for the reported means (default 0.95).

Returns:

The ANOVA table, Tukey and Dunnett contrasts, compact-letter display and per-level means; see the class docstring.

Return type:

ComparisonResult

Examples

>>> res = compare_products(
...     panel, factors=["formulation", "condition"], within="condition", control="Control"
... )
>>> res.letters.query("condition == 'REF'").head()

Recipes and agent-callable tools#

Kevin Dunn, 2010-2026. MIT License.

Analysis recipes for the descriptive sensory pipeline.

Each recipe is a guided, multi-step workflow the agent follows, chaining the sensory tools (sensory_reshape_to_long, sensory_validate_descriptive, sensory_panel_check, sensory_analyze_descriptive) in the right order. The recipes are registered into the package-wide catalog on import; see process_improve.recipes for the framework and the general select_analysis_recipe tool.

Kevin Dunn, 2010-2026. MIT License.

Agent-callable tool wrappers for the descriptive panel-data pipeline.

Each function is decorated with @tool_spec so it can be passed straight to an LLM tool-use API. Inputs are plain JSON (lists of row-records), and every result is a JSON-serialisable dict. analyze_descriptive validates its input first and refuses to run when validation fails, mirroring the in-process gate enforced by process_improve.sensory.validate_descriptive().

process_improve.sensory.tools.sensory_reshape_to_long(spec)[source]#

Reshape to descriptive_long with round-trip checks; see tool spec for details.

Parameters:: spec (_ReshapeInput)
Return type:: dict

process_improve.sensory.tools.sensory_validate_descriptive(spec)[source]#

Validate panel + covariate inputs; see tool spec for details.

Parameters:: spec (_ValidateInput)
Return type:: dict

process_improve.sensory.tools.sensory_analyze_descriptive(spec)[source]#

Validate then analyse; see tool spec for details.

Parameters:: spec (_AnalyzeInput)
Return type:: dict

process_improve.sensory.tools.sensory_panel_check(spec)[source]#

Panel scorecard plus Mixed Assessor Model; see tool spec for details.

Parameters:: spec (_PanelCheckInput)
Return type:: dict

process_improve.sensory.tools.get_sensory_tool_specs()[source]#

Return tool specs for all sensory tools registered in this module.

Return type:: list[dict]