# Suite5 Failure-Aware Council Design

## Goal
Upgrade Suite4's best-performing four-expert Council by making competence estimation explicitly failure-aware and boundary-aware, while preserving expert diversity and restoring the complete V2/Suite3 operational threshold protocol.

## Backbone
Keep the four Suite4 experts unchanged:
1. committee = mean RF/ET/HGB probability,
2. global all-history HGB meta expert,
3. recent-4-block HGB meta expert,
4. relational expert.

All expert predictions must be chronological/OOF when used to train competence models.

## Competence targets
Train two alternative competence systems for controlled comparison.

### Symmetric baseline
Retain Suite4 log-loss regret target:
`r_j = logloss(q) - logloss(p_j)`.

### Failure-aware utility
For specialist j, combine dense log-loss regret with explicit fixed-budget decision utility on each historical validation block:
- positive bonus when specialist rescues a failure missed by committee,
- larger penalty when specialist loses an already-detected failure,
- smaller bonus for removing a committee false positive,
- penalty for introducing a new false positive.

Training samples receive additional weight for failures, committee-missed failures, and rows close to the committee warning boundary.

## Boundary-aware features
Add unlabeled batch-relative decision features to the existing Council feature vector:
- committee percentile rank,
- committee distance to warning-budget percentile cutoff,
- committee warning flag,
- each expert percentile rank,
- each expert rank gap vs committee,
- expert vote count inside top warning budget.

These use predictions only, never target labels.

## Controlled specialization
Do not force non-uniform weights. Compare:
- uniform council,
- symmetric competence,
- failure-aware competence,
with temperature values including sharp and smooth regimes.

Measure entropy and effective number of experts. A sharp gate is accepted only through prequential validation.

## Score space
Compare raw expert probabilities against blockwise rank-preserving alignment of specialist scores to the committee distribution. Alignment uses scores only, preserves ties, and never uses labels.

## Policy selection
Select gate mode, raw/aligned score space, outer committee authority alpha, and temperature on recent historical forward blocks. Selection prioritizes median AP/F2 improvement with safeguards on worst-block AP/F2 loss. Exact committee fallback is always available.

## Operational evaluation
Using pooled development Suite5 predictions:
- tune F2 threshold once,
- freeze threshold for holdout folds 6-7,
- freeze warning-ratio policy,
- compute drift-aware rate sequentially,
- print/save precision, recall, F1, F2, warning rate, TN/FP/FN/TP,
- report equal 20% warning-budget metrics,
- include committee and Suite4-style uniform/symmetric counterfactuals.

Holdout labels never influence expert fitting, competence fitting, gate selection, score-space selection, threshold tuning, or warning-budget tuning.

## Diagnostics
Per fold save/print:
- AP/AUC and F2@20 for every expert and council counterfactual,
- selected gate mode/alpha/temperature/score space,
- mean and quantiles of expert weights,
- entropy/effective expert count,
- failure vs normal mean weights (diagnostic only),
- committee-missed failure vs already-detected failure weights (diagnostic only),
- top-1 competence accuracy and confusion of predicted-best vs hindsight-best expert (diagnostic only),
- non-deployable best-expert oracle ceiling,
- rescued failures, lost existing TPs, FP added/removed at 20% budget,
- history-depth regime metrics.
