# Re-evaluation of Suite26 and rationale for the rewrite

## What the original result establishes

Source: user-supplied `suite26.log`, SHA256 `ff67a85d0da1ef4f6376b041a9b8c2188731130cdcbab67ee87026fff2be42a2`. The original completed seven folds/14 faculties in 5.25 hours on a V100 32GB. Its printed full holdout summary gives:

| Model | Holdout macro AP | Holdout macro AUC | Faculty coverage |
|---|---:|---:|---:|
| Original tree | 0.312259 | 0.843597 | 14 |
| Original tree + memory | 0.287641 | 0.827734 | 14 |
| Original TAIR primary | 0.302149 | 0.830961 | 14 |
| Original joint neural head | 0.339302 | 0.834148 | 14 |
| Committee | 0.350152 | 0.843399 | 14 |
| Suite22 final | 0.369159 | 0.855570 | 14 |
| Council | 0.444927 | 0.833026 | 1 (IT only) |

The old memory addition loses 0.024618 AP against its tree. The ability component gains 0.014508 AP against the old memory model, with 22 wins among 28 faculty-fold cells, but the resulting primary remains 0.010110 AP below its tree and 0.048003 below Committee. IT's original primary averages 0.353780 AP, versus Committee 0.412578 and Council 0.444927. Council's one-faculty number must not be compared with a 14-faculty macro as if coverage were equal.

The intervention audit is informative but does not reverse that result. Shuffling student histories loses 0.034472 AP; masking direct labels loses 0.018684; removing ability loses 0.004954; yet removing the entire memory path gains 0.010110 on average. The memory carries relevant information, while its learned integration imposes a net cost. A wrong or shuffled memory hurting a model does not show that adding the correct memory beats a simpler model.

F7 is particularly unfavorable for the original episodic-memory increment: 11 of the 14 printed component cells have negative B-minus-tree AP. In IT, failure prevalence rises from 0.099628 in F6 to 0.151314 in F7. IT also has 2,145 new-student target rows in F7 (1,591 seen-course and 554 new-course), so cold histories and distribution change matter. These observations motivate explicit cold-history and per-fold reporting. They do not identify a single causal source of the loss.

The old history-K sensitivity slightly favors 4 events (AP 0.304879) over 32 (0.302149). This is inference sensitivity for the previously trained model, not evidence that retraining a new model at K=4 is optimal. The rewrite retains outcome-independent recent candidates and reports K sensitivity; it does not choose a winning K using the new target-fold labels.

The printed fold tables show only part of the faculties, so their visible rows must not be averaged to recreate the full macro. `validation/original_result_summary.json` extracts the printed complete macro and premise-audit sections. No original row predictions were supplied here; no new confidence interval for the old run is claimed.

## The architectural mismatch

The original source computes `primary = logit(tree_score) + bounded_delta`. Leaf embeddings, historical attention and ability processing are trainable, but sklearn's fitted split choices/thresholds are not updated by that final prediction loss. Its jointly trained auxiliary head is another prediction path, optimized through the combined loss and optimizer. Consequently, the original implementation does not fulfill the user's clarified request that memory change tree learning internally.

This source fact supports a rewrite. The unfavorable log does **not** prove that a differentiable forest will outperform it. The original joint head's 0.339302 AP suggests an integration bottleneck, but its below-Committee result also leaves model capacity, training protocol, sampling, drift and generalization as competing explanations.

## Why this particular joint model

MINT makes a conditional tree node the interaction site. A node's selected feature, margin and incoming path reach form a memory query. Retrieved history changes that node's threshold, which changes the next depth's reach and hence its next query. This is a direct dependence of memory retrieval on the tree's evolving decision context, and of tree routing on memory.

For tree t and node j, let the current selected feature be f, base threshold b, temperature tau, current/course context c, retrieved memory m, and readiness-deficit vector g:

```
effective_threshold = b + 2*tanh(u_static · c)
                        + 2*tanh(u_memory · m)
                        + 2*tanh(u_gap · g)
right_probability   = sigmoid((x[f] - effective_threshold) / tau)
```

The history terms are exactly zero for empty history. Parent reach splits into left/right mass, and the final probability is the mean over trees of `sum(leaf_reach * sigmoid(leaf_logit))`. There is one prediction mechanism. The conventional forest initializes it; its scores are not an input to a second estimator.

| Component | Assigned role | Constraint preventing role confusion |
|---|---|---|
| Feature selector/base threshold | Current-state test and reusable tree partition | Trainable internal parameters; hard forward feature identity |
| Node query | Retrieve evidence relevant to this decision/path | Uses tree margin and incoming reach, not target outcome |
| Episodic keys | Historical state/course/recency matching | Direct grades/failures excluded from keys |
| Episodic values | Carry observed historical evidence | Only earlier-semester events; missingness and masks explicit |
| Readiness | Pool history into target-independent performance coordinates | Positive association weights and a finite prior |
| Requirement/importance | Express target-course demands in those coordinates | Bounded requirements; normalized nonnegative importance |
| Conditional thresholds | Let context/history change the forest's decision boundaries | Bounded shifts, reach-weighted regularization; no risk residual |
| Leaves | Produce probability from routed examples | Updated by the same final label loss |

One main supervised objective supplies gradients through leaf mass, split probabilities and attention. The optional ranking and small shift penalty serve ranking/stability, not extra prediction heads. Explicit gradient tests cover selector, threshold, leaf, memory key/value, association, requirement, importance and gap parameters. Intervention tests verify that historical outcomes change routes and that target labels never enter the forward computation.

## What is theoretically sound, and what remains empirical

Path mass is conserved by construction: a parent's two children sum to the parent. Sigmoid leaf values and averaging keep the final probability in [0,1]. All-empty attention is exactly zero and readiness has a defined prior. Strict earlier-semester retrieval makes history causally available at semester granularity under the stated grade-release assumption.

Readiness dimension k is `(2*0.5 + sum(w_i,k * performance_i)) / (2 + sum(w_i,k))`, with positive w. Conditional on fixed historical course associations, increasing a grade increases readiness. Historical grade/10 is used when present; otherwise observed pass/failure is the fallback. Target requirements do not affect readiness. These statements are testable and enforced.

The model is still a hypothesis. Latent dimensions and requirements are not identified academic skills or established item-response parameters. Learned node-gap coefficients can use a deficit differently at different branches, so the final risk is not guaranteed globally monotone in every historical grade. Requiring that monotonicity would restrict the classifier substantially and is not asserted here.

The feature-choice gradient is a straight-through surrogate for a discrete decision; it is not an exact derivative of argmax or a guarantee of finding the best feature. Continuous routing is differentiable, while normal prediction remains soft. No proof of global optimality, theoretical perfection, generalization superiority, or substantial AP gain follows from these design properties.

Conditional thresholds depend on the full encoded current/course context. Therefore these are conditional neural trees, not static axis-aligned sklearn trees with unchanged interpretation. Geometry is fixed at the selected depth; split parameters/feature identities/leaf values learn, but tree topology is not grown or pruned by memory.

## Concrete changes prompted by review

1. **Latest one-semester validation by default.** Fixed-origin memory alone cannot freeze arbitrary cached row-time features over several validation semesters. The default removes that reconstruction-specific mismatch; cache provenance remains an explicit assumption.
2. **Full historical bank and explicit missing metadata.** The old CF-covered-only bank discarded early events. The new bank retains all canonical history without borrowing future course metadata.
3. **No direct failure-prioritized retrieval.** Recent candidate selection does not change when labels/grades change. Memory learns what matters among permitted events.
4. **Matched controls.** Tree-only, episodic-memory and full MINT start from the same seed geometry/shared initialization. The conventional seed is also reported, so joint tree learning is distinguishable from memory addition.
5. **Balanced split discovery, natural leaf counts.** Rare failures influence seed partition discovery without importing class-balanced probabilities into the neural objective.
6. **Equal selection/refit examples.** Below the sampling cap, refit must not silently multiply the selected optimizer budget when more historical rows become available.
7. **Meaningful historical comparisons.** Default directories map to actual score aliases; alignment uses identifiers and verifies labels, not matching label arrays. Missing/ambiguous comparisons remain auditable.
8. **Precision-safe seed conversion.** Rounding thresholds downward preserves original sklearn branches for float32 inputs. An adjacent-float regression test caught and fixed an actual conversion error.

The appropriate verdict is **architecturally aligned and executable, with performance unproven**. The next AQ run should be judged by paired component improvements, uncertainty, cold regimes, calibration and IT operational results. Because original F6/F7 results informed this review, they are reused historical benchmarks; a fresh later semester or other untouched cohort is needed for a stronger final generalization claim.
