# AQ early-warning research: retrospective and next directions

Research decision brief • 11 October 2026

This brief wraps up the recovered experiment history through Suite30 and proposes a smaller, more discriminating research programme. No new AQ models were trained during this research. Published results establish reasons to test candidates; they do not predict their performance on AQ.

## 1. Decision

Retire the Suite29–30 course-set/joint-portfolio architecture as the main performance programme. Preserve its code, controls and findings. The evidence supports moving research effort elsewhere; it does not establish that all information in concurrent enrolments is useless, or that the task has reached an unavoidable performance ceiling.

The next priority should change the source of statistical strength, rather than add another interaction mechanism trained on the same AQ binary labels. Three directions deserve serious attention:

1. **External tabular pretraining**, initially on exactly the existing feature contract, followed by chronological adaptation only if useful transfer is observed.
2. **Verified additional information from the source database**, first tested as simple aggregates, with a relational pretrained model considered only when richer records actually exist.
3. **Observed failure-subtype supervision**, if trustworthy historical outcome-status fields can separate different kinds of grade-below-five outcomes.

Direct macro-AP optimization is a controlled extension to a promising backbone. Training-only academic trajectories are a conditional supervision opportunity. Neither should become another large architecture before its prerequisites are established.

The best near-term experiment is therefore a foundation-model transfer audit, accompanied by a bounded source-data audit. This direction was already raised in the Suite28 research brief; it remains an unresolved candidate, not a discovery that should be presented as new here.

## 2. What the task and evidence actually support

The prediction unit is a student × enrolled course × semester. The retained binary target is:

\[
Y_{s,c,t}=\mathbb{1}[\mathrm{grade10}_{s,c,t}<5].
\]

Prediction happens at the semester origin. Historical grades, enrolments and other records must have been available before that origin. A record belonging to an earlier semester can still leak information if its grade or status was entered after the prediction date. Semester membership alone is not an availability timestamp.

The main evaluation is the equally weighted mean of eligible faculty–fold AP values across 14 faculties and seven expanding chronological folds. It is not pooled registration AP and not student-level warning AP. The Suite28 complete evaluation covered 324,420 target registrations and 12,590 failures. F6/F7 covered 97,004 registrations, 6,574 failures and 28 faculty–fold cells. These are evaluation counts, not a certified count of the full deduplicated source dataset.

F6/F7 have been repeatedly consulted during research. They are useful retrospective comparisons, but cannot now be called untouched confirmatory holdouts. A genuinely later semester would be needed for a stronger prospective claim.

The historical feature contracts also differ. The earlier V2 family used roughly 275 engineered fields; later comparisons often use 136 common raw features plus 17 course-related features. These dimensions describe different pipelines. A common-column intersection can remove faculty-specific information, but its contribution to the observed performance gap has not been measured.

The recovered material includes Suite27 and Suite28 reviews, Suite30's final log and README, several early suite READMEs, earlier experiment records and database dictionaries. Some older outcomes are available through archived summaries rather than fresh execution logs. This is a research synthesis and partial implementation audit, not a complete replay or certification of every Suite1–30 implementation. No claim of exhaustive upstream leakage verification is warranted.

## 3. What we have tried

The following catalogue groups related hypotheses. It avoids treating every renamed combination as a new paradigm.

| Stage | Main implementations or hypotheses | What the work has taught us |
| --- | --- | --- |
| Initial baseline / V2 | Neural classifier; RF, ExtraTrees and HGB; Committee and stacked combinations | Strong tabular predictors remain essential operational references. IT-only results and later all-faculty results are different estimands. |
| Suites2–3 | Global and recent-window experts; history-support specialists; relational expert; Council/meta prediction; prequential fusion and score alignment | Temporal views and complementary predictors can produce useful combinations. Selecting the right contribution prospectively is harder than identifying it retrospectively. |
| Suite4 | Failure-rescue RankNet; log-loss-regret competence prediction; cross-fitted historical-oracle distillation; several grade-threshold targets; OOF nearest-neighbour memory | Ranking, grade-severity supervision, distillation and memory have already been attempted. They cannot be presented as entirely untested techniques now. |
| Suites5–7 | Failure-aware Council; cross-faculty residual MoE; shared encoder with private faculty bottlenecks | Sharing and specialization have a real tradeoff. More private capacity does not automatically yield reliable complementary corrections. Some runs used restricted faculty coverage. |
| Suites8–9 | Expert feature ownership; sparse responsibility competition; utility supervision; observable router distillation | Breaking expert collapse during training is insufficient. A router must identify useful expertise using variables available at deployment. Label-conditioned responsibility can exaggerate apparent opportunity. |
| Suites10–14 | Observable student/course domains; temporal and relational authority; historical error-signature domains; trajectory/curriculum/compatibility/mastery experts; Committee-preserving variants | Several notions of domain, competence and correction were tested. The bottleneck often becomes deployable authority estimation rather than a shortage of specialist architectures. |
| Suites15–18 | Conditional information and tree corrections; robust forward memory/relational/local mechanisms; Newton-style HGB corrections; temporal admission; fuzzy features | Residual learning and carefully guarded corrections are already well explored. Historical correction opportunity is not a guarantee of stable future gains. |
| Suite19 | Official TabM; LaplaceRFM/AGOP; internal normalized ModernNCA, including hard and semantic variants; neural representations with tree readouts | Modern tabular architectures were genuinely investigated. Isolated representation-plus-tree gains did not establish a broad standalone replacement. |
| Suites20–24 | Semester transformer; temporal residual; SFTR-TabM; relational FT-TabM; sparse/hierarchical feature graph; target-conditioned structural history; safe/posthoc fusion | Learned representations can contribute to a strong blend while remaining weaker as standalone predictors. A sparse feature graph is not the same experiment as pretraining over the original relational database. |
| Suite25 | Structure-aware chronological retrieval (SACR), standalone and Committee blend | Retrieval can supply complementary information, but standalone SACR still trails the stronger historical blend. |
| Suite26 | TAIR with current-state trees, same-student labelled memory and ability/course information; a separate joint tree–memory MINT branch | These are distinct branches and should not be merged in the retrospective. Memory availability and its temporal stability matter. |
| Suite27 | Native memory-conditioned GBDT splits and internal memory feedback (MCGB) | An elaborate integration of memory into tree topology did not deliver a robust late-period advantage. In F7, removing memory could improve performance even among students with history. |
| Suite28 | Compensatory MIRT; noncompensatory DRRM; difficulty-adjusted history, shared readiness, course requirements and ordinal grade likelihood | Historical academic information helps versus a restricted context model. The extra readiness bottleneck loses to simpler MIRT and richer tabular controls. Different information contracts confound an architecture-only conclusion. |
| Suite29 | Unordered concurrent-course set attention and masked course-wise classification | Concurrent enrolments can help a weak self-only model, but that does not establish superiority over strong full-feature predictors. |
| Suite30 | Shared latent risk, leave-one-out course interactions, normalized joint portfolio prediction, matched placebo and interventions | Some collective-risk structure is learned, but it has not produced the substantial marginal course-level AP improvement required by the task. |

There is enough breadth here to reject “we simply have not tried a sufficiently complex model” as the default explanation. We have tried substantial complexity. A particular pretrained or differently supervised model can still help; its rationale needs to explain what changes beyond complexity.

## 4. Quantitative current position

The most useful operational comparison is late-period performance. Scores below preserve the variants identified in the archived reviews; they are not all new reruns under one identical software environment.

| Predictor | Role | F6/F7 macro AP |
| --- | --- | ---: |
| Suite25 Committee blend | Strong historical operational reference | 0.369526 |
| Suite22 safe blend | Strong historical operational reference | 0.369159 |
| Committee | Baseline composite | 0.350152 |
| Suite28 full-feature HGB | Rich tabular standalone control | 0.343467 |
| Suite27 MCGB | Memory-conditioned tree predictor | 0.341348 |
| Suite25 standalone SACR | Chronological retrieval predictor | 0.341269 |
| Suite22 relational FT-TabM | Standalone relational representation | 0.340764 |
| Suite26 MINT | Joint tree–memory predictor | 0.338258 |
| Suite22 standalone SFTR-TabM | Standalone tabular representation | 0.336369 |
| Suite30 no-attention model | Three-seed mean | 0.332260 |
| Suite29 set model | Three-seed mean | 0.331077 |
| Suite30 joint model | Three-seed mean | 0.324514 |
| Suite28 MIRT | Simpler structured grade model | 0.291853 |
| Suite28 DRRM | Noncompensatory readiness model | 0.286328 |

The first nine references and the Suite28 controls are documented in `Suite28_review_and_next_research.md`. The Suite29–30 values are arithmetic means of the three seed rows in `suite30_joint_portfolio.log`. That log repeats the same Committee predictions under the three seed labels; those rows are not three independent Committee trainings.

Suite30's seven-fold results make the stopping decision particularly clear:

| Model | Three-seed mean AP | Difference from Suite30 joint |
| --- | ---: | ---: |
| Committee | 0.353855 | +0.049064 |
| Suite29 set | 0.307811 | +0.003021 |
| Suite30 joint | 0.304791 | 0.000000 |
| Suite30 raw, uncalibrated | 0.304365 | −0.000426 |
| Suite30 no attention | 0.303166 | −0.001625 |
| Suite30 matched placebo | 0.299764 | −0.005026 |
| Suite30 course only | 0.290712 | −0.014078 |

Joint modelling improves its weak course-only control by approximately 0.0141 AP. However, the advantage over the fitted no-attention control is about 0.0016, and the model remains below Suite29 and far below Committee. On F6/F7, the no-attention variant is better than the joint variant. These comparisons do not justify scaling the same mechanism in the hope of a large gain.

Some collective outcomes are more favourable. The uncalibrated joint model's overall mean joint NLL is 0.567146, compared with 0.595631 for independent Committee probabilities. But Committee remains slightly better for any-failure Brier and better for multiple-failure Brier in that table. The calibrated joint model's NLL worsens to 0.603282. A same-marginal joint-versus-independent comparison is also necessary to isolate the value of dependence modelling from the quality of the marginal probabilities. This is a useful research result, not demonstrated success on the primary course-level ranking task.

Attention mass and active gates are not evidence of useful reasoning. A module can be active because its normalization requires activity. Its contribution must be judged by fitted controls, interventions, placebos and the primary held-out metric.

## 5. What the negative results mean

### 5.1 We repeatedly found information, without learning to use it reliably

History, relational statistics, recent windows and concurrent enrolments can all contain signal. The difficult step has been converting that signal into a broad chronological advantage over well-tuned tabular predictors. This distinction explains why a mechanism can be scientifically informative while failing the practical goal.

### 5.2 Complexity often changes the problem before improving it

Several structured models restricted the information path, compressed student state, changed the target likelihood, or introduced a router. A comparison against a richer HGB therefore changes information access, inductive bias and optimization together. It cannot identify which change caused the loss.

We should preserve the full feature path in the next main experiment. An interpretation-friendly latent state should earn its restriction through an ablation, rather than define the predictor by assumption.

### 5.3 Prospective competence is a harder problem than retrospective complementarity

Knowing which expert would rescue a failure after seeing the label is useful for diagnosis. Predicting that expert before the semester starts is another supervised problem, often with weaker signal and noisier labels. This recurrent difficulty makes another oracle-trained router a poor default next step.

### 5.4 Temporal instability is central

The recovered reviews report a substantial increase in unavailable history in F7: roughly 27.7%, versus 0.79% in F6 in the relevant pipeline. Unavailable history should not automatically be interpreted as a truly new student. It can reflect extraction coverage or transcript availability. Failure prevalence also rises from approximately 5.52% to 7.79% between these folds.

These changes are consistent with altered observation support and outcome distribution. They do not establish one specific cause of drift. The practical implication is to compare models by chronological origin and history availability, not only by an overall historical mean.

### 5.5 Better joint probability is not automatically better marginal ranking

Our deployment objective ranks course failures. A joint model can improve the distribution of total failures without improving which individual course is ranked above another. It can still help through parameter sharing or additional slate information, but the benefit is empirical rather than automatic.

### 5.6 We have not identified a Bayes ceiling

A deterministic transformation of the current inputs cannot create new information about the target, though it can improve finite-sample learning. Genuine improvement can still come from a better learned prior, a better objective, richer training supervision, or additional origin-available records. None of the recovered experiments measures the irreducible uncertainty of this task.

Conversely, future semester shocks that are invisible at the origin cannot simply be recovered by increasing model size. An in-semester warning task could use later activity, but that is a different prediction horizon and should not silently replace the current benchmark.

## 6. What the literature changes—and what it does not

The sources below are primary papers or official implementation documents. Dates and capabilities were checked through 11 October 2026.

| Research family | Relevant evidence | Meaning for AQ |
| --- | --- | --- |
| Synthetic-data tabular pretraining | TabPFN in Nature [1]; TabICLv2, ICML 2026 [2]; current TabPFN-3.5 technical report and official release [3,4] | A pretrained inference procedure is a materially different source of statistical strength from an AQ-trained transformer. Published tabular rankings do not establish chronological AQ superiority. |
| Temporal limits and adaptation | TabReD, ICLR 2025 [5]; On Fine-Tuning Tabular Foundation Models [6] | Rich feature sets and temporal splits can favour simpler predictors. Chronological adaptation is a testable response, not a guaranteed solution. |
| Alternative pretraining | TabDPT, NeurIPS 2025 [7] | Real-data pretraining offers a useful contrast to synthetic pretraining. It should enter after a bounded initial audit, rather than expand the first grid indefinitely. |
| Table semantics and heterogeneous schemas | CARTE, ICML 2024 [8]; TabSTAR [9] | Native fields and meaningful text could support transfer. Course identifiers alone are not rich semantic descriptions. These methods are conditional on genuine metadata. |
| Original relational databases | RelBench [10]; RelGNN, ICML 2025 [11]; Relational Transformer, ICLR 2026 [12,13] | A database-level pretrained learner is distinct from the sparse feature graphs already tried. Its advantage must be separated from merely adding new raw information. |
| Precision–recall optimization | SOAP, NeurIPS 2021 [14] | A direct AP-oriented loss is an established technique and a plausible controlled extension. It does not create information or automatically preserve calibration. |
| Training-only longitudinal supervision | Time-series privileged information, AISTATS 2022 [15] | Intermediate historical observations may improve learning while keeping origin-only inference. The theory is restricted; it is not a guarantee for educational outcomes. |
| Label shift | BBSE, ICML 2018 [16]; newer posterior-prior adjustment work [17] | Prior correction can improve probabilities under appropriate assumptions. A single monotone probability correction within a faculty–semester preserves ranking, so it cannot by itself deliver a large within-cell AP gain. |

The caution about temporal evaluation is substantive. TabPFN-3.5's strongest published variants are not all equivalent to the locally available core model: the report also evaluates enhanced proprietary variants. Its reported hardware is not our V100. The same report retains difficult regimes in which tuned neural baselines lead. We should test the local model we can actually run, measure its context and memory costs, and avoid borrowing a headline result from a different variant.

TabICLv2 is a practical first candidate because a local implementation and fine-tuning interface are available [2,18]. Its classifier's documented fine-tuning metrics do not directly provide the AQ faculty–fold macro-AP objective. A future AP-specific adaptation would require explicit implementation and validation rather than a metric-name substitution.

## 7. Ranked research proposals

| Priority | Proposal | What is materially different | Potential | Principal uncertainty |
| --- | --- | --- | --- | --- |
| 1 | External pretrained tabular inference, then chronological adaptation | A prior learned before AQ fitting; support-to-query learning rather than another AQ-only interaction layer | Best near-term candidate for a unified predictor | Temporal transfer, imbalance, categorical semantics and context cost |
| 2 | Recover verified source information, then test relational pretraining | Additional origin-available observations and an externally pretrained database learner | Largest conditional information ceiling | Whether useful records are populated, historical and absent from current features |
| 3 | Observed failure-subtype supervision | Richer historical labels while preserving grade-below-five as the final target | Most distinctive task-specific hypothesis | Reliable subtype coverage and predictability from origin inputs |
| 4 | Faculty–semester macro-AP training | Match optimization to the actual comparison metric | Plausible incremental improvement on a strong backbone | Ranking/calibration tradeoff; noisy rare-positive estimates |
| 5 | Training-only academic trajectories | More learning supervision without later inputs at deployment | Conditional complement to priorities 1 or 3 | Trajectories may not exist; teacher information may be unpredictable at origin |

These priorities reflect research judgment, not measured AQ probabilities of success. Priority 2 could become first if the audit discovers a major omitted, valid data source. Priority 3 should be rejected quickly if outcome statuses cannot be reconstructed consistently.

### Proposal 1: a chronologically adapted pretrained tabular predictor

**Hypothesis.** The AQ-only architectures have paid a high estimation cost for learning their own representations. External pretraining may supply useful small-data and nonlinear inference structure that the present training labels have not been sufficient to learn.

The conceptual predictor is:

\[
\widehat p_{s,c,t}=f_\theta\bigl(x_{s,c,t},\mathcal D_{<t}^{\mathrm{available}}\bigr),
\]

where the support set contains only features and labels available before origin \(t\), and \(\theta\) starts from an externally pretrained model. This is different from merely retrieving neighbours or learning memory weights on AQ.

**First test.** Run local TabICLv2 and one current local TabPFN model on the existing 153-feature contract. Compare with strong, matched-information HGB and other established tabular controls already available in the project. Use natural-prevalence queries and the existing chronological metric. This first test requires neither a new router nor a new feature bottleneck.

Measure context size, GPU memory, latency and categorical handling on the actual V100. Do not assume published million-row capabilities translate into direct all-history inference on this hardware. Start with two prespecified support budgets; avoid a large search over sampling policies. A runtime allocation is a budget, not a predicted completion time.

**Second test, only after a useful signal.** Adapt through historical support-to-future-query episodes. For each historical origin \(u\), use support labels released before \(u\) and query covariates available at \(u\). Query labels are used for training only after their release and before the final deployment origin. Across faculties, the global calendar cutoff must hold; matching fold numbers does not certify matching dates.

Compare ordinary historical training episodes with chronological episodes under the same backbone, features, support budgets and compute allocation. This identifies whether the temporal adaptation helps, rather than attributing every gain to pretraining.

**Separate feature question.** Use a factorial comparison:

| | Existing common feature contract | Faculty-native valid feature contract |
| --- | --- | --- |
| Strong tabular control | A | B |
| Pretrained tabular learner | C | D |

The gain C−A tests model transfer with matched information. B−A tests richer fields with the tree control. D−C tests those fields with pretraining. D−B tests whether pretraining remains valuable with richer information. Different schemas require explicit type, missingness and category handling; names or meanings should not be guessed from encoded values.

**Failure condition.** If the pretrained models consistently trail strong same-feature controls on chronological development origins, across reasonable support sizes, and any adaptation benefit is confined to a few selected faculties, stop this route. A large IID benchmark advantage is not a reason to continue despite AQ evidence.

**Why this is promising but not proven.** It changes the learned prior, a gap not filled by TabM, ModernNCA or the previously trained memory architectures. The negative temporal literature is exactly why the experiment must be chronological from its first comparison.

### Proposal 2: information recovery before a relational foundation model

**Hypothesis.** Important parts of academic participation, registration state, historical observation timing or course constraints were lost when the original database was flattened into common features. Better models cannot recover missing observations from the existing table.

The dictionary gives this hypothesis concrete, but limited, support:

| Dictionary entry | Potential research use | Availability question |
| --- | --- | --- |
| `MKQDK`; `NgayNhapDiem`; grade/status notes | Grade release timing; historical result availability; outcome-status reconstruction | Is there historical versioning, or only the final row? |
| `MHuyDK`; `MNKHDKHP`; `NgayDK`, `NGAYHUYDK`, `LyDoHuyDK` | Prior registration changes, cancellations and disruption patterns | Which changes were visible before each origin? |
| `DTONNOCT`; `SoTien`, `DaDong`, `ConNo` | Prior outstanding-debt state, if allowed and relevant to the task | Are amounts timestamped snapshots or later current balances? |
| `KTKBTuan`, `KTKBNGCT`, `KTKBLog` | Published timetable constraints and historical scheduling structure | Was the relevant timetable published by the origin? |
| `LDSVangThiTam`; `NgayThem`; absence notes | Historical absence outcomes; possible subtype supervision | It is a temporary table: does an archive survive? |
| Component grading-scale tables and course catalogue | Actual course requirements or grading policy | Were those definitions stable and historically available? |

These are schema entries. They do not prove populated, usable historical records exist, or that current features omit their content. In particular, a temporary exam-absence table could contain only present operational data and be unusable for retrospective prediction.

**First test.** Inspect a small number of promising tables for coverage, keys, date semantics, versioning and overlap with existing features. Build a few origin-valid aggregates only when the record chronology is defensible. Feed them into an unchanged strong tabular baseline. If this does not improve broad chronological performance, adding a relational architecture is a weak next step.

**Second test, conditional.** Construct a task table of target registrations and verified joins to students, prior registrations, course definitions and other valid entities. Relational Transformer offers a released route for masked database/task-table learning and transfer [12,13]. RelGNN is a useful supervised relational control [11]. Neither should see target outcomes or post-origin fields through neighbouring records.

Compare: existing features; existing features plus new aggregates; relational learner with the same information; relational learner with the additional verified records. Removing the new information from the relational model is necessary to distinguish data gains from representation gains.

**Text extension.** Real course descriptions, syllabus requirements or stable curriculum metadata could support semantic transfer to sparse or new courses. CARTE or TabSTAR then become relevant comparators [8,9]. Do not generate descriptions from course codes and count them as independent information. Verify that useful text exists first.

**Failure condition.** Stop if records cannot be reconstructed as of each origin, if coverage is poor, or if they are already fully represented by current features. Do not use later attendance, examinations, debt updates or grades as origin predictors merely because the database exposes them.

### Proposal 3: preserve the binary task while learning observed failure subtypes

**Hypothesis.** Grade-below-five pools outcomes with different predictive relationships: an attempted assessment with low achievement, recorded exam absence or ineligibility, and potentially other verified statuses. A single binary label gives the learner no direct indication of that heterogeneity. Observed historical subtype labels may improve sample efficiency and temporal stability without an MoE router or a compulsory mastery bottleneck.

This is conditional on actual status records. A zero grade is not evidence of exam absence. A withdrawal is not automatically a failure in the current benchmark. Unknown or missing grades must not be relabelled as failures. The original eligible rows and grade-below-five definition remain unchanged.

Define mutually exclusive, documented failure types only when the database supports them. Let class 0 be pass and classes 1…K be verified failure subtypes. Use one shared full-feature predictor with a normalized head:

\[
(p_0,p_1,\ldots,p_K)=\operatorname{softmax}(h_\theta(x)),
\qquad \widehat p_{\mathrm{fail}}=\sum_{k=1}^{K}p_k=1-p_0.
\]

The subtype taxonomy must cover all retained failure outcomes; include an appropriate residual category if verified statuses are not exhaustive. Partial historical labels can then be retained without pretending their type is known:

\[
\ell(x)=
\begin{cases}
-\log p_0,&\text{observed pass},\\
-\log p_k,&\text{verified failure subtype }k,\\
-\log\sum_{k=1}^{K}p_k,&\text{failure with unknown subtype}.
\end{cases}
\]

The last line preserves the binary failure supervision for all unknown-type positives. It does not create an artificial “unknown mechanism” class. If subtype fields overlap, a documented mutually exclusive taxonomy or a different multi-label objective is needed before using this head.

**First test.** Quantify subtype coverage, stability and agreement with the grade label by faculty and semester. Compare binary-only and subtype-supervised models using identical full features, a closely matched backbone and a fixed training budget. A simple backbone is enough to test the label-information hypothesis. Custom foundation-model head changes should come later.

Subtype availability can itself be selective. If certain faculties or outcomes have systematically better documentation, the known-subtype loss can bias learning. Audit that pattern and test coverage-restricted comparisons or justified weighting; do not assume missing subtype labels are random.

**Decisive controls.** Compare with grade-bin supervision, because grade severity was already explored in Suite4 and ordinal grades in Suite28. Randomize subtype labels within historical faculty–semester and failure groups while preserving counts. A gain that survives randomization may reflect generic regularization or extra capacity rather than meaningful subtype supervision.

**Extension only after success.** Adapt a useful pretrained representation with the partial-label loss. Test whether subtype supervision improves that backbone beyond ordinary binary chronological adaptation. Keep the marginal failure probability as the deployed output.

**Failure condition.** Stop if subtype labels are inconsistent, sparse, predominantly inferred from the same grade value, or unpredictable from origin features. Better fit to the subtype labels alone does not count as success.

**Novelty boundary.** The ingredients are established. The potentially distinctive contribution is an empirically validated, availability-aware outcome-process representation for this academic task. It is not evidence of a globally first method or proof of causal mechanisms.

### Proposal 4: optimize the actual macro-AP comparison

**Hypothesis.** A promising model may lose performance because training prioritizes pooled BCE or AUC-like ranking while selection uses equally weighted faculty–semester AP. The remedy is a controlled objective comparison, not another complete architecture.

For a development set of eligible chronological cells \(\mathcal C\), use a mixed objective of the form:

\[
\mathcal L=\frac{1}{|\mathcal C|}\sum_{(f,u)\in\mathcal C}
\left[\mathcal L_{\mathrm{BCE}}^{f,u}
+\lambda\mathcal L_{\mathrm{AP-surrogate}}^{f,u}\right].
\]

The AP term ranks within each faculty–origin cell. Pooling rankings between faculties or semesters optimizes a different objective. Large cells should not silently dominate the macro metric. Cells without both outcome classes need a prespecified treatment consistent with evaluation eligibility.

SOAP supplies an established starting point [14]. The specific educational adaptation and cell weighting would be our experiment; the AP surrogate itself is not new. Suite4's failure-rescue pairwise ranking and other ranking losses mean that “try ranking” is not a novel proposal.

Compare ordinary BCE, the available historical pairwise loss, and an AP-specific loss on the same useful backbone. Match data, epochs and support contexts. Rare-positive sampling requires careful handling of the natural query distribution; simple prior corrections do not undo arbitrary feature-dependent or retrieval-dependent sampling.

Calibration should be assessed separately. A monotone scalar mapping within a cell preserves its ranking and generally its AP, except for introduced ties. Better probabilities can be operationally valuable, but do not satisfy the AP objective by themselves. Use a historical calibration block with the same fitted scoring model; a checkpoint/refit mismatch is a separate protocol issue to investigate.

**Failure condition.** Stop if gains are confined to the development objective, vanish against an equally tuned pairwise control, or cause unacceptable operational deterioration. This is more likely an incremental enhancement than a new information source.

### Proposal 5: historical training trajectories as privileged supervision

**Hypothesis.** Historical intermediate assessments, attendance or academic participation may give more informative training targets than one final binary outcome, even though only origin features are used at deployment.

This requires genuinely recorded trajectories. A teacher can use intermediate information from completed historical semesters to produce training targets, while the student receives only origin features. Teacher training and target generation must be cross-fitted or otherwise guarded against memorized in-sample targets. Every training record and label must have become available before the final deployment origin. The final evaluation student receives no within-target-semester activity, teacher output based on future observations, or later calibration data.

Time-series privileged-information research provides a statistical rationale in restricted settings [15]. It does not imply that a student can reconstruct unpredictable future shocks. A teacher's future-assisted performance is a nondeployable diagnostic, not the deployed model's score.

Distillation itself is not new to this project: Suite4 and Suite12 already used it. What would be new is verified richer historical trajectories that were not part of those teachers. Compare binary-only training, direct auxiliary trajectory supervision and teacher distillation before combining this with a foundation model.

**Failure condition.** Stop if the trajectories do not exist, if they mainly encode final outcomes without adding learnable structure, or if the origin-only student has no broad gain. An impressive future-assisted teacher does not justify continued investment on its own.

## 8. A research plan that makes weak ideas inexpensive to reject

### Gate 0: freeze the comparison and locate recoverable opportunity

Retain the current feature contract, eligible-row definition, origin cutoffs and metric implementation. Preserve existing reference predictions. Identify errors shared by Committee, the strongest blend and strong standalone predictors. Describe those errors by faculty, origin, history availability, course support and verified outcome status. This is diagnosis of where gains might exist, not a deployable oracle filter.

Run the limited database audit in parallel with preparation of the transfer experiment, without building a new pipeline first. Establish whether native columns, true course metadata, timestamped events or reliable subtype labels exist. A schema name alone does not pass this gate.

### Gate 1: a bounded local transfer pilot

Choose three faculties in advance using sample size and observation support, not favourable candidate scores. Use earlier development origins inside F1–F5. Run one local TabICLv2 configuration and one local TabPFN comparator with two support budgets, plus matched tabular controls. Record memory, runtime and AP. An initial allocation such as four GPU-hours is a proposed budget, not a runtime prediction; change it only with a documented resource reason.

A pilot is a feasibility and signal test, not a publishable result. If one candidate is viable, advance it. If both are clearly uncompetitive across the selected chronological origins and reasonable contexts, do not spend weeks on elaborate adaptation.

### Gate 2: all-faculty development and one addition at a time

Use all 14 faculties and at least three eligible development origins where history permits. Compare the pretrained baseline with matched strong controls. Then add chronological adaptation. Then test native valid fields separately. Then, only if justified, test the AP objective or subtype supervision.

Do not introduce pretraining, new fields, subtype labels, temporal adaptation and a new loss simultaneously. One improvement should identify one research question. Across-faculty support must obey actual dates and label-release times.

### Gate 3: freeze and perform one retrospective late-period comparison

After development selection, freeze the model and protocol before an F6/F7 comparison. Report seed or support-context variability, all eligible faculty results, recent-origin results, warning-budget recall, Brier/log loss and runtime. A strong result there remains retrospective because those periods have informed the research agenda.

For uncertainty, average AP across repeated model seeds within each cell before computing the equally weighted cell metric. Do not count three seeds × 98 cells as 294 independent trials. A paired resampling procedure should preserve student dependence across courses and, where applicable, origins, and recompute cell AP and macro AP. It estimates uncertainty conditional on observed periods, not uncertainty about arbitrary future semesters. Report the separate seed variability as well.

### Gate 4: test a genuinely later period when available

Evaluate a frozen predictor on a later semester that has not driven candidate choice. Confirm the original horizon and label availability. This is the clearest way to distinguish a stable improvement from selection on the repeatedly inspected history.

### Proposed decision rules

The user has not specified a minimum useful AP gain. As a research allocation rule, a gain of approximately 0.015–0.020 macro AP over the strongest matched reference, supported broadly across faculties and not purchased through a major loss in a large faculty, would merit substantial expansion. This is a proposed practical threshold, not a forecast or a universal statistical rule.

Beating strong same-information standalone controls is the first milestone. Approaching the historical 0.3695 blend with a unified predictor is the second. Substantially surpassing that blend, with a later-period confirmation, is the breakthrough claim. If the reference pipeline changes, recompute its score on the exact new test rows; do not compare against an old number on different data.

A small win against a weak restricted control does not satisfy these milestones. A useful isolated scientific finding should be recorded and the branch closed when it fails the practical test.

## 9. Ideas to defer

| Tempting next step | Why it is lower priority now |
| --- | --- |
| Larger set attention or a more elaborate joint failure-count model | The Suite29–30 mechanism has not earned a large marginal-AP investment. Revisit only if the target becomes collective warning quality. |
| Another Council, residual MoE or memory router | These mechanisms have extensive history here. A genuinely different information source would be needed to justify reopening them. |
| A new mastery/readiness bottleneck | Suite28 shows the cost of an imposed information path. Keep full features unless a controlled restriction improves the desired metric. |
| A sequence-model name change: Mamba, another Transformer, or KAN | A model family name does not explain the data limitation it solves. Verified raw event sequences could change this assessment. |
| Generic diffusion augmentation or GAN/SMOTE experiments | Synthetic rows do not create independent observations of future failure. Their value would need a specific supported minority-structure hypothesis. |
| Reinforcement learning for the current static predictor | There is no defined action/reward interaction in this task. Intervention policy learning would require another problem and suitable data. |
| Causal or counterfactual claims from the current observational benchmark | Prediction gains do not identify effects of interventions. Such a study needs explicit estimands and identification assumptions. |
| Pure probability-prior correction as an AP breakthrough | Useful for calibration under the right assumptions; a scalar monotone correction within a cell does not change its ranking. |
| Unrestricted large grids or several new mechanisms at once | They obscure which hypothesis worked and increase selection on the same historical periods. |

These are allocation decisions, not claims that the techniques are universally ineffective.

## 10. What would count as a defensible novel contribution

There are three different novelty claims to keep separate:

* **New to this project:** external tabular inference and original-database pretraining are gaps in the completed evidence recovered here.
* **A distinctive application or combination:** availability-aware chronological adaptation, native schemas and observed failure-subtype supervision could form a meaningful educational research contribution if ablations show which ingredient matters.
* **A new general method:** this requires a narrower technical invention and broader comparisons. No literature search here warrants a “first ever” claim.

A promising eventual combination is a pretrained full-feature predictor adapted through historical origins and observed failure subtypes. Its rationale has three separate parts: external prior, deployment-aligned adaptation and richer outcome supervision. Each part must first show a benefit on its own comparison. Native source information can then be added if verified. AP-specific training is an extension after a strong predictor exists.

This combination is a hypothesis to test, not the next architecture to implement wholesale.

## 11. Final research recommendation

Close the Suite29–30 performance branch, retain its collective-risk findings, and begin with a controlled external-pretraining test. At the same time, answer a small set of concrete data questions about historical availability, native feature coverage and outcome-status labels. Advance a richer paradigm only when those observations justify it.

The main lesson from the experiment history is that we have repeatedly increased modelling machinery without securing a new, reliable source of predictive strength. The next phase should make that source explicit and measurable: a better prior, additional valid observations, or richer historical supervision. That gives us a stronger basis for expecting improvement and a clearer reason to stop when it does not occur.

## Evidence register: project files

| Project source | Main use in this brief |
| --- | --- |
| `suite30_joint_portfolio.log` | Final three-seed seven-fold and F6/F7 scores, controls, collective metrics and calibration results |
| `README_suite30.md` | Intended mechanism, label/input contract and controls |
| `Suite28_review_and_next_research.md` | Late-period reference table, feature-contract limitations, history availability and earlier foundation-model recommendation |
| `Suite27_complete_experiment_review.md`; `suite27_DESIGN.md` | Memory-conditioned tree findings and broader comparison history |
| Recovered Suite2–10 READMEs; Suite11–12 logs; Suite19 README and prior experiment records | Earlier techniques, repeated mechanisms and limits of retrospective coverage |
| `AQDMFile.csv`; `VFilefield.csv` | Database table/field inventory; evidence of schema entries, not certification of usable historical observations |

## Primary literature and official implementations

1. Hollmann et al. **Accurate predictions on small data with a tabular foundation model.** Nature, 2025. [Paper](https://www.nature.com/articles/s41586-024-08328-6).
2. Qu et al. **TabICLv2: A Better, Faster, Scalable, and Open Tabular Foundation Model.** ICML, 2026. [Proceedings](https://proceedings.mlr.press/v306/qu26a.html); [full technical text](https://arxiv.org/html/2602.11139v1).
3. **TabPFN-3.5 technical report.** September 2026; preprint. [Technical report](https://arxiv.org/html/2609.17895v2).
4. Prior Labs. **TabPFN official implementation and 3.5 release documentation.** [Repository](https://github.com/PriorLabs/TabPFN); [release documentation](https://docs.priorlabs.ai/changelog/tabpfn-3.5).
5. **TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks.** ICLR, 2025. [Proceedings](https://proceedings.iclr.cc/paper_files/paper/2025/hash/571799482291411607c54984153190b0-Abstract-Conference.html).
6. **On Fine-Tuning Tabular Foundation Models.** June 2025; preprint. [Paper](https://arxiv.org/abs/2506.08982).
7. **TabDPT.** NeurIPS, 2025. [Proceedings](https://papers.nips.cc/paper_files/paper/2025/hash/fc0e3f908a2116ba529ad0a1530a3675-Abstract-Conference.html).
8. Kim et al. **CARTE: Pretraining and Transfer for Tabular Learning.** ICML, 2024. [Proceedings](https://proceedings.mlr.press/v235/kim24d.html).
9. **TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields.** [Technical text](https://arxiv.org/html/2505.18125v1).
10. **RelBench: A Benchmark for Deep Learning on Relational Databases.** NeurIPS Datasets and Benchmarks, 2024. [Proceedings](https://papers.nips.cc/paper_files/paper/2024/hash/25cd345233c65fac1fec0ce61d0f7836-Abstract-Datasets_and_Benchmarks_Track.html).
11. Chen et al. **RelGNN.** ICML, 2025. [Proceedings](https://proceedings.mlr.press/v267/chen25ad.html); [technical text](https://arxiv.org/html/2502.06784v1).
12. **Relational Transformer.** ICLR, 2026. [Proceedings](https://proceedings.iclr.cc/paper_files/paper/2026/hash/7107d4d2e837bde2171c6b71b5bde954-Abstract-Conference.html).
13. Stanford STAR. **Relational Transformer official implementation.** [Repository](https://github.com/stanford-star/relational-transformer).
14. Qi et al. **Stochastic Optimization of Areas Under Precision-Recall Curves with Provable Convergence.** NeurIPS, 2021. [Proceedings](https://proceedings.neurips.cc/paper_files/paper/2021/hash/0dd1bc593a91620daecf7723d2235624-Abstract.html).
15. Karlsson et al. **Using time-series privileged information for provably efficient learning of prediction models.** AISTATS, 2022. [Proceedings](https://proceedings.mlr.press/v151/k-a-karlsson22a.html).
16. Lipton et al. **Detecting and Correcting for Label Shift with Black Box Predictors.** ICML, 2018. [Proceedings](https://proceedings.mlr.press/v80/lipton18a.html).
17. Lee. **Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment (DistPFN).** ICML, 2026. [Proceedings](https://proceedings.mlr.press/v306/lee26aw.html).
18. SODA/Inria. **TabICL official implementation.** [Repository](https://github.com/soda-inria/tabicl).

Paper families with only a preprint cited are described as research evidence rather than established AQ validation. Repository capabilities are current as of this brief's date and may change.
