Your neural network's black-box decisions just got a discoverable memory

Modern neural networks guide high-stakes decisions in medical diagnosis, credit approval, and energy bidding. But when a model scores one action over another, the reasoning stays opaque: the training experience is absorbed into learned parameters, not exposed as an inspectable case memory. A preprint from Huazhong University of Science and Technology now supplies an exact algebraic bridge between neural action scores and the training cases that shape them.

In From Neural Network Decisions to Training Cases: An Exact Account via Case-Based Decision Theory, researchers Manli Yan, Yuebin Lin, Yaowen Yu, and Yong Zhao show that for fixed-representation ordinary least squares (OLS) readouts, each action score is exactly a weighted sum of training-case returns. The weights come from the empirical Gram geometry of the representation, a purely algebraic relationship that makes case-level evidence explicit. This is a different kind of transparency problem from the one studied in agent failure analysis, but it targets the same core issue: what is the model actually relying on?

An exact case-based decomposition

The key insight builds on case-based decision theory (CBDT), which models decision-making as aggregating outcome support from remembered cases. CBDT formalizes a familiar human pattern: we evaluate current actions by comparing the present problem with past experiences and summing the outcomes of similar cases.

The authors demonstrate that a fixed neural representation, equipped with an OLS top-layer probe, produces action scores that admit the same structural form. For a test input and candidate action, the score equals a sum over training cases of a coefficient times the return for that action. The coefficient is determined by the inner-product geometry of the representation: α_i(x) = φ(x)^⊤G⁻¹φ(x_i), where G is the empirical Gram matrix.

This decomposition is exact when the representation matrix has full column rank, and admits a Tikhonov-regularized version when ill-conditioned. The coefficient vector is the unique minimum-norm solution to a consistent linear system, making it canonical rather than arbitrary.

Semantics: similarity or signed influence?

The algebraic form matches CBDT's aggregation structure, but the semantics differ in a crucial way. CBDT uses non-negative similarity weights, a constraint reflecting how humans retrieve and weigh past experiences. The OLS-induced coefficients, by contrast, are signed, unnormalized, and derived from Gram geometry rather than similarity.

The authors identify a sufficient regime under which the coefficients reduce to scaled inner products and can match CBDT similarity semantics: when the representation is whitened so that G = n I_H. Outside this regime, which is rare in practice, the coefficients are best interpreted as signed Gram-geometric influence terms, not similarity weights. Positive coefficients correspond to nearby cases that raise the action score; negative coefficients act as far-field corrections that offset it.

This distinction prevents users from conflating influence with similarity and clarifies what retrieved cases actually mean. The challenge of interpreting what an AI system retrieves has parallels in visibility over magic in coding agents: knowing what the system is looking at is not the same as knowing why it chose that.

Audit signals without retraining

The decomposition yields three practical audit signals for post-training inspection:

• Case audit score: For a test point and candidate action, each training case's signed contribution ψ_i(a, x) = α_i(x) r_i,a traces which cases raise, lower, or offset the score. Sorting by absolute contribution reveals the most influential cases.
• Influence entropy for action coherence: By building a probability distribution over the optimal actions of the top-k influential cases, weighted by absolute influence mass, the method produces an entropy measure. Low entropy means the influential cases point to a single action; high entropy reveals dispersed or conflicting evidence.
• Case-influence risk diagnostic: Combining influence entropy with a Gini disagreement term among the top 10 influential cases produces a risk score that flags weakly supported predictions, those where the strongest training-case influences provide conflicting action evidence.

All three signals require only fitting an OLS top-layer probe on the fixed representation. No retraining of the representation, no access to the optimization trajectory, and no knowledge of the original loss function needed. This auditability without access to the full training pipeline is increasingly relevant as models get deployed in high-stakes systems where black-box decisions carry real consequences.

Experimental validation

The experiments cover a synthetic CBDT generator with known ground truth, PJM energy bidding with real load observations, and two UCI proxy-decision tasks (Adult Income and Default Credit). Across five random seeds, the method achieves 96% agreement with the CBDT optimal action on synthetic data and strong score-reconstruction correlations on real tasks (0.9998, 0.9972, and 0.9592 for PJM actions).

In the attribution comparison against influence functions, representer point selection, TracIn, and an inner-product baseline, the proposed coefficient achieves the highest Top-30 action consistency (0.941 ± 0.030) while remaining competitive on Pearson correlation with ground-truth case contributions (0.368 ± 0.172).

The audit cost averages 58.6 ms per query in the cached implementation, higher than sub-millisecond baselines, but acceptable for post-training audit scenarios where decision-relevant case retrieval is the priority.

Boundaries and future work

The authors are clear about the framework's limitations. The account is exact only for fixed-representation OLS readouts; end-to-end nonlinear heads are not covered. The real-covariate experiments use synthetic or proxy action returns, and large training memories may require Nyström approximations. Extending the analysis to nonlinear probes and estimated counterfactual returns remains open.

Despite these boundaries, the work provides a practical post-training audit lens for neural decisions through explicit case-based evidence. When a model scores a loan application, a diagnosis, or an energy bid, the framework can now answer: which training cases drove that score, what outcomes they carried, and how coherent their support is. In a landscape where AI systems struggle to remember user preferences from one session to the next, the ability to trace a decision back to its training antecedents marks a meaningful step in a direction that matters more than raw benchmark scores.