nMAS automates heart-failure EHR features, tested only on 500 dummy patients

Before any model gets trained on a heart-failure study, someone has to turn raw electronic health records into features a model can learn from: medication histories, lab trajectories, the small pieces that actually carry the clinical signal. That work absorbs 39 to 45% of data scientists' time on EHR projects, the preprint reports. For heart failure, which affects an estimated 6.7 million U.S. adults, it is worse, because the evidence is scattered across fragmented records and has to be reconciled with disease-specific, guideline-based clinical reasoning. It is the kind of workload the productivity paradox is about: automation that was supposed to save time, and often just moves the burden around.

Posted to arXiv on August 6, the paper argues that rule-based systems and LLM-based approaches automate only part of this, with limited maintainability and weak traceability of where each feature came from. Its answer is the Nimblemind Multi-Agent System, nMAS, an evidence-linked pipeline that grounds every feature in the source data and scores its own output against a rubric as it goes.

On 500 dummy patient records drawn from nine EHR source tables, nMAS generated 132 structured features and 70 rubric-scored aggregated features, with checks for structural integrity, provenance, and rubric compliance at every stage. A restricted LLM audited the output, and an independent LLM-based rubric assessment of evidence support and methodological soundness scored the features at 81.5% of the maximum points.

What the AUROC gains actually show

The headline results are the phenotyping lifts. Adding nMAS's aggregated features to the held-out evaluation raised AUROC from 0.895 to 0.963 for HFrEF, heart failure with reduced ejection fraction, and from 0.870 to 0.910 for HFpEF, the preserved form. The distinction matters because the two forms are treated differently.

Those are meaningful moves on a held-out set. The catch is that the whole evaluation ran on dummy records. There is no real patient data here, no multi-site cohort, and the paper says external validation is still needed. A pipeline that performs on dummy data has proven it can follow its own rules. It has not proven it survives the mess of real hospitals, a familiar failure mode for agents that work in demos and stall in production.

The abstract reports no comparison against features built by hand. So the workload argument rests on an assumption: that automated features are good enough to replace that human effort, not just supplement it.

Auditability is the story, not the AUROC

The deeper claim is traceability. Most automated feature engineering is a black box: the downstream model cannot explain the features, and the features carry no history. nMAS is built so each feature traces back to evidence in the source tables, carries a rubric score, and has been checked by a restricted LLM auditor. For clinical work, that trail is the difference between a model a regulator can inspect and a model nobody can approve. It is the clinical version of a gap enterprise AI already knows: nobody can say what their AI coding agents actually did today.

nMAS lands in a crowded stretch of clinical AI preprints pushing the same direction. ClinPRISM, posted in late July, applies a cost-effective multimodal LLM reasoning framework to question answering over irregular clinical time series, the same sparsity and asynchrony problem EHR data throws at language models. BioSecBench-Surveillance, also from July, is a verifiable benchmark of 100 evaluations testing whether AI agents can infer the right analysis pipeline from raw sequencing data and surveillance context. Verification, not raw accuracy, is the thread running through all of them. It is the same instinct behind BAAI's research agents that check their own homework.

What's missing before real deployment

The paper is upfront about the gap. Evaluation stopped at a single institution's dummy cohort. Real patient records from multiple sites are the obvious next step, along with an audit trail that outsiders can inspect rather than take on the authors' word.

The 81.5% rubric score is the clever part and the fragile part. An independent LLM judged the features on evidence support and methodological soundness, which is a machine grading a machine's homework. Better than nothing, and still one step short of a clinician's sign-off. The agent world keeps rediscovering that making agents trustworthy is not the same thing as making them work.

The pipeline may not need a human hand on every feature anymore. It still needs real patients to prove itself, and that part is not automated yet.