The most defensible finding across recent research is that model accuracy alone does not determine whether an AI coding workflow succeeds operationally. A 2026 deployment study in npj Digital Medicine reports that an HL7-CDA-aligned LLM reduced coding time in a 13-week randomized controlled trial while maintaining accuracy, but satisfaction varied by experience, certification, and generational cohort [1][2]. That split between technical performance and user acceptance recurs across audit tools, retrieval architectures, and agentic systems. For health informatics and revenue cycle leaders, the practical question is not which model scores highest on a public benchmark. It is which workflow can be specified, measured, and audited inside constrained clinical documentation environments where datasets are often closed and coding experts are scarce.
The Named Workflow: A Modular, HL7-Aligned Pipeline
The npj Digital Medicine study describes a modular pipeline rather than a single model endpoint. It moves through data acquisition, redundancy-aware sampling, LLM-as-judge base-model selection with Plackett-Luce ranking, section-aware fine-tuning, and deployment into a hospital coding workflow [1][2]. Base-model selection was formulated as a pairwise preference task rather than relying on lexical overlap or cosine similarity [1]. This sequence is materially different from conventional pipelines that apply ad hoc model selection and exhaustive fine-tuning of all candidates, which the authors characterize as resource-inefficient and hard to scale [2].
The fine-tuning step is also specific. The authors applied section-wise instruction tuning using templates derived from HL7-CDA R2-aligned discharge summaries, marking clinical sections such as DischgDiag with an explicit prefix followed by section name and content [1]. Parameter-level supervised fine-tuning, rather than prompt-only inference, incorporated structured HL7 content into training [1]. For a technology team evaluating this design, the decision point is whether existing discharge documentation is sufficiently structured to support section-aware prompts, and whether the training corpus can represent the institution's actual code distribution.
Where Evaluation Happens: Closed Data and Human Factors
A recurring limitation is access. The hospital deployment dataset is not publicly available due to ethical and legal restrictions related to patient privacy and IRB approval; the authors state that no public or third-party access is permitted under the approved ethical protocol [2][3]. This constrains independent replication and means that reported reductions in coding time cannot be re-derived from open records. Any procurement team comparing this workflow to a vendor claim should treat the absence of a shared evaluation set as a constraint on verification, not a sign of weaker science.
The same study shows that human factors are measurable but uneven. AI-assisted workflows reduced coding time while maintaining accuracy across ten certified coding specialists in a 13-week human-in-the-loop randomized controlled trial, yet satisfaction varied by experience, certification, and generational cohort [1][2]. The authors conclude that successful adoption operates across documentation infrastructure, workflow uptake, and individual user acceptance, which is why accuracy alone is insufficient [2]. This finding should discipline any rollout plan that assumes uniform coder readiness.
The audit study from the International Journal of Medical Informatics introduces a different issue: benchmark quality. Three professional coders using an interpretable LLM pipeline estimated a 76.3 percent undercoding rate in MDACE and a 29.7 percent error rate in CodiEsp [5]. Re-evaluating existing models on the corrected datasets improved performance consistently across all metrics on MDACE and in recall on CodiEsp [5]. The implication for workflow research is that evaluation numbers inherited from flawed human-annotated datasets may encode systematic errors rather than model capability.
What to Require Before Pilot Selection
Require the vendor or internal team to specify the evidence trace for every predicted code, not a single aggregate accuracy figure. If the workflow auto-assigns codes, require a confidence threshold with a documented routing rule for low-confidence codes sent to human review [8]. If the workflow uses retrieval, require disclosure of the embedding model and code description source, because the MedCodER results show material variation across embedding choices [7].
Require a benchmark audit before accepting any reported F1. The audit study's finding that re-evaluating existing models on the corrected datasets improved performance consistently across all metrics on MDACE and in recall on CodiEsp means inherited benchmark numbers may not transfer to your documentation [5]. Request a sample-level error taxonomy separating undercoding, overcoding, and mutually exclusive code conflicts, and ask whether the training data follows the same section conventions as your EHR output [1].
Plan for heterogeneous coder response. The npj Digital Medicine study found satisfaction varied by experience, certification, and generational cohort even when coding time improved [1]. A pilot should measure time, accuracy, and coder-reported satisfaction separately, and segment results by coder background rather than averaging them.
Technical Decision Points: Retrieval, Audit, and Agentic Design
Industry and academic systems make different choices about how to connect a model to the code space. MedCodER, presented at NAACL 2025, extracts diagnoses and then generates a candidate set of ICD-10 codes through semantic search against descriptions from the ICD-10 ontology and UMLS Metathesaurus, retrieving top-k codes by cosine distance before a code-to-record re-ranking step [7]. This design mitigates the large label space by constraining candidates rather than prompting the model to emit any code from the full ontology. The reported F1 for MedCodER at k=1 on ICD-10 coding was 0.62 with GPT-4o, falling to 0.49 when MedCPT embeddings replaced the standard retrieval embeddings [7].
By contrast, the Symphony system from Corti, described in an arXiv preprint, implements a staged agent sequence: evidence extraction, code selection through tabular hierarchy navigation, and code reconciliation to remove mutually exclusive codes, with each stage implemented as an LLM-powered agent accessing structured coding resources [4]. The authors claim span-level evidence attribution for every predicted code and evaluation across five datasets spanning inpatient and outpatient settings in the United States and United Kingdom [4]. These are vendor-reported claims from a preprint and should be treated as unverified until peer review or customer audit.
The ACL 2025 position paper from University of Sydney and CSIRO Data61 recommends separating two workflow designs: present all predicted codes to a human expert, or auto-assign high-confidence codes and route low-confidence codes to an expert [8]. The authors note that the second design changes the optimization objective because some top-confidence errors are acceptable as long as correct options are included [8]. They also recommend explainability metrics such as matching annotated evidence spans, and hierarchical metrics that account for code structure rather than penalizing all mispredictions equally [8].
A Concrete Evaluation Example, Without Invented Results
A feasible evaluation could compare the HL7-aligned pipeline from Dai et al. [1] with the audit pipeline from Khadka et al. [5] on a de-identified sample of discharge summaries from a single hospital. The unit of comparison would not be a single F1 score, but three paired measures: coding time per record with coder review, the rate of undercoding detected by a second coder, and the proportion of predicted codes that carry a retrievable evidence span. The audit pipeline is designed to identify undercoding and data quality problems [5], while the HL7-aligned pipeline is designed to fit sectioned discharge documentation [1]. Neither source reports a head-to-head result.
The interpretability requirement should be explicit in any evaluation protocol. The audit study reports a three-component structure: LLM-based evidence extraction, multiclass classification, and evidence-supported code verification [5]. Symphony claims span-level evidence attribution, but this is a preprint claim [4]. The ACL review argues that evidence span matching is particularly useful when system output is intended to inform human decision-making [8]. A strict evaluation would therefore require coders to adjudicate whether each predicted code is traceable to a specific section or phrase, and would record disagreement rates rather than treating traceability as a binary property.
This example is a suggestion, not a tested protocol. No source in the review reports such a three-way comparison, and the closed hospital dataset from the npj Digital Medicine study cannot be reused under its ethical protocol [2][3]. A team running this evaluation would need its own IRB approval and a coding specialist panel large enough to produce stable inter-annotator agreement, following the calibration approach described in the audit study, where agreement coefficients were computed on a 10 percent calibration sample independently annotated by all three coders [5].
Sources & references
- Evaluating real-world deployment of an HL7-CDA-aligned LLM for ICD-10-CM coding | npj Digital Medicine · www.nature.com
- Evaluating real-world deployment of an HL7-CDA-aligned LLM for ICD-10-CM coding | npj Digital Medicine · www.nature.com
- Evaluating real-world deployment of an HL7-CDA-aligned LLM for ICD-10-CM coding | npj Digital Medicine · www.nature.com
- Symphony for Medical Coding: A Next-Generation Agentic System for Scalable and Explainable Medical Coding · arxiv.org
- LLM-assisted clinical coding audit through an interpretable coding pipeline - ScienceDirect · www.sciencedirect.com
- Generative AI enabled Medical Coding on AWS | AWS for Industries · aws.amazon.com
- [PDF] MedCodER: A Generative AI Assistant for Medical Coding · aclanthology.org
- Aligning AI Research with the Needs of Clinical Coding Workflows: Eight Recommendations Based on US Data Analysis and Critical Review · aclanthology.org