The Last Mile of Coding Agents for Long-Document Tasks: An Empirical Study on Clinical Trial Protocol Digitization

Jiacheng Lin1, Geoffrey Low2, Nishit Agarwal2, Rahul Jain2, Afrah Shafquat2, Mandis Beigi2, Jimeng Sun1
1University of Illinois Urbana-Champaign 2Medidata Solutions
Paper Code Benchmark
Overview of the study: coding agents must localize evidence, interpret complex layouts, and link information across long protocols; overall score against cost per run; and TrialScribe with Qwen3.8-27B against the same model without it, Opus 5, and GPT-5.6-Sol.

(a–c) Coding agents must localize evidence, interpret complex layouts, and link information across long protocols; output quality varies across tasks. (d) Overall score on TrialPD-Bench against average cost per run. (e) TrialScribe with Qwen3.8-27B versus the same model without it, Opus 5, and GPT-5.6-Sol.

Tasks

The input to every task is the entire protocol PDF, not a pre-selected excerpt. Protocols in TrialPD-Bench run to a median of 122 pages and up to 368, and finding the few pages that hold the answer is part of the task. Each example below marks where the answer sits in the PDF, shows that page, and gives the reference output; long outputs are shortened for display.

Find the disease or condition the study investigates and return it in the protocol's own wording, keeping qualifiers such as severity or stage.

Input the entire protocol PDF

Protocol PDF74 pages
p. 1p. 74
answer on pages 1, 9similar text on pages 10, 16, 19, 21, 23, 69–70, 73–74 (objectives, background, amendment history, references)
page 1 of 74Page 1 of the protocol PDF

Output indications.json · reference annotation

{
  "descriptions": [
    "Wilson disease"
  ]
}

Evaluation: Indication F1 · Test set: 40 protocols

Abstract

Clinical trial protocols serve as the source of truth for study design, but they are mostly written and used as unstructured PDF documents. Digitizing protocols turns static content into structured, machine-readable data that clinical software can analyze, validate, and execute. Ultimately, protocol digitization transforms protocols from readable documents into actionable digital assets that streamline trial execution. Coding agents, which select tools, execute code, and revise their approach as they work, are increasingly applied to such long-document tasks, yet it remains unclear how reliably they turn complete documents into accurate structured outputs. We study this question using protocol digitization as a testbed. We introduce TrialPD-Bench, an expert-annotated benchmark of 150 publicly available protocol PDFs spanning four task categories. Beyond final-answer performance, our evaluation measures evidence recall from agent traces, agent behavior, and cost. Across nine proprietary and open-weight models in five coding-agent harnesses, even the strongest agents do not fully solve protocol digitization. Our study further sheds light on where coding agents fall short on long documents: agents usually locate the required evidence but fail in the last mile of transcribing and structuring it. Building on these findings, we develop TrialScribe, a domain-specific harness that equips coding agents with task skills and a verification loop. TrialScribe improves a locally served open-weight model by 6.2–59.4 points across eight metrics, enabling it to match or surpass proprietary models such as GPT-5.6-Sol.

TrialPD-Bench

Every task starts from a complete protocol PDF from ClinicalTrials.gov, so locating the relevant section is part of the task. Sponsors organize protocols differently: a Schedule of Activities (SoA) table may appear near the beginning or in an appendix, with its own arrangement of study periods, visit windows, multi-level headers, and footnotes. Five experts in the clinical trial domain produced the reference annotations, and output schemas follow the CDISC Unified Study Definitions Model (USDM).

150 protocol PDFs 17,471 pages 106 sponsors 18 therapeutic areas median 122 pages 9 models × 5 agent harnesses
TaskInputSamplesReference annotationsMain metrics
Disease indicationProtocol PDF4041 indication descriptions; 1–2 per sampleIndication F1
Eligibility criteriaProtocol PDF1343,974 criteria (1,337 inclusion, 2,637 exclusion)Eligibility criteria F1
Objectives and endpointsProtocol PDF89665 objectives, 1,484 endpoints; 60 objectives without endpointsObjective, endpoint, objective-linked endpoint F1
SoA tablesProtocol PDF3770 tables; 10 protocols with multiple SoA pagesActivity–Visit F1, Sub/sup F1
Structured timelineProtocol PDF314 epochs, 69 encounters, 115 activitiesEpoch accuracy, activity and encounter recall

Test-split statistics; the development protocols used to build TrialScribe are held out.

Overview of TrialPD-Bench: source documents and the four task categories, therapeutic-area coverage, document-length distribution, and study counts across sponsors.

Overview of TrialPD-Bench: source documents and the four task categories, therapeutic-area coverage, document-length distribution (median 122 pages), and study counts across 106 sponsors.

Text-based and scanned PDFs

The protocols come as they were posted. Some have a usable text layer. Others are scans, or their text layer drops or garbles characters, so the content can be read only from the rendered pages or with OCR; 16 of the 150 documents have no extractable text at all. We call a PDF vision-required for a task when the content that task needs cannot be recovered from the text layer.

Text-basedVision-required (scanned or broken text layer)text / vision
Disease indication 33 / 7
Eligibility criteria 94 / 40
Objectives and endpoints 61 / 28
SoA table, single page 61 / 25
SoA table, full PDF 29 / 8
Structured timeline 3 / 0

What the Tasks Require

Protocol digitization combines five capabilities. Our failure analysis groups agent errors into the same five categories, and the dominant one differs by task: transcription fidelity for eligibility criteria, relational structure for SoA tables, and domain concept grounding for disease indication and structured timelines.

Transcription fidelity

Copy the source text exactly, including numbers, units, and comparison operators.

e.g., “≥ 100 meters” written as “> 100 meters”

Long-document localization

Find every relevant passage in a protocol of a hundred pages or more, across sections and page breaks.

e.g., 9 of 11 objectives missed on one protocol

Relational structure

Rebuild how items relate: marks in the right SoA cells, endpoints under the right objective, epochs in order.

e.g., one misresolved merged cell shifts every mark after it

Domain concept grounding

Pick the clinical entity, category, or controlled term the standard asks for.

e.g., “Non-Valvular Atrial Fibrillation” for a study of recurrent embolic stroke

Execution reliability

Finish the multi-step run and write a complete, valid output file.

e.g., missing, empty, or unparseable output

Main Results

  • Even the strongest coding agents do not fully solve protocol digitization. The best agents reach 87.8 eligibility criteria F1 and 86.8 SoA Activity–Visit F1, but only 67.4 endpoint F1, 58.7 objective-linked endpoint F1, and 23.2 encounter recall. The largest gaps lie in the relational structure of a protocol rather than in its surface content.
  • Locating the evidence is not the main bottleneck. Except for Kimi-K3 on objectives and endpoints, the seven frontier agents visit at least 84.8% of the required evidence pages on every task, yet higher evidence recall does not translate into better extraction. On structured timelines, every agent visits at least 98.3% of the evidence pages while encounter recall stays at or below 23.2. Agents fail in the last mile of transcribing and structuring the evidence.
  • Tool errors and interaction turns scale with task complexity. Averaged over the nine agents, SoA table extraction takes 39.8 turns per run, against 7.1 for indication extraction, and weaker models incur more tool errors and use visual evidence less often when it is required. Longer trajectories do not produce better answers.
  • Similar accuracy can come at very different costs. On eligibility criteria, the seven frontier agents lie within 1.4 F1 points of each other while the cost per run ranges from $0.40 to $1.35; DeepSeek-V4-Flash reaches 80.2 F1 at $0.04 per run.
Main results of nine coding agents on TrialPD-Bench: final-answer metrics and evidence recall for each task.

Main results (%) on TrialPD-Bench. Each task reports its final-answer metrics followed by evidence recall (ER), the recall of the required evidence pages visited in the agent trace.

SoA Activity-Visit F1 against model release date.

SoA Activity–Visit F1 on complete protocols against model release date. Hollow markers denote open-weight models.

Page-level SoA extraction: coding agents against OCR and document-understanding baselines.

Single SoA pages: coding agents against OCR and document-understanding baselines.

Failure Analysis

We classify the failures in each failing agent run into five main categories: transcription fidelity, long-document localization and coverage, relational structure reconstruction, domain concept grounding, and execution reliability. The dominant category differs by task. Transcription fidelity accounts for 88.0% of failing eligibility-criteria runs: agents read the right section but do not reproduce it exactly, and among near-verbatim criteria, 167 alter a comparison operator and 308 alter a number. Relational structure accounts for 58.9% of failing SoA runs, most often because scheduling marks land in the wrong cells. Domain concept grounding dominates disease indication and structured timelines, while execution reliability accounts for at most 3.6% of failing runs in any task.

Protocol≥ 100 meters Agent> 100 meters
Distribution of failure categories within each task, and examples of the five categories.

(a) Distribution of failure categories within each task. (b) Examples of the five categories, with incorrect or missing predictions in red.

TrialScribe

Protocols are often confidential, which favors compact open-weight models that run locally, and most failures arise after the evidence is found. TrialScribe is a domain-specific harness layer that equips an existing coding agent with task skills, which supply domain knowledge and output conventions, and a verification loop, in which a fresh agent session audits the draft against the protocol and the agent applies minimal source-supported fixes. We run it with Qwen3.8-27B in the Pi harness, served locally on two NVIDIA A100 GPUs.

+9.2Indication F1
+15.2Eligibility criteria F1
+22.2Objective F1
+18.8Endpoint F1
+42.6SoA Activity–Visit F1
+37.1Epoch accuracy
+6.2Activity recall
+59.4Encounter recall

With TrialScribe, the local model surpasses GPT-5.6-Sol on endpoint extraction (65.5 vs. 63.6) and comes within 1.9 points of Opus 5. On structured timelines, it exceeds both proprietary agents on activity recall (88.6 vs. 88.1 and 87.2) and encounter recall (65.2 vs. 23.2 and 0.0). Verification helps most where transcription fidelity dominates, raising eligibility criteria F1 from 72.2 to 82.3. On SoA tables, TrialScribe reaches 75.1 Activity–Visit F1 but still trails GPT-5.6-Sol and Opus 5 (83.7 and 86.0).

Qwen3.8-27B with and without TrialScribe, compared with Opus 5 and GPT-5.6-Sol, on eight metrics.

Qwen3.8-27B with TrialScribe, the same model without it, Opus 5, and GPT-5.6-Sol. The span above each Qwen bar gives the gain in points.

SoA Case Study

A two-page Schedule of Activities with merged visit headers and footnote markers. Choose a system to compare its HTML output, annotated against the reference, with the source pages.

Source SoA page 14
Source page 14
Source SoA page 15
Source page 15
Correct mark Mark not in the reference Missed reference mark
TrialScribe output for the two SoA pages, annotated against the reference

This protocol was used to develop the TrialScribe SoA workflow and is not part of the SoA evaluation set.

BibTeX

@article{lin2026lastmile,
  title={The Last Mile of Coding Agents for Long-Document Tasks: An Empirical Study on Clinical Trial Protocol Digitization},
  author={Lin, Jiacheng and Low, Geoffrey and Agarwal, Nishit and Jain, Rahul and Shafquat, Afrah and Beigi, Mandis and Sun, Jimeng},
  journal={arXiv preprint arXiv:2610.18507},
  year={2026}
}