Overview

Four developments make the unit of AI evaluation harder to fake. Rare-disease leads stop short of diagnoses, coding challenges run for days, metered agents turn retries into customer expense, and prompt injections cross boundaries between cooperating agents.

1. AI reanalysis surfaces candidate diagnoses in 18 rare-disease cases

OpenAI reports reanalyzing 376 previously unsolved rare-disease cases and surfacing candidate diagnoses for 18 of them. The workflow revisited genetic and clinical evidence after conventional diagnostic processes had failed to produce an answer.

Each lead remains a hypothesis for specialist review and confirmatory testing, with false leads affecting the value of the workflow as directly as recovered diagnoses. The study evaluates case reanalysis under research conditions, not independent diagnosis or prospective patient outcomes.

Sources: OpenAI's rare-disease study

2. Terminal-Bench moves coding-agent evaluation from patches to whole systems

Terminal-Bench Challenges introduces three projects designed to run for days, cost at least about $1,000, and produce 10,000 to 500,000 lines of code. Unlike issue-level benchmarks, each task asks an agent to build or transform a complete system without human intervention.

Initial 12-hour trials found no significant Rust compiler speedup and no correct sub-25-kilobyte inference engine. On the WebGL task, one Fable 5 run reached 96.4% WebGL 1.0 coverage but only 20.5% on WebGL 2.0, exposing progress hidden by a single pass label.

Sources: Terminal-Bench Challenges · Challenge repository

3. Metered AI pricing transfers branching and retry variance to customers

AI software vendors are moving some products from flat subscriptions toward charges based on tokens or work performed, Bloomberg reports. The model aligns revenue with inference consumption while exposing customers to variable context, branching, tool calls, and retries.

A seat-price comparison cannot represent that variance. Cost per accepted task combines model rates with cache policy, context growth, retry limits, failed branches, and human review, making an apparently cheap token expensive when an agent repeatedly misses the completion criterion.

Sources: Bloomberg on token pricing

4. Deep-XPIA finds stacked defenses trade attack reduction for false positives

Deep-XPIA measures prompt injections that cross memory, tool, and delegation boundaries in multi-agent AI systems. In a 300-case Claude Haiku run, 69% of undefended attacks succeeded; intent verification reduced success to 55% with a 1% false-positive rate.

Stacking five defenses reduced attack success to 12% but raised false positives to 31%. Registry injection remained the blind spot because malicious instructions entered during tool discovery, upstream of defenses applied to the prompt stream.

Sources: Deep-XPIA repository and results · Benchmark site