Overview

Three evaluations show why aggregate capability scores hide the mechanism of success or failure. Adaptive jailbreaks outperform static attacks, hierarchical demonstrations help only under ambiguous instructions, and fast robot setup does not translate into precise physical control.

1. Red-team study examines Anthropic's latest model behavior

An independent white paper generated hundreds of thousands of attacks against Fable 5 and Opus 4.8 across 7,826 harmful intents and four jailbreak families. Three judge models re-adjudicated each apparent success by majority vote.

Static obfuscation was almost entirely blocked, but adaptive tree-of-attacks reached 11.5% success against Opus and 6.1% against Fable in the worst tested condition. The panel confirmed 1,620 harmful Opus completions and 702 Fable completions, all under one automated attack framework.

Sources: arXiv

2. Hierarchical demonstrations lift ambiguous web-task pass rates by 14 points

Researchers held action sequences constant across five prompting conditions for 85 web-automation tasks. On 43 vaguely described tasks, grouping the demonstration into named hierarchical subgoals raised pass rates from 76.7% to 90.7%; flat logs produced a smaller, statistically insignificant change.

None of the demonstration formats improved the 42 tasks with precise descriptions. An ablation attributed the ambiguous-task gain to subgoal grouping alone, with preconditions, postconditions, and parameter annotations adding no measurable benefit.

Sources: Hierarchical-demonstration preprint

3. Opus accelerates robot setup but fails the final closed-loop control task

Across three trials, Opus 4.7 completed five software and sensor tasks for an off-the-shelf robot dog in an average 12 minutes 7 seconds. A prior Claude-assisted human team took 264 minutes, and Opus used 1,045 lines of code compared with that team's 10,309.

The model connected sensors, wrote controls, and detected a beach ball, but could not autonomously nudge the ball back to its starting point. The trial distinguishes rapid software integration from feedback-sensitive physical manipulation without measuring general robotic autonomy.

Sources: Anthropic's Project Fetch report