Owner announces the Verified upgrade on July 28, 2025. Historical OSWorld scores must remain version-separated.
Definition refers to the July 2025 Verified revision, not OSWorld 1.0 or V2. Exact task denominator and run configuration must be supplied for every future result.
OSWorld Verified evaluates desktop tasks. Its reported metric is task success under versioned execution checks. Read the release-specific task definitions before treating this as a proxy for your own workflows.
Results belong to the exact system and evaluation setup named in the source. A vendor’s current API is not automatically the system used in a historical run. Hosted browser reachability, task completion and full desktop interaction are different measurements.
Owner methodology and measurement boundary
Owner announces the Verified upgrade on July 28, 2025. Historical OSWorld scores must remain version-separated.
Reproducibility requirements
Resolve exact task denominator, release and run configuration for each result.
Version history and interpretation
Verified was announced July 28, 2025. OSWorld V2 introduces a separate long-horizon task family; neither name permits migration of scores from earlier evaluations.
Attributed results
Numeric results are withheld: a version-matched result has not passed our evidence checks. This methodology guide remains useful for understanding scope and reproducibility.
Can these results be compared?
No unified ranking is published. These records do not establish a controlled comparison across current tools.
Match benchmark code, tasks, assets and environment versions.
Check whether resets, retries, task exclusions and human interventions are counted.
Compare the same metric: task success is different from reachability or cost per attempt.
Re-run representative tasks using your own constraints before a purchase or deployment decision.
Owner announces the Verified upgrade on July 28, 2025. Historical OSWorld scores must remain version-separated.
Scope: desktop; Definition refers to the July 2025 Verified revision, not OSWorld 1.0 or V2. Exact task denominator and run configuration must be supplied for every future result. · Unit: Task success under versioned execution checks · Version: Verified revision · 2025-07-28
Limitation: Definition refers to the July 2025 Verified revision, not OSWorld 1.0 or V2. Exact task denominator and run configuration must be supplied for every future result.