What OpenAI o1’s performance means for exams and expert benchmarks

Machine-assisted translation of the archived article. Read the Chinese original.

With o1, the author sees the industry’s focus shifting from parameter counts to time spent reasoning. The article presents that shift as a possible step towards artificial general intelligence.

What OpenAI o1’s performance means for exams and expert benchmarks — original illustration

As an early model, o1 had several feature limitations at launch.

The author recorded the following errors when testing unsupported features.

system field — HTTP 400

tools field — HTTP 400

Image input — HTTP 400

json_object output — HTTP 500

Structured output — HTTP 400

logprobs output — HTTP 403

Streaming output — HTTP 400

The article reports limits of 20 RPM and 150,000,000 TPM for o1. Requests over a limit could receive HTTP 429.

The temperature, top_p and n parameters were fixed at 1.

The presence_penalty and frequency_penalty parameters were fixed at 0.


What OpenAI o1’s performance means for exams and expert benchmarks — original illustration

The o1 family and o1-preview

OpenAI reported that its reasoning model performed at a level comparable to PhD students on challenging physics, chemistry and biology benchmarks.

On a qualifying exam for the International Mathematical Olympiad, GPT-4o solved 13% of the problems and the reasoning model solved 83%. In competitive programming, it reached the 89th percentile on Codeforces.

OpenAI noted that the early model lacked several ChatGPT features, including browsing and uploading files or images. It expected GPT-4o to remain the stronger option for many everyday tasks in the near term.

OpenAI described the result as a significant advance for complex reasoning, reset its model numbering and named the new series OpenAI o1.

What OpenAI o1’s performance means for exams and expert benchmarks — original illustration

The examples OpenAI gave for o1 included annotating cell sequencing data, generating mathematical expressions for quantum optics and helping developers carry out multi-step workflows.

At launch, developers in API usage tier 5 could prototype with o1 at 20 requests per minute. OpenAI said it would work to raise those limits after more testing.