Skip to content
For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL.
Primary navigation
Theme

Suggested
Overview Models Agents Tools Audio & voice Production API reference

Evals

Manage and run evals in the OpenAI platform.

Create eval
POST
/evals
Delete an eval
DELETE
/evals/{eval_id}
List evals
GET
/evals
Get an eval
GET
/evals/{eval_id}
Update an eval
POST
/evals/{eval_id}
Models
EvalCreateResponse object { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra
EvalCustomDataSourceConfig object { schema, type }

A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

  • Used to define your testing criteria and
  • What data is required when creating a run
schema: map[unknown]

The json schema for the run data source items. Learn how to build JSON schemas here.

type: "custom" (default: "custom")

The type of data source. Always custom.

EvalDeleteResponse object { deleted, eval_id, object }
deleted: boolean
eval_id: string
object: string
EvalListResponse object { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra
EvalRetrieveResponse object { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra
EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }

Deprecated in favor of LogsDataSourceConfig.

EvalUpdateResponse object { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

  • Improve the quality of my chatbot
  • See how well my chatbot handles customer support
  • Check if o4-mini is better at my usecase than gpt-6-astra

EvalsRuns

Manage and run evals in the OpenAI platform.

Cancel eval run
POST
/evals/{eval_id}/runs/{run_id}/cancel
Create eval run
POST
/evals/{eval_id}/runs
Delete eval run
DELETE
/evals/{eval_id}/runs/{run_id}
Get eval runs
GET
/evals/{eval_id}/runs
Get an eval run
GET
/evals/{eval_id}/runs/{run_id}
Models
CreateEvalCompletionsRunDataSource object { source, type, input_messages, 2 more }

A CompletionsRunDataSource object describing a model sampling configuration.

CreateEvalJSONLRunDataSource object { source, type }

A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

EvalAPIError object { code, message }

An object representing an error response from the Eval API.

code: string

The error code.

message: string

The error message.

RunCancelResponse object { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

RunCreateResponse object { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

RunDeleteResponse object { deleted, object, run_id }
deleted: optional boolean
object: optional string
run_id: optional string
RunListResponse object { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

RunRetrieveResponse object { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

EvalsRunsOutput Items

Manage and run evals in the OpenAI platform.

Get eval run output items
GET
/evals/{eval_id}/runs/{run_id}/output_items
Get an output item of an eval run
GET
/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}
Models
OutputItemListResponse object { id, created_at, datasource_item, 7 more }

A schema representing an evaluation run output item.

OutputItemRetrieveResponse object { id, created_at, datasource_item, 7 more }

A schema representing an evaluation run output item.