API Reference

Copy Markdown

Open in Claude

Open in ChatGPT

Open in Cursor

---

Copy Markdown

View as Markdown

Evals

Manage and run evals in the OpenAI platform.

Create eval

evals.create(**kwargs) -> EvalCreateResponse { id, created_at, data_source_config, 4 more }

POST/evals

Delete an eval

evals.delete(eval_id) -> EvalDeleteResponse { deleted, eval_id, object }

DELETE/evals/{eval_id}

List evals

evals.list(**kwargs) -> CursorPage<EvalListResponse { id, created_at, data_source_config, 4 more } >

GET/evals

Get an eval

evals.retrieve(eval_id) -> EvalRetrieveResponse { id, created_at, data_source_config, 4 more }

GET/evals/{eval_id}

Update an eval

evals.update(eval_id, **kwargs) -> EvalUpdateResponse { id, created_at, data_source_config, 4 more }

POST/evals/{eval_id}

ModelsExpand Collapse

class EvalCreateResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

• Improve the quality of my chatbot
• See how well my chatbot handles customer support
• Check if o4-mini is better at my usecase than gpt-4o

class EvalCustomDataSourceConfig { schema, type }

A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

• Used to define your testing criteria and
• What data is required when creating a run

class EvalDeleteResponse { deleted, eval_id, object }

class EvalListResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

• Improve the quality of my chatbot
• See how well my chatbot handles customer support
• Check if o4-mini is better at my usecase than gpt-4o

class EvalRetrieveResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

• Improve the quality of my chatbot
• See how well my chatbot handles customer support
• Check if o4-mini is better at my usecase than gpt-4o

class EvalStoredCompletionsDataSourceConfig { schema, type, metadata }

Deprecated in favor of LogsDataSourceConfig.

class EvalUpdateResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

• Improve the quality of my chatbot
• See how well my chatbot handles customer support
• Check if o4-mini is better at my usecase than gpt-4o

EvalsRuns

Manage and run evals in the OpenAI platform.

Cancel eval run

evals.runs.cancel(run_id, **kwargs) -> RunCancelResponse { id, created_at, data_source, 11 more }

POST/evals/{eval_id}/runs/{run_id}

Create eval run

evals.runs.create(eval_id, **kwargs) -> RunCreateResponse { id, created_at, data_source, 11 more }

POST/evals/{eval_id}/runs

Delete eval run

evals.runs.delete(run_id, **kwargs) -> RunDeleteResponse { deleted, object, run_id }

DELETE/evals/{eval_id}/runs/{run_id}

Get eval runs

evals.runs.list(eval_id, **kwargs) -> CursorPage<RunListResponse { id, created_at, data_source, 11 more } >

GET/evals/{eval_id}/runs

Get an eval run

evals.runs.retrieve(run_id, **kwargs) -> RunRetrieveResponse { id, created_at, data_source, 11 more }

GET/evals/{eval_id}/runs/{run_id}

ModelsExpand Collapse

class CreateEvalCompletionsRunDataSource { source, type, input_messages, 2 more }

A CompletionsRunDataSource object describing a model sampling configuration.

class CreateEvalJSONLRunDataSource { source, type }

A JsonlRunDataSource object with that specifies a JSONL file that matches the eval

class EvalAPIError { code, message }

An object representing an error response from the Eval API.

class RunCancelResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

class RunCreateResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

class RunDeleteResponse { deleted, object, run_id }

class RunListResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

class RunRetrieveResponse { id, created_at, data_source, 11 more }

A schema representing an evaluation run.

EvalsRunsOutput Items

Manage and run evals in the OpenAI platform.

Get eval run output items

evals.runs.output_items.list(run_id, **kwargs) -> CursorPage<OutputItemListResponse { id, created_at, datasource_item, 7 more } >

GET/evals/{eval_id}/runs/{run_id}/output_items

Get an output item of an eval run

evals.runs.output_items.retrieve(output_item_id, **kwargs) -> OutputItemRetrieveResponse { id, created_at, datasource_item, 7 more }

GET/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}

ModelsExpand Collapse

class OutputItemListResponse { id, created_at, datasource_item, 7 more }

A schema representing an evaluation run output item.

class OutputItemRetrieveResponse { id, created_at, datasource_item, 7 more }

A schema representing an evaluation run output item.