API Reference

Evals

Copy Markdown

Open in Claude

Open in ChatGPT

Open in Cursor

---

Copy Markdown

View as Markdown

Get an eval

evals.retrieve(eval_id) -> EvalRetrieveResponse { id, created_at, data_source_config, 4 more }

GET/evals/{eval_id}

Get an evaluation by ID.

ParametersExpand Collapse

eval_id: String

ReturnsExpand Collapse

class EvalRetrieveResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

• Improve the quality of my chatbot
• See how well my chatbot handles customer support
• Check if o4-mini is better at my usecase than gpt-4o

id: String

Unique identifier for the evaluation.

created_at: Integer

The Unix timestamp (in seconds) for when the eval was created.

formatunixtime

data_source_config: EvalCustomDataSourceConfig { schema, type } | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig { schema, type, metadata }

Configuration of data sources used in runs of the evaluation.

metadata: Metadata

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

name: String

The name of the evaluation.

object: :eval

The object type.

testing_criteria: Array[LabelModelGrader { input, labels, model, 3 more } | StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } & { pass_threshold} | 2 more]

A list of testing criteria.

Get an eval

Ruby

HTTP

Python

TypeScript

Go

Ruby

Java

CLI Tool

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

eval_ = openai.evals.retrieve("eval_id")

puts(eval_)

{
  "object": "eval",
  "id": "eval_67abd54d9b0081909a86353f6fb9317a",
  "data_source_config": {
    "type": "custom",
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object",
          "properties": {
            "input": {
              "type": "string"
            },
            "ground_truth": {
              "type": "string"
            }
          },
          "required": [
            "input",
            "ground_truth"
          ]
        }
      },
      "required": [
        "item"
      ]
    }
  },
  "testing_criteria": [
    {
      "name": "String check",
      "id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
      "type": "string_check",
      "input": "{{item.input}}",
      "reference": "{{item.ground_truth}}",
      "operation": "eq"
    }
  ],
  "name": "External Data Eval",
  "created_at": 1739314509,
  "metadata": {},
}

Returns Examples

{
  "object": "eval",
  "id": "eval_67abd54d9b0081909a86353f6fb9317a",
  "data_source_config": {
    "type": "custom",
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object",
          "properties": {
            "input": {
              "type": "string"
            },
            "ground_truth": {
              "type": "string"
            }
          },
          "required": [
            "input",
            "ground_truth"
          ]
        }
      },
      "required": [
        "item"
      ]
    }
  },
  "testing_criteria": [
    {
      "name": "String check",
      "id": "String check-2eaf2d8d-d649-4335-8148-9535a7ca73c2",
      "type": "string_check",
      "input": "{{item.input}}",
      "reference": "{{item.ground_truth}}",
      "operation": "eq"
    }
  ],
  "name": "External Data Eval",
  "created_at": 1739314509,
  "metadata": {},
}