API Reference

Evals

Copy Markdown

Open in Claude

Open in ChatGPT

Open in Cursor

---

Copy Markdown

View as Markdown

Create eval

evals.create(**kwargs) -> EvalCreateResponse { id, created_at, data_source_config, 4 more }

POST/evals

Create the structure of an evaluation that can be used to test a model’s performance. An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the Evals guide.

ParametersExpand Collapse

data_source_config: Custom{ item_schema, type, include_sample_schema} | Logs{ type, metadata} | StoredCompletions{ type, metadata}

The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.

testing_criteria: Array[LabelModel{ input, labels, model, 3 more} | StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } & { pass_threshold} | 2 more]

A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like {{item.variable_name}}. To reference the model’s output, use the sample namespace (ie, {{sample.output_text}}).

metadata: Metadata

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

name: String

The name of the evaluation.

ReturnsExpand Collapse

class EvalCreateResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

• Improve the quality of my chatbot
• See how well my chatbot handles customer support
• Check if o4-mini is better at my usecase than gpt-4o

id: String

Unique identifier for the evaluation.

created_at: Integer

The Unix timestamp (in seconds) for when the eval was created.

formatunixtime

data_source_config: EvalCustomDataSourceConfig { schema, type } | Logs{ schema, type, metadata} | EvalStoredCompletionsDataSourceConfig { schema, type, metadata }

Configuration of data sources used in runs of the evaluation.

metadata: Metadata

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

name: String

The name of the evaluation.

object: :eval

The object type.

testing_criteria: Array[LabelModelGrader { input, labels, model, 3 more } | StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } & { pass_threshold} | 2 more]

A list of testing criteria.

Create eval

Ruby

HTTP

Python

TypeScript

Go

Ruby

Java

CLI Tool

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

eval_ = openai.evals.create(
  data_source_config: {item_schema: {foo: "bar"}, type: :custom},
  testing_criteria: [
    {
      input: [{content: "content", role: "role"}],
      labels: ["string"],
      model: "model",
      name: "name",
      passing_labels: ["string"],
      type: :label_model
    }
  ]
)

puts(eval_)

{
  "object": "eval",
  "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
  "data_source_config": {
    "type": "stored_completions",
    "metadata": {
      "usecase": "chatbot"
    },
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object"
        },
        "sample": {
          "type": "object"
        }
      },
      "required": [
        "item",
        "sample"
      ]
  },
  "testing_criteria": [
    {
      "name": "Example label grader",
      "type": "label_model",
      "model": "o3-mini",
      "input": [
        {
          "type": "message",
          "role": "developer",
          "content": {
            "type": "input_text",
            "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
          }
        },
        {
          "type": "message",
          "role": "user",
          "content": {
            "type": "input_text",
            "text": "Statement: {{item.input}}"
          }
        }
      ],
      "passing_labels": [
        "positive"
      ],
      "labels": [
        "positive",
        "neutral",
        "negative"
      ]
    }
  ],
  "name": "Sentiment",
  "created_at": 1740110490,
  "metadata": {
    "description": "An eval for sentiment analysis"
  }
}

Returns Examples

{
  "object": "eval",
  "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
  "data_source_config": {
    "type": "stored_completions",
    "metadata": {
      "usecase": "chatbot"
    },
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object"
        },
        "sample": {
          "type": "object"
        }
      },
      "required": [
        "item",
        "sample"
      ]
  },
  "testing_criteria": [
    {
      "name": "Example label grader",
      "type": "label_model",
      "model": "o3-mini",
      "input": [
        {
          "type": "message",
          "role": "developer",
          "content": {
            "type": "input_text",
            "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
          }
        },
        {
          "type": "message",
          "role": "user",
          "content": {
            "type": "input_text",
            "text": "Statement: {{item.input}}"
          }
        }
      ],
      "passing_labels": [
        "positive"
      ],
      "labels": [
        "positive",
        "neutral",
        "negative"
      ]
    }
  ],
  "name": "Sentiment",
  "created_at": 1740110490,
  "metadata": {
    "description": "An eval for sentiment analysis"
  }
}