Create eval

client.evals.create(, ?): EvalCreateResponse { id, created_at, data_source_config, 4 more }

POST/evals

Create eval

ParametersExpand Collapse

body: EvalCreateParams { data_source_config, testing_criteria, metadata, name }

data_source_config: Custom { item_schema, type, include_sample_schema } | Logs { type, metadata } | StoredCompletions { type, metadata }

The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.

One of the following:

Custom { item_schema, type, include_sample_schema }

A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs. This schema is used to define the shape of the data that will be:

Used to define your testing criteria and
What data is required when creating a run

item_schema: Record<string, unknown>

The json schema for each row in the data source.

type: "custom"

The type of data source. Always custom.

include_sample_schema?: boolean

Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)

Logs { type, metadata }

A data source config which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc.

type: "logs"

The type of data source. Always logs.

metadata?: Record<string, unknown>

Metadata filters for the logs data source.

StoredCompletions { type, metadata }

Deprecated in favor of LogsDataSourceConfig.

type: "stored_completions"

The type of data source. Always stored_completions.

metadata?: Record<string, unknown>

Metadata filters for the stored completions data source.

testing_criteria: Array<LabelModel { input, labels, model, 3 more } | StringCheckGrader { input, name, operation, 2 more } | TextSimilarity { pass_threshold } | 2 more>

A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like {{item.variable_name}}. To reference the model’s output, use the sample namespace (ie, {{sample.output_text}}).

One of the following:

LabelModel { input, labels, model, 3 more }

A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

input: Array<SimpleInputMessage { content, role } | EvalItem { content, role, type } >

A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

One of the following:

SimpleInputMessage { content, role }

content: string

The content of the message.

role: string

The role of the message (e.g. “system”, “assistant”, “user”).

EvalItem { content, role, type }

A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

content: string | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText { text, type } | 3 more

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

One of the following:

string

ResponseInputText { text, type, prompt_cache_breakpoint }

A text input to the model.

text: string

The text input to the model.

type: "input_text"

The type of the input item. Always input_text.

prompt_cache_breakpoint?: PromptCacheBreakpoint { mode }

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: "explicit"

The breakpoint mode. Always explicit.

OutputText { text, type }

A text output from the model.

text: string

The text output from the model.

type: "output_text"

The type of the output text. Always output_text.

InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: string

The URL of the image input.

formaturi

type: "input_image"

The type of the image input. Always input_image.

detail?: string

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: InputAudio { data, format }

data: string

Base64-encoded audio data.

format: "mp3" | "wav"

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:

"mp3"

"wav"

type: "input_audio"

The type of the input item. Always input_audio.

GraderInputs = Array<string | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText { text, type } | 2 more>

A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

One of the following:

string

ResponseInputText { text, type, prompt_cache_breakpoint }

A text input to the model.

text: string

The text input to the model.

type: "input_text"

The type of the input item. Always input_text.

prompt_cache_breakpoint?: PromptCacheBreakpoint { mode }

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: "explicit"

The breakpoint mode. Always explicit.

OutputText { text, type }

A text output from the model.

text: string

The text output from the model.

type: "output_text"

The type of the output text. Always output_text.

InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: string

The URL of the image input.

formaturi

type: "input_image"

The type of the image input. Always input_image.

detail?: string

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: InputAudio { data, format }

data: string

Base64-encoded audio data.

format: "mp3" | "wav"

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:

"mp3"

"wav"

type: "input_audio"

The type of the input item. Always input_audio.

role: "user" | "assistant" | "system" | "developer"

The role of the message input. One of user, assistant, system, or developer.

One of the following:

"user"

"assistant"

"system"

"developer"

type?: "message"

The type of the message input. Always message.

labels: Array<string>

The labels to classify to each item in the evaluation.

model: string

The model to use for the evaluation. Must support structured outputs.

The name of the grader.

passing_labels: Array<string>

The labels that indicate a passing result. Must be a subset of labels.

type: "label_model"

The object type, which is always label_model.

StringCheckGrader { input, name, operation, 2 more }

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

input: string

The input text. This may include template strings.

The name of the grader.

operation: "eq" | "ne" | "like" | "ilike"

The string check operation to perform. One of eq, ne, like, or ilike.

One of the following:

"eq"

"ne"

"like"

"ilike"

reference: string

The reference text. This may include template strings.

type: "string_check"

The object type, which is always string_check.

TextSimilarity extends TextSimilarityGrader { evaluation_metric, input, name, 2 more } { pass_threshold }

A TextSimilarityGrader object which grades text based on similarity metrics.

pass_threshold: number

The threshold for the score.

Python extends PythonGrader { name, source, type, image_tag } { pass_threshold }

A PythonGrader object that runs a python script on the input.

pass_threshold?: number

The threshold for the score.

ScoreModel extends ScoreModelGrader { input, model, name, 3 more } { pass_threshold }

A ScoreModelGrader object that uses a model to assign a score to the input.

pass_threshold?: number

The threshold for the score.

metadata?: Metadata | null

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

name?: string

The name of the evaluation.

ReturnsExpand Collapse

EvalCreateResponse { id, created_at, data_source_config, 4 more }

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

Improve the quality of my chatbot
See how well my chatbot handles customer support
Check if o4-mini is better at my usecase than gpt-4o

id: string

Unique identifier for the evaluation.

created_at: number

The Unix timestamp (in seconds) for when the eval was created.

formatunixtime

data_source_config: EvalCustomDataSourceConfig { schema, type } | Logs { schema, type, metadata } | EvalStoredCompletionsDataSourceConfig { schema, type, metadata }

Configuration of data sources used in runs of the evaluation.

One of the following:

EvalCustomDataSourceConfig { schema, type }

A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:

Used to define your testing criteria and
What data is required when creating a run

schema: Record<string, unknown>

The json schema for the run data source items. Learn how to build JSON schemas here.

type: "custom"

The type of data source. Always custom.

Logs { schema, type, metadata }

A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.

schema: Record<string, unknown>

The json schema for the run data source items. Learn how to build JSON schemas here.

type: "logs"

The type of data source. Always logs.

metadata?: Metadata | null

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

EvalStoredCompletionsDataSourceConfig { schema, type, metadata }

Deprecated in favor of LogsDataSourceConfig.

schema: Record<string, unknown>

The json schema for the run data source items. Learn how to build JSON schemas here.

type: "stored_completions"

The type of data source. Always stored_completions.

metadata?: Metadata | null

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

metadata: Metadata | null

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

The name of the evaluation.

object: "eval"

The object type.

testing_criteria: Array<LabelModelGrader { input, labels, model, 3 more } | StringCheckGrader { input, name, operation, 2 more } | EvalGraderTextSimilarity { pass_threshold } | 2 more>

A list of testing criteria.

One of the following:

LabelModelGrader { input, labels, model, 3 more }

A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

input: Array<Input>

content: string | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText { text, type } | 3 more

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

One of the following:

string

ResponseInputText { text, type, prompt_cache_breakpoint }

A text input to the model.

text: string

The text input to the model.

type: "input_text"

The type of the input item. Always input_text.

prompt_cache_breakpoint?: PromptCacheBreakpoint { mode }

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: "explicit"

The breakpoint mode. Always explicit.

OutputText { text, type }

A text output from the model.

text: string

The text output from the model.

type: "output_text"

The type of the output text. Always output_text.

InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: string

The URL of the image input.

formaturi

type: "input_image"

The type of the image input. Always input_image.

detail?: string

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: InputAudio { data, format }

data: string

Base64-encoded audio data.

format: "mp3" | "wav"

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:

"mp3"

"wav"

type: "input_audio"

The type of the input item. Always input_audio.

GraderInputs = Array<string | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText { text, type } | 2 more>

A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

One of the following:

string

ResponseInputText { text, type, prompt_cache_breakpoint }

A text input to the model.

text: string

The text input to the model.

type: "input_text"

The type of the input item. Always input_text.

prompt_cache_breakpoint?: PromptCacheBreakpoint { mode }

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: "explicit"

The breakpoint mode. Always explicit.

OutputText { text, type }

A text output from the model.

text: string

The text output from the model.

type: "output_text"

The type of the output text. Always output_text.

InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: string

The URL of the image input.

formaturi

type: "input_image"

The type of the image input. Always input_image.

detail?: string

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: InputAudio { data, format }

data: string

Base64-encoded audio data.

format: "mp3" | "wav"

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:

"mp3"

"wav"

type: "input_audio"

The type of the input item. Always input_audio.

role: "user" | "assistant" | "system" | "developer"

The role of the message input. One of user, assistant, system, or developer.

One of the following:

"user"

"assistant"

"system"

"developer"

type?: "message"

The type of the message input. Always message.

labels: Array<string>

The labels to assign to each item in the evaluation.

model: string

The model to use for the evaluation. Must support structured outputs.

The name of the grader.

passing_labels: Array<string>

The labels that indicate a passing result. Must be a subset of labels.

type: "label_model"

The object type, which is always label_model.

StringCheckGrader { input, name, operation, 2 more }

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

input: string

The input text. This may include template strings.

The name of the grader.

operation: "eq" | "ne" | "like" | "ilike"

The string check operation to perform. One of eq, ne, like, or ilike.

One of the following:

"eq"

"ne"

"like"

"ilike"

reference: string

The reference text. This may include template strings.

type: "string_check"

The object type, which is always string_check.

EvalGraderTextSimilarity extends TextSimilarityGrader { evaluation_metric, input, name, 2 more } { pass_threshold }

A TextSimilarityGrader object which grades text based on similarity metrics.

pass_threshold: number

The threshold for the score.

EvalGraderPython extends PythonGrader { name, source, type, image_tag } { pass_threshold }

A PythonGrader object that runs a python script on the input.

pass_threshold?: number

The threshold for the score.

EvalGraderScoreModel extends ScoreModelGrader { input, model, name, 3 more } { pass_threshold }

A ScoreModelGrader object that uses a model to assign a score to the input.

pass_threshold?: number

The threshold for the score.

Create eval

import OpenAI from "openai";

const openai = new OpenAI();

const evalObj = await openai.evals.create({
  name: "Sentiment",
  data_source_config: {
    type: "stored_completions",
    metadata: { usecase: "chatbot" }
  },
  testing_criteria: [
    {
      type: "label_model",
      model: "o3-mini",
      input: [
        { role: "developer", content: "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'" },
        { role: "user", content: "Statement: {{item.input}}" }
      ],
      passing_labels: ["positive"],
      labels: ["positive", "neutral", "negative"],
      name: "Example label grader"
    }
  ]
});
console.log(evalObj);

{
  "object": "eval",
  "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
  "data_source_config": {
    "type": "stored_completions",
    "metadata": {
      "usecase": "chatbot"
    },
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object"
        },
        "sample": {
          "type": "object"
        }
      },
      "required": [
        "item",
        "sample"
      ]
  },
  "testing_criteria": [
    {
      "name": "Example label grader",
      "type": "label_model",
      "model": "o3-mini",
      "input": [
        {
          "type": "message",
          "role": "developer",
          "content": {
            "type": "input_text",
            "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
          }
        },
        {
          "type": "message",
          "role": "user",
          "content": {
            "type": "input_text",
            "text": "Statement: {{item.input}}"
          }
        }
      ],
      "passing_labels": [
        "positive"
      ],
      "labels": [
        "positive",
        "neutral",
        "negative"
      ]
    }
  ],
  "name": "Sentiment",
  "created_at": 1740110490,
  "metadata": {
    "description": "An eval for sentiment analysis"
  }
}

Returns Examples

{
  "object": "eval",
  "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
  "data_source_config": {
    "type": "stored_completions",
    "metadata": {
      "usecase": "chatbot"
    },
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object"
        },
        "sample": {
          "type": "object"
        }
      },
      "required": [
        "item",
        "sample"
      ]
  },
  "testing_criteria": [
    {
      "name": "Example label grader",
      "type": "label_model",
      "model": "o3-mini",
      "input": [
        {
          "type": "message",
          "role": "developer",
          "content": {
            "type": "input_text",
            "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
          }
        },
        {
          "type": "message",
          "role": "user",
          "content": {
            "type": "input_text",
            "text": "Statement: {{item.input}}"
          }
        }
      ],
      "passing_labels": [
        "positive"
      ],
      "labels": [
        "positive",
        "neutral",
        "negative"
      ]
    }
  ],
  "name": "Sentiment",
  "created_at": 1740110490,
  "metadata": {
    "description": "An eval for sentiment analysis"
  }
}

Suggested

Create eval

ParametersExpand Collapse

ReturnsExpand Collapse

Create eval

Returns Examples