One of the following:
class LabelModelGrader { input, labels, model, 3 more }

A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

input: Array[Input{ content, role, type}]
content: String | ResponseInputText { text, type, prompt_cache_breakpoint } | OutputText{ text, type} | 3 more

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

One of the following:
String = String

A text input to the model.

class ResponseInputText { text, type, prompt_cache_breakpoint }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

prompt_cache_breakpoint: PromptCacheBreakpoint{ mode, ttl}

Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: :explicit

The breakpoint mode. Always explicit.

ttl: :"30m" | :"12h"

The minimum lifetime of this cache breakpoint. Defaults to the request TTL.

One of the following:
:"30m"
:"12h"
class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

formaturi
type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: InputAudio{ data, format_}
data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
:mp3
:wav
type: :input_audio

The type of the input item. Always input_audio.

GraderInputs = Array[GraderInputItem]

A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

One of the following:
String = String

A text input to the model.

class ResponseInputText { text, type, prompt_cache_breakpoint }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

prompt_cache_breakpoint: PromptCacheBreakpoint{ mode, ttl}

Marks the exact end of a reusable prompt prefix. When ttl is omitted, the breakpoint inherits the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: :explicit

The breakpoint mode. Always explicit.

ttl: :"30m" | :"12h"

The minimum lifetime of this cache breakpoint. Defaults to the request TTL.

One of the following:
:"30m"
:"12h"
class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

formaturi
type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: InputAudio{ data, format_}
data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
:mp3
:wav
type: :input_audio

The type of the input item. Always input_audio.

role: :user | :assistant | :system | :developer

The role of the message input. One of user, assistant, system, or developer.

One of the following:
:user
:assistant
:system
:developer
type: :message

The type of the message input. Always message.

labels: Array[String]

The labels to assign to each item in the evaluation.

model: String

The model to use for the evaluation. Must support structured outputs.

name: String

The name of the grader.

passing_labels: Array[String]

The labels that indicate a passing result. Must be a subset of labels.

type: :label_model

The object type, which is always label_model.

class StringCheckGrader { input, name, operation, 2 more }

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

input: String

The input text. This may include template strings.

name: String

The name of the grader.

operation: :eq | :ne | :like | :ilike

The string check operation to perform. One of eq, ne, like, or ilike.

One of the following:
:eq
:ne
:like
:ilike
reference: String

The reference text. This may include template strings.

type: :string_check

The object type, which is always string_check.

class EvalGraderTextSimilarity { pass_threshold }

A TextSimilarityGrader object which grades text based on similarity metrics.

pass_threshold: Float

The threshold for the score.

class EvalGraderPython { pass_threshold }

A PythonGrader object that runs a python script on the input.

pass_threshold: Float

The threshold for the score.

class EvalGraderScoreModel { pass_threshold }

A ScoreModelGrader object that uses a model to assign a score to the input.

pass_threshold: Float

The threshold for the score.