Run grader

fine_tuning.alpha.graders.run(**kwargs) -> GraderRunResponse { metadata, model_grader_token_usage_per_model, reward, sub_rewards }

POST/fine_tuning/alpha/graders/run

Run a grader.

ParametersExpand Collapse

grader: StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } | PythonGrader { name, source, type, image_tag } | 2 more

The grader used for the fine-tuning job.

Accepts one of the following:

class StringCheckGrader { input, name, operation, 2 more }

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

input: String

The input text. This may include template strings.

The name of the grader.

operation: :eq | :ne | :like | :ilike

The string check operation to perform. One of eq, ne, like, or ilike.

Accepts one of the following:

:eq

:ne

:like

:ilike

reference: String

The reference text. This may include template strings.

type: :string_check

The object type, which is always string_check.

class TextSimilarityGrader { evaluation_metric, input, name, 2 more }

A TextSimilarityGrader object which grades text based on similarity metrics.

evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more

The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.

Accepts one of the following:

:cosine

:fuzzy_match

:bleu

:gleu

:meteor

:rouge_1

:rouge_2

:rouge_3

:rouge_4

:rouge_5

:rouge_l

input: String

The text being graded.

The name of the grader.

reference: String

The text being graded against.

type: :text_similarity

The type of grader.

class PythonGrader { name, source, type, image_tag }

A PythonGrader object that runs a python script on the input.

The name of the grader.

source: String

The source code of the python script.

type: :python

The object type, which is always python.

image_tag: String

The image tag to use for the python script.

class ScoreModelGrader { input, model, name, 3 more }

A ScoreModelGrader object that uses a model to assign a score to the input.

input: Array[{ content, role, type}]

The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

content: String | ResponseInputText { text, type } | { text, type} | 3 more

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

Accepts one of the following:

String

A text input to the model.

class ResponseInputText { text, type }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: { data, format_}

data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

:mp3

:wav

type: :input_audio

The type of the input item. Always input_audio.

Array[GraderInputItem]

A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

Accepts one of the following:

String

A text input to the model.

class ResponseInputText { text, type }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: { data, format_}

data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

:mp3

:wav

type: :input_audio

The type of the input item. Always input_audio.

role: :user | :assistant | :system | :developer

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

:user

:assistant

:system

:developer

type: :message

The type of the message input. Always message.

model: String

The model to use for the evaluation.

The name of the grader.

type: :score_model

The object type, which is always score_model.

range: Array[Float]

The range of the score. Defaults to [0, 1].

sampling_params: { max_completions_tokens, reasoning_effort, seed, 2 more}

The sampling parameters for the model.

max_completions_tokens: Integer

The maximum number of tokens the grader model may generate in its response.

minimum1

reasoning_effort: ReasoningEffort

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

:none

:minimal

:low

:medium

:high

:xhigh

seed: Integer

A seed value to initialize the randomness, during sampling.

temperature: Float

A higher temperature increases randomness in the outputs.

top_p: Float

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

class MultiGrader { calculate_output, graders, name, type }

A MultiGrader object combines the output of multiple graders to produce a single score.

calculate_output: String

A formula to calculate the output based on grader results.

graders: StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } | PythonGrader { name, source, type, image_tag } | 2 more

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

Accepts one of the following:

class StringCheckGrader { input, name, operation, 2 more }

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

input: String

The input text. This may include template strings.

The name of the grader.

operation: :eq | :ne | :like | :ilike

The string check operation to perform. One of eq, ne, like, or ilike.

Accepts one of the following:

:eq

:ne

:like

:ilike

reference: String

The reference text. This may include template strings.

type: :string_check

The object type, which is always string_check.

class TextSimilarityGrader { evaluation_metric, input, name, 2 more }

A TextSimilarityGrader object which grades text based on similarity metrics.

evaluation_metric: :cosine | :fuzzy_match | :bleu | 8 more

The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.

Accepts one of the following:

:cosine

:fuzzy_match

:bleu

:gleu

:meteor

:rouge_1

:rouge_2

:rouge_3

:rouge_4

:rouge_5

:rouge_l

input: String

The text being graded.

The name of the grader.

reference: String

The text being graded against.

type: :text_similarity

The type of grader.

class PythonGrader { name, source, type, image_tag }

A PythonGrader object that runs a python script on the input.

The name of the grader.

source: String

The source code of the python script.

type: :python

The object type, which is always python.

image_tag: String

The image tag to use for the python script.

class ScoreModelGrader { input, model, name, 3 more }

A ScoreModelGrader object that uses a model to assign a score to the input.

input: Array[{ content, role, type}]

The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

content: String | ResponseInputText { text, type } | { text, type} | 3 more

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

Accepts one of the following:

String

A text input to the model.

class ResponseInputText { text, type }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: { data, format_}

data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

:mp3

:wav

type: :input_audio

The type of the input item. Always input_audio.

Array[GraderInputItem]

A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

Accepts one of the following:

String

A text input to the model.

class ResponseInputText { text, type }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: { data, format_}

data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

:mp3

:wav

type: :input_audio

The type of the input item. Always input_audio.

role: :user | :assistant | :system | :developer

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

:user

:assistant

:system

:developer

type: :message

The type of the message input. Always message.

model: String

The model to use for the evaluation.

The name of the grader.

type: :score_model

The object type, which is always score_model.

range: Array[Float]

The range of the score. Defaults to [0, 1].

sampling_params: { max_completions_tokens, reasoning_effort, seed, 2 more}

The sampling parameters for the model.

max_completions_tokens: Integer

The maximum number of tokens the grader model may generate in its response.

minimum1

reasoning_effort: ReasoningEffort

gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.
All models before gpt-5.1 default to medium reasoning effort, and do not support none.
The gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.

Accepts one of the following:

:none

:minimal

:low

:medium

:high

:xhigh

seed: Integer

A seed value to initialize the randomness, during sampling.

temperature: Float

A higher temperature increases randomness in the outputs.

top_p: Float

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

class LabelModelGrader { input, labels, model, 3 more }

A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

input: Array[{ content, role, type}]

content: String | ResponseInputText { text, type } | { text, type} | 3 more

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

Accepts one of the following:

String

A text input to the model.

class ResponseInputText { text, type }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: { data, format_}

data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

:mp3

:wav

type: :input_audio

The type of the input item. Always input_audio.

Array[GraderInputItem]

A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

Accepts one of the following:

String

A text input to the model.

class ResponseInputText { text, type }

A text input to the model.

text: String

The text input to the model.

type: :input_text

The type of the input item. Always input_text.

class OutputText { text, type }

A text output from the model.

text: String

The text output from the model.

type: :output_text

The type of the output text. Always output_text.

class InputImage { image_url, type, detail }

An image input block used within EvalItem content arrays.

image_url: String

The URL of the image input.

type: :input_image

The type of the image input. Always input_image.

detail: String

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio { input_audio, type }

An audio input to the model.

input_audio: { data, format_}

data: String

Base64-encoded audio data.

format_: :mp3 | :wav

The format of the audio data. Currently supported formats are mp3 and wav.

Accepts one of the following:

:mp3

:wav

type: :input_audio

The type of the input item. Always input_audio.

role: :user | :assistant | :system | :developer

The role of the message input. One of user, assistant, system, or developer.

Accepts one of the following:

:user

:assistant

:system

:developer

type: :message

The type of the message input. Always message.

labels: Array[String]

The labels to assign to each item in the evaluation.

model: String

The model to use for the evaluation. Must support structured outputs.

The name of the grader.

passing_labels: Array[String]

The labels that indicate a passing result. Must be a subset of labels.

type: :label_model

The object type, which is always label_model.

The name of the grader.

type: :multi

The object type, which is always multi.

model_sample: String

The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.

item: untyped

The dataset item provided to the grader. This will be used to populate the item namespace. See the guide for more details.

ReturnsExpand Collapse

class GraderRunResponse { metadata, model_grader_token_usage_per_model, reward, sub_rewards }

metadata: { errors, execution_time, name, 4 more}

errors: { formula_parse_error, invalid_variable_error, model_grader_parse_error, 11 more}

formula_parse_error: bool

invalid_variable_error: bool

model_grader_parse_error: bool

model_grader_refusal_error: bool

model_grader_server_error: bool

model_grader_server_error_details: String

other_error: bool

python_grader_runtime_error: bool

python_grader_runtime_error_details: String

python_grader_server_error: bool

python_grader_server_error_type: String

sample_parse_error: bool

truncated_observation_error: bool

unresponsive_reward_error: bool

execution_time: Float

sampled_model_name: String

scores: Hash[Symbol, untyped]

token_usage: Integer

type: String

model_grader_token_usage_per_model: Hash[Symbol, untyped]

reward: Float

sub_rewards: Hash[Symbol, untyped]

Run grader

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

response = openai.fine_tuning.alpha.graders.run(
  grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check},
  model_sample: "model_sample"
)

puts(response)

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}

Returns Examples

{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}