Validate grader
Validate a grader.
Body ParametersJSONExpand Collapse
grader: StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or PythonGrader { name, source, type, image_tag } or 2 moreThe grader used for the fine-tuning job.
The grader used for the fine-tuning job.
StringCheckGrader = object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
TextSimilarityGrader = object { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics.
A TextSimilarityGrader object which grades text based on similarity metrics.
PythonGrader = object { name, source, type, image_tag } A PythonGrader object that runs a python script on the input.
A PythonGrader object that runs a python script on the input.
ScoreModelGrader = object { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input.
A ScoreModelGrader object that uses a model to assign a score to the input.
input: array of object { content, role, type } The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more } The sampling parameters for the model.
The sampling parameters for the model.
The maximum number of tokens the grader model may generate in its response.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1 default to medium reasoning effort, and do not support none.
- The
gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
MultiGrader = object { calculate_output, graders, name, type } A MultiGrader object combines the output of multiple graders to produce a single score.
A MultiGrader object combines the output of multiple graders to produce a single score.
graders: StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or PythonGrader { name, source, type, image_tag } or 2 moreA StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
StringCheckGrader = object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
TextSimilarityGrader = object { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics.
A TextSimilarityGrader object which grades text based on similarity metrics.
PythonGrader = object { name, source, type, image_tag } A PythonGrader object that runs a python script on the input.
A PythonGrader object that runs a python script on the input.
ScoreModelGrader = object { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input.
A ScoreModelGrader object that uses a model to assign a score to the input.
input: array of object { content, role, type } The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more } The sampling parameters for the model.
The sampling parameters for the model.
The maximum number of tokens the grader model may generate in its response.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1 default to medium reasoning effort, and do not support none.
- The
gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
LabelModelGrader = object { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: array of object { content, role, type }
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
ReturnsExpand Collapse
grader: optional StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or PythonGrader { name, source, type, image_tag } or 2 moreThe grader used for the fine-tuning job.
The grader used for the fine-tuning job.
StringCheckGrader = object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
TextSimilarityGrader = object { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics.
A TextSimilarityGrader object which grades text based on similarity metrics.
PythonGrader = object { name, source, type, image_tag } A PythonGrader object that runs a python script on the input.
A PythonGrader object that runs a python script on the input.
ScoreModelGrader = object { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input.
A ScoreModelGrader object that uses a model to assign a score to the input.
input: array of object { content, role, type } The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more } The sampling parameters for the model.
The sampling parameters for the model.
The maximum number of tokens the grader model may generate in its response.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1 default to medium reasoning effort, and do not support none.
- The
gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
MultiGrader = object { calculate_output, graders, name, type } A MultiGrader object combines the output of multiple graders to produce a single score.
A MultiGrader object combines the output of multiple graders to produce a single score.
graders: StringCheckGrader { input, name, operation, 2 more } or TextSimilarityGrader { evaluation_metric, input, name, 2 more } or PythonGrader { name, source, type, image_tag } or 2 moreA StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
StringCheckGrader = object { input, name, operation, 2 more } A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
TextSimilarityGrader = object { evaluation_metric, input, name, 2 more } A TextSimilarityGrader object which grades text based on similarity metrics.
A TextSimilarityGrader object which grades text based on similarity metrics.
PythonGrader = object { name, source, type, image_tag } A PythonGrader object that runs a python script on the input.
A PythonGrader object that runs a python script on the input.
ScoreModelGrader = object { input, model, name, 3 more } A ScoreModelGrader object that uses a model to assign a score to the input.
A ScoreModelGrader object that uses a model to assign a score to the input.
input: array of object { content, role, type } The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
sampling_params: optional object { max_completions_tokens, reasoning_effort, seed, 2 more } The sampling parameters for the model.
The sampling parameters for the model.
The maximum number of tokens the grader model may generate in its response.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1 defaults to none, which does not perform reasoning. The supported reasoning values for gpt-5.1 are none, low, medium, and high. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1 default to medium reasoning effort, and do not support none.
- The
gpt-5-pro model defaults to (and only supports) high reasoning effort.
xhigh is supported for all models after gpt-5.1-codex-max.
Constrains effort on reasoning for
reasoning models.
Currently supported values are none, minimal, low, medium, high, and xhigh. Reducing
reasoning effort can result in faster responses and fewer tokens used
on reasoning in a response.
gpt-5.1defaults tonone, which does not perform reasoning. The supported reasoning values forgpt-5.1arenone,low,medium, andhigh. Tool calls are supported for all reasoning values in gpt-5.1.- All models before
gpt-5.1default tomediumreasoning effort, and do not supportnone. - The
gpt-5-promodel defaults to (and only supports)highreasoning effort. xhighis supported for all models aftergpt-5.1-codex-max.
LabelModelGrader = object { input, labels, model, 3 more } A LabelModelGrader object which uses a model to assign labels to each item
in the evaluation.
A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
input: array of object { content, role, type }
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
GraderInputs = array of string or ResponseInputText { text, type } or object { text, type } or 2 moreA list of inputs, each of which may be either an input text, output text, input
image, or input audio object.
A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
InputImage = object { image_url, type, detail } An image input block used within EvalItem content arrays.
An image input block used within EvalItem content arrays.
Validate grader
curl https://api.openai.com/v1/fine_tuning/alpha/graders/validate \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"grader": {
"type": "string_check",
"name": "Example string check grader",
"input": "{{sample.output_text}}",
"reference": "{{item.label}}",
"operation": "eq"
}
}'
{
"grader": {
"type": "string_check",
"name": "Example string check grader",
"input": "{{sample.output_text}}",
"reference": "{{item.label}}",
"operation": "eq"
}
}
Returns Examples
{
"grader": {
"type": "string_check",
"name": "Example string check grader",
"input": "{{sample.output_text}}",
"reference": "{{item.label}}",
"operation": "eq"
}
}