grader: Grader

The grader used for the fine-tuning job.

One of the following:
class StringCheckGrader: …

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

input: str

The input text. This may include template strings.

name: str

The name of the grader.

operation: Literal["eq", "ne", "like", "ilike"]

The string check operation to perform. One of eq, ne, like, or ilike.

One of the following:
"eq"
"ne"
"like"
"ilike"
reference: str

The reference text. This may include template strings.

type: Literal["string_check"]

The object type, which is always string_check.

class TextSimilarityGrader: …

A TextSimilarityGrader object which grades text based on similarity metrics.

evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]

The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.

One of the following:
"cosine"
"fuzzy_match"
"bleu"
"gleu"
"meteor"
"rouge_1"
"rouge_2"
"rouge_3"
"rouge_4"
"rouge_5"
"rouge_l"
input: str

The text being graded.

name: str

The name of the grader.

reference: str

The text being graded against.

type: Literal["text_similarity"]

The type of grader.

class PythonGrader: …

A PythonGrader object that runs a python script on the input.

name: str

The name of the grader.

source: str

The source code of the python script.

type: Literal["python"]

The object type, which is always python.

image_tag: Optional[str]

The image tag to use for the python script.

class ScoreModelGrader: …

A ScoreModelGrader object that uses a model to assign a score to the input.

input: List[Input]

The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

content: InputContent

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

One of the following:
str

A text input to the model.

class ResponseInputText: …

A text input to the model.

text: str

The text input to the model.

type: Literal["input_text"]

The type of the input item. Always input_text.

prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: Literal["explicit"]

The breakpoint mode. Always explicit.

class InputContentOutputText: …

A text output from the model.

text: str

The text output from the model.

type: Literal["output_text"]

The type of the output text. Always output_text.

class InputContentInputImage: …

An image input block used within EvalItem content arrays.

image_url: str

The URL of the image input.

formaturi
type: Literal["input_image"]

The type of the image input. Always input_image.

detail: Optional[str]

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio: …

An audio input to the model.

input_audio: InputAudio
data: str

Base64-encoded audio data.

format: Literal["mp3", "wav"]

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
"mp3"
"wav"
type: Literal["input_audio"]

The type of the input item. Always input_audio.

List[GraderInputItem]
One of the following:
str

A text input to the model.

class ResponseInputText: …

A text input to the model.

text: str

The text input to the model.

type: Literal["input_text"]

The type of the input item. Always input_text.

prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: Literal["explicit"]

The breakpoint mode. Always explicit.

class GraderInputItemOutputText: …

A text output from the model.

text: str

The text output from the model.

type: Literal["output_text"]

The type of the output text. Always output_text.

class GraderInputItemInputImage: …

An image input block used within EvalItem content arrays.

image_url: str

The URL of the image input.

formaturi
type: Literal["input_image"]

The type of the image input. Always input_image.

detail: Optional[str]

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio: …

An audio input to the model.

input_audio: InputAudio
data: str

Base64-encoded audio data.

format: Literal["mp3", "wav"]

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
"mp3"
"wav"
type: Literal["input_audio"]

The type of the input item. Always input_audio.

role: Literal["user", "assistant", "system", "developer"]

The role of the message input. One of user, assistant, system, or developer.

One of the following:
"user"
"assistant"
"system"
"developer"
type: Optional[Literal["message"]]

The type of the message input. Always message.

model: str

The model to use for the evaluation.

name: str

The name of the grader.

type: Literal["score_model"]

The object type, which is always score_model.

range: Optional[List[float]]

The range of the score. Defaults to [0, 1].

sampling_params: Optional[SamplingParams]

The sampling parameters for the model.

max_completions_tokens: Optional[int]

The maximum number of tokens the grader model may generate in its response.

minimum1
reasoning_effort: Optional[ReasoningEffort]

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

seed: Optional[int]

A seed value to initialize the randomness, during sampling.

temperature: Optional[float]

A higher temperature increases randomness in the outputs.

top_p: Optional[float]

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

class MultiGrader: …

A MultiGrader object combines the output of multiple graders to produce a single score.

calculate_output: str

A formula to calculate the output based on grader results.

graders: Graders

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

One of the following:
class StringCheckGrader: …

A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.

input: str

The input text. This may include template strings.

name: str

The name of the grader.

operation: Literal["eq", "ne", "like", "ilike"]

The string check operation to perform. One of eq, ne, like, or ilike.

One of the following:
"eq"
"ne"
"like"
"ilike"
reference: str

The reference text. This may include template strings.

type: Literal["string_check"]

The object type, which is always string_check.

class TextSimilarityGrader: …

A TextSimilarityGrader object which grades text based on similarity metrics.

evaluation_metric: Literal["cosine", "fuzzy_match", "bleu", 8 more]

The evaluation metric to use. One of cosine, fuzzy_match, bleu, gleu, meteor, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, or rouge_l.

One of the following:
"cosine"
"fuzzy_match"
"bleu"
"gleu"
"meteor"
"rouge_1"
"rouge_2"
"rouge_3"
"rouge_4"
"rouge_5"
"rouge_l"
input: str

The text being graded.

name: str

The name of the grader.

reference: str

The text being graded against.

type: Literal["text_similarity"]

The type of grader.

class PythonGrader: …

A PythonGrader object that runs a python script on the input.

name: str

The name of the grader.

source: str

The source code of the python script.

type: Literal["python"]

The object type, which is always python.

image_tag: Optional[str]

The image tag to use for the python script.

class ScoreModelGrader: …

A ScoreModelGrader object that uses a model to assign a score to the input.

input: List[Input]

The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings.

content: InputContent

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

One of the following:
str

A text input to the model.

class ResponseInputText: …

A text input to the model.

text: str

The text input to the model.

type: Literal["input_text"]

The type of the input item. Always input_text.

prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: Literal["explicit"]

The breakpoint mode. Always explicit.

class InputContentOutputText: …

A text output from the model.

text: str

The text output from the model.

type: Literal["output_text"]

The type of the output text. Always output_text.

class InputContentInputImage: …

An image input block used within EvalItem content arrays.

image_url: str

The URL of the image input.

formaturi
type: Literal["input_image"]

The type of the image input. Always input_image.

detail: Optional[str]

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio: …

An audio input to the model.

input_audio: InputAudio
data: str

Base64-encoded audio data.

format: Literal["mp3", "wav"]

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
"mp3"
"wav"
type: Literal["input_audio"]

The type of the input item. Always input_audio.

List[GraderInputItem]
One of the following:
str

A text input to the model.

class ResponseInputText: …

A text input to the model.

text: str

The text input to the model.

type: Literal["input_text"]

The type of the input item. Always input_text.

prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: Literal["explicit"]

The breakpoint mode. Always explicit.

class GraderInputItemOutputText: …

A text output from the model.

text: str

The text output from the model.

type: Literal["output_text"]

The type of the output text. Always output_text.

class GraderInputItemInputImage: …

An image input block used within EvalItem content arrays.

image_url: str

The URL of the image input.

formaturi
type: Literal["input_image"]

The type of the image input. Always input_image.

detail: Optional[str]

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio: …

An audio input to the model.

input_audio: InputAudio
data: str

Base64-encoded audio data.

format: Literal["mp3", "wav"]

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
"mp3"
"wav"
type: Literal["input_audio"]

The type of the input item. Always input_audio.

role: Literal["user", "assistant", "system", "developer"]

The role of the message input. One of user, assistant, system, or developer.

One of the following:
"user"
"assistant"
"system"
"developer"
type: Optional[Literal["message"]]

The type of the message input. Always message.

model: str

The model to use for the evaluation.

name: str

The name of the grader.

type: Literal["score_model"]

The object type, which is always score_model.

range: Optional[List[float]]

The range of the score. Defaults to [0, 1].

sampling_params: Optional[SamplingParams]

The sampling parameters for the model.

max_completions_tokens: Optional[int]

The maximum number of tokens the grader model may generate in its response.

minimum1
reasoning_effort: Optional[ReasoningEffort]

Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

seed: Optional[int]

A seed value to initialize the randomness, during sampling.

temperature: Optional[float]

A higher temperature increases randomness in the outputs.

top_p: Optional[float]

An alternative to temperature for nucleus sampling; 1.0 includes all tokens.

class LabelModelGrader: …

A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.

input: List[Input]
content: InputContent

Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

One of the following:
str

A text input to the model.

class ResponseInputText: …

A text input to the model.

text: str

The text input to the model.

type: Literal["input_text"]

The type of the input item. Always input_text.

prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: Literal["explicit"]

The breakpoint mode. Always explicit.

class InputContentOutputText: …

A text output from the model.

text: str

The text output from the model.

type: Literal["output_text"]

The type of the output text. Always output_text.

class InputContentInputImage: …

An image input block used within EvalItem content arrays.

image_url: str

The URL of the image input.

formaturi
type: Literal["input_image"]

The type of the image input. Always input_image.

detail: Optional[str]

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio: …

An audio input to the model.

input_audio: InputAudio
data: str

Base64-encoded audio data.

format: Literal["mp3", "wav"]

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
"mp3"
"wav"
type: Literal["input_audio"]

The type of the input item. Always input_audio.

List[GraderInputItem]
One of the following:
str

A text input to the model.

class ResponseInputText: …

A text input to the model.

text: str

The text input to the model.

type: Literal["input_text"]

The type of the input item. Always input_text.

prompt_cache_breakpoint: Optional[PromptCacheBreakpoint]

Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request’s prompt_cache_options.ttl; the boundary is not rounded to a token block.

mode: Literal["explicit"]

The breakpoint mode. Always explicit.

class GraderInputItemOutputText: …

A text output from the model.

text: str

The text output from the model.

type: Literal["output_text"]

The type of the output text. Always output_text.

class GraderInputItemInputImage: …

An image input block used within EvalItem content arrays.

image_url: str

The URL of the image input.

formaturi
type: Literal["input_image"]

The type of the image input. Always input_image.

detail: Optional[str]

The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

class ResponseInputAudio: …

An audio input to the model.

input_audio: InputAudio
data: str

Base64-encoded audio data.

format: Literal["mp3", "wav"]

The format of the audio data. Currently supported formats are mp3 and wav.

One of the following:
"mp3"
"wav"
type: Literal["input_audio"]

The type of the input item. Always input_audio.

role: Literal["user", "assistant", "system", "developer"]

The role of the message input. One of user, assistant, system, or developer.

One of the following:
"user"
"assistant"
"system"
"developer"
type: Optional[Literal["message"]]

The type of the message input. Always message.

labels: List[str]

The labels to assign to each item in the evaluation.

model: str

The model to use for the evaluation. Must support structured outputs.

name: str

The name of the grader.

passing_labels: List[str]

The labels that indicate a passing result. Must be a subset of labels.

type: Literal["label_model"]

The object type, which is always label_model.

name: str

The name of the grader.

type: Literal["multi"]

The object type, which is always multi.

hyperparameters: Optional[ReinforcementHyperparameters]

The hyperparameters used for the reinforcement fine-tuning job.