Skip to content
For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL.
Primary navigation

Suggested
Overview Models Agents Tools Audio & voice Production API reference

Run grader

client.fineTuning.alpha.graders.run(body, options?): GraderRunResponse { metadata, model_grader_token_usage_per_model, reward, sub_rewards }
POST
/fine_tuning/alpha/graders/run

Run a grader.

Parameters
body: GraderRunParams { grader, model_sample, item }
grader: StringCheckGrader { input, name, operation, 2 more } | TextSimilarityGrader { evaluation_metric, input, name, 2 more } | PythonGrader { name, source, type, image_tag } | 2 more

The grader used for the fine-tuning job.

model_sample: string

The model sample to be evaluated. This value will be used to populate the sample namespace. See the guide for more details. The output_json variable will be populated if the model sample is a valid JSON string.

item?: unknown

The dataset item provided to the grader. This will be used to populate the item namespace. See the guide for more details.

Returns
GraderRunResponse { metadata, model_grader_token_usage_per_model, reward, sub_rewards }
metadata: Metadata { errors, execution_time, name, 4 more }
model_grader_token_usage_per_model: Record<string, unknown>
reward: number
sub_rewards: Record<string, unknown>

Run grader

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted
});

const response = await client.fineTuning.alpha.graders.run({
  grader: {
    input: 'input',
    name: 'name',
    operation: 'eq',
    reference: 'reference',
    type: 'string_check',
  },
  model_sample: 'model_sample',
});

console.log(response.metadata);
200 example
{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}
Returns Examples
200 example
{
  "metadata": {
    "errors": {
      "formula_parse_error": true,
      "invalid_variable_error": true,
      "model_grader_parse_error": true,
      "model_grader_refusal_error": true,
      "model_grader_server_error": true,
      "model_grader_server_error_details": "model_grader_server_error_details",
      "other_error": true,
      "python_grader_runtime_error": true,
      "python_grader_runtime_error_details": "python_grader_runtime_error_details",
      "python_grader_server_error": true,
      "python_grader_server_error_type": "python_grader_server_error_type",
      "sample_parse_error": true,
      "truncated_observation_error": true,
      "unresponsive_reward_error": true
    },
    "execution_time": 0,
    "name": "name",
    "sampled_model_name": "sampled_model_name",
    "scores": {
      "foo": "bar"
    },
    "token_usage": 0,
    "type": "type"
  },
  "model_grader_token_usage_per_model": {
    "foo": "bar"
  },
  "reward": 0,
  "sub_rewards": {
    "foo": "bar"
  }
}