Skip to content
Primary navigation

Suggested
API Dashboard
API Reference
Evals
Runs
Output Items

Get eval run output items

evals.runs.output_items.list(run_id, **kwargs) -> CursorPage<OutputItemListResponse { id, created_at, datasource_item, 7 more } >
GET/evals/{eval_id}/runs/{run_id}/output_items

Get a list of output items for an evaluation run.

ParametersExpand Collapse
eval_id: String
run_id: String
after: String

Identifier for the last output item from the previous pagination request.

limit: Integer

Number of output items to retrieve.

order: :asc | :desc

Sort order for output items by timestamp. Use asc for ascending order or desc for descending order. Defaults to asc.

One of the following:
:asc
:desc
status: :fail | :pass

Filter output items by status. Use failed to filter by failed output items or pass to filter by passed output items.

One of the following:
:fail
:pass
ReturnsExpand Collapse
class OutputItemListResponse { id, created_at, datasource_item, 7 more }

A schema representing an evaluation run output item.

id: String

Unique identifier for the evaluation run output item.

created_at: Integer

Unix timestamp (in seconds) when the evaluation run was created.

datasource_item: Hash[Symbol, untyped]

Details of the input data source item.

datasource_item_id: Integer

The identifier for the data source item.

eval_id: String

The identifier of the evaluation group.

object: :"eval.run.output_item"

The type of the object. Always "eval.run.output_item".

results: Array[{ name, passed, score, 2 more}]

A list of grader results for this output item.

name: String

The name of the grader.

passed: bool

Whether the grader considered the output a pass.

score: Float

The numeric score produced by the grader.

sample: Hash[Symbol, untyped]

Optional sample or intermediate data produced by the grader.

type: String

The grader type (for example, "string-check-grader").

run_id: String

The identifier of the evaluation run associated with this output item.

sample: { error, finish_reason, input, 7 more}

A sample containing the input and output of the evaluation run.

error: EvalAPIError { code, message }

An object representing an error response from the Eval API.

code: String

The error code.

message: String

The error message.

finish_reason: String

The reason why the sample generation was finished.

input: Array[{ content, role}]

An array of input messages.

content: String

The content of the message.

role: String

The role of the message sender (e.g., system, user, developer).

max_completion_tokens: Integer

The maximum number of tokens allowed for completion.

model: String

The model used for generating the sample.

output: Array[{ content, role}]

An array of output messages.

content: String

The content of the message.

role: String

The role of the message (e.g. "system", "assistant", "user").

seed: Integer

The seed used for generating the sample.

temperature: Float

The sampling temperature used.

top_p: Float

The top_p value used for sampling.

usage: { cached_tokens, completion_tokens, prompt_tokens, total_tokens}

Token usage details for the sample.

cached_tokens: Integer

The number of tokens retrieved from cache.

completion_tokens: Integer

The number of completion tokens generated.

prompt_tokens: Integer

The number of prompt tokens used.

total_tokens: Integer

The total number of tokens used.

status: String

The status of the evaluation run.

Get eval run output items

require "openai"

openai = OpenAI::Client.new(api_key: "My API Key")

page = openai.evals.runs.output_items.list("run_id", eval_id: "eval_id")

puts(page)
{
  "object": "list",
  "data": [
    {
      "object": "eval.run.output_item",
      "id": "outputitem_67e5796c28e081909917bf79f6e6214d",
      "created_at": 1743092076,
      "run_id": "evalrun_67abd54d60ec8190832b46859da808f7",
      "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
      "status": "pass",
      "datasource_item_id": 5,
      "datasource_item": {
        "input": "Stock Markets Rally After Positive Economic Data Released",
        "ground_truth": "Markets"
      },
      "results": [
        {
          "name": "String check-a2486074-d803-4445-b431-ad2262e85d47",
          "sample": null,
          "passed": true,
          "score": 1.0
        }
      ],
      "sample": {
        "input": [
          {
            "role": "developer",
            "content": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\"  \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\"  \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\"  \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\"  \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\"  \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n",
            "tool_call_id": null,
            "tool_calls": null,
            "function_call": null
          },
          {
            "role": "user",
            "content": "Stock Markets Rally After Positive Economic Data Released",
            "tool_call_id": null,
            "tool_calls": null,
            "function_call": null
          }
        ],
        "output": [
          {
            "role": "assistant",
            "content": "Markets",
            "tool_call_id": null,
            "tool_calls": null,
            "function_call": null
          }
        ],
        "finish_reason": "stop",
        "model": "gpt-4o-mini-2024-07-18",
        "usage": {
          "total_tokens": 325,
          "completion_tokens": 2,
          "prompt_tokens": 323,
          "cached_tokens": 0
        },
        "error": null,
        "temperature": 1.0,
        "max_completion_tokens": 2048,
        "top_p": 1.0,
        "seed": 42
      }
    }
  ],
  "first_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
  "last_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
  "has_more": true
}
Returns Examples
{
  "object": "list",
  "data": [
    {
      "object": "eval.run.output_item",
      "id": "outputitem_67e5796c28e081909917bf79f6e6214d",
      "created_at": 1743092076,
      "run_id": "evalrun_67abd54d60ec8190832b46859da808f7",
      "eval_id": "eval_67abd54d9b0081909a86353f6fb9317a",
      "status": "pass",
      "datasource_item_id": 5,
      "datasource_item": {
        "input": "Stock Markets Rally After Positive Economic Data Released",
        "ground_truth": "Markets"
      },
      "results": [
        {
          "name": "String check-a2486074-d803-4445-b431-ad2262e85d47",
          "sample": null,
          "passed": true,
          "score": 1.0
        }
      ],
      "sample": {
        "input": [
          {
            "role": "developer",
            "content": "Categorize a given news headline into one of the following topics: Technology, Markets, World, Business, or Sports.\n\n# Steps\n\n1. Analyze the content of the news headline to understand its primary focus.\n2. Extract the subject matter, identifying any key indicators or keywords.\n3. Use the identified indicators to determine the most suitable category out of the five options: Technology, Markets, World, Business, or Sports.\n4. Ensure only one category is selected per headline.\n\n# Output Format\n\nRespond with the chosen category as a single word. For instance: \"Technology\", \"Markets\", \"World\", \"Business\", or \"Sports\".\n\n# Examples\n\n**Input**: \"Apple Unveils New iPhone Model, Featuring Advanced AI Features\"  \n**Output**: \"Technology\"\n\n**Input**: \"Global Stocks Mixed as Investors Await Central Bank Decisions\"  \n**Output**: \"Markets\"\n\n**Input**: \"War in Ukraine: Latest Updates on Negotiation Status\"  \n**Output**: \"World\"\n\n**Input**: \"Microsoft in Talks to Acquire Gaming Company for $2 Billion\"  \n**Output**: \"Business\"\n\n**Input**: \"Manchester United Secures Win in Premier League Football Match\"  \n**Output**: \"Sports\" \n\n# Notes\n\n- If the headline appears to fit into more than one category, choose the most dominant theme.\n- Keywords or phrases such as \"stocks\", \"company acquisition\", \"match\", or technological brands can be good indicators for classification.\n",
            "tool_call_id": null,
            "tool_calls": null,
            "function_call": null
          },
          {
            "role": "user",
            "content": "Stock Markets Rally After Positive Economic Data Released",
            "tool_call_id": null,
            "tool_calls": null,
            "function_call": null
          }
        ],
        "output": [
          {
            "role": "assistant",
            "content": "Markets",
            "tool_call_id": null,
            "tool_calls": null,
            "function_call": null
          }
        ],
        "finish_reason": "stop",
        "model": "gpt-4o-mini-2024-07-18",
        "usage": {
          "total_tokens": 325,
          "completion_tokens": 2,
          "prompt_tokens": 323,
          "cached_tokens": 0
        },
        "error": null,
        "temperature": 1.0,
        "max_completion_tokens": 2048,
        "top_p": 1.0,
        "seed": 42
      }
    }
  ],
  "first_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
  "last_id": "outputitem_67e5796c28e081909917bf79f6e6214d",
  "has_more": true
}