Iterate on difficult problems

Introduction

Some tasks are easy to verify in one shot: the build passes, the tests go green, and you are done. But there are some optimization problems that are difficult to solve, and need many iterations with a tight evaluation loop. To know which direction to go in, Codex needs to inspect the current output, score it, decide the next change, and repeat until the result is actually good.

This type of use case pairs well with a custom UI that lets you inspect progress visually, by having Codex log the outputs and generated artifacts for each iteration. You can watch Codex continue working in the app while the target artifact, model output, or generated asset keeps improving. The key is to give Codex the necessary scripts to generate the evaluation metrics and the artifacts to inspect.

Start with evals

Before the task begins, define how success will be measured. The best setup usually combines:

• Deterministic checks: things the scripts can score directly, such as constraint violations or deterministic metrics computed with code
• LLM-as-a-judge checks: rubric-based scores for qualities that are harder to encode exactly, such as resemblance, readability, usefulness, or overall quality - this can rely on text or image outputs

If the subjective part matters, give Codex a script that can call a model for example using the Responses API and return structured scores. The point is not to replace deterministic checks, it’s to supplement them with a consistent judge for the part humans would otherwise assess by eye.

The loop works best when the eval output is machine-readable, saved after every run, and easy to compare over time.

Give Codex a stopping rule

Hard tasks often drift because the prompt says “keep improving” without saying when to stop. Make the stopping rule explicit.

A practical pattern is:

1. Set a target for the overall score.
2. Set a separate target for the LLM-judge average.
3. Tell Codex to continue until both are above the threshold, not just one.

For example, if the goal is a high-quality artifact, ask Codex to keep going until both the overall score and the LLM average are above 90%. That makes the task legible: Codex can tell whether it is still below target, where the gap is, and whether the latest change helped.

Keep a running log of the loop

Long-running work is much more reliable when Codex keeps notes about the loop instead of trying to remember everything from the thread.

That running log should record:

• the current best scores
• what changed on the last iteration
• what the eval said got better or worse
• what Codex plans to try next

This is especially important when the task runs for a long time. The log becomes the handoff point for the next session and the self-evaluation record for the current one.

Inspect the artifact, not just the logs

For some difficult tasks, the code diff and metric output are not enough. Codex should look at the artifact it produced.

If the output is visual, such as a generated image, layout, or rendered state, let Codex inspect that artifact directly, for example when the output lives on disk as an image and compare the current result to the prior best result or to the intended rubric.

This makes the loop stronger:

• the eval script reports the score
• the artifact shows what the score missed
• the next change is grounded in both

That combination is much more effective than changing code blindly between runs.

Make every iteration explicit

Ask Codex to follow the same loop every time:

1. Run the evals on the current baseline.
2. Identify the biggest failure mode from the scores and artifacts.
3. Make one focused change that addresses that bottleneck.
4. Re-run the evals.
5. Log the new scores and whether the change helped.
6. Continue until the thresholds are met.

This discipline matters. If each iteration changes too many things at once, Codex cannot tell which idea improved the score. If it skips logging, the session becomes hard to trust and hard to resume.