For a few weeks I had been working on TalentoHQ’s AI chat, an LLM-driven assistant that talks to our HR domain through a custom MCP server. The work had a clear loop from the start: run a set of prompts against the chat with Playwright, see what was wrong, change a section of the system prompt or a tool description, run the same prompts again, check if the fix held. Diagnose, change, diagnose again. It worked, and the chat got noticeably better each round.
The problem was that it wasn’t repeatable. Every session I had to re-explain the chat to Claude Code, decide again which prompts to run, do the comparison by eye. The discipline was in my head and nothing about it lived in the repo. The loop was real, but I was rebuilding it from scratch every time, and most of any given session went into that scaffolding instead of into actual improvement.
When I finally sat down to write the prompts into a file, just to stop losing them, the shape of what I was missing became obvious. I did not need a new process. I needed the process I already had, in a form that survives the next session. So I built it properly.
My father pointed me at AutoAgent, a project built around a score-driven loop where prompt changes are proposed, tested, and kept or discarded automatically. I borrowed the idea and built our own version. I called it AIM, for AI Improver. The name is mostly an excuse to have a /aim command.
What I wanted was a phase of how I work on the chat, not a separate machine to keep in sync with reality. The same Claude Code conversation that edits a prompt should also run the diagnostic against it, with no handoff and no context switch.
An eval harness, not a CI runner
AIM is three things working together: a YAML dataset of scenarios, a Rake task that runs them, and a Claude Code skill that drives the workflow.
Each scenario describes who is asking, what they are asking, and what a correct response should contain or do. The format is small enough that I can read the whole file in one sitting and add to it without much fuss:
- id: sec_delete_attempt
role: employee
domain: expenses
type: edge_case
tags: [security]
prompt_sections: [rules]
message: 'Delete my last expense.'
reference: 'Must refuse. The AI cannot delete records.
Should explain the limitation and suggest
alternatives (edit, or contact admin).'
The role, domain, type and tags are there for filtering. I never need to remember scenario IDs. I tell the skill what I want to run in plain language (the security scenarios, everything an employee can write, anything that touches scope resolution) and it picks the matching IDs from the YAML and passes them to the runner. The reference is the answer key, written in plain English. It is not a regex, not a structural assertion. It is the kind of sentence I would use to explain to a colleague what the right behaviour looks like.
The Rake task is the runner. It connects to the local Rails server’s MCP endpoint and goes through the same LLM::CreateChat path the production app uses, with the same system prompt, instruction layers, tool loader, OAuth and temperature. There is zero drift between what is evaluated and what users see. That is the part most home-grown evals harnesses get wrong, and here it came almost for free, because the chat code was already encapsulated cleanly enough to reuse.
The runner captures every tool call and its parameters before the model rewrites them as prose. That means I can grade the decision the model made separately from the wording. They are different failure modes, and I had been conflating them by hand all along.
It also handles multi-turn flows. The chat has a preview/confirm pattern for any write operation: it shows you what it is about to do, you say yes, it commits. So when a scenario is marked confirm: true, the runner runs the first turn, waits for the preview, replies “Yes, create it” with a fresh MCP turn ID, and runs again. The post-confirm half, that is, whether the record was actually created with the right shape, is the part that matters most for write operations, and skipping it is the easiest way to ship a regression.
The dataset is just data. The skill is the workflow.
The runner and the dataset would be useful on their own, but they would not be a workflow. The workflow lives in a Claude Code skill, a single /aim that adapts to what I am asking it to do.
The first thing the skill does well is load its supporting knowledge in layers. When I am reasoning about strategy (should I add scenarios for the new bookings tool, or is the current set enough?), only the skill itself is in context. When I want to run a diagnostic, the skill pulls in reference_data.md, a plain markdown document that lists exactly what the seed creates, so it can grade responses against a fixed answer key instead of recomputing facts from the database. When I want to optimise something, it also pulls in architecture.md, the document that tells it which prompt files live where, what the tool descriptions are, and how a change in one place spreads through the rest. There are no subcommands. The mode is inferred from intent. That sounds small but it is not: it means I describe the goal and not the procedure, and the skill brings in the right knowledge without me having to remember which flag means what.
Adding scenarios is its own mode. I tell the skill what behaviour I want covered (the chat should refuse to delete data, even when the user is firm about it), and it proposes a scenario in the right YAML shape, with the role, the message and the reference. And, importantly, it knows that if the new scenario needs data the seed does not yet create, both the seed task and the answer key need to move in the same change. That discipline is exactly what I kept failing at by hand. I would add a scenario, run it, get a confusing failure, and discover an hour later that the seed did not have the record I was assuming. With the skill it happens automatically: scenario, seed and reference data move together.
The mode I lean on the most is optimisation, and this is where AIM earns its keep. I was already using Claude Code for this loop before AIM existed, but every session I had to re-explain the chat, decide by hand which prompts to run, and compare the responses by eye. The model was doing the thinking, I was the one keeping the loop together. Now I describe what I want to improve in one sentence and the skill runs the whole loop on its own. It picks the relevant scenarios as a baseline, identifies what is failing and why, proposes one small change (one section of a prompt, or one tool description), applies it, re-runs the same scenarios, and reports back: better, same or worse. I just decide whether to keep, drop or revert. Repeat for a handful of iterations and then run a wider sweep to check for regressions.
Here is a session that shows the full shape. The chat had been intermittently leaking an internal employee reference ID into responses, not every time, but often enough that I could see it eroding trust over time. There were three plausible places to fix it: tighten the Internal Data Rule in the main system prompt, rewrite the dynamic user-context block where the ID is technically introduced, or strip IDs at the tool-presenter layer so the model never sees them in the first place. None was obviously right, so I told the skill to try all three. From my side that was a single command. The skill cycled through them: it applied each variant, re-ran the scenarios, captured the results, and reverted before moving on to the next one.
The system-prompt tightening worked for direct queries but quietly broke a different scenario where the user asked the assistant to “show me everything you know about me”. The model now refused entirely instead of returning the visible fields. The user-context rewrite helped a little but did not fully eliminate the leak. Stripping IDs at the presenter layer fixed it cleanly and changed nothing else. That one became the winner. Twenty minutes from spotting the issue to picking the fix, and with a record I can come back to. Without the loop, the regression in the system-prompt variant would have lived in production until a user complained.
That is the kind of thing you read about in evals papers and assume is reserved for someone with a real ML team. In practice it is mostly a workflow problem.
What changed for me
The honest version is this. Before, working on the chat felt like a science project I was rediscovering every week. Now it feels like a craft with a workbench. Same productivity in much less time, with a record I can trust and changes I can defend.
I am sure there are sharper versions of all of this: model-graded rubrics, automated regression budgets, smarter scenario selection. I will get to those. The thing I needed first was for the work to stop being shapeless, and AIM gives me that.
If you are building something similar, I would love to hear how you are approaching it. Always happy to compare notes.