Anatomy of an AI agent: model, harness, context, memory, tools and hooks
In short: an AI agent is a model running inside a loop. Around the model sits the harness: the context it reads, the memory it keeps, the tools it uses and the hooks that check each step.
Many failures come from the system around the model, not the model itself: a rule that never reached the context, a step no check stopped, a test that read the reply instead of what actually changed.
Your business rules have to enter the loop twice, once as guidance in the context and once as a check in the hooks, and then be tested from outside.
Why this matters now
Agents are moving out of demos and into real operations. Support desks, claims, onboarding, access requests and field teams are all trying them. The people who run those operations want to know what they would be buying and what could go wrong.
The people building agents have a short way of putting it: an agent is a model plus a harness. The model is the part everyone talks about. The harness is the software around it, and it is where much of the work, and much of the risk, now sits.
The model
The model is the reasoning engine: GPT, Claude, Gemini, Llama and the rest. On each turn of the loop it reads what is in front of it and decides the next step: answer, ask a question, or use a tool.
Models are good at language and at choosing a sensible next step. What they do not know is your business as it runs today. A model has never seen your current refund policy, your approval limits or the exception your team made for a large customer last spring. The harness has to supply that private, current knowledge.
Models also change underneath you. New versions and cheaper options arrive every few months, and a cheaper model often looks like an easy win. Switching models without re-testing can quietly introduce new rule failures in an agent that worked last month.
What is an agent harness?
The harness is the software around the model. It controls what the model sees, what happens after each decision and what is remembered. Two agents built on the same model can behave very differently because their harnesses differ.
The harness has four working parts: context, memory, tools and hooks. A framework usually helps developers build the loop that connects them.
Context: what the agent sees
Context is the information sent to the model for one turn: its instructions, your rules, the conversation so far, and anything fetched from your systems or from memory. Context engineering means choosing which of these the model receives, in what order, and what is left out.
It matters because a model's attention is limited. Put too little in and the agent guesses. Put too much in and the model may give less weight to the one rule that matters.
This is where your rules first enter the loop, and it can make a big difference. In one re-run of Sierra's public telecom benchmark, adding the written telecom policy to the GPT-4.1 agent's context raised its "actions completed" score from 43.8% to 73.8%. The run covered 24 cases, each repeated five times, with the same simulated customer setup in both versions. Most of the score checks whether the agent called the right tool, not every value it passed. Same model, same tools; the only change was the rules in its context.
Memory: what the agent keeps
Memory is what the harness keeps, as distinct from what the model sees on a given turn. Within one conversation, the harness keeps the history and other working state, and chooses which parts to place in the context each turn. Across conversations, it can store facts about a customer, preferences, past cases or lessons from earlier mistakes, and bring them back later.
Memory makes an agent feel continuous, and it creates its own risks. A detail remembered from one case can leak into the next. A note that was true last month can be wrong today. A good harness decides deliberately what to keep, for how long, and when to forget.
Tools: what the agent can do
An agent becomes useful when it can act: look up an account, change a plan, issue a credit, grant access, open a ticket. Each of those capabilities is a tool. Some tools only read or calculate; others change a real system.
Tools are where mistakes become real. A wrong sentence can be corrected. A credit issued to the wrong account, or access granted to the wrong person, has to be found and reversed. And unless their inputs are checked, some tools will accept values your policy forbids.
Agent hooks: where rules are checked and enforced
Hooks, sometimes called callbacks or guardrails depending on the framework, are places where the harness runs your own code at fixed points: before or after the model is called, or before or after a tool runs. They run every time, whatever the model decided. A hook that applies a plain rule gives the same answer for the same inputs; some hooks also call a model, and those need testing like any other model.
A hook before a tool call can block it, change it, or stop and ask a person. A hook after a tool call can check the result, remove personal data before the result goes back to the model, or write an audit record. For example:
- Block any refund above the limit, however the customer asked.
- Require a manager's approval before access to a production system is granted.
- Remove card numbers and personal details from tool results before they reach the model's context.
- Log every change to an account, with the rule that allowed it.
This is where your rules enter the loop the second time. Instructions in the context ask the agent to behave. A check before the action can stop it when it does not, as long as the rule is written correctly and the check covers every path to that action. An agent has no conscience and no fear of a bad review, so the rules that matter most should not depend on it remembering them.
The framework
Frameworks such as Google's Agent Development Kit, OpenAI's Agents SDK and LangGraph help developers build and run the loop. They pass context to the model, run the tools it asks for, keep state and offer places to attach checks.
A framework is plumbing, and good plumbing matters. But it does not know your business rules, and it only checks what your team writes and attaches. "We built it on a well-known framework" tells you how the agent is assembled. It tells you nothing about whether it does the right thing.
Evals: checking from outside the loop
Evals are the tests that tell you whether the whole loop does what your business needs, including in the awkward cases. They come in several flavours: code checks that look for exact facts, a second model acting as a judge, and human review. An eval can check one step, the sequence of actions or the final outcome.
The most important question is what the eval actually looks at. Many check the reply: did it sound polite, did it mention the right policy. For an agent that takes actions, the test has to include what changed in your systems. Was the right account touched? Was the credit within the limit? Was the case handed to a person when the rule said it must be? We will look at the flavours of evals, and when each one can be trusted, in a later article.
Where agents actually fail
When an agent goes wrong, the model is rarely the whole story. Look for the failure in the harness too:
- A rule exists in someone's head but never made it into the context.
- A rule that must never be broken was left to the instructions, with no check before the action.
- A tool accepts a value the rules forbid.
- A memory from an earlier case quietly shapes the current one.
- An eval checks the wording of the reply, so a wrong action passes.
- A model is swapped for a cheaper one and nobody re-runs the evals.
What you own, even if you never write code
If you run the operation the agent will work in, two things stay yours even when a technical team builds the agent: the business rules and the definition of a correct result.
The rules are yours, because you understand how the work should be done. They decide what goes into the context and what the hooks check. The evals are yours too, because you are best placed to say what "right" looks like, with input from risk, compliance and engineering. The model, the framework and the tools can be bought or built. The rules and the evals decide whether the agent can be trusted.
That is the work we do at Constolve. We take the rules from the people who know them, turn them into checks that look at what the agent actually did, and run them again whenever the model, the rules or the tools change.
If you have an agent in pilot or production, we can test it against your own rules and show you where it breaks them. Write to us at hello@constolve.com.
Questions people ask
What is an AI agent, in one sentence?
A language model running in a loop, with a harness that gives it context, memory and tools, and that checks its steps with hooks.
What is an agent harness?
Everything around the model: the context it reads, the memory it keeps, the tools it can use and the hooks that check each step.
What is context engineering?
Choosing what the model sees on each turn: which instructions, rules, history and data, in what order, and what to leave out.
What are hooks in an AI agent?
Places where the harness runs your own code before or after a step, for example to block a tool call, ask a person for approval or remove personal data. Hooks that apply a plain rule give the same answer every time.
Does a framework like Google ADK or LangGraph make an agent reliable?
Not on its own. Frameworks help build the loop and offer places to attach checks. They do not know your business rules; your team has to write and attach those checks.
How should an AI agent be tested?
Against your own rules, by checking what changed in your systems after each conversation, not only what the agent said. Test again whenever the model, the rules or the tools change.