01 · Foundation
Agent Engineering
An agent is a model in a loop with tools, pursuing a goal it was given but choosing its own path there. Agent engineering is the discipline of building that loop well: what the model sees, what it can touch, when it stops, and how you know it worked.
In 2023 the scarce skill was prompting. By 2026 the model does the reasoning — the leverage moved to the harness around it. The same model scores wildly differently depending on the loop it runs in, which is why agent engineering displaced prompt engineering as the core AI job skill.
Working definition
Workflow: you write the control flow, the model fills in steps. Agent: the model owns the control flow — it decides which tool to call next based on what it just observed. Most production systems are a spectrum between the two, and picking the point on that spectrum is design decision #1.
The Job in Four Questions
👁️
What does it see?
Context engineering — instructions, history, tool results, memory. The window is the agent's entire world.
🛠️
What can it do?
Tool design — the action surface. Tools are the API contract between model intent and the real world.
🛑
When does it stop?
Stop conditions, budgets, approval gates. Autonomy is a dial you set, not a property you hope for.
📊
Did it work?
Evals — objective outcome checks plus trajectory inspection. Without them you are shipping vibes.
🧯
What if it fails?
Error recovery and sandboxing decide whether a bad turn is a self-correction or an incident.
💸
What does it cost?
Tokens × turns × model tier. Latency and spend are architecture outputs, not billing surprises.
01 · Foundation
The Agent Loop
Strip away every framework and the same four-beat loop remains. The model gathers context, takes an action through a tool call, observes the result, and repeats — until it decides the goal is met or a guardrail says otherwise.
📥
Gather
Assemble the context window
🧠
Decide
Model picks a tool + args
⚡
Act
Harness executes the call
🔍
Observe
Result feeds the next turn
🏁
Stop
Done, blocked, or budget hit
The Loop in Code
Everything else in this guide is elaboration on one of these lines — what goes in messages, what belongs in tools, and what execute is allowed to do.
agent-loop.ts
// The whole discipline in ~20 lines
async function runAgent(goal: string, tools: Tool[]) {
const messages: Message[] = [
{ role: 'user', content: goal },
];
for (let turn = 0; turn < MAX_TURNS; turn++) {
const response = await model.generate({
messages,
tools, // schemas, not implementations
});
if (response.stopReason === 'end_turn') {
return response.text; // agent decided it is done
}
for (const call of response.toolCalls) {
const result = await execute(call); // the only real-world I/O
messages.push(toolResult(call.id, result));
}
}
throw new Error('Turn budget exhausted before goal was met');
}
Key property
The model never touches the world directly — every side effect passes through
execute(). That single choke point is where you attach permissions, logging, sandboxing, and approval gates. Guard it jealously.
01 · Foundation
Models as Engines
The model is the engine you bolt the harness onto. For agents, raw benchmark IQ matters less than three engine properties: tool-call reliability (does it emit valid calls with sane arguments?), long-horizon coherence (does turn 30 still remember the plan from turn 3?), and self-correction (does an error message change its behavior?).
Picking a Tier
Frontier tier everywhere
Every subtask pays flagship price
Latency compounds per turn
Overkill for mechanical steps
One rate limit throttles all work
Tiered routing
Frontier model plans and verifies
Small model runs bulk extraction
Escalate on failure, not by default
Cost scales with difficulty
Compounding errors
A 95%-reliable step is a 36% failure rate over 20 turns. Agents amplify small per-turn defects exponentially — which is why a model that is 3 points better on tool-use benchmarks can be the difference between a demo and a product.
What to Benchmark Yourself
valid tool-call ratearg hallucination rateturns-to-goal$ per solved taskrecovery after injected errorcache hit rate
Public leaderboards rank chat quality. Your harness, your tools, and your task distribution are the only benchmark that predicts production behavior — see the evals section.
02 · Architecture
Context Management
The context window is the agent's working memory, and it degrades before it overflows — models attend less reliably to the middle of a bloated window long before the hard token limit. Treat window space as the scarcest resource in the system.
The Per-Turn Budget
build-context.ts
// Every turn re-decides what deserves window space
function buildContext(state: AgentState): Message[] {
return [
system(state.instructions), // stable, cache-friendly prefix
...state.pinnedFacts, // decisions, constraints
summarize(state.oldTurns), // compacted history
...state.recentTurns.slice(-8), // full fidelity recency
latestToolResults(state, { maxTokens: 4_000 }),
];
}
// Sub-agents keep bulk out of the parent window entirely:
// spawn('find every call site of createUser') returns a
// 10-line report, not 300 file reads.
Three Pressure-Relief Valves
🗜️
Compaction
Summarize old turns into a durable brief: decisions made, constraints discovered, work remaining. Drop the raw transcript.
🕵️
Sub-agents
Delegate search-heavy work. The sub-agent burns its own window on file dumps and returns a conclusion, not the evidence.
📝
External memory
Notes files, scratchpads, task lists. State that must survive compaction lives outside the window and is re-read on demand.
Cache economics
Providers price cached prefix tokens at ~10% of fresh ones. A stable system prompt + append-only history can cut agent cost 5–10×. Anything that rewrites earlier messages — dynamic timestamps, reordered tools — silently invalidates the cache and multiplies your bill.
02 · Architecture
Stop Conditions & Guardrails
An agent that cannot stop is a bug with an API key. Every production loop needs explicit answers to: how many turns, how many tokens, how much wall-clock, and which actions require a human.
guardrails.ts
const limits = {
maxTurns: 40, // hard iteration ceiling
maxTokens: 500_000, // budget, not a suggestion
maxWallClock: 15 * 60_000, // agents should not run forever
requireApproval: [
'delete_*', 'send_email', 'deploy_*', 'payments.*',
],
};
async function gate(call: ToolCall) {
if (limits.requireApproval.some((p) => matches(p, call.name))) {
const ok = await askHuman(call); // human-in-the-loop
if (!ok) return denied(call, 'User declined. Propose an alternative.');
}
return execute(call);
}
The Autonomy Dial
Low autonomy
Every action previewed + approved
Read-only tools by default
Right for: prod ops, payments
High autonomy
Auto-approve inside a sandbox
Review outcomes, not every step
Right for: dev envs, research
Design rule
Autonomy should be scoped per
action class, not per agent. The same coding agent can freely edit files in its worktree, need confirmation to push, and be flatly unable to touch production secrets. One global "YOLO mode" flag is an anti-pattern.
03 · Production
Error Recovery
Agents fail constantly in the small — a bad path, a malformed query, a stale assumption. That is fine. The design question is whether a failed tool call becomes an observation the model can react to or an exception that kills the run.
tool-errors.ts
// ❌ A thrown exception kills the loop — the model never sees it
const result = await db.query(call.args.sql); // throws on bad SQL
// ✓ Errors are observations. Return them and let the model adapt
try {
return await db.query(call.args.sql);
} catch (err) {
return {
isError: true,
content:
'query failed: ' + errMessage(err) +
'. Check column names with describe_table before retrying.',
};
}
// Well-written error text turns a dead end into a self-correction.
Recovery Ladder
🔁
Retry
Transient: backoff + retry silently
💬
Inform
Return error text to the model
🔀
Replan
Model tries a different approach
🙋
Escalate
Surface to a human with context
Loop detection
Track repeated identical tool calls. Three attempts at the same failing call with the same arguments means the error message is not teaching — inject a nudge ("this approach has failed 3×, try another route") or escalate. Budget-burn from doom loops is the most common silent agent failure.
03 · Production
Sandboxing & Permissions
Treat the agent like untrusted code that happens to be smart. It executes text it read from the internet; prompt injection means any content it ingests is a potential instruction. The defense is not a smarter prompt — it is blast-radius control.
sandbox.yaml
# Least privilege, from the outside in
container:
filesystem: workspace-only # no $HOME, no secrets dir
network: allowlist # api.internal, registry.npmjs.org
cpu: 2
memory: 4Gi
timeout: 900s
permissions:
read: auto-approve
write: auto-approve-in-workspace
execute: prompt # shell commands ask first
irreversible: always-prompt # deploy, delete, send
Layered Defense
📦
Isolation
Containers, VMs, or worktrees. The agent gets a copy of the world it can safely wreck.
🔑
Least privilege
Scoped tokens, short TTLs, no ambient credentials. The agent holds capabilities, not passwords.
🧾
Audit trail
Every tool call logged with args and results. When something goes wrong you replay the trajectory, not guess.
🙋
Human gates
Irreversible or outward-facing actions — deploys, emails, deletes — always pause for sign-off.
🕸️
Egress control
Network allowlists stop exfiltration even if injection succeeds. Deny-by-default outbound.
⏱️
Resource caps
CPU, memory, disk, wall-clock. A runaway agent should hit a ceiling, not a cloud bill.
03 · Production
Evals for Agents
Chat evals grade one response. Agent evals grade a trajectory — dozens of decisions ending in a real-world outcome. The gold standard is an objective outcome check the agent cannot see: hidden tests, a verifiable artifact, a ground-truth answer.
agent-eval.ts
// Outcome eval: did the agent actually finish the job?
const suite = defineEval({
name: 'fix-failing-test',
cases: loadCases('evals/swe/*.json'), // repo + bug + hidden test
run: async (c) => runAgent(c.prompt, { repo: c.repo }),
score: async (c, trajectory) => ({
outcome: await hiddenTestsPass(c.repo), // binary, objective
turns: trajectory.turns.length, // efficiency signal
violations: countPolicyViolations(trajectory),
}),
});
// pass@1 on outcomes is the headline number.
// Trajectory metrics explain WHY it moved.
Two Layers of Signal
Outcome (headline)
Hidden tests pass
Task completed end-to-end
pass@1 across the suite
Trajectory (diagnosis)
Turns and tokens to goal
Wrong-tool + doom-loop rate
Guardrail violations
Practice
Every production incident becomes an eval case. Run the suite on every harness change — prompt tweaks and tool renames shift agent behavior as much as model swaps do. Track pass rate over time like you track test coverage; an LLM judge can grade soft criteria, but anchor the suite in objective outcomes.
04 · Execution
Framework Landscape 2026
The 2026 convergence: every serious stack is a thin, typed loop over provider APIs plus opinions about tools, memory, and human gates. Pick by ecosystem fit, not feature checklists — the loop itself is ~50 lines everywhere.
The Short List
Claude Agent SDK
The harness behind Claude Code, productized: sub-agents, hooks, MCP, permissions, compaction out of the box. TS + Python.
Vercel AI SDK
Provider-agnostic TS toolkit — generateText, tool(), agent loops, UI streaming. The default in the React/Next world.
OpenAI Agents SDK
Lightweight handoff-centric loops with built-in tracing and guardrail hooks. Python-first.
LangGraph
Explicit state-machine graphs with checkpointing. Shines when control flow must be auditable and resumable.
smolagents
Hugging Face minimalism — agents that write code as their actions. Great for research and open-model stacks.
Roll your own
The loop is 20 lines. Legitimate when you need total control — you will reinvent evals, tracing, and gates, so budget for it.
Selection heuristic
Building product UI around an agent → Vercel AI SDK. Deep coding agents or Claude-first → Claude Agent SDK. Compliance-grade resumability → LangGraph or durable-execution engines. Research on open models → smolagents.
04 · Execution
Ship Checklist
The gap between an agent demo and an agent product is this list. Green across the board before real users touch it.
Loop & Tools
turn / token / time budgetsloop detection≤ 15 task-shaped toolserrors returned, not thrownstable prompt prefix (cache)
Safety
sandboxed executionegress allowlistapproval gates on irreversiblesscoped, short-lived credentialsfull tool-call audit log
Quality
outcome eval suite in CItrajectory metrics dashboardsincidents become eval casescost per solved task tracked
Where to start
If you adopt only three: return errors to the model, put an outcome eval in CI, and gate irreversible actions behind a human. Those three catch the majority of real-world agent incidents.