We Log Every Agent Call. The Night the Log Itself Broke

We Log Every Agent Call. The Night the Log Itself Broke

Tool calls kept working. Nothing was logged.

What should you log when agents call tools, and how do you know the log is still writing? Five things. Store the full request and response per session. Never store secrets or approval codes. Delete everything after 90 days. Add thumbs up and down. Replay a real session on dev with read-only calls only. A monitor on call-log failures tells you when that log stops writing, even if the tools still work.

AI agent observability is the record of what an agent was asked and what it did, plus a way to know that record is still being written.

The hub for this seat is fractional CTO services for mid-market and PE portcos. That page is the operator who owns the release when about eight agents are building at once. If the product in front of customers is the AI feature, the same ownership question sits with a fractional chief AI officer. The two seats beside it are what a fractional CIO owns and what a CTO of a company owns.

What did I ask to see?

For a media AI platform client, I asked to see what users actually ask through the AI connector, so the tools can be improved.

What did the call log store?

The team shipped a call log. It stores the full request and the full response for each session. It adds thumbs up and thumbs down.

What does the log never keep?

It never stores secrets or approval codes. It deletes everything after 90 days.

What can a replay do?

The log can replay a real session on dev with read-only calls only. It refuses anything that writes, spends, approves, or sends.

What went out the same night?

The same night, a change that consolidated database connection pools went out in a production release without its follow-up fix. Tool calls kept working. Nothing was logged.

What caught it?

A Datadog monitor on call-log failures caught it. A hotfix with only that fix shipped. The alert recovered within minutes.

What did the pool change do to connections?

The pool work cut each Lambda from 9 separate pools, up to 27 connections, to 2 shared pools, at most 7. A 15-minute production watch peaked at 24 connections with 0 errors.

What did we decide after that?

With about eight agents building in parallel, every release now goes through one release owner.

What does AI agent observability check at each stage?

1. Before build. Capture the full request and the full response per session. Never store secrets or approval codes. Delete everything after 90 days. Add thumbs up and down. A replay on dev is read-only, and it refuses anything that writes, spends, approves, or sends.
2. Before ship. The follow-up fix ships in the same release as the pool change.
3. After ship. Watch production. On this file, a 15-minute watch peaked at 24 connections with 0 errors after each Lambda moved from 9 separate pools (up to 27 connections) to 2 shared pools (at most 7).
4. When the monitor fires. Ship a hotfix that contains only that fix. With about eight agents building in parallel, every release goes through one release owner.

What do you capture, and what do you never store?

Capture

• The full request and the full response, per session.
• Thumbs up and thumbs down.
• A replay of a real session on dev, read-only calls only.

Never store

• Secrets.
• Approval codes.

How long it stays

• Delete everything after 90 days.

What replay refuses

• Anything that writes, spends, approves, or sends.

What changed in the pools?

Before

• 9 separate pools on each Lambda.
• Up to 27 connections.

After

• 2 shared pools on each Lambda.
• At most 7 connections.
• A 15-minute production watch peaked at 24 connections with 0 errors.

Why are teams starting to talk about a harness?

Teams are starting to talk about harness engineering. The work is controlling what an agent can see, and verifying what it produced. Agents are running for hours with less oversight.

The operator version is shorter. Controlling what an agent sees does not tell you the log is still writing. When the tools still answer and the record is empty, you have a feature that works.

What will an acquirer ask?

If an AI feature is in front of customers, an acquirer will ask what it was asked and what it did. A log with retention rules and secret redaction, plus an alert when that log stops writing, answers both.

That question belongs on a technology due diligence consulting file. The control on what an agent may do is AI governance consulting. The labels on the tools the log is watching are the AI governance checklist before an MCP connector. The loop those calls sit inside is what an AI agent is.

Don't trust the feature that works. Trust the monitor on the log.

If an AI feature is in front of customers tonight, what was it asked, what did it do, and which alert fires when that log stops writing?

If that is the file you want before someone asks, talk to us.

Related reading

• Fractional CTO for mid-market and PE portcos
• What a fractional CIO owns
• What is a CTO of a company
• AI governance checklist before an MCP connector
• What is an AI agent
• Technology due diligence consulting
• What AI governance consulting covers
• Fractional chief AI officer