Finding the AI spend nobody could explain

Problem

As Technical Lead at MindFortress I owned environment health and deploys for two AI products. One bucket of their AI spend had no owner: about $16 a day that no user or project accounted for.

A leak that size is easy to live with. It is also a sign that something in production is doing work nobody asked for — and whatever that is, it will not stay small on its own.

Approach

I treated it as a bug with a number attached. Instead of guessing where the money might be going, I followed the unattributed spend back until it led to its source: a stale staging worker running against the production database.

1. 01Spend~$16/day, no owner
2. 02Traceback, not guessed
3. 03Sourcea stale staging worker
4. 04Guardguard + split credentials

1. 01One bucket of AI spend had no owner: about $16 a day that no user or project accounted for.
2. 02Instead of guessing where the money went, I followed the unattributed spend back to its source.
3. 03The source was a stale staging worker running against the production database.
4. 04A guard in the code keeps it from happening again, and production and staging no longer share credentials.

Step 1 of 4

One bucket of AI spend had no owner: about $16 a day that no user or project accounted for.

Stopping it once would not have been a fix, because the conditions that started it could return. So I shipped a guard in the code to keep the same situation from happening again, and split the production and staging credentials.

Result

The leak was traced to the worker behind it, guarded in the code, and production and staging stopped sharing credentials. It was part of a team effort to attribute every LLM call to a user or project.

The guard turns a quiet, recurring cost into a failure that would show up immediately. The broader habit is the one I apply to any AI system I run: spend is a metric like latency, and a number nobody can explain is an incident waiting to be named.