Most cost investigation is a search problem. You have a number, and you go looking for the thing that produced it. The instrument you reach for is usually the one matched to the service: request graphs for an API, logs for a function, a savings report for the estate.
This post is about three cases where that instrument is structurally incapable of showing you the cost. Not misconfigured. Incapable. The charge exists in a dimension the graph does not plot.
One of the three is a gap in my own detection engine, which I will get to, because it is the clearest example of the pattern I know.
1. A dead REST API that is not free
API Gateway bills per call. So a REST API nobody calls costs nothing in request charges, and that is genuinely true. It is the basis for a common and correct piece of advice: an idle REST API is worth cleaning up for hygiene, but it is not costing you money.
Unless something provisioned a cache on the stage.
A REST API cache bills hourly by cache size, regardless of traffic. At $0.038 per hour for 1.6 GB that is about $27 a month ($0.038 x 730 = $27.74) with nobody calling the API at all.
How the cache gets left on
The usual route is a load test. AWS's own caching guidance recommends running a 10-minute load test after provisioning, and the stage control is a persistent setting labelled "Provision API cache". So the sequence is: someone provisions a cache to test throughput, runs the test, gets their answer, and moves on. The stage keeps the cache. The meter keeps running.
Two years later the API has no traffic, and everyone looking at it correctly concludes that a zero-request REST API is free.
Why no dashboard catches it
This is the structural part. The cache charge is hourly by cache size, not per request. It therefore does not appear in request metrics at all. You can stare at an invocation graph reading a flat zero for a month while the stage bills $27 every one of those months.
There is no anomaly either, which rules out the other common instrument. The charge is perfectly flat. It has been flat since the day of the load test.
2. The rule I got wrong
I build a detection engine for this category, and it has a rule for idle REST APIs: APIGW-O002, which flags REST APIs with zero requests in 30 days and reports a saving of $0.
That $0 is deliberate. Per-request pricing means an unused REST API genuinely costs nothing in request charges, so reporting a dollar saving would be inventing one. The rule exists to tell you the API is dead, not to claim money back. I stand by that design.
The problem is that the rule reports $0 unconditionally. It does not check whether the stage has a cache provisioned. So on exactly the case described above, an idle stage with a cache left on, my own rule under-reports. It tells you there is nothing to save, and there is $27 a month sitting there.
That one I still owe a fix.
I am publishing it because rule count is a weak headline, mine included. Adding the next detection rule is the easy part of building an engine like this. Working out which findings are worth putting in front of someone, and being honest about where a finding is incomplete, is the part I keep getting wrong. A tool that only ever tells you what it found is less useful than one that tells you what it cannot see.
Practical takeaway that does not depend on my roadmap: if you run REST APIs, check whether any idle stage still has caching provisioned. Request graphs will not show it. Neither will my rule.
3. Lambda bills for startup, and sometimes does not log it
On 1 August 2025, AWS standardised billing for the initialization (INIT) phase across all Lambda function configurations. INIT is the startup code that runs outside your handler: imports, SDK clients, config and secrets loaded at module scope.
Before that date, that phase ran unbilled on functions packaged as ZIP files using managed runtimes and invoked on demand. Three setups were already paying for it: custom runtimes, Provisioned Concurrency and OCI (container image) packaging all already included the INIT phase duration in their Billed Duration.
AWS is measured about the impact, and it is worth repeating rather than sensationalising: most users will see minimal impact on their overall Lambda bill, because INIT typically occurs for a very small fraction of function invocations, and cold starts typically occur in under 1% of invocations.
So for most functions this is a rounding error. It bites in one specific shape: where startup is a large share of a short invocation. Heavy SDK imports, a config fetch, a secrets call, all outside the handler, wrapped around 40ms of actual work. My read, not AWS's: your exposure tracks how many execution environments you create, not how much traffic you serve.
Where it hides in the logs
INIT never appears as its own charge. On the invocation that initialises a fresh environment, the time is folded into Billed Duration, the milliseconds you already pay for. The log REPORT line does carry an Init Duration field on a normal cold start, so you can at least see it there. The bill cannot show it separately.
The version with no log line at all
Now the case that is genuinely invisible.
If a Lambda function crashes or times out during the Invoke phase, Lambda resets the execution environment. If that environment is then used for a new invocation, Lambda re-initialises the runtime along with that next invocation. AWS has a name for this: a suppressed init.
And this is the part to sit with:
When suppressed inits occur, Lambda doesn't explicitly report an additional INIT phase in CloudWatch Logs.
No Init Duration field appears. AWS's own worked example of a suppressed-init REPORT line carries Duration, Billed Duration, Memory Size and Max Memory Used, and no Init Duration at all. The startup time lands inside that invocation's reported duration instead, so the call simply looks slower than it should be.
And you pay for it. Duration charges apply to code that runs in the handler as well as initialization code declared outside it, and AWS states directly that you are charged for this time and that it adds latency to your overall invocation duration.
The compounding effect matters here: a function that times out is a function that will re-run INIT on the next call. So a timeout problem is quietly also a billing problem, and the billing half is not visible in the place you would look for it.
"Always warm" is not the exit
The instinctive fix is to keep environments alive with scheduled pings. That does not close it:
Lambda terminates execution environments every few hours to allow for runtime updates and maintenance, even for functions that are invoked continuously.
Those re-inits are billable. And you cannot schedule around it, because AWS publishes no execution environment idle timeout, describes retention as a non-deterministic period of time, and says explicitly that you should not assume the execution environment will persist indefinitely.
The real fix is the boring one: move work out of module scope, load secrets lazily, and stop importing the entire SDK when you need one client.
4. Savings reports do not add up, literally
The last one is not a hidden charge but a hidden error, and it is in the instrument itself.
Take one db.m5 RDS instance on gp2 storage, with low CPU and few connections over 14 days. In my engine it trips three rules simultaneously:
- RDS-O006: the storage should move from gp2 to gp3.
- RDS-O013: current-generation Intel families, db.m5 named explicitly, can migrate to Graviton. Its description states 10-11% savings.
- RDS-O007: low CPU and connection utilisation over 14 days, so it can be downsized.
All three findings are correct. Their savings cannot simply be added.
RDS-O013 and RDS-O007 both price the same instance-hours. Graviton takes a percentage off the hours that one instance runs; downsizing reprices those identical hours at a smaller size. Add both and one dollar is counted twice. RDS-O006 touches storage, which is a separate line on the bill, so that one does stack cleanly.
Handling this is the actual engineering in a rules engine, and it is not the part anyone puts in marketing. Writing the next rule was never the hard part. Making sure a new rule does not count a dollar an older rule already counted is. My scan folds overlapping findings together so no dollar is counted twice.
None of that is special to my rules. It is arithmetic. Two recommendations that price the same hours on the same instance cannot be added, whoever produced them, whether that is a vendor, a cloud provider's own console, or a spreadsheet someone built. It is worth checking any savings list for overlap before you total it and take the total to a finance conversation.
The shape of all four
Each one breaks the link between the cost and the instrument you would naturally use to find it:
- The API Gateway cache bills hourly, so a per-request graph cannot plot it.
- A suppressed init is folded into invocation duration, so a log field that would name it is never emitted.
- Overlapping recommendations are individually correct, so reviewing each finding will never reveal the double count. Only the total is wrong.
In every case the instrument is working exactly as designed and telling you the truth about the thing it measures. It just is not measuring the thing that is costing you money.
What to check
- List REST API stages with caching provisioned, then cross-reference against stages with zero requests over 30 days. That intersection is pure waste and no request dashboard will surface it.
- Find functions with a high timeout or error rate and treat them as a billing item, not only a reliability item. Every reset environment re-runs INIT on the next call, unlogged.
- Look at what your functions do at module scope. Startup cost is now billed on every configuration, and it scales with environment churn rather than traffic.
- Before totalling any savings report, group findings by the bill line they price. Same instance-hours means pick one, not both. Different lines stack.
Sources
AWS statements are from AWS documentation and pricing pages. The monthly cache figure is arithmetic on AWS's published hourly rate. The APIGW-O002 and RDS rule behaviour is from my own engine's rule registry, quoted as it stands today including the known gap.