CloudWatch is rarely anyone's first guess when a bill looks wrong, because it does not feel like infrastructure. Nobody provisions it. It accumulates.
Which is exactly the problem. Both halves of it, logs and metrics, have defaults that grow without anyone choosing to grow them, and in both cases the growth is driven by something that looks like good engineering practice.
Part one: logs keep everything, forever
By default, log data is stored in CloudWatch Logs indefinitely. On a log group where nobody set retention, the console control reads Never Expire, which is at least honest about it.
That default is defensible for an audit log. It is expensive for the debug output of a service that has been running for four years.
Retention is per log group and does not propagate
This is the mechanical reason estates drift. Retention is set per log group. The 30 days you set on one group says nothing about the next one, or about the group that gets created next sprint.
And the number of groups grows with the application:
- By default, a Lambda function's logs are stored in a log group named
/aws/lambda/<function-name>. One group per function, created automatically the first time it runs. - On ECS the log group is named per container definition. AWS's own example task definition writes to
awslogs-wordpressandawslogs-mysql: two containers, two groups, one task.
So the thing you have to remember to configure is created for you, silently, every time someone ships a function or adds a container. A retention policy applied by hand is a policy that covers the estate as it looked on the day someone applied it.
The charge stops before the data does
This part is genuinely good news and worth knowing precisely, because it changes how quickly a cleanup pays off.
When log events reach their retention setting they are marked for deletion. CloudWatch Logs does not immediately delete them. But AWS is explicit about the billing consequence:
After they are marked for deletion, they do not add to your archival storage costs anymore, even if they are not actually deleted until later.
So cutting retention on a large log group stops the storage charge at the retention mark rather than whenever the bytes physically go away. Actual deletion typically takes up to 72 hours, and in rare situations longer, but you are not paying during that window.
The 72-hour trap
There is a sharp edge in that same window. If you raise retention on a group that currently holds events past their expiration date but not yet actually deleted, those events survive to the new retention date instead of going away.
Which means a hasty sequence, cut retention to 7 days, panic, put it back to 90, does not restore the previous state. It resurrects data you had already stopped paying for and starts paying for it again. If you are going to cut retention, cut it deliberately and leave it.
Retention does nothing to ingestion
The most common mistake I see here is assuming a retention policy is a general CloudWatch Logs cost fix. It is not. It only touches one of two lines:
- Archival storage, the keep-it charge, $0.03 per GB per month in us-east-1 and higher in other regions. Retention cuts this.
- Ingestion, what AWS charges to accept the logs in the first place. Retention does nothing to this at all.
Which of the two dominates depends entirely on your volume and how long you have been keeping things. A chatty service with 7-day retention can be almost pure ingestion cost, in which case tuning retention further is wasted effort and the real fix is logging less. A quiet service with Never Expire set five years ago is almost pure storage. You cannot tell which you have without looking at both lines separately.
Part two: metrics multiply
The logs half is a housekeeping problem. The metrics half is more interesting, because the cost is created in your code and there is no idle resource anywhere to find.
A dimension is a name/value pair that is part of the identity of a metric, like Endpoint=/checkout. And metrics are uniquely defined by a name, a namespace and zero or more dimensions. That identity rule is also the billing rule:
CloudWatch treats each unique combination of dimensions as a separate metric, even if the metrics have the same metric name.
So one metric name with one extra dimension carrying 100 values is 100 metrics on the bill.
What that costs
Custom metrics are the ones your code publishes, as distinct from the ones AWS emits for you. In us-east-1 the first 10,000 are $0.30 per metric.
Take a hypothetical: 100 customer IDs on one dimension, 5 endpoints on another. That is up to 500 metrics, one per combination you actually publish, and 500 x $0.30 = $150 a month. Both of those numbers are stipulated for the example rather than measured from anything, and the multiplication is mine.
Treat $150 as a ceiling rather than a forecast. All custom metrics and Detailed Monitoring charges are prorated by the hour, and charges are incurred only for hours in which metrics are actually sent. A dimension value that reports for two hours a day does not cost a full month.
One thing not to lean on: the 10 free metrics are a shared pool across custom metrics and Detailed Monitoring, so you cannot assume they are available to absorb the first ten of anything.
You cannot delete a metric
There is no cleanup operation here, which surprises people who go looking for one:
Metrics cannot be deleted, but they automatically expire after 15 months if no new data is published to them.
The saving grace is the hourly meter. Because you are only charged for hours in which a metric reports, the charge stops when the code stops publishing. You fix this at the write site, in the code emitting the dimension, not by tidying anything up in the console.
The ones you cannot see are the ones you already fixed
A useful and slightly disorienting detail: metrics that have not had any new data points in the past two weeks do not appear in the console, and are not returned in the results of a list-metrics command.
So the inventory only ever shows you live cardinality. A dimension explosion that someone already removed becomes invisible within a fortnight, which is convenient, but it also means you cannot audit historical cardinality after the fact to explain a past bill.
To find which dimension is responsible while it is still live:
aws cloudwatch list-metrics --namespace <your-namespace>Then count distinct values per dimension name. The one that exploded is usually obvious the moment you group it: a customer ID, a request ID, a user ID, a full URL path with parameters in it.
Why this one is hard to catch
Nothing in either half looks like waste. An unattached EBS volume looks abandoned the second you see it. A log group with Never Expire looks like retention nobody configured, which is exactly what it is. And a high-cardinality metric looks like a team that instruments its code well.
That last one is the real trap. Adding a customer ID dimension so you can slice latency per tenant is a good engineering instinct. It is the kind of thing that gets praised in review. It just happens to bill per combination, and nobody reviewing the pull request is looking at a price list.
What to check
- List log groups with no retention set or retention beyond 30 days. This is a list, not a dollar figure: what it costs depends entirely on volume per group. My own scanner has a rule for exactly this, CWL-O001, and it deliberately reports no dollar amount for the same reason.
- Split your CloudWatch Logs spend into ingestion and archival storage before deciding what to do. They respond to completely different fixes.
- Run
list-metricsagainst your own namespaces and count values per dimension name. Anything unbounded, IDs, paths, request identifiers, is a cardinality problem waiting on traffic growth. - If you cut retention, cut it once and leave it. Raising it back inside the 72-hour window revives data you had stopped paying for.
Sources
AWS statements are from AWS documentation and the CloudWatch pricing page. The $150 monthly figure is arithmetic on a hypothetical scenario, labelled as such in the text.