Amazon has told its engineers that token counts, AI adoption rates and code volume are poor measures of how well teams use AI. The same memo tells managers to keep studying the teams that use AI least, the kind of pressure that led employees to game an internal leaderboard in May.
Amazon has told its engineers that token usage, AI tool adoption rates and code changes are "deeply flawed" ways to measure how teams use AI, CNBC's Annie Palmer reported Tuesday. The memo went to software engineers and their managers in September, and Amazon posted it on its website the same day CNBC's story ran.
Senior engineering leaders wrote it, including the company's Distinguished and Senior Principal Engineers, as part of an update to the job guideline for Amazon software engineers. They argue that these metrics have little connection to results for customers or the business, and that using them as targets raises costs and makes systems more complicated. The memo links each problem to an Amazon leadership principle, saying tokenmaxxing conflicts with frugality and counting lines of code conflicts with invent and simplify.
It also tells engineers who don't see a benefit from AI tools that this is "a finding, not a failing." They're asked to document why, share it with peers and bring it to their manager. The memo says engineers who can explain where AI doesn't help yet are showing the judgment Amazon wants.
At State of AI, we think this is the right correction from one of the companies that pushed hardest on AI usage targets, and other enterprises should copy most of it. We also think the memo hedges in a way that could undercut its own advice, and managers reading it should notice where.
Amazon already saw what usage targets do
In May, Amazon shut down KiroRank, a leaderboard some employees built to rank colleagues by their activity on Kiro, the company's AI coding tool. People had been running AI agents on pointless tasks to move up the rankings, which added to Amazon's compute costs, CIO reported, citing the Financial Times. Senior vice president Dave Treadwell told staff, "Please don't use AI just for the sake of using AI," HR Brew reported.
The leaderboard showed up while Amazon was pushing toward a goal of having more than 80% of its developers use AI every week. After shutting it down, the company started measuring how much AI-assisted work gets deployed, Heise reported.
Amazon wasn't the only company with this problem. Meta ended a similar employee-built ranking in April, according to CIO, and Uber's CTO said the company had spent its whole year's Claude Code budget by April, Shopifreaks reported.
The memo keeps the metrics it criticizes
Much of today's coverage reads the memo as Amazon dropping token metrics, and that isn't what it says. According to the memo, these input metrics have tracked well with real results so far, and Amazon expects that to continue. It says teams and people who aren't adopting AI are falling behind, and it tells managers they can keep using the numbers as long as they're careful about it.
Managers are also asked to study the 10% of teams with the lowest AI adoption to find out why, and the top 10% to check whether heavy use has led to more delivered work. Every engineer who hasn't started using AI is told to start now.
Marc Brooker, a vice president and distinguished engineer at AWS, told CNBC that Amazon also tracks how quickly teams ship features. A spokesperson told CNBC that Amazon hasn't capped how many tokens employees can use.
The memo holds managers to a standard as well. They're expected to have used AI tools on their own production work and to be able to describe a task where the tools helped and one where they didn't.
Engineers will remember the bottom-10% instruction
An Amazon engineer reading this memo gets two messages. One says it's fine to skip AI when there's a good reason. The other says managers will be looking closely at the teams that use AI least. When a company sends both, employees usually act on the one they think will come up in their review, and that instinct is what produced KiroRank.
We'd have liked the memo to say plainly that a team at the bottom of the adoption chart, with documented reasons, is doing what Amazon asked. As written, "a finding, not a failing" holds only as long as each manager decides to honor it.
AWS sells the tokens Amazon is telling staff to use carefully
Amazon published the memo partly for customers, saying it hopes its view of the engineer's role can serve as a model for them. People online have pointed out the irony. AWS earns more when customers use more AI compute, and Amazon is now telling its own staff that running up tokens goes against frugality.
We don't read it as hypocrisy. Amazon pays for its employees' AI use just as its customers pay for theirs, and KiroRank showed how fast that bill grows when usage becomes the goal. Customers should take the advice seriously because it comes from a company that knows exactly what heavy usage costs.
Our take
Enterprise leaders should take three things from this memo and change one.
Take the "finding, not a failing" standard and make the documentation mandatory. A short note on what an engineer tried, on which task, and why it didn't work is useful data. Collected across a team, those notes show where AI fits your business far better than a token dashboard does.
Take the manager requirement. Anyone setting AI targets should have used the tools on their own work and be able to say where they helped and where they didn't.
Take the cost argument to finance and judge token spend against what shipped, which is where Amazon ended up after KiroRank. One engineering leader told us recently that he uses spending caps and simple metrics to keep agent costs in line with what the agents deliver.
Change the bottom-10% review. If you track adoption, say openly that low usage with a documented reason counts as good work, and make sure it's treated that way in performance reviews. If that isn't explicit, expect engineers to keep using AI to protect their standing, and expect to pay for those tokens.
