James Shore, writing on his Successful Software blog and republished by O'Reilly Radar, argues that companies need to measure the return on investment from AI in software development before deciding how to spend on it. He describes a CIO with 170 people who measured a 2.1x increase in productivity from AI. The same organization's top AI users each spent $21,000 per month on tokens. Another engineering leader reported spending six to eight thousand dollars per engineer per month.
Shore says these costs will likely come from personnel budgets eventually, which makes the trade-offs important. He warns that gut feelings are unreliable. He cites a METR study in which engineers believed AI made them 49% faster than it actually did.
On costs, Shore notes that large AI firms are offering substantial discounts to gain market share, so current prices may not last. He lists factors that could push token costs down, including open-weight models, local inference, dedicated inference chips, routers that send prompts to the cheapest suitable model, and models that need fewer tokens. He recommends stating assumptions explicitly and modeling both optimistic and pessimistic price scenarios.
Shore also says adoption takes time, and productivity may fall during a transition before it rises, which can make early measurements misleading. He says