Banks increasingly buy dedicated GPU clusters for on-premises AI workloads, but utilization remains stubbornly low. The Singapore bank profiled here purchased eight H100s to keep sensitive customer documents on-site and avoid cloud vendor lock-in, a rational security and independence play. Yet finance teams discover these expensive chips sit idle most nights, raising hard questions about ROI.
This pattern reflects a broader tension in enterprise AI deployment. On-premises compute solves real problems: regulatory compliance, data sovereignty, and vendor independence matter for regulated institutions. However, the economics don't work at scale. A single H100 costs roughly $40,000. Eight units represent $320,000 in capital expenditure, plus ongoing power, cooling, and maintenance costs. Running at 20-30% utilization turns that investment into expensive overhead.
The core issue surfaces a mismatch between demand and infrastructure. Organizations buy peak capacity to handle occasional workload spikes, then face months of underutilized hardware. Finance teams watch the red ink accumulate while IT teams defend the purchase as necessary for sovereignty.
Three patterns emerge. First, companies underestimate the effort required to fill expensive clusters with productive work. Second, dedicated infrastructure locks teams into specific hardware architectures and vendors, creating different but equally binding constraints as cloud lock-in. Third, the economics of small-scale on-premises compute remain poor compared to cloud alternatives unless utilization consistently exceeds 70%.
Some banks address this by mixing strategies. They reserve on-premises compute for sensitive workloads (customer data processing, model training on proprietary datasets) while using cloud capacity for bursty, non-sensitive tasks. This hybrid model requires discipline around data classification and workload routing.
The Singapore bank's situation represents a broader wake-up call. Buying sovereign compute for compliance reasons remains legitimate, but teams must right-size purchases to realistic utilization forecasts