Weka, a data infrastructure company, is launching NeuralMesh 6, a storage platform designed to reduce GPU memory pressure in production AI systems by caching pre-calculated tokens. The approach addresses a fundamental bottleneck in large language models: repeatedly recomputing information during long context windows and multi-turn conversations wastes expensive GPU resources.
The platform uses cheap NAND flash storage to extend GPU memory through what Weka calls an Augmented Memory Grid. Instead of forcing models to regenerate tokens they have already processed, the system pre-calculates and caches tokens in flash, allowing GPUs to retrieve them on demand. This shifts computational load from the most expensive resource in AI infrastructure to significantly cheaper storage hardware.
GPU memory remains the primary constraint limiting production AI deployment. Each additional context window or conversation turn forces recomputation of previously processed data, consuming both memory and compute cycles that could serve more users or generate new responses faster. By externally caching these pre-calculated tokens, Weka's platform allows organizations to operate larger models or serve more concurrent users without purchasing additional GPUs.
The NeuralMesh 6 launch includes Wekapod 3, Weka's first self-designed hardware line built specifically for this token-caching workload. The hardware focuses on flash-based storage optimized for the sequential read patterns common in token retrieval.
This approach represents a shift in how the industry thinks about AI infrastructure optimization. Rather than scaling vertically by adding more GPUs, companies can scale horizontally using cheaper flash storage. The economics are compelling: flash storage costs a fraction of GPU memory, making this strategy attractive for enterprises running large models at scale.
Weka targets organizations operating production LLMs where GPU memory constraints limit model size or user concurrency. The platform essentially treats GPU memory as a cache for a larger virtual memory pool backed by flash, following principles from