Agentic AI represents a clear evolution from stateless chatbots to complex workflows, and its expansion requires new memory architectures.
As underlying models scale up to trillions of parameters and context windows reach millions of tokens, the computational cost of remembering history is increasing faster than the ability to process it.
Organizations deploying these systems currently face a bottleneck where vast amounts of “long-term memory” (technically known as key-value (KV) cache) overwhelm existing hardware architectures.
Today’s infrastructure forces you to choose between storing inference context in scarce high-bandwidth GPU memory (HBM) or allocating it to slower general-purpose storage. The former is prohibitively expensive in large-scale contexts. The latter introduces delays and prevents real-time agent interactions.
To address this growing disparity that is hindering the scaling of agent AI, NVIDIA introduced the Inference Context Memory Storage (ICMS) platform within the Rubin architecture, proposing a new storage layer specifically designed to handle the ephemeral and fast nature of AI memory.
“AI is revolutionizing the entire computing stack and now storage,” Huang said. “AI is no longer a one-shot chatbot, but an intelligent collaborator who understands the physical world, reasons with a long-term view, acts on facts, uses tools to do real work, and retains both short-term and long-term memory.”
The operational challenge lies in the specific behavior of transformer-based models. To avoid recomputing the entire conversation history every time a new word is generated, the model stores the previous state in the KV cache. In agent workflows, this cache acts as persistent memory across tools and sessions, and grows linearly with the length of the sequence.
This creates separate data classes. Unlike financial records or customer logs, KV Cash is derived data. This is essential for immediate performance, but does not require the strong durability guarantees of enterprise file systems. General-purpose storage stacks running on standard CPUs spend energy on metadata management and replication that is not needed by agent workloads.
The current hierarchy, ranging from GPU HBM (G1) to shared storage (G4), is becoming inefficient.
Efficiency drops sharply as context leaks from the GPU (G1) to system RAM (G2) and eventually to shared storage (G4). Moving active contexts to the G4 layer introduces millisecond-level latency, increases per-token power cost, and leaves expensive GPUs idle while waiting for data.
For enterprises, this manifests itself as a bloated total cost of ownership (TCO), with power wasted on infrastructure overhead rather than active inference.
New memory layer for AI factories
The industry response includes inserting a dedicated layer into this hierarchy. The ICMS platform establishes a “G3.5” layer, an Ethernet-attached flash layer explicitly designed for gigascale inference.
This approach integrates storage directly into compute pods. By leveraging the NVIDIA BlueField-4 data processor, the platform offloads the management of this context data from the host CPU. The system facilitates agent AI scaling by providing petabytes of shared capacity per pod, allowing agents to maintain large amounts of history without hogging expensive HBM.
Operational benefits can be quantified in throughput and energy. By preserving the context associated with this middle layer (faster than standard storage but cheaper than HBM), the system can “prestige” memory back to the GPU before it is needed. This reduces GPU decoder idle time and enables up to 5x more tokens per second (TPS) for long-context workloads.
From an energy perspective, the impact is equally measurable. This architecture eliminates the overhead of general purpose storage protocols and is therefore 5x more power efficient than traditional methods.
Data plane integration
Implementing this architecture requires IT teams to look at storage networks differently. The ICMS platform leverages NVIDIA Spectrum-X Ethernet to provide the high-bandwidth, low-jitter connectivity needed to treat flash storage as if it were local memory.
For enterprise infrastructure teams, the integration point is the orchestration layer. Frameworks such as NVIDIA Dynamo and Inference Transfer Library (NIXL) manage the movement of KV blocks between tiers.
These tools work with the storage layer to ensure that the correct context is loaded into GPU memory (G1) or host memory (G2) exactly when the AI model needs it. The NVIDIA DOCA framework further supports this by providing a KV communication layer that treats context caches as first-class resources.
Major storage vendors already support this architecture. Companies such as AIC, Cloudian, DDN, Dell Technologies, HPE, Hitachi Vantara, IBM, Nutanix, Pure Storage, Supermicro, VAST Data, and WEKA are building platforms using BlueField-4. These solutions are expected to be available in the second half of this year.
Redefining your infrastructure to scale agent AI
Adopting a dedicated context memory layer has implications for capacity planning and data center design.
Data reclassification: CIOs must recognize KV cache as a unique data type. This is “temporary but delay-sensitive,” as opposed to “persistent and cold” compliance data. The G3.5 layer handles the former, allowing durable G4 storage to focus on long-term logs and artifacts. Orchestration maturity: Success depends on software that can intelligently place workloads. The system uses topology-aware orchestration (via NVIDIA Grove) to place jobs near cached contexts and minimize data movement across the fabric. Power density: By fitting more usable capacity into the same rack footprint, organizations can extend the life of their existing facilities. However, this increases the computing density per square meter, which requires proper cooling and power distribution planning.
Migration to agent AI requires physical reconfiguration of your data center. The common model of completely separating computing from slow persistent storage is incompatible with the real-time search needs of agents with photographic memories.
By introducing a specialized context layer, companies can decouple model memory growth from GPU HBM costs. This architecture for Agent AI allows multiple agents to share a large, low-power memory pool to reduce the cost of processing complex queries and improve scaling by enabling high-throughput inference.
As organizations plan their next cycle of infrastructure investments, evaluating the efficiency of their memory hierarchy will be as important as choosing the GPUs themselves.
SEE ALSO: The AI chip wars of 2025: What business leaders learned about supply chain realities
Want to learn more about AI and big data from industry leaders? Check out the AI & Big Data Expos in Amsterdam, California, and London. This comprehensive event is part of TechEx and co-located with other major technology events. Click here for more information.
AI News is brought to you by TechForge Media. Learn about other upcoming enterprise technology events and webinars.
