\n
![](\"https:\/\/images.ctfassets.net\/jdtwqhzvc2n1\/4oxKXvOOiNDzKlG7ycGI78\/fa4c22e954e072347b2c26d0f0e3ef06\/VB-WEKA-AI-Impact-2025-093.jpg?w=300&q=30\")
<\/p>\n
\n<\/p>\n
As agentic AI moves from experiments to real production workloads, a quiet but serious infrastructure problem is coming into focus: memory. Not compute. Not models. Memory.<\/p>\n
Under the hood, today\u2019s GPUs simply don\u2019t have enough space to hold the Key-Value (KV) caches that modern, long-running AI agents depend on to maintain context. The result is a lot of invisible waste \u2014 GPUs redoing work they\u2019ve already done, cloud costs climbing, and performance taking a hit. It\u2019s a problem that\u2019s already showing up in production environments, even if most people haven\u2019t named it yet.<\/p>\n
At a recent stop on the VentureBeat AI Impact Series, WEKA CTO Shimon Ben-David joined VentureBeat CEO Matt Marshall to unpack the industry\u2019s emerging \u201cmemory wall,\u201d and why it\u2019s becoming one of the biggest blockers to scaling truly stateful agentic AI \u2014 systems that can remember and build on context over time. The conversation didn\u2019t just diagnose the issue; it laid out a new way to think about memory entirely, through an approach WEKA calls token warehousing.<\/p>\n
\u201cWhen we're looking at the infrastructure of inferencing, it is not a GPU cycles challenge. It's mostly a GPU memory problem,\u201d said Ben-David. <\/p>\n
The root of the issue comes down to how transformer models work. To generate responses, they rely on KV caches that store contextual information for every token in a conversation. The longer the context window, the more memory those caches consume, and it adds up fast. A single 100,000-token sequence can require roughly 40GB of GPU memory, noted Ben-David.<\/p>\n
That wouldn\u2019t be a problem if GPUs had unlimited memory. But they don\u2019t. Even the most advanced GPUs top out at around 288GB of high-bandwidth memory (HBM), and that space also has to hold the model itself. <\/p>\n
In real-world, multi-tenant inference environments, this becomes painful quickly. Workloads like code development or processing tax returns rely heavily on KV-cache for context. <\/p>\n
\u201cIf I'm loading three or four 100,000-token PDFs into a model, that's it \u2014 I've exhausted the KV cache capacity on HBM,\u201d said Ben-David. This is what\u2019s known as the memory wall. \u201cSuddenly, what the inference environment is forced to do is drop data," he added. <\/p>\n
That means GPUs are constantly throwing away context they\u2019ll soon need again, preventing agents from being stateful and maintaining conversations and context over time<\/p>\n
\u201cWe constantly see GPUs in inference environments recalculating things they already did,\u201d Ben-David said. Systems prefill the KV cache, start decoding, then run out of space and evict earlier data. When that context is needed again, the whole process repeats \u2014 prefill, decode, prefill again. At scale, that\u2019s an enormous amount of wasted work. It also means wasted energy, added latency, and degraded user experience \u2014 all while margins get squeezed.<\/p>\n
That GPU recalculation waste shows up directly on the balance sheet. Organizations can suffer nearly 40% overhead just from redundant prefill cycles This is creating ripple effects in the inference market.<\/p>\n
\u201cIf you look at the pricing of large model providers like Anthropic and OpenAI, they are actually teaching users to structure their prompts in ways that increase the likelihood of hitting the same GPU that has their KV cache stored,\u201d said Ben-David. \u201cIf you hit that GPU, the system can skip the prefill phase and start decoding immediately, which lets them generate more tokens efficiently.\u201d <\/p>\n
But this still doesn't solve the underlying infrastructure problem of extremely limited GPU memory capacity. <\/p>\n
\u201cHow do you climb over that memory wall? How do you surpass it? That's the key for modern, cost- effective inferencing,\u201d Ben-David said. \u201cWe see multiple companies trying to solve that in different ways.\u201d<\/p>\n
Some organizations are deploying new linear models that try to create smaller KV caches. Others are focused on tackling cache efficiency. <\/p>\n
\u201cTo be more efficient, companies are using environments that calculate the KV cache on one GPU and then try to copy it from GPU memory or use a local environment for that,\u201d Ben-David explained. \u201cBut how do you do that at scale in a cost-effective manner that doesn't strain your memory and doesn't strain your networking? That's something that WEKA is helping our customers with.\u201d<\/p>\n
Simply throwing more GPUs at the problem doesn\u2019t solve the AI memory barrier. \u201cThere are some problems that you cannot throw enough money at to solve," Ben-David said. <\/p>\n
WEKA\u2019s answer is what it calls augmented memory and token warehousing \u2014 a way to rethink where and how KV cache data lives. Instead of forcing everything to fit inside GPU memory, WEKA\u2019s Augmented Memory Grid extends the KV cache into a fast, shared \u201cwarehouse\u201d within its NeuralMesh architecture.<\/p>\n
In practice, this turns memory from a hard constraint into a scalable resource \u2014 without adding inference latency. WEKA says customers see KV cache hit rates jump to 96\u201399% for agentic workloads, along with efficiency gains of up to 4.2x more tokens produced per GPU.<\/p>\n
Ben-David put it simply: "Imagine that you have 100 GPUs producing a certain amount of tokens. Now imagine that those hundred GPUs are working as if they're 420 GPUs."<\/p>\n
For large inference providers, the result isn\u2019t just better performance \u2014 it translates directly to real economic impact. <\/p>\n
\u201cJust by adding that accelerated KV cache layer, we're looking at some use cases where the savings amount would be millions of dollars per day,\u201d said Ben-David<\/p>\n
This efficiency multiplier also opens up new strategic options for businesses. Platform teams can design stateful agents without worrying about blowing up memory budgets. Service providers can offer pricing tiers based on persistent context, with cached inference delivered at dramatically lower cost. <\/p>\n
NVIDIA projects a 100x increase in inference demand as agentic AI becomes the dominant workload. That pressure is already trickling down from hyperscalers to everyday enterprise deployments\u2014 this isn\u2019t just a \u201cbig tech\u201d problem anymore.<\/p>\n
As enterprises move from proofs of concept into real production systems, memory persistence is becoming a core infrastructure concern. Organizations that treat it as an architectural priority rather than an afterthought will gain a clear advantage in both cost and performance.<\/p>\n
The memory wall is not something organizations can simply outspend to overcome. As agentic AI scales, it is one of the first AI infrastructure limits that forces a deeper rethink, and as Ben-David\u2019s insights made clear, memory may also be where the next wave of competitive differentiation begins.<\/p>\n