\n
![](\"https:\/\/images.ctfassets.net\/jdtwqhzvc2n1\/2VCnNmP0CWrEo1sgvW4EaV\/fdf8524a18ae6c74af92aaf6cbaf90c5\/Brownie_recipe.png?w=300&q=30\")
<\/p>\n
\n<\/p>\n
Today\u2019s LLMs excel at reasoning, but can still struggle with context. This is particularly true in real-time ordering systems like Instacart.\u00a0<\/p>\n
Instacart CTO Anirban Kundu calls it the "brownie recipe problem." <\/p>\n
It's not as simple as telling an LLM \u2018I want to make brownies.\u2019 To be truly assistive when planning the meal, the model must go beyond that simple directive to understand what\u2019s available in the user\u2019s market based on their preferences \u2014 say, organic eggs versus regular eggs \u2014 and factor that into what\u2019s deliverable in their geography so food doesn\u2019t spoil. This among other critical factors.\u00a0<\/p>\n
For Instacart, the challenge is juggling latency with the right mix of context to provide experiences in, ideally, less than one second\u2019s time.\u00a0<\/p>\n
\u201cIf reasoning itself takes 15 seconds, and if every interaction is that slow, you're gonna lose the user,\u201d Kundu said at a recent VB event.\u00a0<\/p>\n
In grocery delivery, there\u2019s a \u201cworld of reasoning\u201d and a \u201cworld of state\u201d (what\u2019s available in the real world), Kundu noted, both of which must be understood by an LLM along with user preference. But it\u2019s not as simple as loading the entirety of a user\u2019s purchase history and known interests into a reasoning model.\u00a0<\/p>\n
\u201cYour LLM is gonna blow up into a size that will be unmanageable,\u201d said Kundu.\u00a0<\/p>\n
To get around this, Instacart splits processing into chunks. First, data is fed into a large foundational model that can understand intent and categorize products. That processed data is then routed to small language models (SLMs) designed for catalog context (the types of food or other items that work together) and semantic understanding.\u00a0<\/p>\n
In the case of catalog context, the SLM must be able to process multiple levels of details around the order itself as well as the different products. For instance, what products go together and what are their relevant replacements if the first choice isn't in stock? These substitutions are \u201cvery, very important\u201d for a company like Instacart, which Kundu said has \u201cover double digit cases\u201d where a product isn\u2019t available in a local market.\u00a0<\/p>\n
In terms of semantic understanding, say a shopper is looking to buy healthy snacks for children. The model needs to understand what a healthy snack is and what foods are appropriate for, and appeal to, an 8 year old, then identify relevant products. And, when those particular products aren\u2019t available in a given market, the model has to also find related subsets of products.\u00a0<\/p>\n
Then there\u2019s the logistical element. For example, a product like ice cream melts quickly, and frozen vegetables also don\u2019t fare well when left out in warmer temperatures. The model must have this context and calculate an acceptable deliverability time.\u00a0<\/p>\n
\u201cSo you have this intent understanding, you have this categorization, then you have this other portion about logistically, how do you do it?\u201d, Kundu noted. <\/p>\n
Like many other companies, Instacart is experimenting with AI agents, finding that a mix of agents works better than a \u201csingle monolith\u201d that does multiple different tasks. The Unix philosophy of a modular operating system with smaller, focused tools helps address different payment systems, for instance, that have varying failure modes, Kundu explained.\u00a0<\/p>\n
\u201cHaving to build all of that within a single environment was very unwieldy,\u201d he said. Further, agents on the back end talk to many third-party platforms, including point-of-sale (POS) and catalog systems. Naturally, not all of them behave the same way; some are more reliable than others, and they have different update intervals and feeds.\u00a0<\/p>\n
\u201cSo being able to handle all of those things, we've gone down this route of microagents rather than agents that are dominantly large in nature,\u201d said Kundu.\u00a0<\/p>\n
To manage agents, Instacart has integrated with OpenAI\u2019s model context protocol (MCP), which standardizes and simplifies the process of connecting AI models to different tools and data sources. <\/p>\n
The company also uses Google\u2019s Universal Commerce Protocol (UCP) open standard, which allows AI agents to directly interact with merchant systems.\u00a0<\/p>\n
However, Kundu's team still deals with challenges. As he noted, it's not about whether integration is possible, but how reliably those integrations behave and how well they're understood by users. Discovery can be difficult, not just in identifying available services, but understanding which ones are appropriate for which task.<\/p>\n
Instacart has had to implement MCP and UCP in \u201cvery different\u201d cases, and the biggest problems they\u2019ve run into are failure modes and latency, Kundu noted. \u201cThe response times and understandings of both of those services are very, very different I would say we spend probably two thirds of the time fixing those error cases.\u201d\u00a0<\/p>\n