Someone types "winter gloves heated" into Agora.
It is a short query, but answering it well takes several decisions. Does this person want battery-heated gloves, or just warm ones? Which of the thousands of glove listings across our merchants actually have a heating element? Which ones are replacement batteries that happen to say "gloves" in the title? And of the real matches, which ones should come first?
Agora searches millions of listings from thousands of independent stores. Every merchant writes titles differently, and shoppers write queries differently again. We could not hand-tune our way to good results across all of that. So we built a search system that learns from its own traffic.
Once a day, it takes the last 24 hours of searches, compares what it returned with what Amazon returned for the same queries, works out which stage of the pipeline fell short, and trains new versions of the models that interpret queries and rank products. Tomorrow's searches run on the new models and produce the data for the next round.
This post explains how that loop works, what we mean (and do not mean) when we call it reinforcement learning, and why we think it compounds as more stores and shoppers join.
The serving path
A search goes through four stages before anything reaches the shopper:
Planning. A fine-tuned Qwen 2.5 model turns the shopper's words into a structured search plan.
Retrieval. Typesense pulls 100 candidates using keyword and vector search together.
Reranking. A cross-encoder scores every candidate against the request and keeps the best.
Checks. Deterministic code verifies the models' output against data we own.
The models in this path are self-hosted and served through Ollama. The one exception is our deep research mode, used for gift ideas and use-case questions, which calls Claude to write longer summaries and buying guides. Even there, the candidates come from the same retrieval and ranking pipeline.
Self-hosting matters for the loop. We can retrain the models every day, the search logs never leave our infrastructure, and the cost of a search does not depend on someone else's API pricing.
Planning
The planner reads the query and produces something a search engine can run. For the gloves example, the plan looks roughly like this (simplified):
{
"resolvedQuery": "heated gloves",
"embeddingQuery": "heated winter gloves battery powered",
"searchTypes": ["product_type"],
"filters": { "categories": ["gloves"] },
"identifiers": [],
"sort": null
}There are two queries, not one. resolvedQuery goes to keyword search and should only contain words that are likely to appear in a product title. embeddingQuery goes to vector search and can carry context that titles never state.
This split matters more than it looks. If a shopper asks for sandals that go with a tan dress, "tan dress" belongs in the semantic query and the reranker, not the keyword query. Otherwise keyword search will happily return dresses.
The search type also matters. It sets how much retrieval leans on keywords versus vectors. Exact-product and compatibility queries ("iPhone 15 case") lean lexical. Gift and use-case queries lean semantic. Generic product types sit in between.
Retrieval
Our index runs on Typesense across ten shards. Each listing has a 256-dimensional embedding from Qwen3-VL, built from its image and text, so a text query and a product photo live in the same vector space.
Retrieval runs keyword and vector search in one hybrid query. Each shard returns 20 hits, and they merge into a pool of 100. Hard constraints like price, category and rating are applied as filters in Typesense, not left to a model to remember. If the shopper named a store or brand, a second search runs against that catalog alone, so the store's products reach the pool even when their titles do not repeat the store's name.
Reranking
The reranker is a cross-encoder based on Qwen3-Reranker-4B. The embedding model encodes the query and each product separately. A cross-encoder reads them together and returns a relevance score between 0 and 1. That is slower, but much better at the distinctions shopping depends on: a heated glove versus an insulated one described as "extra warm," or the glove versus its replacement battery.
Candidates scoring below 0.4 are dropped. If the shopper named a store or brand, its listings move up, but only when they score within 0.15 of the best result on the page. A wrong item from the right store should not outrank a strong match from somewhere else.
If fewer than three candidates survive, the planner gets one more attempt. It rewrites the plan using the failed candidates as evidence, retrieval runs again, and the stronger of the two pools wins.
On staging, retrieval takes about 150 ms and reranking about 1.5 seconds. The language model calls around them take longer, and a full conversational turn currently takes several seconds. That is the main cost of this design, and we are working on it.
Models propose, data decides
Small self-hosted models make mistakes. We decided early not to patch those mistakes with phrase lists or regexes over the shopper's words. Rules like that grow forever and break quietly. Instead, we check each model output against data the system already owns.
A few examples:
Invented constraints. A price filter is accepted only with evidence: a number in the message, a remembered budget, or the limit of the search being refined. Otherwise the planner made it up, so we drop it and log it.
Misread products. If the planner decides the shopper named a product type, we check that word against the catalog. In product titles, the type usually comes last ("Baffin Tundra Boots") and brands and modifiers come first. A word counts as a product type only if it is the head noun in at least 30% of the titles that contain it. "Boots" passes. "Fine," from the typo "fine me coats," does not.
Missing products. If the shopper named a product and none of the returned listings is that product, we search for it directly and keep whichever page has it.
These checks protect the shopper. They also protect the training data. A system that learns from its own logs will learn its own mistakes, unless something catches them first.
The feedback loop
Serving produces logs. Learning turns logs into new models.
For each search, we record the query, the plan, the pool of 100 candidates, and the ranked page. Then we run the same query on Amazon and store its top results next to ours. Amazon's products are embedded into the same vector space as our catalog, so we can compare the two result sets even when they share no listings.
From that comparison we compute a few signals:
Coverage: how many of the benchmark's products, or close equivalents, appear in our results.
Visual overlap: how many of our results sit near the benchmark's results in embedding space.
Title overlap: how much the two result sets agree lexically.
Missed lookalikes: products that were in our pool, closely matched the benchmark, and were ranked out anyway.
Each signal is computed twice: once on the final page and once on the whole pool. That is the most useful part of the record, because it tells us which stage failed.
What the record shows | Where it failed | What fixes it |
|---|---|---|
Good matches in the pool, not on the page | Reranker | Reranker training |
No good matches in the pool, but comparable products in our index | Planner or retrieval | Planner training |
Nothing comparable in our index at all | Catalog | More stores |
A concrete example. In September we pulled a set of 65 searches where Amazon's results looked better than ours. Across that set, our candidate pools averaged 19.5% visual overlap with Amazon's results. Our final pages averaged 6.2%. In 48 of the 65 searches, the pool scored better than the page. The right kind of products were there, and the reranker was not choosing them. That is a ranking problem, and it is the cheapest kind to fix.
To be clear, that set was chosen because it failed. Those numbers say nothing about our average quality. They show where the headroom was.
Daily training
Once a day, the last 24 hours of logs become training data for new versions of the planner and the reranker.
Retraining on logs has a subtle failure mode. A model retrained only on old logs will forget any output field the old logs did not contain. When we add a new field to the planner, such as a shopper's standing requirements or a relative price ("cheaper than that one"), the training rows need to be labeled with it, or the next model quietly unlearns it. That labeling work is in progress.
The harder question is not whether we can produce a new model every day. It is whether each one is better than the last.
Is this reinforcement learning?
Someone will ask, so we should answer directly.
In the textbook sense, reinforcement learning means an agent takes actions, the environment returns a reward, and the agent updates its policy to earn more reward. Our loop has that shape. The planner and reranker are the policy. Each search is an action. The comparison with the benchmark, and the diagnosis of which stage failed, is the reward signal. Each day the policy updates, and the new policy generates the next day's data.
What it is not, today, is online RL. Models do not update during a session. We do not explore by deliberately showing worse results to see what happens. The update step itself looks a lot like supervised fine-tuning and preference training on logged data. A more precise description would be batch, off-policy learning from a weak external reward.
That comes with known problems. Logs only show outcomes for what the previous model chose, not for everything it could have chosen. And a reward based on another marketplace can, at best, teach us to look like that marketplace.
We call it reinforcement learning because the loop is the important part: the system's own behavior produces the signal that improves it, every day, without anyone writing new rules. But we would rather be precise about the mechanism than borrow a term that suggests more.
Why Amazon is only the first teacher
It may seem odd for us to benchmark against Amazon. In one test of 50 product searches, Consumers' Checkbook found that 54% of the first 25 Amazon results were paid placements. We are not trying to copy that.
But Amazon has something a young search engine lacks: years of shopper behavior across an enormous range of queries. Its results contain a lot of learned knowledge about what people mean when they type "winter gloves heated." That makes it a useful reference for query understanding, as long as we remember what else is in it.
So we treat it as evidence, not ground truth. Disagreeing with Amazon is not automatically a mistake. Its catalog is not ours. Its ranking reflects its own incentives. Two different products can meet the same need equally well. What we learn from a comparison is not "rank like Amazon." It is "our reranker threw away a heated glove that was sitting in the pool."
The longer-term plan is to replace this teacher with a better one. The bottom of our diagram says "future user actions" for a reason. Today the loop learns mostly from the benchmark. As usage grows, it will learn more from what shoppers actually do: what they open, compare, buy, and return. Those signals measure our goal directly instead of through someone else's results.
Why it compounds
Most search systems get better when engineers make them better. This one is built to get better as it is used. There are three reasons.
More stores mean better candidates. A catalog gap is the one failure no model can fix. Every new merchant adds products that can enter the pool, and turns some of yesterday's catalog gaps into ranking problems the loop can solve.
More searches mean better coverage of intent. Each day's training data is made of the queries people actually typed. The long tail of shopping queries is very long, and the only way to cover it is to see it.
More shoppers mean better rewards. A benchmark comparison is a proxy. Shopper behavior is the real thing. The more of it we have, the less we depend on anyone else's idea of relevance.
There is a fourth reason, and it is the one we care about most. Agora takes no money for ranking and sells no products of its own. So the reward our system learns from has one term in it: did the shopper find the right product. A marketplace that earns from ads, or from its own private-label goods, has other terms in its objective, whether it means to or not. Ours does not have them to learn.
This is why we believe Agora can become the strongest e-commerce search in the world. Not because any one of our models is better than everyone else's, but because the loop gets stronger with every store and every shopper, and it points at only one thing. That is a goal, not a result. A loop that compounds can compound its mistakes too, and the only protection is measurement we take seriously.
What we have not solved
We are early, and there is a lot left to do.
Imitation ceiling. Until shopper behavior carries more of the reward, a benchmark-driven loop can only get us close to the benchmark. Getting past it requires our own signal.
Off-policy bias. We only see outcomes for results we chose to show. Correcting for that properly is ongoing work.
Broad queries. For "Nike shoes," we and Amazon often make different assumptions about whether the shopper means running, lifestyle or basketball. Neither is obviously right, and we need an explicit policy for ambiguous intent rather than inheriting one.
Multi-category requests. "An outfit for a wedding" still becomes a single query.
Latency. Several seconds per conversational turn is too slow. Streaming the response will help, and the response format is already built for it.
We will write about each of these as we make progress, including when an approach does not work.
Closing
The first version of Agora was a single search box over millions of Shopify products, built in a weekend. It was useful, but it was only as good as the day it shipped.
What we have built since is different in one important way. Every search now teaches the next one. The models that understand queries and rank products are retrained every day, on our own infrastructure, from what happened the day before. As more stores join and more people shop, the system has more to learn from.
We think that is how search should work when its only job is to find the best product for the person asking. We will keep sharing what we learn here, including the parts that do not work yet.
