The lineup problem: an opportunity for late interaction in recsys
A guy from Boston, now living in DC, walks into a bar and gets talking basketball with the bartender, who before long congratulates him on the Knicks’ first title in 53 years. Safe to say this Celtics fan doesn’t take it well. Not the reaction you want from your customer, is it?
The bartender’s mistake was averaging the only two things he knew — where the guy’s from (Boston) and where he’s now (DC) — into the city halfway between them (New York). Representing a complex thing with a single embedding vector does something similar: it squeezes many details into one mean point. And yet, to keep candidate retrieval scalable, many recommender systems lean on exactly this: the popular two-tower model encodes the user as one vector and each item as another, then retrieves the items nearest the user vector. With the explosion of LLMs, we can even skip the learned vectors and just describe a user’s preferences in text, the way we’d describe an item (for example, "A Celtics fan who never misses a Jokić game"), then embed that. Either way, users and items are each represented by a single vector: fast and scalable, but not without limits.
There’s an alternative, one that’s been brewing in the IR/search community for a few years now and is gaining real traction. In 2020, the ColBERT paper introduced late interaction: instead of one vector per item, you keep one vector per token, and score a query against a document by letting every query token find its most similar token in that document (hence MaxSim). Token-level matching is much more precise. I haven’t seen it used in recsys, though (the closest is probably multi-interest retrieval, which gives the user several vectors but still typically keeps one per item, not one per token). The likely reason is cost 💲: many more vectors to store, and a scoring function that was assumed to be prohibitively expensive for retrieval. But this is changing: recent work, including MUVERA, TACHIOM, and our SMVE algorithm, has made late interaction viable for first-stage retrieval at scale. Which is why I think it’s worth a look:
What can late interaction actually do for a recsys task, on real data?
To find out, I turned to the NBA. If you’re running a sports-streaming platform, the stakes are concrete. Miss a fan’s team and they churn. But this isn’t really about sports. The same failures could show up in any recommender whose items are rich in attributes and whose users want something specific (products, listings, profiles; most of recsys, in other words). The NBA just makes them easy to illustrate (and the finals are still fresh, so it feels timely). So I set up three simple situations a streaming-platform recommender could run into. In one, the dense vector breaks down but keyword search holds; in another, keyword search breaks but the dense vector holds; in the third, both break. But late interaction stays robust in all three, and not by chance.
The setup
I took every game on ESPN’s 2025/26 NBA schedule (1,235, a bit above the 1,230-game regular season since the feed includes the in-season NBA Cup) and turned it into a small retrieval dataset. Each item is a game, described by the team names and their rosters:
Oklahoma City Thunder @ Los Angeles Lakers. Lineups -- Oklahoma City Thunder:
Alex Caruso, Luguentz Dort, Shai Gilgeous-Alexander, ...; Los Angeles Lakers:
Deandre Ayton, Luka Doncic, LeBron James, ...The example item just lists the two teams and their lineups. Simple.
Each query is a simple user profile*, expressed as a short text. That keeps it easy to audit and lets us define relevance with fixed rules, no model in the loop. There are three kinds:
"fan of {player}": relevant games are the ones that player plays in."lives in {state}": relevant games are those involving a team from that state."lives in {state}, fan of {star}": a game is relevant only when the star’s team faces a team from that state.
*Not a claim that this is a smart way to model user profiles; it’s deliberately minimal and transparent. The point is only that profiles 1 and 2 hinge on a single attribute, while profile 3 needs two matched at once.
It’s a deliberately small and plain dataset: 1,235 items, each just a couple of rosters, about as structurally simple as a catalog gets. And it’s still enough to surface systematic failures in the two standard retrieval methods, keyword search and a single dense vector.
I then indexed those items three ways, each with a different off-the-shelf textual retriever:
- BM25, the staple of keyword search.
Qwen/Qwen3-Embedding-0.6B, a popular dense-retrieval baseline: one vector per query and item, scored with cosine similarity.topk-io/Iso-ModernColBERT, a 150M-parameter ColBERT-style model that produces one vector for each query/document token, scored with MaxSim.
I used TopK to ingest and query all three representations, one index each, so the only thing that changes is how each item is represented: keyword_index for BM25 (no model needed), vector_index for the Qwen3 vectors, and multi_vector_index (MaxSim) for Iso-ModernColBERT’s per-token vectors. The embedding models run locally; TopK stores and searches what they produce. TopK also has a managed semantic_index that runs Iso-ModernColBERT and handles the embedding for you, but since the dense Qwen3 model runs locally anyway (TopK doesn’t offer a managed single-vector embedder, only multi-vector), I ran Iso-ModernColBERT locally too and used the multi_vector_index directly, keeping the comparison apples-to-apples.
Each profile type runs over many instances: one query per player (sampled across all teams) for "fan of {player}"; one per home state for "lives in {state}"; and one per pairing of a home state with one of last season’s top-10 scorers for the compound. I report two numbers. MRR (mean reciprocal rank) is the average of 1/(rank of the first relevant game); 1.0 means the right game is always first, 0.5 means second on average. Recall@R asks the complementary question: of the R relevant games, how many land in the top R? It catches a method that surfaces a good game but misses the rest.
Profile 1 — “fan of Anthony Edwards”
Find a player’s games. Each of his ~82 games literally contains his name, so this should be easy, and for two of the three it is: BM25 0.99, late interaction 1.00.
The single dense vector gets 0.30. Averaged across players, it surfaces only ~17% of a given player’s games.
A player’s name is one of ~30 in the item, averaged into a single point, and a pooled vector has only so much room: most of that detail blurs together, and a given player’s games are hard to surface. BM25 and late interaction don’t have that problem; a token is a token, however crowded the item.
Profile 2 — “lives in California”
The bartender’s task, basically: you know roughly where someone is (often just their state), so surface their local teams. The catch: there’s no “California” among the team names. Only Los Angeles (the Lakers and Clippers), Golden State (the Warriors), and Sacramento (the Kings).
BM25 can only match a state that’s also a team name (New York, Indiana, Oklahoma); for everyone else, California included, it has nothing to grab. Across all states that nets 0.37 MRR and 0.20 recall, blind to most of the map. Dense does better by MRR (0.85), because “California” is a broad concept and one local game floats up; but by recall it finds only about half. Late interaction matches California → Los Angeles, Golden State, Sacramento at the token level: a perfect 1.00 MRR, and by far the best recall.
Profile 3 — the stress test
"lives in California, fan of Shai Gilgeous-Alexander"
An ordinary profile: a state plus a favorite player. To serve it you need a semantic match (“California” is in no title; the teams sit in Los Angeles, Golden State, and Sacramento) and a conjunction (those teams hosting Gilgeous-Alexander’s Thunder). Here’s the real top-3 from each method (✓ = the Thunder playing a California team):
| retriever | its top 3 games |
|---|---|
| dense | Mavericks @ Lakers · Mavericks @ Lakers · Lakers @ Grizzlies |
| BM25 | Pelicans @ Mavericks · Thunder @ Mavericks · Thunder @ Celtics |
| late interaction | Thunder @ Lakers ✓ · Thunder @ Lakers ✓ · Thunder @ Kings ✓ |
Dense can match by meaning but lacks the capacity to remember every name; BM25 matches the name but not the meaning; late interaction gets both.
The two baselines fail in opposite ways. Dense nails the place (every result is a Lakers game) but loses the player; it matched “California” and forgot Gilgeous-Alexander. BM25 nails the player (the Thunder, plus a stray game matching “Alexander”) but never California, because there’s no such token to match. Only late interaction holds both at once: Gilgeous-Alexander on Oklahoma City’s roster and California on the Lakers and Kings. In aggregate over every state-and-star pairing: dense 0.20 MRR (0.10 recall), BM25 0.16 (0.06), late interaction 1.00 (0.94), the only method that doesn’t break when a profile needs two clues at once.
Why this happens
This isn’t a quirk of one model; it’s a ceiling. A result at ICLR 2026 shows that how many distinct relevance patterns a d-dimensional vector can represent is capped by d itself, a limit no amount of training removes. On a stress test built to expose it, BM25 scores 93.6% recall@100 while an 8-billion-parameter single-vector model manages 4.8% (Table 5 in the paper): lexical, high-dimensional search sidesteps the ceiling that dense pooling slams into. A recent follow-up proof goes further, constructing multi-vector similarities that no single vector can even approximate without exponentially more dimensions.
Our setup is exactly that pressure: ~30 roster names crammed into one pooled vector, and a compound profile is just the kind of conjunction a single point provably can’t hold. Late interaction keeps a vector per token, so nothing gets averaged away and each clue keeps its own evidence.
This is not about basketball
I picked the NBA because I’m a fan and it’s the first example I could write a story about. But the shape is common, and it’s probably sitting in your catalog, too. Two questions to ask yourself: do my items pack many things into their description? Do my users’ interests read as conjunctions or descriptions? If both are yes, you might have the lineup problem.
- Job listings. Skills, tools, seniority, location, comp, all in one blob. The candidate is “backend, Berlin, Rust, payments”: a conjunction with a geographic clause and no guaranteed literal token.
- Travel. A rental is a bunch of amenities, a neighborhood, a view, a vibe. The guest wants “quiet, near the old town, with a workspace and a balcony.”
- Streaming. Cast, setting, era, and themes all live in the synopsis; the mood you’re in is “slow-burn thriller, Scandinavia, that actor from the heist movie.”
- E-commerce. The product page lists every attribute, and you’re shopping for “a lightweight two-person tent that packs small and handles wind.”
Recruiting, real estate, scientific-paper search: same problem. The moment an item is rich and an interest is “A and B that feels like C,” a single pooled vector blurs and pure keyword search misses whatever isn’t written literally. Late interaction, by contrast, gives every token its own point. And it isn’t limited to text: it works in any modality where you can produce tokens, for example images (ColPali). Recsys looks like a particularly good fit, since both user profiles and item data tend to be complex and many-sided. It’s worth a try.