When Semantic Embeddings Break: Why No Cosine Threshold Will Save You

By Amer Alsabbagh. Amer builds on the Intelligence team at zaimler, the unified context layer for agentic AI.

Everyone is pointing agents at a vector store of their enterprise data. Measured across four models and forty term pairs, the similarity score at the bottom of that stack can rank what is related but cannot decide what is the same, and agents live on decisions.

The short version

• A cosine score ranks what is related and cannot decide what is the same. On all-mpnet, invoice/bill scores 0.42 and buyer/seller scores 0.85.
• No threshold works. Across a 40-pair benchmark and four model families, at least 14 of the 16 true synonym pairs score below the highest-scoring converse pair, so a cutoff set just above that pair, which is where you would set it to avoid corrupting records, rejects nearly every true synonym.
• Stronger encoders, richer input and task instructions move the score without changing what it measures. The best embedding configuration reaches 0.703 AUC and still leaves 9 of 16 synonyms under the top converse pair.
• Completeness is a second wall. A 2025 DeepMind proof shows that at any fixed dimension some result sets cannot be represented at all under dot-product retrieval.
• The fix is a hand-off: embeddings for recall, a reader (a cross-encoder or a small generative judge) for the same-or-different decision, and the verdicts kept as edges in a structural layer over the data.

In short

The 2026 stack hides an identity decision inside similarity retrieval, and the scores interleave: one true synonym sits below the converse pair buyer/seller, the other a hair above it.

The default enterprise AI recipe in 2026 goes like this. Take everything you know about your data, the table names, the column names, the descriptions, the glossary entries, and embed all of that metadata into a vector store with an agent on top. A user asks a question, the agent retrieves the relevant metadata by similarity, then queries the actual systems using what it found. Every "chat with your data" product is a variant of this.

For the answer to come back right, something in that loop has to decide that client in the CRM and customer_ref in billing name the same thing, and that buyer and seller do not. Where that decision lands depends on whether anything downstream reads the shortlist.

Many pipelines have no reader in the loop at all: deduplication jobs, entity resolution, vector-store upserts that skip records already "known," agent memory writes. There the threshold is the verdict. Score above the bar and two names silently become one thing. Call that a merge.

The stack just described does have a reader. The LLM sees what comes back, and an LLM reading two names together can in fact tell buyer from seller. But it only reads what survives the similarity cut. Invoice sits at 0.42 from bill, so it never makes the shortlist, and the reader never learns what it never saw. The threshold decides by omission, and fetching the right metadata from the right tables fails before any reasoning starts.

Here is what the score at the bottom of that stack actually says.

All-mpnet, bare terms

cosine(Invoice, Bill) = 0.42

cosine(Buyer, Seller) = 0.85

cosine(Doctor, Physician) = 0.87

Three numbers from a production embedding model (all-mpnet, one of the most-downloaded sentence embedders on Hugging Face), each pair encoded as a bare term.

Invoice and bill are one document with two names, and under any reasonable target schema they belong together: 0.42. Doctor and physician are one person with two names: 0.87. The same relationship, at opposite ends of the range. Buyer and seller are opposite roles in one transaction and must never be treated as one thing: 0.85, above one true synonym and a hair below the other.

An agent that links buyer to seller does not look wrong. It produces clean-looking joins where every row is about the wrong party.

We first hit these three pairs unifying database schemas, deciding whether a column in one system means the same thing as a column in another. You meet the same wall anywhere a similarity score becomes a decision: deduplication, entity matching, merging retrieved chunks, catalog cleanup. Keep the three of them in view, because they follow us through the whole post.

This post is about why that wall exists, why a better embedder will not move it, and what actually does.

01 · Retrieval answers "related." Agents need "same, how, and all."

In short

Semantic retrieval is genuinely good at the job it was built for, fuzzy lookup with a human in the loop, but an agent needs identity, structure, and completeness, and a similarity score ranks rather than decides.

None of that makes the retrieval step useless, and the concession is real. For fuzzy lookup and document-grounded question answering, where a human reads the retrieved text and repairs the small errors on the way past, semantic retrieval is genuinely good. That is what it was built for and it earns its place in the stack.

An agent operating on enterprise data asks three questions that a relatedness ranking does not answer. Is this the same thing as that? Call that identity. How do these two relate, and in which direction? Structure. Is this all of them? Completeness.

A similarity score is a ranking signal. It was never built as a decision procedure. Agents chain decisions, so every silent same-or-related call compounds into the next one.

Here is the map. Identity gets measured, over the next four sections, because it is the wall people hit first. Completeness gets a theorem. Structure falls out of the diagnosis, and the fix falls out of which failures are recoverable and where.

02 · There is no threshold that works

In short

Across a 40-pair benchmark and four model families, the best separation any model reaches in either term-level condition is gte-modernbert's 0.574, and in every model and condition at least 14 of 16 synonym pairs fall below the top-scoring converse pair (14 to 16 depending on model and condition, 15 for the all-mpnet run in the figure), so a safe cutoff rejects nearly every true synonym.

The instinct is to reach for a cutoff. Merge above 0.8, say. Watch it fail on the three pairs we already have.

To catch Invoice/Bill you need the bar at or below 0.42. Buyer/Seller sits at 0.85, so that same bar merges the buyer into the seller. Push the bar above 0.85 to block Buyer/Seller and Invoice/Bill goes out with it. No setting gets all three right, because the should-merge pairs alone span 0.42 to 0.87, and the must-not-merge pair lands inside that span, just under the top.

Three pairs could be cherry-picked, so we built a benchmark anyone can run: 40 pairs, made of 16 synonym pairs, 16 converse-role pairs like buyer/seller and landlord/tenant, and 8 related-but-different pairs like invoice/receipt. Four embedding model families score all of them, with every term encoded twice, once bare and once inside a sentence frame.

Rank the synonym pairs against the converse pairs by cosine and, across all four models and both term-level conditions, the best separation anywhere is gte-modernbert's 0.574, on sentence frames. The per-model numbers are in the sweep table below. Full written descriptions on their eight-pair subset reach 0.750, and task instructions do better still and get their own section.

On bare terms two of the four sit below a coin flip, mpnet (2021) at 0.414 and bge (2023) at 0.434. Qwen3 (2025) is statistically at it, 0.508. The best, gte-modernbert (2025), manages 0.555. Chance is 0.5, and at this sample size differences of a few hundredths are noise, so trust the count ahead of the AUC (the area under the ROC curve, the probability that a random synonym pair outscores a random converse pair).

Below chance sounds like a hidden signal you could simply invert. It is not one. Flip mpnet's 0.414 and you get 0.586, and the merge gate you just built fires on the least similar pairs first, which is absurd on its face. Read it either way. There is no usable signal here.

The operative number is blunter than any AUC. In every model and every condition, at least 14 of the 16 synonym pairs score below the highest-scoring converse pair. The exact count runs from 14 to 16 depending on model and condition, all of it in the sweep table below; the figure above is all-mpnet on bare terms, where it is 15. So a threshold set just above the top converse pair, which is exactly where you would set it if you cared about not corrupting records, rejects nearly every true synonym you have.

One more thing about how the benchmark is built, because it cuts against us if you read it wrong. We built the converse class to be hard on purpose, because the hardest class is exactly what a merge gate has to survive. The related-but-different pairs are the easy negatives; their class mean is the lowest of the three everywhere. The converse roles are the hard ones. A system that validates its threshold on easy negatives, which is what a hand-assembled eval set usually contains, will overestimate its own safety exactly where a bad merge is most destructive.

The score misses in the harmless direction too. Shipper/consignee, a converse pair, comes in at 0.328, below invoice/bill. This is noise in both directions. A bias would point one way.

03 · Trained on company, asked about identity

In short

From Firth to Word2Vec to all-mpnet's 1.17 billion training pairs to the instruction-aware LLM embedders topping the 2026 leaderboards, the objective grades co-occurrence and never labels which relation made a pair co-occur, which is why converses read as near-synonyms, while the false low on Invoice/Bill turns out to be a polysemy artifact instead.

In 1957 the linguist J.R. Firth wrote the line that ended up running the field: "You shall know a word by the company it keeps." For decades that idea powered count-based methods; latent semantic analysis and its relatives built word vectors straight out of co-occurrence tables through the 1990s. Then in 2013, Word2Vec turned it into a neural training objective simple enough to state in one breath: give every word a vector, pull together the vectors of words that co-occur, push apart the vectors of random pairs. Train that on enough text and you get the demo everyone remembers, king minus man plus woman landing near queen.

Levy and Goldberg showed in 2014 that the objective is implicitly factorizing a matrix of co-occurrence statistics (shifted pointwise mutual information), which makes "cosine measures co-occurrence" closer to literal than metaphor. That result is proven for word2vec specifically, but it names the family's center of gravity.

Modern sentence embedders kept that recipe and scaled it. all-mpnet, the model behind our three numbers, was contrastively tuned on nearly 1.2 billion text pairs, 1,170,060,424 by its own model card, with in-batch negatives: pull the pair together, push the strangers apart.

It matters what those pairs are, and the split that follows is our categorization of the model card's own table: duplicate-question and paraphrase datasets counted as sameness, question-answer, title-body, citation, and comment-reply datasets counted as relatedness. About 93 percent of them are relatedness signals: a question with its answer, a title with its body, a citation with the paper it cites, a comment with its reply. Reddit comment-with-reply alone is 62 percent of the whole mixture. The other 7 percent genuinely are sameness signals, duplicate questions and paraphrases. So the model has seen sameness. What it never saw is a label marking which relation made a pair co-occur, so it blends sameness and relatedness into a single score. The score is doing its job accurately. The question it was trained on is the blurred one.

The current generation moved the same recipe onto bigger brains. As of mid-2026 the top of the embedding leaderboards is LLMs converted into embedders: the Qwen3-Embedding family on the open-source side, Gemini Embedding on the API side. All of them are trained contrastively on paired text, and all of them are instruction-aware, meaning you prepend a sentence telling the model what task the embedding is for. A bigger model and an instruction slot, with the same graded question underneath.

Why dissect all-mpnet rather than one of those? Its 1.17-billion-pair mixture is fully public, so the anatomy can be shown with receipts, and the newest models mostly do not publish theirs. We use all-mpnet for the anatomy and the 2025 generation, below, to show the anatomy has not changed.

Our cast walks straight into that objective. Buyer and Seller keep identical company: "the ___ signed the agreement," "the ___ agreed on a price," "the ___ backed out of the deal." Every sentence that holds one could hold the other, so the objective files them as near-neighbors and out comes 0.85. The model learned exactly what it was asked to learn.

The false low has a different and much more mundane cause. Bill is one token wearing four hats: an invoice, a piece of proposed legislation, a banknote, a duck's beak. Measured on the same model, bill sits 0.418 from invoice, 0.368 from legislation, and 0.337 from banknote. Three senses crowd into one vector and none of them wins.

If that crowding is what holds the pair down, then giving the token a financial context should release it. It does: put both terms in a sentence frame and invoice/bill jumps from 0.418 to 0.788. That jump is the evidence, and it makes the 0.42 largely a polysemy artifact. Hold that thought, because it is an artifact context can fix, which is not true of the false high, as the next section shows.

The field formalized this gap in 2015. SimLex-999 deliberately rates associated-but-opposed pairs as dissimilar, and embedding models have always scored far worse on it than on relatedness benchmarks. What our benchmark measures is that same gap, on the vocabulary enterprise schemas are actually made of.

One mechanism, every relation type. Synonyms, converses, siblings, hierarchy, and merely topical pairs all keep similar company, so co-occurrence training lifts all of them together and sorts none of them apart.

04 · The escape hatches, closed

In short

A stronger encoder lifts the synonym pair and lifts the converse pair with it; richer input rescues the false low while feeding the false high (buyer/seller reaches 0.940) and full written descriptions top out at 0.750 AUC; and giving an instruction-aware 2025 model our exact task in its own instruction slot is the biggest lever available without leaving the embedding paradigm, moving AUC from 0.508 to 0.703 while still leaving 9 of 16 synonyms under the top converse pair.

Four rebuttals arrive on schedule. Each is worth testing, so we tested all four.

Use a stronger encoder. We swept the cast across four families.

Table 1 — The sweep across four model families

The three cast pairs, bare terms

The whole benchmark, both term-level conditions

Table 1 notes — AUC ranks the 16 synonym pairs against the 16 converse pairs; 0.574 in bold is the best separation any model reaches in either term-level condition. The counts are how many of the 16 synonym pairs fall below the highest-scoring converse pair, which is where a safe cutoff would have to sit. Qwen3-Embedding is LLM-based and instruction-aware; it runs here without instructions, and the instructed run is below.

The newer models do lift the synonym pair, and gte-modernbert takes Invoice/Bill from 0.42 to 0.71. In every one of them the converse pair still rides above one of the two true synonyms. The ordering pathology is family-wide.

Give the model richer input. This is what "add metadata to your chunks" amounts to. Put every term in a sentence frame and the polysemy artifact melts: on all-mpnet, Invoice/Bill goes from 0.418 to 0.788. Buyer/Seller, same model and same frame, goes to 0.940. Context rescues the false low and feeds the false high, which is precisely the wrong trade. On the full benchmark the frame compresses everything upward: bge's synonym class mean lands at 0.923 and its converse class mean at 0.926, three thousandths apart.

So try harder at it. Attach full descriptions to every term instead, written independently in two registers, one dictionary and one database documentation, so that the two sides of a pair never mirror each other's phrasing. That run covers an eight-pair subset, four synonym pairs and four converse pairs, with the full texts in the published harness. The failure changes shape and does not close. The best model-and-register cell reaches only 0.750 AUC. One model inverts outright. And in every cell, at least one of the four synonym pairs still lands under the top converse pair.

The pairs that hurt most are the ones enterprise schemas are full of: debtor/creditor at 0.957, lessor/lessee at 0.951, payer/payee at 0.955, each from at least one major model. Morphological converses, the ones built by swapping a suffix, read as near-duplicates to every encoder we tested.

Tell the model the task. The newest embedders accept instructions, so we gave one exactly our question, in its own format. The model is Qwen3-Embedding (2025, the 0.6B sibling of the family that tops the open-source leaderboard). The instruction string is Given a term from a database schema, retrieve terms that denote exactly the same real-world concept, prepended to both sides of every pair.

Qwen3-Embedding-0.6B, synonym vs converse AUC

bare terms 0.508

task instruction 0.703

query/document mode 0.605

instruction + frame 0.664

This is the biggest improvement available without leaving the embedding paradigm. AUC on synonym versus converse rises from 0.508 uninstructed to 0.703 instructed. The instruction slot is real.

And it is still nowhere near a decision. Nine of the 16 synonym pairs still score below the top converse pair, debtor/creditor at 0.874. Buyer/seller at 0.793 still outscores invoice/bill at 0.746. A threshold set to block every converse pair keeps only 7 of the 16 true synonyms. Two more measured notes: the asymmetric query-versus-document mode these models ship for retrieval does worse here, 0.605, with all 16 synonyms below the top converse pair; and stacking the sentence frame on top of the instruction does not stack the gains, 0.664.

Each generation moves the score. None of them has changed what the score is. An instruction names the task at inference time, but a name is not a training signal, and the geometry underneath was still graded on relatedness.

Use more dimensions. That one gets its own section, because it has its own theorem.

05 · What is actually broken (and what is not)

In short

One number is enough for a decision in principle, and specialization methods proved cosine geometry can be taught synonymy, so what stays broken is narrower and more specific: the quantity being estimated, the single global geometry, the pre-committed vector, and an interface that drops relation type and direction.

Before the list, two concessions. Both are easy to get wrong, and both are load-bearing.

First, one number is enough. In principle a single scalar is plenty for a binary decision, and every classifier ever shipped ends in a threshold on a score. The count of numbers is beside the point. What settles it is what the number was trained to estimate.

Second, embeddings can be taught this distinction. A specialization literature did it a decade ago: counter-fitting, and then ATTRACT-REPEL, injected synonym and antonym constraints directly into word vectors and roughly doubled performance on genuine-similarity benchmarks. Cosine geometry can hold synonyms close and opposites far. Something has to train it to.

So here is what actually stays broken for a generic embedding pipeline.

The score estimates the wrong quantity. Off-the-shelf encoders are tuned on a 93-to-7 blend of relatedness and sameness, and nothing anywhere in a retrieval stack retrains them for your merge decision. You inherit the blend, whole. The instruction slot on the newest models lets you name your task at inference time, and naming it helps, as measured above, but the verdict the score renders is still a relatedness verdict with the volume adjusted.

One vector per term means one global geometry. Whether doctor and physician are the same thing depends on the target schema. One schema files dentists under doctor and the next does not; an order and a purchase order can be one workflow object or two. A fixed embedding freezes a single answer and hands it to every context, and the escape hatches above already showed that stuffing the context into the input does not rescue the hard cases.

A bi-encoder must commit before the comparison. Each term is squeezed into its vector before the model knows what it will be measured against. "A buyer is the counterparty of a seller in the same transaction" is pair-conditional reasoning, and pair-conditional reasoning is precisely what a pre-committed vector cannot do and precisely what a model reading both terms together can.

The interface discards relation type and direction. cosine(x, y) equals cosine(y, x) by construction, and exactly one scalar per pair survives the comparison. Whatever the vectors internally encode about hierarchy, or about who pays whom, the score has no channel to say it. Converses do not merge because of this: a symmetric score could perfectly well hold buyer and seller far apart, and sameness is itself a symmetric question. What the interface costs is how: even a perfect similarity, reduced to one scalar, cannot tell you how two things relate or in which direction, which is the part an agent acts on.

The labels-and-policy bill comes due wherever the decision is made. Fine-tune a bi-encoder on equivalence labels with converse pairs as hard negatives and it will separate buyer from seller: you have built a decision model whose output happens to be read through cosine. Prompt or train a reader with the same policy and you have built a decision model that states its verdict in words. Either way, the deciding is done by the labels, the policy, and a model shaped to the question, and the off-the-shelf similarity score carries none of it.

The practical difference is what you can do afterwards. The reader sees both terms together and hands you a verdict you can audit. The geometry hides the policy inside coordinates.

A similarity score means exactly what its training loss graded, and nothing more.

06 · The completeness wall

In short

A 2025 DeepMind result proves that at any fixed dimension some result sets are impossible to represent under dot-product retrieval, measured from dimension 4 through 45 with free vectors optimized on the answer sheet; a schema-only store of a few thousand entries sits below that wall today, but content-level retrieval and multi-condition agent queries cross it, and there 95 percent recall is a wrong answer that looks right.

Identity is the failure you hit first. Completeness is the one you cannot engineer around.

Retrieval is geometry. Documents are points, the query is a point, you return the nearest few. In 2025, researchers at Google DeepMind proved that for any fixed embedding dimension there are result sets that no arrangement of points can produce under dot-product retrieval. Those sets are impossible to represent, which is a stronger claim than hard to learn. The proof is straight math about which top-k patterns a d-dimensional space can express, with no margin tricks and no approximation caveats to argue with.

The experiment underneath the theorem is what makes it hard to wave off. The authors drop language entirely and optimize raw vectors directly against the answer sheet, the best case that could exist for an embedding. The wall shows up at every dimension they ran, 4 through 45, and at 45 dimensions it arrives by 626 documents.

Extrapolating outward with their cubic fit over the full 4-to-45 sweep (r squared 0.999) puts it at roughly 500 thousand documents at 512 dimensions, 4 million at 1024, and 250 million at 4096. The measured region shows the shape; those three numbers are that curve read outward, and should be read as extrapolation.

The sharper fact is measured, not extrapolated. In a separate experiment on their small test collection, a real embedding model, trained on the test set, at 64 dimensions, still could not solve a task that free vectors solve at 12. Real models come in several times more limited than the theoretical best, because they have to model language and generalize instead of memorizing an answer sheet.

The paper aims this squarely at instruction-following retrieval and search agents, where a query carries multiple conditions and any subset of the collection might be the right answer. As agents push embeddings toward "any query, any definition of relevance," the combinatorics outrun the dimension. Their recommendation is ours too: cross-encoders or multi-vector setups for the queries that need them.

So where does that leave you? If your vector store holds only schema metadata, a few thousand entries, you are likely below these walls today. The limit becomes yours the moment the store holds the content itself, every ticket, every chunk, every log line, or the moment your agent starts issuing the multi-condition queries just described.

At that level the questions are set-shaped. "All invoices unpaid past 90 days, across both systems" is a set, and a set is either complete or it is wrong. For a human reading an answer, 95 percent recall is a good day. For a reconciliation, it is a wrong answer that looks right. Similarity retrieval has no knob for completeness, at any setting. Past a certain scale, completeness leaves the menu entirely.

07 · The fix: retrieve, then decide, then keep the decision

In short

Both remedies leave the embedding contract for inference-time reading: a cross-encoder lifts AUC to 0.820 while still scoring debtor/creditor 0.998, and a 4B generative judge asked in words never merges a single converse or related pair, 24 of 24, which is the hand-off to make once, offline, and persist as edges.

Everything above stayed inside the embedding contract: compress each term into a vector ahead of time, then make every later decision a distance between precomputed vectors. That contract is what makes embeddings cheap, and it is also what caps them. No embedding configuration we tested produced anything close to decision-grade separation. To do better you have to break the contract, and the moment you break it you have left embedding retrieval for inference-time reading.

There are two ways to break it, and we measured both on the same 40 pairs.

Table 2 — Two ways out of the embedding contract, same 40 pairs

Table 2 notes — The reranker and the judges read both terms at inference time, which is the boundary: nothing is precomputed and nothing is a distance. AUC figures rank the 16 synonym pairs against the 16 converse pairs, as everywhere else here; the judge rows are verdict counts, because a judge returns a decision rather than a score.

Break precomputation, keep the score. bge-reranker-large is a cross-encoder trained on relevance, and it reads each pair jointly instead of comparing two finished vectors. AUC jumps to 0.820. Set against the 0.434 that its own family's embedder scores on the same pairs, that is the largest jump anywhere in this article. And it still scores debtor/creditor and payer/payee at 0.998, because it faithfully answers the question it was trained on. Joint reading changes the failure class. The training question still decides what the number means.

Break the score, ask in words. Qwen3-4B is a small generative model, and zero-shot we simply asked it whether each pair names the same concept. It never merges a single converse or related pair, 24 of 24 on the side where mistakes are catastrophic, while rejecting 12 of 16 true synonyms as not the same thing. Reframe the prompt with the merge policy spelled out, merge columns that store the same kind of business information and keep opposite roles separate, and it accepts all 16 synonyms while wrongly merging 4 boundary pairs: lessor/lessee, host/guest, invoice/receipt, budget/expense. One floor note: a 1.7B judge lands near chance at 21 of 40. Verdict-quality reading has a capability floor.

Neither zero-shot prompt gets both sides at once. But look at what the errors are. They are policy-boundary cases, and two of the four are still converse pairs, so the failure is reduced and not eliminated. What changed is its shape: the embeddings failed on converses systematically, and this judge fails on four named boundary pairs. Getting precision and recall together is exactly the work of writing the policy down and training or few-shotting the judge with labels, which is the decision model this section prescribes.

This is a hand-off between machines. Nothing here asks for a better model. An embedding gives you precomputed geometry: what is nearby, millions of comparisons per second. A reader gives you a verdict: are these the same, under this policy, one pair at a time, at generative-inference prices. The default stack's mistake is asking the first machine the second machine's question.

The cost is the catch. Joint attention means nothing precomputes; every pair is a fresh forward pass, and a million documents against each other is half a trillion of them. A pair-reader judges a shortlist, and retrieval still has to build that shortlist. Embeddings for recall, a reader for the decision: each component doing what its training actually taught it, each failure mode contained by the other. This division of labor is already how serious search stacks work today, retrieving with embeddings and reranking the shortlist with a cross-encoder, and rerankers are standard products for exactly this reason; the argument here extends the same pattern from relevance to identity.

For enterprise data, go one step further and stop re-deriving the same decisions on every query. The verdicts a pair-reader emits, same, different, one contains the other, are edges. An edge is a different kind of object from a score: you can store it, traverse it, and audit who set it. Persist them and you have a structural layer sitting over your data: which columns are the same thing, which entities contain which, which roles must never merge. Make those decisions once, offline, where you can afford the reader and an audit trail, and let agents operate on the structure, keeping semantic retrieval for the fuzzy edges it is genuinely good at.

This is why data teams keep reinventing ontologies. What they rebuild each time is the place where identity decisions live once somebody finally makes them properly.

A similarity score says two things are related. It cannot say they are the same. Nothing in its training ever graded it for that.

08 · Run it yourself

In short

The 40-pair benchmark runs in a few minutes on a laptop across four open models and now carries the instruction-conditioned run, the reranker control, and the judge prompts, with the larger siblings and API embedders one edit away, and it doubles as the test you hand any vendor claiming their embedder fixes this.

The benchmark is 40 pairs and four open models, and it runs in a few minutes on a laptop: 16 synonym pairs, 16 converse pairs, 8 related-but-different pairs, exact AUC with ties counted at 0.5. It includes the instruction-conditioned variant, the reranker control, and the judge prompts, all scored on the same 40 pairs, and the script runs the 0.6B instruction-tuned model on a laptop.

The 4B and 8B siblings and the API embedders drop in with one edit. If any of them turns 0.703 into 0.95 at high precision on hard converse pairs, that is worth publishing. We measured what fits on a laptop and published the harness.

Swap in the converse pairs from your own domain, because every domain has them. Then run the same 40 pairs the next time a vendor tells you their embedder fixes this.

References

• Weller, Boratko, Naim and Lee, "On the Theoretical Limitations of Embedding-Based Retrieval" (arXiv 2508.21038, ICLR 2026).
• Levy and Goldberg, "Neural Word Embedding as Implicit Matrix Factorization" (NeurIPS 2014).
• Hill, Reichart and Korhonen, "SimLex-999" (2015).
• Mrksic et al., "Counter-fitting Word Vectors to Linguistic Constraints" (NAACL 2016) and "Semantic Specialization of Distributional Word Vector Spaces using Monolingual and Cross-lingual Constraints" (ATTRACT-REPEL, TACL 2017), cited together as the specialization literature.
• Mikolov et al., "Distributed Representations of Words and Phrases" (2013).
• Zhang et al., "Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models" (arXiv 2506.05176).
• Firth, "A Synopsis of Linguistic Theory" (1957).

Steck, Ekanadham and Kallus, "Is Cosine-Similarity of Embeddings Really About Similarity?" (arXiv 2403.05440) proves that for a class of linear factorization models the cosine of learned embeddings is arbitrary; modern text embedders train against cosine directly, which is the paper's own first remedy, so we do not lean on that theorem here, but it is worth reading as the sharpest statement that a similarity score means only what its loss graded.