Negation in RAG, Measured: Most Rerankers Do Worse Than a Coin Flip

I gave 13 local retrieval models a simple job. The query says the backup did not complete. Two documents are on the table: one says the backup failed, the other says it completed. Which one goes first? When the correct document used different words than the query, the average embedding model picked the wrong one 72% of the time. The average reranker did slightly worse, at 74%. Gemma 4, running on a laptop, got all 80 decisions right.

I ran this because my own notes were wrong. I am building a fully local RAG pipeline over a mixed Turkish and English archive of technical notes, and an earlier test had convinced me that negation was an embedding problem and a reranker would fix it. I wrote that down as a requirement. Then I measured it as a ranking problem, which is what retrieval actually does, and the reranker did not fix it.

This article covers the test design, the results for 14 models in two languages, a control run that shows what is really going on, where an LLM fixes it and what that costs on a 32 GB Mac, and a small script so you can run the same test against your own models. The test set is published. None of it comes from my private notes.

Scope: local models only, run through Ollama 0.34.0 and sentence-transformers on an Apple M5 with 32 GB of unified memory. 20 hand-written triplets per language, each tested in both directions, so 40 decisions per language per condition. This is a stress test for one failure mode, not a leaderboard.

Why negation is hard for retrieval models

A bi-encoder turns a sentence into one vector before it ever sees your query. "The backup completed" and "The backup did not complete" share almost every token, so their vectors land close together. In my runs the two opposite documents had a cosine similarity between 0.71 and 0.96 depending on the model.

A cross-encoder reranker reads the query and the document together, so in theory it can see the "not". In practice most rerankers are trained to answer "is this passage relevant to this query?" A document about the backup failing is highly relevant to a question about the backup completing. Topic relevance and factual agreement are different things, and the training signal mostly rewards the first one. That is my reading of the results, not something I measured directly.

This is not a new observation. The NevIR benchmark showed in 2024 that most neural retrievers do no better than random on document pairs that differ only by negation. A SIGIR 2025 reproduction extended it to newer models and found the same ordering: bi-encoders at the bottom, cross-encoders in the middle, LLMs on top. What those studies do not cover is a language where negation lives inside the word, how much of the failure comes from word overlap, and what the fix costs on local hardware.

How the test works

Each triplet has a positive query, a negative query, and documents on both sides. It runs twice. The positive query should rank the positive document first, and the negative query should rank the negative document first. A model that always prefers the positive document scores exactly 50%, so 50% is the coin-flip line.

Here is one triplet, in the three conditions I tested:

query: backup did not complete

SAME WORDING
  right  The nightly PostgreSQL backup did not complete
         at 02:00 and nothing was uploaded to S3.
  wrong  The nightly PostgreSQL backup completed
         at 02:00 and was uploaded to S3.

DIFFERENT WORDING
  right  Last night's PostgreSQL dump broke off
         before 02:00, so S3 received no file.
  wrong  The nightly PostgreSQL backup completed
         at 02:00 and was uploaded to S3.

CONTROL
  right  Last night's PostgreSQL dump broke off
         before 02:00, so S3 received no file.
  wrong  Last night's PostgreSQL dump finished
         successfully and the file landed in S3.

The same wording condition is the textbook test, and it is the one I ran first. The different wording condition is the one that looks like a real corpus: the document you want was written by someone else, in their own words, while a document with the opposite outcome happens to reuse your query's phrasing. The control removes that asymmetry. Both documents are paraphrases and only the outcome differs.

The set covers three kinds of flip. Eight triplets use an explicit marker such as not, never or cannot. Eight use antonyms such as succeeded and failed, or allows and blocks. Four change state: resolved and still ongoing, all and none. The Turkish set mirrors the English one, so Turkish negation shows up as a suffix (tamamlandı versus tamamlanmadı) rather than a separate word.

Models with a documented prompt format ran with it: query: and passage: for e5, the task prefixes for EmbeddingGemma, and the instruction prefix for Qwen3-Embedding. The full set is downloadable as JSON.

With the same wording, everything passes

When the right document repeats the query's words, every model family looks excellent. That result is the reason this failure goes unnoticed.

Averages across English and Turkish, 40 decisions per language per model.

If I had stopped after the first column, I would have concluded that negation is solved. The models are not reading the negation in that case. They are matching "did not complete" in the query to "did not complete" in the document. Remove the shared phrase and the picture changes completely.

With different wording, most models pick the wrong document

When the right document paraphrases the query and the wrong one shares its words, 10 of the 13 retrieval models score below 50% in both languages, and none reaches 75% in either. They are not guessing. They are consistently choosing the document with the opposite meaning.

The two tutorial defaults, all-MiniLM-L6-v2 and ms-marco-MiniLM-L6-v2, both score 5% in English. That is 38 wrong answers out of 40. If your pipeline was copied from a getting-started guide, this is the pair you are running.

The type of flip matters. Among rerankers, explicit negation was the worst case at 13%, antonyms reached 24%, and state words such as resolved versus still ongoing reached 55%. Embedding models were weakest on antonyms at 17%. Language made less difference than I expected. Turkish puts negation inside the word as a suffix, yet embedding models averaged 29% in English and 28% in Turkish. Rerankers did drop further in Turkish, from 30% to 22%.

Is it negation, or just word overlap?

It is mostly word overlap winning over meaning. In the control condition, where both documents are paraphrases, the same models reach 72% on average, and the best ones do well: bge-m3 scores 85-88% and bge-reranker-v2-m3 scores 90% in both languages.

So the models are not blind to negation. They carry a polarity signal, and it is weaker than the signal from shared vocabulary. When the two signals point in the same direction, as in the textbook test, you get 96%. When they point in opposite directions, overlap wins most of the time.

That distinction matters for what you do about it. A model that ignored negation entirely would need replacing. A model that underweights it needs a second step that reads the documents properly, or a design that stops relying on text similarity for facts that have a yes-or-no answer.

Does a newer or bigger reranker help?

Not reliably. Newer and higher-ranked on a leaderboard did not mean better at this test.

Qwen3-Reranker-0.6B is the newest reranker here. Its model card reports 66.36 on MMTEB-R against 58.36 for bge-reranker-v2-m3, an eight-point lead on multilingual reranking. On the different-wording test it scored 10% in English. The older bge-reranker-v2-m3 scored 60%. A Turkish fine-tune of that same bge model did best overall, at 57% in Turkish and, less expectedly, 72% in English.

The SIGIR 2025 reproduction points the same way. Its best cross-encoder was jina-reranker-v2-base-multilingual at 65.2% on NevIR, and bge-reranker-v2-m3 scored 43.5%, against a random baseline of 25% on that benchmark's stricter pairwise metric. I did not test the Jina model, because loading it requires trust_remote_code.

I ran into the same gap between leaderboards and reality earlier in this project. On 25 real questions against my own notes, Qwen3-Reranker-0.6B scored an MRR of 0.698 against 0.818 for bge-reranker-v2-m3, and ran four times slower. Leaderboard averages are a reasonable shortlist. They are not a substitute for testing on sentences that look like yours.

Can a local LLM fix it, and what does it cost?

Yes, and the cost is lower than I assumed. Gemma 4 26B-A4B scored 100% on both the same-wording and different-wording tests, and 98% on the control, where it missed two Turkish decisions. Each decision took about 330 ms.

The setup was a plain A/B question, with the right document alternating between position A and B so position bias could not help:

Query: backup did not complete

Document A: The nightly PostgreSQL backup completed at 02:00 and was uploaded to S3.
Document B: Last night's PostgreSQL dump broke off before 02:00, so S3 received no file.

Which document states what the query says? Answer with a single letter: A or B.

That does not make the LLM a good reranker. I tried exactly that on my own notes: Gemma 4 ranking 20 candidates at once returned valid JSON every time, but reached an MRR of 0.679 against 0.768 for the cross-encoder, and took 5.3 seconds per query, 17 times slower. The LLM is good at reading two documents closely. It is worse at ordering twenty.

The practical placement is the generation step, which already reads the retrieved chunks. If it gets five or ten chunks instead of one, a document with the opposite outcome sitting in first place is something it can notice and route around.

How fast is Gemma 4 26B-A4B on a 32 GB Mac?

About 29 tokens per second of output and 600 tokens per second of prompt processing. That makes it roughly four times faster than a dense 27B model at the same quantization on the same machine.

Medians of three runs, Ollama 0.34.0, 16K context, thinking disabled, temperature 0. Each prompt started with a random nonce, because otherwise Ollama's prompt cache reuses the previous run and reports a prefill time close to zero.

Gemma 4 26B-A4B is a mixture-of-experts model: 25.8B parameters in total, about 3.8B active per token. That explains the gap. On this machine Ollama runs both models through llama.cpp with Metal, not the MLX backend, which I confirmed from the running process. The dense model had speculative decoding enabled and was still four times slower.

For RAG, a context of ten chunks at roughly 250 tokens each is about 2,500 prompt tokens, which Gemma 4 processes in around four seconds before it starts writing. That is the realistic price of letting the LLM read more than the top hit.

Where this breaks a real pipeline

The failure only hurts when the wrong document reaches a place where nobody reads it critically. Three places are common.

The first is anything that shows or uses the top hit directly. A search box, an agent tool that returns "the most relevant note", or a pipeline that passes k=1 to save tokens. There is no second reader in that path.

The second is documents that record changing state. Incident notes say ongoing and later resolved. Deploy logs say failed and then succeeded. Runbooks describe the broken state and the fixed state. These corpora are full of near-identical sentences with opposite outcomes, which is precisely the shape of this test.

The third is score thresholds. If you drop anything below a similarity cutoff, the paraphrased right answer can fall under the line while the word-matching wrong answer stays above it.

None of this means you should remove your reranker. On my own notes, with 25 test questions, adding a reranker on top of hybrid search raised MRR from 0.732 to 0.841 at 1,500 chunks, from 0.629 to 0.820 at 4,800 chunks, and from 0.480 to 0.688 at 20,000 chunks. Rerankers are good at ranking relevance. Relevance is simply not the same as agreement.

What to do about it

Pick the fix that matches where the wrong document would do damage.

Metadata is the most reliable of these, and the most often skipped. Whether a backup succeeded is a boolean. Asking a vector space to recover a boolean from prose is harder than storing it once when the document is written. Vector databases such as Qdrant support payload filters for exactly this.

How to run this test on your own models

Download the test set, then swap in your own embedding model and reranker. With the two tutorial defaults below, this script prints the same numbers reported above: 88% and 100% with the same wording, 5% and 5% with different wording.

# pip install sentence-transformers
import json
from sentence_transformers import SentenceTransformer, CrossEncoder

triplets = json.load(open("negation-test-set.json"))["en"]  # or "tr"

def decisions(t, condition):
    """Two decisions per triplet: (query, right_doc, wrong_doc)."""
    if condition == "same_wording":
        pos, neg = t["doc_positive"], t["doc_negative"]
    else:
        # right doc is a paraphrase, wrong doc shares the query's words
        pos, neg = t["paraphrase_positive"], t["paraphrase_negative"]
    return [(t["query_positive"], pos, t["doc_negative"]),
            (t["query_negative"], neg, t["doc_positive"])]

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

def encode(texts):
    return embedder.encode(texts, normalize_embeddings=True)

for condition in ("same_wording", "different_wording"):
    rows = [d for t in triplets for d in decisions(t, condition)]
    q, right, wrong = (encode([r[i] for r in rows]) for i in range(3))
    emb_acc = ((q * right).sum(1) > (q * wrong).sum(1)).mean()

    s_right = reranker.predict([(r[0], r[1]) for r in rows])
    s_wrong = reranker.predict([(r[0], r[2]) for r in rows])
    rr_acc = (s_right > s_wrong).mean()

    print(f"{condition:18} embedding {emb_acc:.0%}  reranker {rr_acc:.0%}")

If your model needs a query prefix, add it to the query strings before encoding. Better still, write ten triplets from your own documents. The phrasing of your corpus is what decides how badly this bites.

Limits of this test

The sample is small. 40 decisions per language gives a 95% confidence interval of roughly ±15 percentage points for scores in the middle of the range, so a 10-point gap between two rerankers is not a meaningful ranking. The gaps between 5% and 50%, or between 26% and 100%, are.

I wrote every sentence myself, and the different-wording condition was built to be hard. Real corpora will not always place a word-for-word opposite next to a paraphrased answer. The A/B format is also easier for an LLM than scoring documents one at a time. Everything ran on one machine, and scores from hosted embedding APIs may differ.

FAQ

Do embedding models understand negation?

Partly. In this test, embedding models ranked the correct document first 72% of the time when both candidates were paraphrases, so they do carry a polarity signal. When the wrong document shared the query's words, their average dropped to 28%, because shared vocabulary outweighs the negation.

Does a reranker fix negation in RAG?

Not on its own. Six local cross-encoder rerankers averaged 26% when the right document used different words and the wrong one shared the query's words, which is worse than a coin flip. The best, a Turkish fine-tune of bge-reranker-v2-m3, reached 72% in English, so it still got more than one in four decisions wrong.

Which reranker handles negation best?

Among local rerankers tested here, bge-reranker-v2-m3 and its Turkish fine-tune did best, at 60% and 72% on English. Qwen3-Reranker-0.6B scored 10% despite a higher MMTEB-R score. A SIGIR 2025 reproduction of NevIR found jina-reranker-v2-base-multilingual to be the strongest cross-encoder in its set. Leaderboard position did not predict negation handling.

Can an LLM be used as a reranker to handle negation?

An LLM handles negation well when it compares documents directly. Gemma 4 26B-A4B chose correctly in 80 of 80 different-wording decisions at about 330 ms each. As a general reranker over 20 candidates it was worse than a cross-encoder, with an MRR of 0.679 against 0.768 and 17 times the latency, so it fits better at the generation step than in the ranking step.

How fast is Gemma 4 26B-A4B on a Mac with 32 GB of memory?

On an Apple M5 with 32 GB, through Ollama 0.34.0, Gemma 4 26B-A4B at Q4_K_M generated 28-33 tokens per second and processed prompts at 540-610 tokens per second. A dense 27B model at the same quantization managed 7-8.5 tokens per second. A 1,900-token RAG prompt with a 256-token answer took 12 seconds with Gemma 4 and 50 seconds with the dense model.

How do I test my RAG pipeline for negation errors?

Write pairs of documents from your own domain that describe the same event with opposite outcomes, and phrase one of them differently from your test query. Run both directions of the query and count how often the matching document ranks first; 50% means the model is not distinguishing them. The test set and script in this article give you a starting point in English and Turkish.

The thing I got wrong first

My project notes said the reranker would solve negation. The evidence behind that was a similarity test: all six embedding models I had compared rated "the backup was taken" and "the backup could not be taken" as more similar than genuine paraphrases, and I reasoned that a cross-encoder, which reads both texts together, would catch the difference. It sounded right, so I wrote it down as a requirement and moved on. When I finally tested ranking, which is the operation retrieval actually performs, the reranker I was using scored 42% in Turkish.

The Gemma 4 speed number in those notes was wrong for a similar reason. I had recorded 11.7 tokens per second and concluded that the mixture-of-experts design was not paying off. That figure was output tokens divided by total time, including the processing of a 2,500-token prompt. Measured separately, decoding runs at 29 tokens per second.

Both mistakes came from measuring something near the question instead of the question itself. Similarity is not ranking, and total time is not generation speed. If you take one habit from this, make your test perform the exact operation your pipeline performs. You can see that operation step by step in the RAG pipeline simulator.