review-sentiment-eval / finding §6.6
source repository

The faithfulness judge ranked the weaker summariser above the stronger one.

An interactive walkthrough of one finding from an LLM evaluation suite of 34 cases, run at 4 model configurations, 3 runs each. This page covers the 16 normal review cases and their Ragas faithfulness scores; the adversarial and judge-calibration cases are analysed in the repository. All scores and case texts below come from the run data and dataset published there.

The mechanism

What the judge actually caught

The faithfulness metric decomposes a summary into atomic statements, then checks each one against the source. The score is the fraction of the statements that pass.

On negative_timeline_shipping, a review about a three-week-late delivery and a refund still missing after 14 days, the local llama3.2 summary averaged 1.00 for faithfulness across three runs and the claude-sonnet-4-6 summary 0.76. Both pass the 0.70 threshold, judged by the same claude-sonnet-4-6 model. The weaker summariser came out ahead here and on four other normal cases, all in the table below, and this 0.24 gap is the widest of them.

Source text — negative_timeline_shipping
Placed the order on March 1st. The estimated delivery was March 5th. On March 8th I was told the item hadn't shipped yet. It shipped March 12th and arrived March 19th — nearly three weeks after I ordered. By then I had already bought a replacement elsewhere. I returned it immediately. The refund still hasn't appeared after 14 days.
The highlighted spans mark the source facts behind the strong summary's two failed claims.
Strong summariser (claude-sonnet-4-6) — 5/7 = 0.71 this run, mean 0.76
The customer experienced significant shipping delays. PASS
The customer's order arrived nearly three weeks late. PASS
The estimated delivery window for the customer's order was 5 days. FAIL
The customer had already purchased a replacement item by the time the order arrived. PASS
The customer returned the order immediately upon arrival. PASS
The customer has been waiting over 14 days for a refund. FAIL
The refund for the customer's returned order has not yet been processed. PASS
Why it failed
"The estimated delivery window … was 5 days."
The source gives an order date and an estimated delivery date, but states no duration. The model converted the delivery estimate into a span of time rather than a single arrival date.
Why it failed
"… waiting over 14 days for a refund."
The elapsed time is pinned at 14 days. One word, "over", claims that more than 14 have passed, which is something the source does not give.
Weak summariser (llama3.2) — 6/6 = 1.00, all three runs
The customer placed an order on March 1st. PASS
The order took nearly three weeks to arrive. PASS
The customer received a shipping update indicating that the item had not shipped yet. PASS
The customer had to return the item. PASS
The customer returned the item due to receiving a replacement elsewhere. PASS
The refund has not appeared after 14 days. PASS
"The metric is working correctly. The Sonnet judge caught genuine hallucinations in the SS output. The issue is not a scoring error — it is a design consequence: faithfulness measures whether every claim is supported by the source, but it does not measure how much of the source is covered."
— §6.6, model configuration analysis
The full picture

Every case, every configuration

Mean faithfulness across 3 runs for each of the 16 review cases under four summariser/judge configurations. Each is a two-letter code where the first letter is the summariser and the second is the judge. S is the strong model (claude-sonnet-4-6) and W is the weak one (llama3.2 summariser, mistral judge), so WS means a llama3.2 summary graded by a claude-sonnet-4-6 judge. Select a row for run detail and the source text.

pass (≥ 0.70) below threshold * unstable across runs (max − min > 0.2) ▲ weak summariser outscored the strong one under the same Sonnet judge (5 of 16 cases)
case SS SW WS WW

The suite also tests prompt-injection robustness and judge calibration. Those are separate findings, covered in the full analysis linked below.