Eval Study #1 — Ending 'Vibe Benchmarking' · Accuracy-based + First LLM-as-Judge Implementation

Starting Eval (Evaluation) study today.

In past projects, I often measured metrics with vague expressions like “this feels faster” or “this seems to give more accurate answers.” That’s fine for a tool made just for myself, but if I’m building a service to show or sell to someone else, it needs to be quantifiable.

I already wrote in the quant post retrospective that I need to move from “vibe benchmarking” to “numeric benchmarking.” Today starts the follow-up study on that.

Table of contents

Why Eval matters — you can’t make decisions on vibes alone

When you evaluate on vibes alone:

• Not reproducible — you can’t recreate the same result next time
• Can’t measure the degree of improvement — you don’t know how much better it got
• Trade-offs get missed — when a code change improves A but degrades B, you’ll miss B’s degradation if you’re not measuring it
• Weak grounds for decisions — deciding “by gut feeling” without numbers doesn’t lead to the best decision

Eval = quantitatively measuring the quality of an LLM / agent system. Numbers, not vibes.

The 4 types of Eval

Today I’m implementing accuracy-based and LLM-as-Judge eval.

• RAG retrieval / agent tool selection → accuracy-based
• RAG answers / agent answers → LLM-as-Judge

1. Accuracy-based — tool selection accuracy

The simplest case — the accuracy of which tool the agent chose.

Test cases

test_cases = [
    {"question": "847 곱하기 2391은?",   "expected_tool": "calculator"},
    {"question": "부산 날씨 어때?",        "expected_tool": "get_weather"},
    {"question": "오늘 주요 뉴스 알려줘",   "expected_tool": "search_web"},
    # ... 총 10개
]

Back when I was doing embedded development, I practiced adopting TDD once, and writing test cases to verify AI quality feels like the same kind of work. If you define input → expected output pairs ahead of time, you can run them like regression tests.

Execution results

This time it was 10/10 = 100%, but if there had been incorrect cases:

• Show the incorrect case itself
• Display which wrong tool was picked
• → Then refine the tool’s description to be clearer to improve it

The degree of improvement becomes visible as numbers. Something like 65% → 80% → 95%. This is the whole reason Eval exists. The results are also saved as JSON so they can be compared later.

2. LLM-as-Judge — quantifying subjective evaluation

This is for cases where “did well / did poorly” — like answer quality — isn’t a number you can verify directly. To quantify this, you assign an LLM the role of evaluator.

Question-answer pairs to evaluate

These are fake question-answer pairs for now, but in a real setting, answers actually generated by the system would go straight into the evaluation.

Judge prompt

The key: make the evaluation criteria explicit + force JSON format.

judge_prompt = """당신은 답변 품질을 평가하는 전문가입니다.

질문: {question}
답변: {answer}

다음 기준으로 1~10점 평가:
- accuracy: 정확성 1-10
- completeness: 완결성 1-10
- clarity: 명확성 1-10
- overall: 종합 1-10
- reason: 평가 이유 한 문장

JSON만 출력하세요."""

Results

Two answers to the same question (“What is RAG?”):

A human looking at this would get the impression that “the first one is better,” roughly speaking. But the LLM evaluates it from multiple angles and makes the score difference explicit. 8 vs 3 → a 5-point gap. This gap is exactly the room for improvement.

The limits of LLM-as-Judge — why human sample verification is still needed

LLM-as-Judge isn’t perfect either. Limitations I noticed while actually working with it:

1. The evaluator itself is an LLM — can’t be trusted 100%
2. Scoring the same answer twice can produce different scores — it’s not deterministic
3. There’s a tendency to score answers generated by the same model more generously (self-preference bias)

→ This is why human sample verification remains the last safeguard. If only LLMs evaluate each other, you fall into a closed loop. You need humans to directly check a subset of cases to verify the reliability of the LLM evaluation itself.

Retrospective

What I got out of today’s study:

1. Organized the pitfalls of vibe benchmarking into 4 categories — not reproducible / can’t measure improvement / missed trade-offs / weak grounds for decisions
2. The value of accuracy-based Eval — an improvement cycle like “refining the description takes it from 65 → 80 → 95%” becomes visible as numbers
3. LLM-as-Judge is a tool for quantifying subjective evaluation — but because of evaluator variance and self-model bias, human sample verification can’t be skipped

To objectively explain what a system I built can do, I ultimately need numeric evidence. Without it, even saying “this system is good” gives the listener no way to verify it. The whole reason Eval exists comes down to that ability to explain objectively.

Things to study further

1. Similarity-based Eval

• Cosine similarity — semantic distance in embedding space
• BLEU / ROUGE — n-gram based text generation evaluation (originated in machine translation)
• BERTScore — embedding-based token matching
• What each metric captures / fails to capture

2. Patterns for mitigating LLM-as-Judge bias

• Order bias (favoring whichever answer is seen first in A/B evaluation) → average across swapped orders
• Self-model scoring generosity → use a judge model different from the one being evaluated (e.g., evaluate gpt-4o’s answers with claude)
• Averaging/median across multiple evaluation rounds → absorbs variance
• The effect of providing a reference answer vs. not providing one

3. RAG retrieval accuracy metrics

• Hit Rate @ K — proportion of correct chunks included within top-K
• MRR (Mean Reciprocal Rank) — average of the reciprocal of the rank where the correct answer appears
• NDCG — reflects rank weighting
• Separately measuring retrieval quality vs. answer quality (RAGAS’s Faithfulness vs Context Precision)

4. The quality of the Eval Set itself

• If the eval set is biased, the evaluation results are biased too — calibrating golden answers
• Auto-generated eval sets (auto-generating questions with an LLM) → the ratio of human review
• Domain / difficulty distribution of the eval set
• Directly connected to the eval set section in the RAG data preparation post

5. Wiring Eval into CI

• Code change → automatically run Eval, compare scores → detect regressions
• Block a PR if the score falls below a threshold
• Visualize A/B scores (compared to previous versions)
• The exact same flow as doing TDD in embedded — except it’s score regression, not binary pass/fail

6. Making human evaluation more efficient

• Since checking everything is inefficient, combine prioritizing low-confidence cases + random sampling
• Securing inter-annotator agreement (consistency across human evaluators)
• Designing evaluation guidelines / rubrics
• Same approach as the pattern in FEMS #2’s automatic readable_ratio detection + manual review of only suspicious pages