Tag: tool-use
All the articles with the tag "tool-use".
-
Eval Study #3 — Agent Eval, the Limits of Single-Step Scoring, and Multi-Step Grading
Expanding from RAG Eval to agent Eval. Agents have 6 axes to judge (tool selection, argument extraction, multi-step trajectory, termination judgment, safety guards, final answer quality), so separating axes—like unit tests before integration tests—makes it faster to pinpoint causes. Today I covered just 3: tool selection, multi-step, and trap regression. I fed in MCP server tool metadata (name, description, schema) and measured against a (question, expected tool, expected args, level) test set → tool selection accuracy 11/13 = 84.6%. Two failure cases were interesting — (1) a single-step eval falsely flagged a case that was actually correct as a multi-step trajectory, showing the limits of single-step scoring, and (2) an ambiguous tool description caused "electricity bill" to wrongly pick consumption, while "how much money is it?" got it right — patching the system prompt instead of fixing the description risks overfitting. I then switched to multi-step eval (did it call all necessary tools, in order, using prior results, with correct termination judgment) → 4/4 = 100%, and the case that failed under single-step scoring now passed.
-
MCP Study #1 — Getting Started with Model Context Protocol · First Call with stdio Server + Inspector
Starting to study MCP (Model Context Protocol) today. It's a standard protocol created by Anthropic that standardizes how LLMs access external systems. Comparing it with Tool Use → building my first MCP server (say_hello / add_numbers) with the Python SDK → verifying the stdio connection with MCP Inspector → building a fake energy management server that connects with FEMS. Connection failure due to venv python.exe path issue + fix.
-
Giving LLMs Tools — Claude Tool Use and the Agent Loop
LLMs don't know large-number arithmetic or today's weather. I tried out Tool Use, which patches that weakness with external function calls. Message flow, automatic multi-tool selection, and the agent loop.