Attempts to improve skill discovery in Claude Code
Adding a standing lookup rule to CLAUDE.md raised Claude Code skill discovery from 7 of 58 tasks to 40 on Sonnet across an 897-skill test repository.
By Dillip Chowdary • Nov 19, 2026 • Source: github.com

At a glance
- Product
- skill-finder
- Author
- Dmitry Yermichev
- Repository
- ydmitry/skill-finder
- Benchmark
- Skill-Use-Bench arXiv 2608.04828
- Haiku baseline
- 2 of 58 skills opened
- Haiku guided
- 47 of 58 skills opened
In this post
Developer Dmitry Yermichev released skill-finder, an open-source tool and configuration framework designed to resolve automated skill discovery bottlenecks inside Anthropic's Claude Code CLI, as documented in github.com's report. The project demonstrates that Claude Code rarely inspects local skill files without explicit user prompting, missing needed files in 56 of 58 benchmark evaluations on Claude 3.5 Haiku and 51 of 58 on Claude 3.5 Sonnet. By adding a single structured instruction block to the local CLAUDE.md file, skill discovery rates increased to 47 of 58 on Haiku and 40 of 58 on Sonnet across identical test suites.
This report examines how skill discovery behaves across large skill libraries, the mechanics of automated command lookups, benchmark scores across three model tiers, and the phrasing techniques needed for tool descriptions. Software engineers, team leads managing standardized repository rules, and developers running complex workflows with Claude Code can use these findings to ensure local project guidelines and reference files load reliably during coding sessions.
01Discovery rates across large skill collections
Claude Code exposes only high-level names and concise descriptions to the underlying model when launching a session, hiding deeper reference documentation until a specific skill file opens. In environments containing 897 public skills alongside custom repository conventions, the model frequently bypasses specialized instructions unless the developer directly inputs the exact skill name in the prompt. This visibility gap prevents agents from enforcing project-specific architecture guidelines, security rules, and code formatting policies that exist inside workspace directories.
The evaluation measured discovery reliability using 58 real-world tasks selected from the Skill-Use-Bench evaluation dataset (arXiv 2608.04828) along with an isolated local rules test. Without dedicated standing instructions, Haiku successfully located the necessary skill in only 2 of 58 tasks, while Sonnet succeeded in 7 of 58 tasks. The deficit stems from compact prompt representations that give models insufficient context to identify relevant tools before generating code.
02The standing instruction lookup protocol
To force skill identification before code generation, the framework installs a mandatory lookup sequence directly into ~/.claude/CLAUDE.md or a repository-level CLAUDE.md file. The rule mandates that Claude Code read the user prompt and referenced files first, isolate technical entities, and execute a dedicated search utility across the skills directory prior to drafting code or architectural plans. The search step runs either through the custom Python script or through standard command-line tools.
- Request inspection Read the prompt and every named file to catalog technical terms, libraries, actions, and end artifacts.
- Batch command lookup Execute a single search command with ordered multi-word phrases to scan skill documentation folders without shell loops.
- Exhaustive document reading Read every matching skill file and bundled reference guide completely before writing plans or code.
- Recursive entity lookup Run follow-up searches for newly discovered data types or system areas uncovered inside the initial reference files.
The protocol mandates subsequent lookups after reading each new document, creating a repeatable discovery loop throughout long sessions. When an initial query returns zero matches, the system requires the model to reformulate search terms using alternative synonyms before proceeding.
03Discovery rates and benchmark rule gains
Empirical testing compared baseline execution against guided discovery across Claude 3.5 Haiku, Claude 3.5 Sonnet, and Claude 3 Opus. The primary test bed hid 12 fabricated organizational skills inside an index of 897 public skills, evaluating 24 tasks over two independent runs to create 144 rule-following opportunities. Each rule carried a unique identifier, such as API-14 requiring partner endpoints to use a /px/ prefix, ensuring adherence could be verified through token matching.
| Metric and Model | Baseline Without Rule | Guided With Instruction | Evaluation Scale |
|---|---|---|---|
| Haiku Benchmark Discovery | 2 of 58 opened | 47 of 58 opened | 58 tasks |
| Sonnet Benchmark Discovery | 7 of 58 opened | 40 of 58 opened | 58 tasks |
| Haiku Chained Rules | 0 of 48 followed | 21 of 48 followed | 48 runs |
| Sonnet Chained Rules | 0 of 48 followed | 35 of 48 followed | 48 runs |
| Opus Chained Rules | 0 of 10 followed | 8 of 10 followed | 10 runs |
| Haiku Benchmark Task Pass | 51% checks passed | 63% checks passed | 21 file checks |
| Sonnet Benchmark Task Pass | 49% checks passed | 64% checks passed | 21 file checks |
The presence of recursive discovery rules allowed models to resolve chained dependencies where a secondary rule was referenced solely within the body of an earlier skill document. In older trial configurations lacking recursive commands, Sonnet followed 75 of 144 rules with passive reminders, 78 of 144 with a single lookup, and 101 of 144 when instructed to search iteratively.
Note: Small gaps of two to five tasks between identical runs occurred across the evaluation runs, and citing an identification number does not guarantee the model fully implemented the underlying design constraint.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
04Trigger wording impact on mid-task invocation
A targeted test examined whether description formatting influences mid-session discovery when Claude Code runs without standing instructions or helper scripts. Eight specialized skills were placed alongside 12 background skills, testing explicit triggers introduced either in the initial request or midway through execution. While initial prompts mentioning the trigger produced an 8 of 8 success rate on Sonnet regardless of phrasing, mid-task triggers exposed significant sensitivity to phrasing structure.
| Description Syntax Pattern | Sonnet Mid-Task Success | Haiku Mid-Task Success | Context Size |
|---|---|---|---|
| Imperative ALWAYS phrasing | 14 of 15 invoked | 8 of 24 invoked | 20 skills |
| Conditional "Use when" syntax | 7 of 15 invoked | Not recorded | 20 skills |
| Descriptive "Rules for X" syntax | 3 of 13 invoked | 0 of 23 invoked | 20 skills |
| Imperative ALWAYS at scale | Not recorded | 1 of 24 invoked | 200 skills |
Imperative descriptions instructing the agent to invoke the tool unconditionally proved essential for Sonnet in small catalogs, though Haiku struggled across all phrasing styles. Furthermore, expanding the total catalog size from 20 to 200 skills reduced Haiku's invocation rate under imperative phrasing to 1 of 24 runs, confirming that prompt wording alone cannot substitute for structured command searches at scale.
05Environment installation and model configuration commands
Developers can deploy the discovery framework by cloning the repository directly into their local configuration directory. The underlying CLI environment should remain updated to ensure compatibility with standing instruction formats.
npm install -g @anthropic-ai/claude-code
claude updateThe dedicated discovery script installs via Git directly into the local skills path for immediate access.
git clone https://github.com/ydmitry/skill-finder ~/.claude/skills/skill-finderTo search without external Python utilities, developers can configure the equivalent search command using standard grep inside CLAUDE.md. On macOS systems with symlinked directories, include the -S flag.
grep -Ril --include=SKILL.md -e "<a name>" -e "<another name>" ~/.claude/skillsInside the CLI interface, developers switch between models or pin specific configurations for individual sessions.
/model sonnet
/model haiku
claude --model sonnetDefault model configurations can be set permanently in the main user configuration file to maintain predictable lookup performance across all repository sessions. Sonnet baseline: 7 of 58 skills opened Sonnet guided: 40 of 58 skills opened
Frequently asked questions
Why does Claude Code fail to open local skills automatically?
Claude Code receives only compact skill names and short descriptions in its session context, which gives the model insufficient information to match complex project rules without dedicated search commands.
How does adding a lookup command to CLAUDE.md improve skill discovery?
It forces the model to extract technical entities from requests and run automated grep or Python searches across skill files before drafting code or architecture plans.
Can skill discovery function without the custom skill-finder Python script?
Yes, standard grep commands targeting SKILL.md files achieve equivalent discovery accuracy when embedded into the standing CLAUDE.md instructions.
What phrasing works best for individual skill descriptions?
Imperative descriptions beginning with instructions to use the skill every time achieved a 14 of 15 invocation rate on Sonnet during mid-task events, outperforming passive descriptions.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement