Home / Blog / Attempts to improve skill discovery in Claude Code
Tech News

Attempts to improve skill discovery in Claude Code

Adding a standing lookup rule to CLAUDE.md raised Claude Code skill discovery from 7 of 58 tasks to 40 on Sonnet across an 897-skill test repository.

By Dillip Chowdary • Nov 19, 2026 • Source: github.com

Attempts to improve skill discovery in Claude Code
Illustration · Pexels

At a glance

Product
skill-finder
Author
Dmitry Yermichev
Repository
ydmitry/skill-finder
Benchmark
Skill-Use-Bench arXiv 2608.04828
Haiku baseline
2 of 58 skills opened
Haiku guided
47 of 58 skills opened

Developer Dmitry Yermichev released skill-finder, an open-source tool and configuration framework designed to resolve automated skill discovery bottlenecks inside Anthropic's Claude Code CLI, as documented in github.com's report. The project demonstrates that Claude Code rarely inspects local skill files without explicit user prompting, missing needed files in 56 of 58 benchmark evaluations on Claude 3.5 Haiku and 51 of 58 on Claude 3.5 Sonnet. By adding a single structured instruction block to the local CLAUDE.md file, skill discovery rates increased to 47 of 58 on Haiku and 40 of 58 on Sonnet across identical test suites.

This report examines how skill discovery behaves across large skill libraries, the mechanics of automated command lookups, benchmark scores across three model tiers, and the phrasing techniques needed for tool descriptions. Software engineers, team leads managing standardized repository rules, and developers running complex workflows with Claude Code can use these findings to ensure local project guidelines and reference files load reliably during coding sessions.

01Discovery rates across large skill collections

Claude Code exposes only high-level names and concise descriptions to the underlying model when launching a session, hiding deeper reference documentation until a specific skill file opens. In environments containing 897 public skills alongside custom repository conventions, the model frequently bypasses specialized instructions unless the developer directly inputs the exact skill name in the prompt. This visibility gap prevents agents from enforcing project-specific architecture guidelines, security rules, and code formatting policies that exist inside workspace directories.

The evaluation measured discovery reliability using 58 real-world tasks selected from the Skill-Use-Bench evaluation dataset (arXiv 2608.04828) along with an isolated local rules test. Without dedicated standing instructions, Haiku successfully located the necessary skill in only 2 of 58 tasks, while Sonnet succeeded in 7 of 58 tasks. The deficit stems from compact prompt representations that give models insufficient context to identify relevant tools before generating code.

02The standing instruction lookup protocol

To force skill identification before code generation, the framework installs a mandatory lookup sequence directly into ~/.claude/CLAUDE.md or a repository-level CLAUDE.md file. The rule mandates that Claude Code read the user prompt and referenced files first, isolate technical entities, and execute a dedicated search utility across the skills directory prior to drafting code or architectural plans. The search step runs either through the custom Python script or through standard command-line tools.

  1. Request inspection Read the prompt and every named file to catalog technical terms, libraries, actions, and end artifacts.
  2. Batch command lookup Execute a single search command with ordered multi-word phrases to scan skill documentation folders without shell loops.
  3. Exhaustive document reading Read every matching skill file and bundled reference guide completely before writing plans or code.
  4. Recursive entity lookup Run follow-up searches for newly discovered data types or system areas uncovered inside the initial reference files.

The protocol mandates subsequent lookups after reading each new document, creating a repeatable discovery loop throughout long sessions. When an initial query returns zero matches, the system requires the model to reformulate search terms using alternative synonyms before proceeding.

03Discovery rates and benchmark rule gains

Empirical testing compared baseline execution against guided discovery across Claude 3.5 Haiku, Claude 3.5 Sonnet, and Claude 3 Opus. The primary test bed hid 12 fabricated organizational skills inside an index of 897 public skills, evaluating 24 tasks over two independent runs to create 144 rule-following opportunities. Each rule carried a unique identifier, such as API-14 requiring partner endpoints to use a /px/ prefix, ensuring adherence could be verified through token matching.

Metric and ModelBaseline Without RuleGuided With InstructionEvaluation Scale
Haiku Benchmark Discovery2 of 58 opened47 of 58 opened58 tasks
Sonnet Benchmark Discovery7 of 58 opened40 of 58 opened58 tasks
Haiku Chained Rules0 of 48 followed21 of 48 followed48 runs
Sonnet Chained Rules0 of 48 followed35 of 48 followed48 runs
Opus Chained Rules0 of 10 followed8 of 10 followed10 runs
Haiku Benchmark Task Pass51% checks passed63% checks passed21 file checks
Sonnet Benchmark Task Pass49% checks passed64% checks passed21 file checks

The presence of recursive discovery rules allowed models to resolve chained dependencies where a secondary rule was referenced solely within the body of an earlier skill document. In older trial configurations lacking recursive commands, Sonnet followed 75 of 144 rules with passive reminders, 78 of 144 with a single lookup, and 101 of 144 when instructed to search iteratively.

Note: Small gaps of two to five tasks between identical runs occurred across the evaluation runs, and citing an identification number does not guarantee the model fully implemented the underlying design constraint.

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

04Trigger wording impact on mid-task invocation

A targeted test examined whether description formatting influences mid-session discovery when Claude Code runs without standing instructions or helper scripts. Eight specialized skills were placed alongside 12 background skills, testing explicit triggers introduced either in the initial request or midway through execution. While initial prompts mentioning the trigger produced an 8 of 8 success rate on Sonnet regardless of phrasing, mid-task triggers exposed significant sensitivity to phrasing structure.

Description Syntax PatternSonnet Mid-Task SuccessHaiku Mid-Task SuccessContext Size
Imperative ALWAYS phrasing14 of 15 invoked8 of 24 invoked20 skills
Conditional "Use when" syntax7 of 15 invokedNot recorded20 skills
Descriptive "Rules for X" syntax3 of 13 invoked0 of 23 invoked20 skills
Imperative ALWAYS at scaleNot recorded1 of 24 invoked200 skills

Imperative descriptions instructing the agent to invoke the tool unconditionally proved essential for Sonnet in small catalogs, though Haiku struggled across all phrasing styles. Furthermore, expanding the total catalog size from 20 to 200 skills reduced Haiku's invocation rate under imperative phrasing to 1 of 24 runs, confirming that prompt wording alone cannot substitute for structured command searches at scale.

05Environment installation and model configuration commands

Developers can deploy the discovery framework by cloning the repository directly into their local configuration directory. The underlying CLI environment should remain updated to ensure compatibility with standing instruction formats.

Command
npm install -g @anthropic-ai/claude-code
claude update

The dedicated discovery script installs via Git directly into the local skills path for immediate access.

Command
git clone https://github.com/ydmitry/skill-finder ~/.claude/skills/skill-finder

To search without external Python utilities, developers can configure the equivalent search command using standard grep inside CLAUDE.md. On macOS systems with symlinked directories, include the -S flag.

Command
grep -Ril --include=SKILL.md -e "<a name>" -e "<another name>" ~/.claude/skills

Inside the CLI interface, developers switch between models or pin specific configurations for individual sessions.

Command
/model sonnet
/model haiku
claude --model sonnet

Default model configurations can be set permanently in the main user configuration file to maintain predictable lookup performance across all repository sessions. Sonnet baseline: 7 of 58 skills opened Sonnet guided: 40 of 58 skills opened

Frequently asked questions

Why does Claude Code fail to open local skills automatically?

Claude Code receives only compact skill names and short descriptions in its session context, which gives the model insufficient information to match complex project rules without dedicated search commands.

How does adding a lookup command to CLAUDE.md improve skill discovery?

It forces the model to extract technical entities from requests and run automated grep or Python searches across skill files before drafting code or architecture plans.

Can skill discovery function without the custom skill-finder Python script?

Yes, standard grep commands targeting SKILL.md files achieve equivalent discovery accuracy when embedded into the standing CLAUDE.md instructions.

What phrasing works best for individual skill descriptions?

Imperative descriptions beginning with instructions to use the skill every time achieved a 14 of 15 invocation rate on Sonnet during mid-task events, outperforming passive descriptions.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →