The method, or how we grade texts that pretend to be programs

A skill for an AI agent looks innocent: a folder, a SKILL.md with a description and instructions, sometimes a script or two. No compiler complains, no CI turns red. Change one word in the description and the agent stops recognising its job; drop a file with a key into the folder and it ships to the catalog with everything else. Across the 48 057 skills we read while calibrating we saw both, and worse: a trading agent instructed to hide gateway refusals from the user, a food-ordering skill that imports Chrome cookies, entire workspaces with memory files published as a "skill". What follows is how all of that becomes one letter from A to F.

Three questions instead of one

We do not try to answer "is this a good skill". We answer three questions that can actually be checked: is it safe for an agent to read, is it written so that it will fire, and does it work. The first two need no model call at all: seconds per skill, for every skill in every catalog. The third costs money and time, so it is a bonus, not an obligation.

1. Safety: 60 percent of the score

Guard reads every text file in the skill and looks for 53 tells. Half the rules exist in two languages, because "игнорируй предыдущие инструкции" turns up in catalogs about as often as its English cousin, and English-only scanners never see it: they look for ASCII word boundaries, and Cyrillic is noise to them. Separate checks catch what eyes cannot: zero-width characters, Unicode tag characters, bidirectional overrides, words that mix Latin and Cyrillic letters. File-level checks look at the folder as a whole: .env and key files, agent memory dumps, binaries, credential variables requested from the Hermes runtime.

The hard part of a scanner is not finding suspicious things; it is not shouting about them. In security documentation "rm -rf /" appears in every other paragraph, and there it is a warning, not a command. So every finding carries a context that can lower its severity: a negation in front of the phrase, a placeholder instead of a key, a detector definition, a table row, an attack demo, a code comment. The downgrades are shown on each skill page as grey chips, so you can argue with us.

The arithmetic is plain: a critical finding costs 45 of 100 points, high 18, medium 5, low 1. One critical finding blocks the skill: grade F, no model run, and a "What is at stake" block appears: what the installer risks and what the author should do. Calibration on live catalogs brought false blocks to zero; real GitHub, Telegram, OpenAI and Slack key leaks are still caught.

2. Quality: 40 percent

Two layers. First, lint against the Agent Skills spec (agentskills.io) and the Hermes dialect: is there a description, does it say when to use the skill; description length (up to 1,024 characters, Hermes up to 60 plus a "When to Use" section); body size; references to files that do not exist; the name format. Each remark subtracts 1 to 45 points.

The second layer arrived after we looked at the first rating and saw 92 percent A grades: most skills had no lint remarks, and nothing told them apart. Now quality starts at 70 and a skill earns its way to 100 with signals of craft, each of which you can verify by eye: quoted example trigger phrases right in the description (6 percent of skills have them), an explicit "when NOT to use" (4 percent), headings, numbered steps, a stated output format, input and output examples, reference files the SKILL.md actually points to, documented scripts, a license, bilingual text. Penalties for the opposite: a nearly empty body, TODOs, absolute paths like C:\Users, localhost URLs, names like "skill-1". The full list with points is on every skill page.

3. Tests: a bonus and a ceiling

An evals.json adds 4 points, a spec.yaml another 2. If the skill went through a model run, ACTIVE adds 8, DEGRADED 2, OBSOLETE subtracts 5. And a ceiling: a skill that missed its thresholds on models cannot wear an A, however well it is written. DEGRADED caps the grade at B, OBSOLETE at C. A separate "Tests" scale shows how far the behaviour has been checked at all, from zero to a hundred.

On top of the grade every skill carries a verdict, the answer to the plain question "install it or not". "Blocked": guard found signs of malicious behaviour. "Not recommended": critical or high findings, a D or F grade, or a live run that showed the skill does not improve model answers. Everything else: "OK to use". The verdict shows in the rating table, on the skill page and in the JSON.

Total

overall = 0.6 × safety + 0.4 × quality + bonus, rounded and clamped to 0…100. A from 90, B from 75, C from 60, D from 40, F below. On equal scores the more downloaded skill ranks higher: a logarithm of downloads, installs and stars, so one giant does not eclipse the rest.

The model run, and where CMRG comes in

A full check does what a fussy colleague would do with a free evening. If the skill ships no tests, a model drafts them: four to six realistic requests in Russian and English, the phrases the skill should fire on, assertions about the answer format. Then every request runs twice: with the skill in the system prompt and without it. The gap between those two numbers is the answer to "why does this skill exist": if the model copes without it, the skill is OBSOLETE, however handsome.

Answers are checked by two judges of different natures. Deterministic assertions: language, JSON, required and forbidden strings, a canary planted in a file with an embedded instruction. And a model judge that compares the answer against a rubric. The model judge is fallible and we know it; its verdict is compared with a configurable threshold, and contested cases are shown in the report with its reasoning. The next step is described by the CMRG protocol (crossmodelrg.org): not one judge but three independent ones, with unanimity as the reliability signal. In the CMRG experiments a unanimous panel was what predicted which claims could be trusted, and disagreement honestly marked "a human is needed here". We are folding that logic in narrowly: a three-judge panel on contested cases, a "needs human" status on a split, and no attempt to squeeze an overall quality score out of the panel, which CMRG itself found unreliable.

Triggers are tested separately: the skill is placed in a catalog next to eight decoys (translation, calendar, SQL, PDF and so on), and the model must pick it for the right phrases and leave it alone for the wrong ones. Statuses: ACTIVE (pass rate of 80 percent or more and a real uplift over baseline), DEGRADED (below threshold or triggers failing), OBSOLETE (the model copes without the skill), ERROR (the run did not finish: the credit is returned).

Models today: openrouter:openai/gpt-5-mini; judge: openrouter:anthropic/claude-haiku-4.5. One run costs a few cents; a daily spending cap keeps surprises off the bill.

Which agents this works for

SKILL.md became an open standard, and the same skill is read by Claude / Claude Code, ChatGPT / Codex, Gemini CLI, Cursor, GitHub Copilot / VS Code, Qwen Code, Kimi Code CLI, Z.ai (GLM), DeepSeek (Deep Code), Mistral Vibe, LangChain (deepagents), OpenClaw, Goose, OpenCode. So every skill in the rating carries "Runs in" chips: the format family plus the platform the catalog says it was made for. The Hermes dialect is read by Hermes Agent. Two caveats people ask about most: CrewAI: Python tools, does not read SKILL.md; a skill body can be pasted into an agent backstory. AutoGen / Microsoft Agent Framework: function tools, does not read SKILL.md. Genspark publishes workspace skills in the same format; we read those too.

Why formatting is not cosmetics

A skill is read by a program, not by a person, so details that look trivial in ordinary prose decide the fate of the whole file. Below are the defects we found in our own catalog, with counts from it. Each one we first took for a bug in our analyzer, and only then confirmed it was a property of the skills themselves.

A colon in the description kills the skill. description: Work with Discord: send messages is invalid YAML: a colon followed by a space inside an unquoted value starts a new key. Strict parsing throws the whole block away, and the skill is left with no name and no description, so the agent never finds it. 9,469 skills in the catalog are in this state. Quoting the value fixes it.

No frontmatter, no skill. A file that starts straight with a heading is invisible to the agent however good the text is. We hold 6 591 of them: usually a README that ended up named SKILL.md.

A reference to a file that is not bundled. The instructions say "follow the rules in references/rules.md" and the file is not there. The process stalls halfway, and this is the only defect for which we apply a hard "will not run" mark. 10 416 skills carry it.

Prose instead of steps. A numbered or bulleted list is not decoration: it tells the agent that the actions have an order and that there are exactly this many. From a paragraph every reader reconstructs a different sequence, and the result stops being reproducible.

Name the headings the ordinary way. "Inputs", "What can go wrong", "Output" are not our taste: they are the signals by which both we and the agent can tell whether the process has inputs, failure handling and a completion criterion. A creatively named section does not exist as far as a program is concerned.

Non-Latin scripts break other people's tooling more often than you would think. In JavaScript a word boundary is defined on ASCII characters, so a pattern like закупк never matches. We walked into this twice ourselves and repaired 153 places; until then the whole Russian-language half of our analysis was blind. If your skill is written in a non-Latin script, expect third-party checkers to underrate it for the same reason, which makes the formatting that needs no regular expressions — frontmatter, lists, ordinary headings — matter even more.

The conclusion is plain. Our quality score is not built from beauty but from the signals that let a program satisfy itself the skill is executable. Careful formatting is the cheapest way to raise it, because it changes nothing in substance and makes the skill visible.

What we do not do, and why it matters

• We never execute scripts or install skills. The analysis is static. It does not prove safety; it makes obvious attacks expensive and visible.
• We never test found keys for liveness and never publish them: evidence on the pages is masked. We tell the author to revoke the key rather than checking whether it works.
• Cases are single-turn: agent tools are not called. A skill that lives by CLI calls will show less on models than it can do. We say so next to the result, not in a footnote.
• The judge is a language model. On contested cases it can be strict or lenient. Threshold and rubrics are configurable in spec.yaml; the CMRG panel is on the roadmap.

Where the data comes from and how often it refreshes

The rating is fed by 32 catalogs: ClawHub through its public API, GitHub repositories as branch archives (no git), mirrors of skills.sh, LobeHub and browse.sh, catalogs for Copilot, Gemini, Qwen, Kimi, Z.ai, DeepSeek, Mistral Vibe, LangChain and Genspark, plus links users submit. Each source is re-read every 3 hours; a skill is re-analyzed when its content or the rules version (currently 13) changes. The exact time of the last crawl sits on the home page under "Data freshness", and every skill keeps a visible score history.

All of it runs offline too: npx skilleval audit ./skills for a catalog and npx skilleval run ./my-skill for one skill. Same engine.