skill-tester

# Skill Tester

**Tier**: POWERFUL · **Category**: Engineering Quality Assurance · **Dependencies**: None (Python stdlib only)

Meta-skill that validates, tests, and scores skills in this repository. Four tools, run from the **repo root** with full paths:

1. **`scripts/skill_validator.py`** — structure + documentation compliance
2. **`scripts/script_tester.py`** — Python script syntax/imports/runtime/output testing
3. **`scripts/quality_scorer.py`** — multi-dimensional scoring with letter grade
4. **`scripts/security_scorer.py`** — security posture scoring (also available via `quality_scorer.py --include-security`)

> **Scope note:** this skill's tier line-count minimums measure *legacy* skills. For authoring *new* skills, `engineering/write-a-skill` (SKILL.md under ~100 lines, Matt Pocock doctrine) is the binding standard — do not pad a new skill to satisfy a tier minimum here.

## Quick Start (exact, runnable from repo root)

```bash
# 1. Validate structure (exit non-zero on failure — usable as a gate)
python3 engineering/skills/skill-tester/scripts/skill_validator.py engineering/skills/self-eval --json

# 2. Test the skill's Python scripts (30s default timeout per script)
python3 engineering/skills/skill-tester/scripts/script_tester.py engineering/skills/self-eval --json

# 3. Score quality (fail CI below threshold with --minimum-score)
python3 engineering/skills/skill-tester/scripts/quality_scorer.py engineering/skills/self-eval --json --detailed --minimum-score 75
```

Consume the JSON: validator emits `overall_score`, `compliance_level`, per-check `checks{}`; scorer emits `overall_score`, `letter_grade`, `tier_recommendation`, `dimensions`, and an `improvement_roadmap` — work the roadmap top-down, then re-run until the target score is met.

For repo-wide auditing prefer `scripts/audit_skills.py` at the repo root (wraps the write-a-skill checklist runner across all skills).

## What Each Tool Checks

### skill_validator.py
- SKILL.md frontmatter parsing, required sections, minimum line counts per tier (`--tier BASIC|STANDARD|POWERFUL`)
- Required structure: SKILL.md, README.md, scripts/, references/, assets/, expected_outputs/
- Python scripts: argparse present, stdlib-only imports

### script_tester.py
- AST-based syntax validation; import analysis (flags external dependencies)
- Controlled execution with timeout protection (`--timeout`, default 30s)
- `--help` functionality verification; sample-data runs compared against expected_outputs/

### quality_scorer.py
Four dimensions, 25% each: **Documentation** (depth, examples, references), **Code Quality** (complexity, error handling, output consistency), **Completeness** (required dirs, sample data, expected outputs), **Usability** (help text, example clarity). Outputs 0-100 + A-F grade + tier recommendation.

## Tier Classification

| Tier | SKILL.md | Scripts | CLI surface |
|---|---|---|---|
| BASIC | ≥ 100 lines | 1 (100-300 LOC) | basic argparse |
| STANDARD | ≥ 200 lines | 1-2 (300-500 LOC) | subcommands, JSON + text output |
| POWERFUL | ≥ 300 lines | 2-3 (500-800 LOC) | multiple modes, CI integration |

(Advisory for legacy skills; new skills follow write-a-skill — see scope note above.)

## CI Integration

```yaml
# GitHub Actions: gate changed skills
- name: "validate-changed-skills"
  run: |
    for skill in $changed_skills; do
      python3 engineering/skills/skill-tester/scripts/skill_validator.py "$skill" --json
      python3 engineering/skills/skill-tester/scripts/script_tester.py "$skill"
      python3 engineering/skills/skill-tester/scripts/quality_scorer.py "$skill" --minimum-score 75
    done
```

Pre-commit hook: run the validator on the staged skill directory and block the commit on non-zero exit.

## Verification Loop

A skill "passes" when, in one run from repo root:

1. `skill_validator.py <skill> --json` exits 0,
2. `script_tester.py <skill>` reports all scripts passing, and
3. `quality_scorer.py <skill> --minimum-score <target>` exits 0.

If any step fails, apply the top `improvement_roadmap` item and re-run all three — never report a partial pass.

## Troubleshooting

- **Timeout errors** → raise `--timeout` or optimize the script under test
- **Import failures** → external deps detected; stdlib-only is the repo policy
- **Tier misclassification** → check line counts/LOC against the tier table; remember the write-a-skill exception for new skills

References: `references/` holds the structure specification, tier requirements matrix, and scoring rubric the tools implement.

{
  "format": "aiskill/v1",
  "skill": {
    "name": "skill-tester",
    "slug": "skill-tester",
    "version": "1.0.0",
    "description": "Validate, test, and score the quality of skills within the claude-skills ecosystem. Comprehensive meta-skill: structure validation, Python script testing (syntax + imports + runtime + output format), multi-dimensional quality scoring with letter grades and tier classification (BASIC/STANDARD/POWERFUL). Use when authoring a new skill, auditing existing skills for tier promotion, setting up pre-commit hooks for skill quality, or integrating skill QA into CI.",
    "category": "Development",
    "tags": [
      "ai-agents",
      "testing",
      "python"
    ],
    "platforms": [
      "chatgpt",
      "claude-code",
      "cursor"
    ],
    "tested_environment": "Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and compatible agents",
    "license": "MIT"
  },
  "prompts": {
    "system": "# Skill Tester\r\n\r\n**Tier**: POWERFUL · **Category**: Engineering Quality Assurance · **Dependencies**: None (Python stdlib only)\r\n\r\nMeta-skill that validates, tests, and scores skills in this repository. Four tools, run from the **repo root** with full paths:\r\n\r\n1. **`scripts/skill_validator.py`** — structure + documentation compliance\r\n2. **`scripts/script_tester.py`** — Python script syntax/imports/runtime/output testing\r\n3. **`scripts/quality_scorer.py`** — multi-dimensional scoring with letter grade\r\n4. **`scripts/security_scorer.py`** — security posture scoring (also available via `quality_scorer.py --include-security`)\r\n\r\n> **Scope note:** this skill's tier line-count minimums measure *legacy* skills. For authoring *new* skills, `engineering/write-a-skill` (SKILL.md under ~100 lines, Matt Pocock doctrine) is the binding standard — do not pad a new skill to satisfy a tier minimum here.\r\n\r\n## Quick Start (exact, runnable from repo root)\r\n\r\n```bash\r\n# 1. Validate structure (exit non-zero on failure — usable as a gate)\r\npython3 engineering/skills/skill-tester/scripts/skill_validator.py engineering/skills/self-eval --json\r\n\r\n# 2. Test the skill's Python scripts (30s default timeout per script)\r\npython3 engineering/skills/skill-tester/scripts/script_tester.py engineering/skills/self-eval --json\r\n\r\n# 3. Score quality (fail CI below threshold with --minimum-score)\r\npython3 engineering/skills/skill-tester/scripts/quality_scorer.py engineering/skills/self-eval --json --detailed --minimum-score 75\r\n```\r\n\r\nConsume the JSON: validator emits `overall_score`, `compliance_level`, per-check `checks{}`; scorer emits `overall_score`, `letter_grade`, `tier_recommendation`, `dimensions`, and an `improvement_roadmap` — work the roadmap top-down, then re-run until the target score is met.\r\n\r\nFor repo-wide auditing prefer `scripts/audit_skills.py` at the repo root (wraps the write-a-skill checklist runner across all skills).\r\n\r\n## What Each Tool Checks\r\n\r\n### skill_validator.py\r\n- SKILL.md frontmatter parsing, required sections, minimum line counts per tier (`--tier BASIC|STANDARD|POWERFUL`)\r\n- Required structure: SKILL.md, README.md, scripts/, references/, assets/, expected_outputs/\r\n- Python scripts: argparse present, stdlib-only imports\r\n\r\n### script_tester.py\r\n- AST-based syntax validation; import analysis (flags external dependencies)\r\n- Controlled execution with timeout protection (`--timeout`, default 30s)\r\n- `--help` functionality verification; sample-data runs compared against expected_outputs/\r\n\r\n### quality_scorer.py\r\nFour dimensions, 25% each: **Documentation** (depth, examples, references), **Code Quality** (complexity, error handling, output consistency), **Completeness** (required dirs, sample data, expected outputs), **Usability** (help text, example clarity). Outputs 0-100 + A-F grade + tier recommendation.\r\n\r\n## Tier Classification\r\n\r\n| Tier | SKILL.md | Scripts | CLI surface |\r\n|---|---|---|---|\r\n| BASIC | ≥ 100 lines | 1 (100-300 LOC) | basic argparse |\r\n| STANDARD | ≥ 200 lines | 1-2 (300-500 LOC) | subcommands, JSON + text output |\r\n| POWERFUL | ≥ 300 lines | 2-3 (500-800 LOC) | multiple modes, CI integration |\r\n\r\n(Advisory for legacy skills; new skills follow write-a-skill — see scope note above.)\r\n\r\n## CI Integration\r\n\r\n```yaml\r\n# GitHub Actions: gate changed skills\r\n- name: \"validate-changed-skills\"\r\n  run: |\r\n    for skill in $changed_skills; do\r\n      python3 engineering/skills/skill-tester/scripts/skill_validator.py \"$skill\" --json\r\n      python3 engineering/skills/skill-tester/scripts/script_tester.py \"$skill\"\r\n      python3 engineering/skills/skill-tester/scripts/quality_scorer.py \"$skill\" --minimum-score 75\r\n    done\r\n```\r\n\r\nPre-commit hook: run the validator on the staged skill directory and block the commit on non-zero exit.\r\n\r\n## Verification Loop\r\n\r\nA skill \"passes\" when, in one run from repo root:\r\n\r\n1. `skill_validator.py <skill> --json` exits 0,\r\n2. `script_tester.py <skill>` reports all scripts passing, and\r\n3. `quality_scorer.py <skill> --minimum-score <target>` exits 0.\r\n\r\nIf any step fails, apply the top `improvement_roadmap` item and re-run all three — never report a partial pass.\r\n\r\n## Troubleshooting\r\n\r\n- **Timeout errors** → raise `--timeout` or optimize the script under test\r\n- **Import failures** → external deps detected; stdlib-only is the repo policy\r\n- **Tier misclassification** → check line counts/LOC against the tier table; remember the write-a-skill exception for new skills\r\n\r\nReferences: `references/` holds the structure specification, tier requirements matrix, and scoring rubric the tools implement.",
    "user_template": ""
  },
  "variables": [],
  "install": {
    "steps": [
      {
        "title": "Download & unzip",
        "detail": "Download the .zip package from SkillVault and unzip it. The skill-tester/ folder contains SKILL.md plus any companion scripts, references and assets."
      },
      {
        "title": "Copy into your agent skills directory",
        "detail": "Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code), .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or reload the session."
      },
      {
        "title": "Use it",
        "detail": "Describe the task in natural language; the agent loads the skill automatically when the request matches the skill description."
      }
    ]
  },
  "usage": {
    "steps": []
  },
  "metadata": {
    "author": "SkillVault",
    "source": "https://skillvault.top/skills/skill-tester",
    "exported_at": "2026-11-03T07:44:22.823Z"
  }
}

format: aiskill/v1
skill:
  name: skill-tester
  slug: skill-tester
  version: 1.0.0
  description: >-
    Validate, test, and score the quality of skills within the claude-skills ecosystem.
    Comprehensive meta-skill: structure validation, Python script testing (syntax + imports +
    runtime + output format), multi-dimensional quality scoring with letter grades and tier
    classification (BASIC/STANDARD/POWERFUL). Use when authoring a new skill, auditing existing
    skills for tier promotion, setting up pre-commit hooks for skill quality, or integrating skill
    QA into CI.
  category: Development
  tags:
    - ai-agents
    - testing
    - python
  platforms:
    - chatgpt
    - claude-code
    - cursor
  tested_environment: >-
    Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and
    compatible agents
  license: MIT
prompts:
  system: "# Skill Tester\r\n\r\n**Tier**: POWERFUL · **Category**: Engineering Quality Assurance · **Dependencies**: None (Python stdlib only)\r\n\r\nMeta-skill that validates, tests, and scores skills in this repository. Four tools, run from the **repo root** with full paths:\r\n\r\n1. **`scripts/skill_validator.py`** — structure + documentation compliance\r\n2. **`scripts/script_tester.py`** — Python script syntax/imports/runtime/output testing\r\n3. **`scripts/quality_scorer.py`** — multi-dimensional scoring with letter grade\r\n4. **`scripts/security_scorer.py`** — security posture scoring (also available via `quality_scorer.py --include-security`)\r\n\r\n> **Scope note:** this skill's tier line-count minimums measure *legacy* skills. For authoring *new* skills, `engineering/write-a-skill` (SKILL.md under ~100 lines, Matt Pocock doctrine) is the binding standard — do not pad a new skill to satisfy a tier minimum here.\r\n\r\n## Quick Start (exact, runnable from repo root)\r\n\r\n```bash\r\n# 1. Validate structure (exit non-zero on failure — usable as a gate)\r\npython3 engineering/skills/skill-tester/scripts/skill_validator.py engineering/skills/self-eval --json\r\n\r\n# 2. Test the skill's Python scripts (30s default timeout per script)\r\npython3 engineering/skills/skill-tester/scripts/script_tester.py engineering/skills/self-eval --json\r\n\r\n# 3. Score quality (fail CI below threshold with --minimum-score)\r\npython3 engineering/skills/skill-tester/scripts/quality_scorer.py engineering/skills/self-eval --json --detailed --minimum-score 75\r\n```\r\n\r\nConsume the JSON: validator emits `overall_score`, `compliance_level`, per-check `checks{}`; scorer emits `overall_score`, `letter_grade`, `tier_recommendation`, `dimensions`, and an `improvement_roadmap` — work the roadmap top-down, then re-run until the target score is met.\r\n\r\nFor repo-wide auditing prefer `scripts/audit_skills.py` at the repo root (wraps the write-a-skill checklist runner across all skills).\r\n\r\n## What Each Tool Checks\r\n\r\n### skill_validator.py\r\n- SKILL.md frontmatter parsing, required sections, minimum line counts per tier (`--tier BASIC|STANDARD|POWERFUL`)\r\n- Required structure: SKILL.md, README.md, scripts/, references/, assets/, expected_outputs/\r\n- Python scripts: argparse present, stdlib-only imports\r\n\r\n### script_tester.py\r\n- AST-based syntax validation; import analysis (flags external dependencies)\r\n- Controlled execution with timeout protection (`--timeout`, default 30s)\r\n- `--help` functionality verification; sample-data runs compared against expected_outputs/\r\n\r\n### quality_scorer.py\r\nFour dimensions, 25% each: **Documentation** (depth, examples, references), **Code Quality** (complexity, error handling, output consistency), **Completeness** (required dirs, sample data, expected outputs), **Usability** (help text, example clarity). Outputs 0-100 + A-F grade + tier recommendation.\r\n\r\n## Tier Classification\r\n\r\n| Tier | SKILL.md | Scripts | CLI surface |\r\n|---|---|---|---|\r\n| BASIC | ≥ 100 lines | 1 (100-300 LOC) | basic argparse |\r\n| STANDARD | ≥ 200 lines | 1-2 (300-500 LOC) | subcommands, JSON + text output |\r\n| POWERFUL | ≥ 300 lines | 2-3 (500-800 LOC) | multiple modes, CI integration |\r\n\r\n(Advisory for legacy skills; new skills follow write-a-skill — see scope note above.)\r\n\r\n## CI Integration\r\n\r\n```yaml\r\n# GitHub Actions: gate changed skills\r\n- name: \"validate-changed-skills\"\r\n  run: |\r\n    for skill in $changed_skills; do\r\n      python3 engineering/skills/skill-tester/scripts/skill_validator.py \"$skill\" --json\r\n      python3 engineering/skills/skill-tester/scripts/script_tester.py \"$skill\"\r\n      python3 engineering/skills/skill-tester/scripts/quality_scorer.py \"$skill\" --minimum-score 75\r\n    done\r\n```\r\n\r\nPre-commit hook: run the validator on the staged skill directory and block the commit on non-zero exit.\r\n\r\n## Verification Loop\r\n\r\nA skill \"passes\" when, in one run from repo root:\r\n\r\n1. `skill_validator.py <skill> --json` exits 0,\r\n2. `script_tester.py <skill>` reports all scripts passing, and\r\n3. `quality_scorer.py <skill> --minimum-score <target>` exits 0.\r\n\r\nIf any step fails, apply the top `improvement_roadmap` item and re-run all three — never report a partial pass.\r\n\r\n## Troubleshooting\r\n\r\n- **Timeout errors** → raise `--timeout` or optimize the script under test\r\n- **Import failures** → external deps detected; stdlib-only is the repo policy\r\n- **Tier misclassification** → check line counts/LOC against the tier table; remember the write-a-skill exception for new skills\r\n\r\nReferences: `references/` holds the structure specification, tier requirements matrix, and scoring rubric the tools implement."
  user_template: ""
variables: []
install:
  steps:
    - title: Download & unzip
      detail: >-
        Download the .zip package from SkillVault and unzip it. The skill-tester/ folder contains
        SKILL.md plus any companion scripts, references and assets.
    - title: Copy into your agent skills directory
      detail: >-
        Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code),
        .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or
        reload the session.
    - title: Use it
      detail: >-
        Describe the task in natural language; the agent loads the skill automatically when the
        request matches the skill description.
usage:
  steps: []
metadata:
  author: SkillVault
  source: https://skillvault.top/skills/skill-tester
  exported_at: "2026-11-03T07:44:22.823Z"