experiment-designer

# Experiment Designer

Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.

## When To Use

Use this skill for:
- A/B and multivariate experiment planning
- Hypothesis writing and success criteria definition
- Sample size and minimum detectable effect planning
- Experiment prioritization with ICE scoring
- Reading statistical output for product decisions

## Core Workflow

1. Write hypothesis in If/Then/Because format
- If we change `[intervention]`
- Then `[metric]` will change by `[expected direction/magnitude]`
- Because `[behavioral mechanism]`

2. Define metrics before running test
- Primary metric: single decision metric
- Guardrail metrics: quality/risk protection
- Secondary metrics: diagnostics only

3. Estimate sample size
- Baseline conversion or baseline mean
- Minimum detectable effect (MDE)
- Significance level (alpha) and power

Use:
```bash
python3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute
```

4. Prioritize experiments with ICE
- Impact: potential upside
- Confidence: evidence quality
- Ease: cost/speed/complexity

ICE Score = (Impact * Confidence * Ease) / 10

5. Launch with stopping rules
- Decide fixed sample size or fixed duration in advance
- Avoid repeated peeking without proper method
- Monitor guardrails continuously

6. Interpret results
- Statistical significance is not business significance
- Compare point estimate + confidence interval to decision threshold
- Investigate novelty effects and segment heterogeneity

## Hypothesis Quality Checklist

- [ ] Contains explicit intervention and audience
- [ ] Specifies measurable metric change
- [ ] States plausible causal reason
- [ ] Includes expected minimum effect
- [ ] Defines failure condition

## Common Experiment Pitfalls

- Underpowered tests leading to false negatives
- Running too many simultaneous changes without isolation
- Changing targeting or implementation mid-test
- Stopping early on random spikes
- Ignoring sample ratio mismatch and instrumentation drift
- Declaring success from p-value without effect-size context

## Statistical Interpretation Guardrails

- p-value < alpha indicates evidence against null, not guaranteed truth.
- Confidence interval crossing zero/no-effect means uncertain directional claim.
- Wide intervals imply low precision even when significant.
- Use practical significance thresholds tied to business impact.

See:
- `references/experiment-playbook.md`
- `references/statistics-reference.md`

## Tooling

### `scripts/sample_size_calculator.py`

Computes required sample size (per variant and total) from:
- baseline rate
- MDE (absolute or relative)
- significance level (alpha)
- statistical power

Example:
```bash
python3 scripts/sample_size_calculator.py \
  --baseline-rate 0.10 \
  --mde 0.015 \
  --mde-type absolute \
  --alpha 0.05 \
  --power 0.8
```

{
  "format": "aiskill/v1",
  "skill": {
    "name": "experiment-designer",
    "slug": "experiment-designer",
    "version": "1.0.0",
    "description": "Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.",
    "category": "Business",
    "tags": [
      "product-business",
      "content"
    ],
    "platforms": [
      "chatgpt",
      "claude-code",
      "cursor"
    ],
    "tested_environment": "Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and compatible agents",
    "license": "MIT"
  },
  "prompts": {
    "system": "# Experiment Designer\r\n\r\nDesign, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.\r\n\r\n## When To Use\r\n\r\nUse this skill for:\r\n- A/B and multivariate experiment planning\r\n- Hypothesis writing and success criteria definition\r\n- Sample size and minimum detectable effect planning\r\n- Experiment prioritization with ICE scoring\r\n- Reading statistical output for product decisions\r\n\r\n## Core Workflow\r\n\r\n1. Write hypothesis in If/Then/Because format\r\n- If we change `[intervention]`\r\n- Then `[metric]` will change by `[expected direction/magnitude]`\r\n- Because `[behavioral mechanism]`\r\n\r\n2. Define metrics before running test\r\n- Primary metric: single decision metric\r\n- Guardrail metrics: quality/risk protection\r\n- Secondary metrics: diagnostics only\r\n\r\n3. Estimate sample size\r\n- Baseline conversion or baseline mean\r\n- Minimum detectable effect (MDE)\r\n- Significance level (alpha) and power\r\n\r\nUse:\r\n```bash\r\npython3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute\r\n```\r\n\r\n4. Prioritize experiments with ICE\r\n- Impact: potential upside\r\n- Confidence: evidence quality\r\n- Ease: cost/speed/complexity\r\n\r\nICE Score = (Impact * Confidence * Ease) / 10\r\n\r\n5. Launch with stopping rules\r\n- Decide fixed sample size or fixed duration in advance\r\n- Avoid repeated peeking without proper method\r\n- Monitor guardrails continuously\r\n\r\n6. Interpret results\r\n- Statistical significance is not business significance\r\n- Compare point estimate + confidence interval to decision threshold\r\n- Investigate novelty effects and segment heterogeneity\r\n\r\n## Hypothesis Quality Checklist\r\n\r\n- [ ] Contains explicit intervention and audience\r\n- [ ] Specifies measurable metric change\r\n- [ ] States plausible causal reason\r\n- [ ] Includes expected minimum effect\r\n- [ ] Defines failure condition\r\n\r\n## Common Experiment Pitfalls\r\n\r\n- Underpowered tests leading to false negatives\r\n- Running too many simultaneous changes without isolation\r\n- Changing targeting or implementation mid-test\r\n- Stopping early on random spikes\r\n- Ignoring sample ratio mismatch and instrumentation drift\r\n- Declaring success from p-value without effect-size context\r\n\r\n## Statistical Interpretation Guardrails\r\n\r\n- p-value < alpha indicates evidence against null, not guaranteed truth.\r\n- Confidence interval crossing zero/no-effect means uncertain directional claim.\r\n- Wide intervals imply low precision even when significant.\r\n- Use practical significance thresholds tied to business impact.\r\n\r\nSee:\r\n- `references/experiment-playbook.md`\r\n- `references/statistics-reference.md`\r\n\r\n## Tooling\r\n\r\n### `scripts/sample_size_calculator.py`\r\n\r\nComputes required sample size (per variant and total) from:\r\n- baseline rate\r\n- MDE (absolute or relative)\r\n- significance level (alpha)\r\n- statistical power\r\n\r\nExample:\r\n```bash\r\npython3 scripts/sample_size_calculator.py \\\r\n  --baseline-rate 0.10 \\\r\n  --mde 0.015 \\\r\n  --mde-type absolute \\\r\n  --alpha 0.05 \\\r\n  --power 0.8\r\n```",
    "user_template": ""
  },
  "variables": [],
  "install": {
    "steps": [
      {
        "title": "Download & unzip",
        "detail": "Download the .zip package from SkillVault and unzip it. The experiment-designer/ folder contains SKILL.md plus any companion scripts, references and assets."
      },
      {
        "title": "Copy into your agent skills directory",
        "detail": "Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code), .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or reload the session."
      },
      {
        "title": "Use it",
        "detail": "Describe the task in natural language; the agent loads the skill automatically when the request matches the skill description."
      }
    ]
  },
  "usage": {
    "steps": []
  },
  "metadata": {
    "author": "SkillVault",
    "source": "https://skillvault.top/skills/experiment-designer",
    "exported_at": "2026-11-03T07:42:38.714Z"
  }
}

format: aiskill/v1
skill:
  name: experiment-designer
  slug: experiment-designer
  version: 1.0.0
  description: >-
    Use when planning product experiments, writing testable hypotheses, estimating sample size,
    prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
  category: Business
  tags:
    - product-business
    - content
  platforms:
    - chatgpt
    - claude-code
    - cursor
  tested_environment: >-
    Agent Skills convention (SKILL.md) — works with Claude Code, Cursor, ChatGPT/Codex and
    compatible agents
  license: MIT
prompts:
  system: "# Experiment Designer\r\n\r\nDesign, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions.\r\n\r\n## When To Use\r\n\r\nUse this skill for:\r\n- A/B and multivariate experiment planning\r\n- Hypothesis writing and success criteria definition\r\n- Sample size and minimum detectable effect planning\r\n- Experiment prioritization with ICE scoring\r\n- Reading statistical output for product decisions\r\n\r\n## Core Workflow\r\n\r\n1. Write hypothesis in If/Then/Because format\r\n- If we change `[intervention]`\r\n- Then `[metric]` will change by `[expected direction/magnitude]`\r\n- Because `[behavioral mechanism]`\r\n\r\n2. Define metrics before running test\r\n- Primary metric: single decision metric\r\n- Guardrail metrics: quality/risk protection\r\n- Secondary metrics: diagnostics only\r\n\r\n3. Estimate sample size\r\n- Baseline conversion or baseline mean\r\n- Minimum detectable effect (MDE)\r\n- Significance level (alpha) and power\r\n\r\nUse:\r\n```bash\r\npython3 scripts/sample_size_calculator.py --baseline-rate 0.12 --mde 0.02 --mde-type absolute\r\n```\r\n\r\n4. Prioritize experiments with ICE\r\n- Impact: potential upside\r\n- Confidence: evidence quality\r\n- Ease: cost/speed/complexity\r\n\r\nICE Score = (Impact * Confidence * Ease) / 10\r\n\r\n5. Launch with stopping rules\r\n- Decide fixed sample size or fixed duration in advance\r\n- Avoid repeated peeking without proper method\r\n- Monitor guardrails continuously\r\n\r\n6. Interpret results\r\n- Statistical significance is not business significance\r\n- Compare point estimate + confidence interval to decision threshold\r\n- Investigate novelty effects and segment heterogeneity\r\n\r\n## Hypothesis Quality Checklist\r\n\r\n- [ ] Contains explicit intervention and audience\r\n- [ ] Specifies measurable metric change\r\n- [ ] States plausible causal reason\r\n- [ ] Includes expected minimum effect\r\n- [ ] Defines failure condition\r\n\r\n## Common Experiment Pitfalls\r\n\r\n- Underpowered tests leading to false negatives\r\n- Running too many simultaneous changes without isolation\r\n- Changing targeting or implementation mid-test\r\n- Stopping early on random spikes\r\n- Ignoring sample ratio mismatch and instrumentation drift\r\n- Declaring success from p-value without effect-size context\r\n\r\n## Statistical Interpretation Guardrails\r\n\r\n- p-value < alpha indicates evidence against null, not guaranteed truth.\r\n- Confidence interval crossing zero/no-effect means uncertain directional claim.\r\n- Wide intervals imply low precision even when significant.\r\n- Use practical significance thresholds tied to business impact.\r\n\r\nSee:\r\n- `references/experiment-playbook.md`\r\n- `references/statistics-reference.md`\r\n\r\n## Tooling\r\n\r\n### `scripts/sample_size_calculator.py`\r\n\r\nComputes required sample size (per variant and total) from:\r\n- baseline rate\r\n- MDE (absolute or relative)\r\n- significance level (alpha)\r\n- statistical power\r\n\r\nExample:\r\n```bash\r\npython3 scripts/sample_size_calculator.py \\\r\n  --baseline-rate 0.10 \\\r\n  --mde 0.015 \\\r\n  --mde-type absolute \\\r\n  --alpha 0.05 \\\r\n  --power 0.8\r\n```"
  user_template: ""
variables: []
install:
  steps:
    - title: Download & unzip
      detail: >-
        Download the .zip package from SkillVault and unzip it. The experiment-designer/ folder
        contains SKILL.md plus any companion scripts, references and assets.
    - title: Copy into your agent skills directory
      detail: >-
        Move the folder into your agent's skills directory — e.g. ~/.claude/skills/ (Claude Code),
        .codex/skills/, .cursor/skills/ or your project's .claude/skills/ folder — then restart or
        reload the session.
    - title: Use it
      detail: >-
        Describe the task in natural language; the agent loads the skill automatically when the
        request matches the skill description.
usage:
  steps: []
metadata:
  author: SkillVault
  source: https://skillvault.top/skills/experiment-designer
  exported_at: "2026-11-03T07:42:38.714Z"