AI-powered tooling: GitHub Action, generate command, evals + leaderboard (#41)

Three features riding 2026 trends (agentic CI, codegen, evals), sharing one dependency-free Anthropic client (bin/lib/anthropic.mjs). 1. GitHub Action (action/) — run any skill in a consumer repo's CI: uses: mohitagw15856/pm-claude-skills/action@main. Composite action + run.mjs (loads the bundled SKILL.md, calls the API, exposes result as a step output / file). Docs with auto-PR-description example. 2. generate command — `npx pm-claude-skills generate --from <url|file>` turns a team's docs into a SKILL.md following the authoring standard (bin/generate.mjs, wired into the CLI; needs ANTHROPIC_API_KEY). 3. Skill evals + Leaderboard — evals/run-evals.mjs runs each case across models and scores output with an LLM judge (structure/completeness/usefulness/ grounding); scripts/build-leaderboard.mjs renders web/leaderboard.html (built in the Pages deploy, falls back to clearly-labelled example data). Linked from README, catalog, and playground. Offline-testable parts verified (prompt building, skill loading, graceful errors, leaderboard render). SkillCheck/audit/exports all green. Claude-Session: https://claude.ai/code/session_016JWn5jRD5tcEFKrubjQ6Px Co-authored-by: Claude <noreply@anthropic.com>
2026-06-18 08:37:40 +01:00
parent 288a340dbe
commit 51bf4be52f
17 changed files with 644 additions and 2 deletions
@@ -0,0 +1,65 @@
+# PM Skills — GitHub Action
+
+Run any skill from this library inside **your** repo's CI. Turn the library's frameworks
+into automation: auto-write PR descriptions, generate release notes and changelogs, or run
+a code-review checklist — on every push or PR.
+
+```yaml
+- uses: mohitagw15856/pm-claude-skills/action@main
+  with:
+    skill: pr-description-writer
+    input: ${{ steps.diff.outputs.text }}
+    api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+```
+
+## Inputs
+
+| Input | Required | Description |
+|---|---|---|
+| `skill` | ✅ | Skill name, e.g. `pr-description-writer`, `changelog-generator`, `code-review-checklist`. |
+| `input` | — | The text/context to run the skill on. |
+| `input_file` | — | Read input from a file instead of `input`. |
+| `api_key` | ✅ | Anthropic API key (store as a repo secret). |
+| `model` | — | Model id (default `claude-sonnet-4-6`). |
+| `output_file` | — | Also write the result to this file. |
+
+**Output:** `result` — the skill's output (use `output_file` for long, multi-line results).
+
+## Example — auto-write a PR description
+
+```yaml
+name: PR description
+on: { pull_request: { types: [opened] } }
+permissions: { contents: read, pull-requests: write }
+jobs:
+  describe:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with: { fetch-depth: 0 }
+      - id: diff
+        run: |
+          echo "text<<EOF" >> "$GITHUB_OUTPUT"
+          git diff origin/${{ github.base_ref }}...HEAD --stat >> "$GITHUB_OUTPUT"
+          echo "EOF" >> "$GITHUB_OUTPUT"
+      - id: skill
+        uses: mohitagw15856/pm-claude-skills/action@main
+        with:
+          skill: pr-description-writer
+          input: ${{ steps.diff.outputs.text }}
+          api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+      - uses: actions/github-script@v7
+        with:
+          script: |
+            github.rest.pulls.update({ owner: context.repo.owner, repo: context.repo.repo,
+              pull_number: context.issue.number, body: process.env.BODY })
+        env: { BODY: ${{ steps.skill.outputs.result }} }
+```
+
+## Other ideas
+
+- `skill: changelog-generator` from `git log` → write `CHANGELOG.md`.
+- `skill: release-notes` on tag push → set the GitHub Release body.
+- `skill: code-review-checklist` → post a review checklist as a PR comment.
+
+Pin to a release tag (e.g. `@v19`) for stability once you've tried `@main`.
@@ -0,0 +1,51 @@
+name: 'PM Skills — Run a Skill'
+description: 'Run any pm-claude-skills SKILL.md in CI — auto PR descriptions, changelogs, release notes, code-review checklists, and more.'
+author: 'Mohit Aggarwal'
+branding:
+  icon: 'cpu'
+  color: 'purple'
+
+inputs:
+  skill:
+    description: 'Skill name to run (e.g. pr-description-writer, changelog-generator, code-review-checklist).'
+    required: true
+  input:
+    description: 'The input/context text the skill should work on.'
+    required: false
+  input_file:
+    description: 'Read the input from this file instead of the `input` string.'
+    required: false
+  api_key:
+    description: 'Anthropic API key (store it as a secret).'
+    required: true
+  model:
+    description: 'Claude model id.'
+    required: false
+    default: 'claude-sonnet-4-6'
+  output_file:
+    description: 'If set, also write the result to this file.'
+    required: false
+  max_tokens:
+    description: 'Max output tokens.'
+    required: false
+    default: '4096'
+
+outputs:
+  result:
+    description: 'The skill output (also use output_file for multi-line results).'
+    value: ${{ steps.run.outputs.result }}
+
+runs:
+  using: composite
+  steps:
+    - id: run
+      shell: bash
+      run: node "$GITHUB_ACTION_PATH/run.mjs"
+      env:
+        INPUT_SKILL: ${{ inputs.skill }}
+        INPUT_INPUT: ${{ inputs.input }}
+        INPUT_INPUT_FILE: ${{ inputs.input_file }}
+        INPUT_API_KEY: ${{ inputs.api_key }}
+        INPUT_MODEL: ${{ inputs.model }}
+        INPUT_OUTPUT_FILE: ${{ inputs.output_file }}
+        INPUT_MAX_TOKENS: ${{ inputs.max_tokens }}
@@ -0,0 +1,58 @@
+#!/usr/bin/env node
+// Runner for the pm-skills GitHub Action. Loads a bundled SKILL.md, runs it on
+// the provided input via the Anthropic API, and exposes the result as a step
+// output (and optionally a file). Inputs arrive as INPUT_* env vars.
+import { readFileSync, existsSync, writeFileSync, appendFileSync } from 'node:fs';
+import { join, dirname } from 'node:path';
+import { fileURLToPath, pathToFileURL } from 'node:url';
+import { complete, parseSkill } from '../bin/lib/anthropic.mjs';
+
+const ACTION_DIR = dirname(fileURLToPath(import.meta.url));
+const REPO_ROOT = join(ACTION_DIR, '..');
+
+const inp = (name, def = '') => (process.env[`INPUT_${name.toUpperCase()}`] ?? def).trim();
+
+// Pure: assemble the system prompt + user message for a skill run (testable offline).
+export function buildRequest(skillBody, userInput) {
+  const system = skillBody +
+    '\n\n---\nExecute this skill now on the input below and produce the complete output. ' +
+    'Do not ask follow-up questions — work with what is given and note any reasonable assumptions. ' +
+    'Output only the finished artifact (no preamble).';
+  return { system, messages: [{ role: 'user', content: userInput }] };
+}
+
+async function main() {
+  const skill = inp('skill');
+  if (!skill) throw new Error('Input `skill` is required.');
+  const apiKey = inp('api_key') || process.env.ANTHROPIC_API_KEY || '';
+  const model = inp('model', 'claude-sonnet-4-6');
+  const maxTokens = parseInt(inp('max_tokens', '4096'), 10) || 4096;
+
+  let input = inp('input');
+  const inputFile = inp('input_file');
+  if (!input && inputFile && existsSync(inputFile)) input = readFileSync(inputFile, 'utf8');
+  if (!input) throw new Error('Provide `input` or `input_file`.');
+
+  const skillFile = join(REPO_ROOT, 'skills', skill, 'SKILL.md');
+  if (!existsSync(skillFile)) throw new Error(`Unknown skill "${skill}" (no skills/${skill}/SKILL.md).`);
+  const { body } = parseSkill(readFileSync(skillFile, 'utf8'));
+
+  const { system, messages } = buildRequest(body, input);
+  console.log(`Running skill "${skill}" with ${model}…`);
+  const result = await complete({ apiKey, model, system, messages, maxTokens });
+
+  // Step output (multiline-safe heredoc) + optional file.
+  if (process.env.GITHUB_OUTPUT) {
+    const d = `EOF_${Math.random().toString(36).slice(2)}`;
+    appendFileSync(process.env.GITHUB_OUTPUT, `result<<${d}\n${result}\n${d}\n`);
+  }
+  const outFile = inp('output_file');
+  if (outFile) { writeFileSync(outFile, result + '\n'); console.log(`Wrote ${outFile}`); }
+
+  console.log('\n----- skill output -----\n' + result);
+}
+
+// Run only when executed directly (so tests can import buildRequest).
+if (import.meta.url === pathToFileURL(process.argv[1] || '').href) {
+  main().catch((e) => { console.error(`Error: ${e.message}`); process.exit(1); });
+}