AI-powered tooling: GitHub Action, generate command, evals + leaderboard (#41)

Three features riding 2026 trends (agentic CI, codegen, evals), sharing one dependency-free Anthropic client (bin/lib/anthropic.mjs). 1. GitHub Action (action/) — run any skill in a consumer repo's CI: uses: mohitagw15856/pm-claude-skills/action@main. Composite action + run.mjs (loads the bundled SKILL.md, calls the API, exposes result as a step output / file). Docs with auto-PR-description example. 2. generate command — `npx pm-claude-skills generate --from <url|file>` turns a team's docs into a SKILL.md following the authoring standard (bin/generate.mjs, wired into the CLI; needs ANTHROPIC_API_KEY). 3. Skill evals + Leaderboard — evals/run-evals.mjs runs each case across models and scores output with an LLM judge (structure/completeness/usefulness/ grounding); scripts/build-leaderboard.mjs renders web/leaderboard.html (built in the Pages deploy, falls back to clearly-labelled example data). Linked from README, catalog, and playground. Offline-testable parts verified (prompt building, skill loading, graceful errors, leaderboard render). SkillCheck/audit/exports all green. Claude-Session: https://claude.ai/code/session_016JWn5jRD5tcEFKrubjQ6Px Co-authored-by: Claude <noreply@anthropic.com>
2026-06-18 08:37:40 +01:00
parent 288a340dbe
commit 51bf4be52f
17 changed files with 644 additions and 2 deletions
@@ -0,0 +1,51 @@
+// Minimal, dependency-free Anthropic Messages API client (Node 18+ global fetch).
+// Shared by the GitHub Action runner, the eval harness, and skill generation.
+// No SDK, no install — just a thin POST wrapper.
+
+const API_URL = 'https://api.anthropic.com/v1/messages';
+
+/**
+ * Call the Anthropic Messages API and return the concatenated text output.
+ * @param {object} o
+ * @param {string} o.apiKey  - Anthropic API key.
+ * @param {string} [o.model] - Model id (default claude-sonnet-4-6).
+ * @param {string} [o.system]- System prompt.
+ * @param {Array}  o.messages- [{role, content}] messages.
+ * @param {number} [o.maxTokens]
+ * @returns {Promise<string>}
+ */
+export async function complete({ apiKey, model = 'claude-sonnet-4-6', system, messages, maxTokens = 4096 }) {
+  if (!apiKey) throw new Error('Missing Anthropic API key (set ANTHROPIC_API_KEY).');
+  const res = await fetch(API_URL, {
+    method: 'POST',
+    headers: {
+      'content-type': 'application/json',
+      'x-api-key': apiKey,
+      'anthropic-version': '2023-06-01',
+    },
+    body: JSON.stringify({ model, max_tokens: maxTokens, ...(system ? { system } : {}), messages }),
+  });
+  if (!res.ok) {
+    const body = await res.text().catch(() => '');
+    throw new Error(`Anthropic API ${res.status}: ${body.slice(0, 500)}`);
+  }
+  const data = await res.json();
+  return (data.content || []).map((c) => c.text || '').join('').trim();
+}
+
+/** Parse "name: value" YAML-ish frontmatter + body from a SKILL.md string. */
+export function parseSkill(text) {
+  const m = text.match(/^---\n([\s\S]*?)\n---\n?([\s\S]*)$/);
+  const meta = {};
+  if (m) {
+    for (const line of m[1].split('\n')) {
+      const kv = line.match(/^(\w[\w-]*):\s*(.*)$/);
+      if (kv) {
+        let v = kv[2].trim();
+        if ((v.startsWith('"') && v.endsWith('"')) || (v.startsWith("'") && v.endsWith("'"))) v = v.slice(1, -1);
+        meta[kv[1]] = v;
+      }
+    }
+  }
+  return { meta, body: m ? m[2].trim() : text.trim() };
+}