SkillCheck validator, Cursor exports, and per-agent installers (#27)
Three more learnings from alirezarezvani/claude-skills, applied: 1. SkillCheck validator (scripts/skillcheck.mjs) — validates every SKILL.md against the authoring standard (frontmatter, name/folder match, trigger + produces clauses, required headings) plus tier referential integrity. Errors fail CI; --strict fails on warnings too. New skillcheck.yml workflow and a SkillCheck status badge in the README. Current: 0 errors / 14 advisory warnings across 172 skills. 2. Cursor export platform — build-exports.mjs now generates exports/cursor/<bundle>/<skill>/<skill>.mdc rule files. The PLATFORMS registry now supports per-skill filenames (file as a function). 3. Per-agent installers — scripts/install.sh unifies install for claude/hermes/codex/openclaw/cursor (--link, --target, --dry-run, --list). Curl-able one-liners codex-install.sh, openclaw-install.sh, and cursor-install.sh clone the library and install in a single command. README documents the one-line installs and Cursor exports; CHANGELOG and the authoring standard updated. Claude-Session: https://claude.ai/code/session_016JWn5jRD5tcEFKrubjQ6Px Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
mohitagw15856
GitHub
@@ -0,0 +1,103 @@
|
||||
---
|
||||
description: "Extract pixel-level data from an image of a chart or graph and produce a structured data table. Use when asked to extract data from a chart image, transcribe numbers from a graph, digitise a chart, or turn a screenshot of data into a table. Produces a structured table with extracted values, confidence levels, and a reconstructed chart source. Best used with Claude Opus 4.7 or newer for reliable chart data extraction."
|
||||
globs:
|
||||
alwaysApply: false
|
||||
---
|
||||
|
||||
# Chart Data Extractor Skill
|
||||
|
||||
Extracts data from images of charts and graphs — bar charts, line charts, pie charts, scatter plots, and tables in images — producing a structured data table that can be used in spreadsheets or rebuilt in any charting tool. Built to leverage Opus 4.7 pixel-level image analysis capabilities.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
Ask the user for these if not provided:
|
||||
- **The chart image** (upload a screenshot or image file)
|
||||
- **Chart type** (if ambiguous — bar / line / pie / scatter / other)
|
||||
- **What matters most** (approximate trends / precise values / specific data points / categorisation)
|
||||
- **Known axis values** (optional — if the user knows the max/min values to anchor the extraction)
|
||||
|
||||
## Output Structure
|
||||
|
||||
### 1. Chart Identification
|
||||
|
||||
| Attribute | Value |
|
||||
|---|---|
|
||||
| Chart type | [Bar / Line / Pie / Scatter / Area / Other] |
|
||||
| Chart title (if visible) | [Title text] |
|
||||
| X-axis label | [Label + unit] |
|
||||
| Y-axis label | [Label + unit] |
|
||||
| Number of series | N |
|
||||
| Legend categories | [List] |
|
||||
| Data period (if time-based) | [Start — End] |
|
||||
|
||||
### 2. Extracted Data Table
|
||||
|
||||
| [X axis] | [Series 1] | [Series 2] | ... |
|
||||
|---|---|---|---|
|
||||
| [Value] | [Value] | [Value] | |
|
||||
|
||||
### 3. Confidence Levels
|
||||
|
||||
For each data point or series, flag confidence:
|
||||
|
||||
- **High confidence:** data points where the value is clearly readable against gridlines or labels
|
||||
- **Medium confidence:** data points where the value is interpolated between gridlines
|
||||
- **Low confidence:** data points where the value is ambiguous or overlaps with other elements
|
||||
|
||||
Low-confidence points should be explicitly listed — not silently included in the main table.
|
||||
|
||||
### 4. Notable Observations
|
||||
|
||||
Observations that the data itself reveals:
|
||||
- Peak value: [Value, when, in which series]
|
||||
- Lowest value: [Value, when, in which series]
|
||||
- Largest delta between series: [Details]
|
||||
- Any anomalies or outliers visible in the chart
|
||||
|
||||
### 5. Reconstructed Source
|
||||
|
||||
CSV format for direct use:
|
||||
|
||||
```csv
|
||||
[x_axis],[series_1],[series_2]
|
||||
[value],[value],[value]
|
||||
```
|
||||
|
||||
### 6. Assumptions and Caveats
|
||||
|
||||
- Grid resolution: [How precisely values could be read — e.g. "Y-axis has major gridlines every 10 units, minor every 2"]
|
||||
- Interpolation used: [Any values that required estimating between gridlines]
|
||||
- Unclear data: [Anything in the chart that could not be read reliably]
|
||||
- Axis scale: [Linear/logarithmic/etc — note if not obvious]
|
||||
|
||||
### 7. Follow-up Options
|
||||
|
||||
Ask the user which of these they want:
|
||||
- Rebuild the chart in a specified format (Excel formula, Python matplotlib, D3, etc.)
|
||||
- Produce a narrative description of what the chart shows
|
||||
- Compare this data against another chart or source
|
||||
- Flag potentially misleading visual choices in the original (truncated axes, misleading scales, etc.)
|
||||
|
||||
## Quality Checks
|
||||
- [ ] Every extracted number specifies which series it belongs to
|
||||
- [ ] Confidence levels are explicit for ambiguous points
|
||||
- [ ] Low-confidence values are flagged separately, not silently included
|
||||
- [ ] Assumptions about axis scale and interpolation are stated
|
||||
- [ ] CSV output is clean and directly usable
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- [ ] Do not silently include low-confidence data points in the main table — flag them separately so the user knows which values to verify
|
||||
- [ ] Do not assume a linear scale without confirming it — logarithmic axes make extracted values incorrect by orders of magnitude if misread
|
||||
- [ ] Do not report extracted values with false precision — if the chart's Y-axis only shows gridlines every 10 units, a reported value of 37 is invented, not extracted
|
||||
- [ ] Do not omit the assumptions and caveats section — partial image quality, overlapping bars, or unlabelled axes must be disclosed
|
||||
|
||||
## Example Trigger Phrases
|
||||
- "Extract the data from this chart"
|
||||
- "Transcribe the numbers in this graph"
|
||||
- "Turn this chart image into a spreadsheet"
|
||||
- "Digitise this chart so I can rebuild it"
|
||||
- "What are the exact values in this bar chart?"
|
||||
|
||||
## Why This Works Better on Opus 4.7
|
||||
Earlier models struggled with pixel-level data transcription from charts, often hallucinating values or misreading gridline positions. Opus 4.7 uses a higher image resolution (2576px vs 1568px) with coordinates mapping 1:1 to pixels, making chart data extraction reliable for practical use.
|
||||
@@ -0,0 +1,196 @@
|
||||
---
|
||||
description: "Structure a cohort analysis for retention, LTV, or behavioural patterns. Use when asked to run a cohort analysis, analyse retention by cohort, segment users by behaviour over time, or calculate lifetime value by acquisition period. Produces a complete cohort analysis framework with methodology, cohort definitions, retention curves, and prioritised interventions."
|
||||
globs:
|
||||
alwaysApply: false
|
||||
---
|
||||
|
||||
# Cohort Analysis Skill
|
||||
|
||||
This skill produces a structured cohort analysis covering retention curves, LTV estimation, behavioural segmentation, and actionable interventions. Output is ready to present to product leadership or share with growth and data teams.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
Ask the user for these if not provided:
|
||||
- **Analysis goal** (retention improvement / LTV modelling / behavioural segmentation / churn prediction)
|
||||
- **Product or feature being analysed**
|
||||
- **Cohort definition** — what groups users? (acquisition month, signup channel, plan tier, feature adoption)
|
||||
- **Observation window** — how many periods to track? (e.g. 12 months, 8 weeks)
|
||||
- **Key metric** — what are you measuring per cohort? (retention rate, revenue, engagement score, feature usage)
|
||||
- **Available data** — what tables/metrics are available? (paste schema or describe)
|
||||
- **Baseline** — any existing retention benchmarks or goals?
|
||||
|
||||
## Output Structure
|
||||
|
||||
---
|
||||
|
||||
# Cohort Analysis: [Product / Feature]
|
||||
|
||||
**Analysis type:** [Retention / LTV / Behavioural / Churn]
|
||||
**Cohort definition:** [Acquisition month / Signup channel / Plan tier / Feature adoption date]
|
||||
**Observation window:** [X months / weeks]
|
||||
**Primary metric:** [Metric name]
|
||||
**Date prepared:** [Date]
|
||||
|
||||
---
|
||||
|
||||
## 1. Cohort Definitions
|
||||
|
||||
| Cohort | Period | Size | Description |
|
||||
|---|---|---|---|
|
||||
| [Cohort 1] | [Jan 2025] | [N users] | [e.g. Users who signed up in Jan 2025 via organic] |
|
||||
| [Cohort 2] | [Feb 2025] | [N users] | [...] |
|
||||
|
||||
**Cohort logic:**
|
||||
- Cohort entry event: [First sign-up / First purchase / Feature activation]
|
||||
- Cohort exit criteria: [Churned / Downgraded / No activity for 30 days]
|
||||
- Exclusions: [Trial users / Internal test accounts / Users with < X days of data]
|
||||
|
||||
---
|
||||
|
||||
## 2. Retention Curve
|
||||
|
||||
**How to read:** Each cell shows what % of the cohort performed the key metric in period N.
|
||||
|
||||
| Cohort | Period 0 | Period 1 | Period 2 | Period 3 | Period 6 | Period 12 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Jan 2025 | 100% | [X%] | [X%] | [X%] | [X%] | [X%] |
|
||||
| Feb 2025 | 100% | [X%] | [X%] | [X%] | [X%] | [X%] |
|
||||
| [Trend] | — | [↑/↓ vs prior] | [...] | [...] | [...] | [...] |
|
||||
|
||||
**Retention plateau:** [At what period does retention flatten? What % does it flatten at?]
|
||||
|
||||
**Key observations:**
|
||||
- [e.g. Period 1 → Period 2 drop is the largest — average X% churn in first 30 days]
|
||||
- [e.g. Cohorts acquired via [channel] retain X% better at Period 6]
|
||||
- [e.g. Retention has improved from X% → Y% at Period 3 comparing oldest to newest cohort]
|
||||
|
||||
---
|
||||
|
||||
## 3. LTV Projection (if applicable)
|
||||
|
||||
**ARPU per period:** [£/$/€ X per active user per month]
|
||||
**Retention curve used:** [Which cohort or blended average]
|
||||
|
||||
| Period | Retained % | Revenue per user | Cumulative LTV |
|
||||
|---|---|---|---|
|
||||
| Month 1 | [X%] | [£X] | [£X] |
|
||||
| Month 3 | [X%] | [£X] | [£X] |
|
||||
| Month 6 | [X%] | [£X] | [£X] |
|
||||
| Month 12 | [X%] | [£X] | [£X] |
|
||||
|
||||
**Blended LTV:** [£X at 12 months — based on blended retention across cohorts]
|
||||
|
||||
**LTV by segment:**
|
||||
| Segment | LTV (12M) | vs Baseline |
|
||||
|---|---|---|
|
||||
| [Organic] | [£X] | [+X%] |
|
||||
| [Paid] | [£X] | [-X%] |
|
||||
| [Enterprise] | [£X] | [+X%] |
|
||||
|
||||
---
|
||||
|
||||
## 4. Behavioural Segmentation
|
||||
|
||||
Group cohorts by behaviour patterns, not just acquisition date:
|
||||
|
||||
| Segment | Definition | Size | Retention (P6) | LTV (12M) |
|
||||
|---|---|---|---|---|
|
||||
| **Power users** | [Used core feature ≥ 3x/week in first 30 days] | [X%] | [X%] | [£X] |
|
||||
| **Casual users** | [Used 1–2x/week in first 30 days] | [X%] | [X%] | [£X] |
|
||||
| **Dormant** | [Logged in but did not use core feature] | [X%] | [X%] | [£X] |
|
||||
| **Never activated** | [Signed up but never completed onboarding] | [X%] | [X%] | [£X] |
|
||||
|
||||
**Activation threshold insight:** [What action — taken within the first X days — most strongly predicts retention? This is the "aha moment" to optimise for.]
|
||||
|
||||
---
|
||||
|
||||
## 5. Leading Indicators of Churn
|
||||
|
||||
List the signals that appear **before** users churn, so teams can intervene:
|
||||
|
||||
| Signal | How early does it appear? | Churn correlation | Intervention |
|
||||
|---|---|---|---|
|
||||
| [No login for 7 days] | [7 days before churn] | [Strong] | [Re-engagement email sequence] |
|
||||
| [Support ticket with escalation] | [14 days before churn] | [Moderate] | [CSM outreach within 48 hours] |
|
||||
| [Feature usage dropped >50% WoW] | [10 days before churn] | [Strong] | [In-app nudge with use-case tutorial] |
|
||||
|
||||
---
|
||||
|
||||
## 6. Cohort Comparison: What's Changed Over Time
|
||||
|
||||
Compare oldest and newest cohorts to assess whether product improvements are showing up in retention:
|
||||
|
||||
| Metric | [Oldest cohort — e.g. Jan 2024] | [Newest cohort — e.g. Jan 2025] | Change |
|
||||
|---|---|---|---|
|
||||
| Period 1 retention | [X%] | [X%] | [↑/↓ X pp] |
|
||||
| Period 3 retention | [X%] | [X%] | [↑/↓ X pp] |
|
||||
| Activation rate | [X%] | [X%] | [↑/↓ X pp] |
|
||||
| Avg. sessions in first 30 days | [X] | [X] | [↑/↓] |
|
||||
|
||||
**Verdict:** [Are more recent cohorts performing better or worse? What shipped in that period that might explain the change?]
|
||||
|
||||
---
|
||||
|
||||
## 7. Recommendations
|
||||
|
||||
Prioritise by impact on retention curve:
|
||||
|
||||
| # | Recommendation | Target segment | Expected impact | Effort | Priority |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | [e.g. Redesign onboarding to hit activation milestone in day 1, not day 7] | [Never-activated segment] | [+X pp P1 retention] | [Medium] | P1 |
|
||||
| 2 | [e.g. Launch re-engagement sequence at day 7 inactivity trigger] | [Dormant segment] | [+X pp P2 retention] | [Low] | P1 |
|
||||
| 3 | [e.g. Introduce power-user features earlier to accelerate habit formation] | [Casual users] | [+X pp P6 LTV] | [High] | P2 |
|
||||
|
||||
---
|
||||
|
||||
## 8. SQL Reference (if applicable)
|
||||
|
||||
Provide the core cohort query so data teams can replicate or extend the analysis:
|
||||
|
||||
```sql
|
||||
-- Retention cohort query
|
||||
SELECT
|
||||
DATE_TRUNC('month', u.created_at) AS cohort_month,
|
||||
DATE_TRUNC('month', e.event_date) AS activity_month,
|
||||
DATEDIFF('month', u.created_at, e.event_date) AS period,
|
||||
COUNT(DISTINCT e.user_id) AS retained_users,
|
||||
COUNT(DISTINCT c.user_id) AS cohort_size,
|
||||
ROUND(COUNT(DISTINCT e.user_id) * 100.0 / COUNT(DISTINCT c.user_id), 1) AS retention_rate
|
||||
FROM users u
|
||||
JOIN events e ON u.user_id = e.user_id
|
||||
JOIN (
|
||||
SELECT user_id, DATE_TRUNC('month', created_at) AS cohort_month
|
||||
FROM users
|
||||
WHERE created_at >= '[start_date]'
|
||||
) c ON u.user_id = c.user_id AND DATE_TRUNC('month', u.created_at) = c.cohort_month
|
||||
WHERE e.event_type = '[key_retention_event]'
|
||||
GROUP BY 1, 2, 3
|
||||
ORDER BY 1, 3;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- [ ] Cohort definition is unambiguous — the same user cannot appear in two cohorts
|
||||
- [ ] Retention curve shows a clear plateau, or the analysis notes that the window is too short to see one
|
||||
- [ ] LTV projection uses observed retention, not assumed
|
||||
- [ ] Behavioural segments are mutually exclusive and exhaustive
|
||||
- [ ] Recommendations are tied to specific cohort or segment findings — not generic growth advice
|
||||
- [ ] Leading indicators are observable in production data, not just in theory
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- [ ] Do not allow the same user to appear in multiple cohorts — overlapping cohorts produce retention numbers that cannot be compared or acted upon
|
||||
- [ ] Do not assume assumed ARPU in LTV projections — use observed revenue per retained user per period, not a blended average that hides segment differences
|
||||
- [ ] Do not draw conclusions from cohorts too small to be statistically meaningful — flag minimum cohort size thresholds and note when a cohort is too small to trust
|
||||
- [ ] Do not conflate retention rate with engagement rate — a user who logs in but does not complete the key retention event is not retained by the definition used
|
||||
- [ ] Do not make recommendations without connecting them to specific cohort or segment findings — generic growth advice that could apply to any product adds no value
|
||||
|
||||
## Example Trigger Phrases
|
||||
|
||||
- "Run a cohort analysis for our SaaS product"
|
||||
- "Analyse retention by acquisition month for the last 12 cohorts"
|
||||
- "What's the LTV of users who came via paid vs organic?"
|
||||
- "Build a cohort retention model showing period 0 through period 12"
|
||||
- "Segment users by behaviour and show me which group retains best"
|
||||
@@ -0,0 +1,131 @@
|
||||
---
|
||||
description: "Convert a business question into a complete dashboard specification. Use when asked to design a dashboard, create a dashboard spec or brief, plan a BI report, or define what charts and metrics a dashboard should include. Produces a structured spec with metrics, dimensions, chart types, filters, and layout guidance."
|
||||
globs:
|
||||
alwaysApply: false
|
||||
---
|
||||
|
||||
# Dashboard Brief Skill
|
||||
|
||||
This skill converts a business question or monitoring need into a complete, implementation-ready dashboard specification. The output gives a data engineer or BI developer everything they need to build without a follow-up meeting.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
Ask the user for these if not provided:
|
||||
- **The business question this dashboard should answer** (e.g. "How is our activation funnel performing this week?")
|
||||
- **Primary audience** (exec / product team / operations / customer success / engineering)
|
||||
- **Refresh cadence** (real-time / hourly / daily / weekly)
|
||||
- **Data sources available** (e.g. Postgres, BigQuery, Mixpanel, Salesforce, Jira)
|
||||
- **BI tool being used** (Looker / Metabase / Tableau / Power BI / Grafana / Custom / Unknown)
|
||||
|
||||
## Output Structure
|
||||
|
||||
---
|
||||
|
||||
# Dashboard Brief: [Dashboard Name]
|
||||
|
||||
**Business Question:** [The question this dashboard answers — verbatim from inputs or refined]
|
||||
**Audience:** [Who uses this]
|
||||
**Refresh Rate:** [Real-time / Hourly / Daily / Weekly]
|
||||
**Data Sources:** [List]
|
||||
**BI Tool:** [Tool or Unknown]
|
||||
|
||||
---
|
||||
|
||||
## Section 1: Key Metrics (KPI Cards)
|
||||
|
||||
List the headline numbers that should appear at the top of the dashboard as KPI cards.
|
||||
|
||||
| Metric | Definition | Data Source | Comparison |
|
||||
|---|---|---|---|
|
||||
| [Metric name] | [How it's calculated] | [Table/source] | [vs. last week / vs. target / MoM] |
|
||||
|
||||
Aim for 3–6 KPI cards. More than 6 is noise.
|
||||
|
||||
---
|
||||
|
||||
## Section 2: Charts & Visualisations
|
||||
|
||||
For each chart, specify:
|
||||
|
||||
### Chart [N]: [Chart Title]
|
||||
|
||||
- **Chart type:** [Line / Bar / Stacked bar / Pie / Funnel / Heatmap / Table / Scatter]
|
||||
- **Why this chart type:** [One sentence — why this type suits this data]
|
||||
- **X-axis / Rows:** [Dimension — e.g. Date, User segment, Product]
|
||||
- **Y-axis / Values:** [Metric — e.g. Count of active users, Revenue]
|
||||
- **Breakdown/colour:** [Optional secondary dimension — e.g. by Plan tier, by Channel]
|
||||
- **Data source:** [Table or source]
|
||||
- **Filters:** [Any default filters applied — e.g. "Exclude internal test accounts"]
|
||||
- **Key insight to surface:** [What pattern or signal this chart should help the viewer spot]
|
||||
|
||||
---
|
||||
|
||||
## Section 3: Filters & Controls
|
||||
|
||||
Global filters available to dashboard viewers:
|
||||
|
||||
| Filter | Type | Default | Options |
|
||||
|---|---|---|---|
|
||||
| Date range | Date picker | Last 30 days | Custom |
|
||||
| [Segment filter] | Dropdown | All | [List relevant values] |
|
||||
| [Other filter] | Multi-select | All | [List relevant values] |
|
||||
|
||||
---
|
||||
|
||||
## Section 4: Layout Recommendation
|
||||
|
||||
Describe the dashboard layout in plain terms:
|
||||
|
||||
```
|
||||
[ROW 1 — KPI Cards]: [Metric 1] | [Metric 2] | [Metric 3] | [Metric 4]
|
||||
[ROW 2 — Primary chart, full width]: [Chart name]
|
||||
[ROW 3 — Two charts side by side]: [Chart A] | [Chart B]
|
||||
[ROW 4 — Supporting table, full width]: [Table name]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Section 5: Data Requirements
|
||||
|
||||
List any data transformations, joins, or derived fields needed:
|
||||
|
||||
| Derived Field | Logic | Source Tables |
|
||||
|---|---|---|
|
||||
| [Field name] | [How it's calculated] | [Tables involved] |
|
||||
|
||||
Flag any fields that may not exist in current data infrastructure.
|
||||
|
||||
---
|
||||
|
||||
## Section 6: Access & Ownership
|
||||
|
||||
- **Dashboard owner:** [Leave for user to fill]
|
||||
- **Who can edit:** [Leave for user to fill]
|
||||
- **Who can view:** [Leave for user to fill]
|
||||
- **Review cadence:** [When should this dashboard be reviewed for relevance?]
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- [ ] Every chart has a stated "key insight to surface" — not just "show the data"
|
||||
- [ ] KPI cards are 3–6 (not more)
|
||||
- [ ] Chart types are justified
|
||||
- [ ] Layout follows visual hierarchy (summary → detail)
|
||||
- [ ] Data requirements section flags any missing fields
|
||||
- [ ] Filters are practical and don't require IT to configure
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- [ ] Do not specify metrics that the available data sources cannot actually support — always validate data availability
|
||||
- [ ] Do not include more than 8–10 primary metrics on a single dashboard — more creates noise, not insight
|
||||
- [ ] Do not skip the primary business question — a dashboard without a north-star question becomes a vanity metrics display
|
||||
- [ ] Do not choose chart types for aesthetic reasons — every chart type must match the data relationship it represents
|
||||
- [ ] Do not leave filter configurations vague — specify exact filter values, not just filter categories
|
||||
|
||||
## Example Trigger Phrases
|
||||
|
||||
- "Design a dashboard to track [business process]"
|
||||
- "Give me a spec for a [team] performance dashboard"
|
||||
- "What should go on a [topic] dashboard?"
|
||||
- "Write a dashboard brief for our [metric] monitoring"
|
||||
@@ -0,0 +1,230 @@
|
||||
---
|
||||
description: "Design an ETL/ELT data pipeline specification. Use when asked to design a data pipeline, spec an ETL or ELT process, document a data ingestion workflow, or plan a data integration. Produces a complete pipeline spec with sources, transforms, destinations, SLAs, error handling, and data quality rules."
|
||||
globs:
|
||||
alwaysApply: false
|
||||
---
|
||||
|
||||
# Data Pipeline Spec Skill
|
||||
|
||||
This skill produces a complete data pipeline specification covering sources, transformations, destinations, scheduling, SLAs, error handling, data quality checks, and monitoring requirements. Output is ready for engineering handoff or architecture review.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
Ask the user for these if not provided:
|
||||
- **Pipeline purpose** — what business question or workflow does this pipeline serve?
|
||||
- **Source systems** — where does data come from? (databases, APIs, files, event streams)
|
||||
- **Destination** — where does data land? (data warehouse, data lake, downstream DB, reporting tool)
|
||||
- **Transformation type** — ETL (transform before loading) or ELT (load raw, transform in warehouse)?
|
||||
- **Frequency / SLA** — how often must data be fresh? (real-time / hourly / daily / weekly)
|
||||
- **Volume estimate** — approximate rows/events per run
|
||||
- **Data quality requirements** — completeness, deduplication, freshness, schema enforcement
|
||||
- **Team or stack** — any specific tools in use? (Airflow, dbt, Fivetran, Spark, Kafka, etc.)
|
||||
|
||||
## Output Structure
|
||||
|
||||
---
|
||||
|
||||
# Data Pipeline Spec: [Pipeline Name]
|
||||
|
||||
**Purpose:** [One sentence — what decision or workflow does this pipeline enable?]
|
||||
**Type:** [ETL / ELT / Streaming / Batch]
|
||||
**Owner:** [Team or individual]
|
||||
**Version:** [1.0]
|
||||
**Date:** [Date]
|
||||
**Status:** [Draft / Under Review / Approved]
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
[2–3 sentences describing the pipeline end-to-end: what data moves, from where to where, at what cadence, and why.]
|
||||
|
||||
**Architecture diagram (text):**
|
||||
|
||||
```
|
||||
[Source A] ──┐
|
||||
[Source B] ──┤──► [Ingestion Layer] ──► [Transform Layer] ──► [Destination] ──► [Consumers]
|
||||
[Source C] ──┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Sources
|
||||
|
||||
| Source | System | Connection type | Data format | Update pattern | Volume |
|
||||
|---|---|---|---|---|---|
|
||||
| [Source 1] | [PostgreSQL / Salesforce / S3 / Kafka] | [JDBC / REST API / SDK / Webhook] | [JSON / CSV / Parquet / CDC] | [Append / Full refresh / Incremental] | [X rows/day] |
|
||||
| [Source 2] | [...] | [...] | [...] | [...] | [...] |
|
||||
|
||||
**Incremental key (if applicable):** [The column used to identify new or changed records — e.g. `updated_at`, `event_id`]
|
||||
|
||||
**Authentication:** [API key / OAuth / IAM role / connection string — note where credentials are stored]
|
||||
|
||||
---
|
||||
|
||||
## 3. Ingestion Layer
|
||||
|
||||
**Tool:** [Fivetran / Airbyte / Kafka Connect / custom script / dbt source]
|
||||
|
||||
**Ingestion method:**
|
||||
- [ ] Full extract (full table refresh each run)
|
||||
- [ ] Incremental extract (only new/changed rows since last run)
|
||||
- [ ] CDC (change data capture from database transaction log)
|
||||
- [ ] Event streaming (continuous ingestion from Kafka/Kinesis)
|
||||
|
||||
**Raw landing zone:** [Where raw data lands before transformation — e.g. `raw.salesforce_opportunities` in Snowflake, S3 bucket `s3://data-raw/crm/`]
|
||||
|
||||
**Schema handling:** [Strict schema enforcement / Schema evolution allowed / Union schema]
|
||||
|
||||
---
|
||||
|
||||
## 4. Transformation Logic
|
||||
|
||||
List each transformation in execution order. For ELT pipelines, this is the dbt model or SQL layer.
|
||||
|
||||
| Step | Name | Description | Input | Output | Tool |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | [Deduplicate events] | [Remove duplicate event rows based on event_id] | `raw.events` | `staging.events_deduped` | [dbt / SQL / Spark] |
|
||||
| 2 | [Join user profile] | [Enrich events with user attributes from CRM] | `staging.events_deduped`, `raw.users` | `staging.events_enriched` | [...] |
|
||||
| 3 | [Aggregate to daily] | [Roll up to user×day grain] | `staging.events_enriched` | `mart.user_daily_activity` | [...] |
|
||||
|
||||
**Business logic rules:**
|
||||
- [e.g. Revenue is recognised on `payment_confirmed_at`, not `payment_initiated_at`]
|
||||
- [e.g. Users in the `internal@company.com` domain are excluded from all metrics]
|
||||
- [e.g. Currency conversion uses the ECB rate from the first business day of each month]
|
||||
|
||||
**Slowly Changing Dimensions (SCD) — if applicable:**
|
||||
- [e.g. `users.plan_tier` is SCD Type 2 — keep history of plan changes with `valid_from` / `valid_to`]
|
||||
|
||||
---
|
||||
|
||||
## 5. Destination
|
||||
|
||||
| Destination | System | Schema / Table | Write mode | Consumers |
|
||||
|---|---|---|---|---|
|
||||
| [Primary] | [Snowflake / BigQuery / Redshift / PostgreSQL] | [`analytics.mart_user_activity`] | [Append / Upsert / Full replace] | [Looker / Metabase / downstream pipeline] |
|
||||
| [Secondary] | [...] | [...] | [...] | [...] |
|
||||
|
||||
**Partitioning / Clustering:** [e.g. Partitioned by `event_date`, clustered by `user_id` — reduces query cost for time-range scans]
|
||||
|
||||
**Retention policy:** [e.g. Raw data retained for 90 days; mart tables retained indefinitely]
|
||||
|
||||
---
|
||||
|
||||
## 6. Scheduling & SLAs
|
||||
|
||||
| SLA | Target | Breach action |
|
||||
|---|---|---|
|
||||
| **Data freshness** | [Data must be ≤ X hours old by HH:MM UTC] | [Page on-call / alert Slack channel] |
|
||||
| **Pipeline completion** | [Must complete within X minutes of trigger] | [Alert and auto-retry] |
|
||||
| **Availability** | [Pipeline must run successfully X% of days per month] | [Incident review] |
|
||||
|
||||
**Schedule:** [Cron expression and human description — e.g. `0 6 * * *` — daily at 06:00 UTC]
|
||||
|
||||
**Trigger type:**
|
||||
- [ ] Time-based (cron)
|
||||
- [ ] Event-based (triggered by upstream pipeline success / file arrival / Kafka lag)
|
||||
- [ ] Manual (ad hoc runs only)
|
||||
|
||||
**Backfill strategy:** [How to reprocess historical data if the pipeline fails or logic changes — e.g. parameterised date range, full drop-and-reload]
|
||||
|
||||
---
|
||||
|
||||
## 7. Data Quality Rules
|
||||
|
||||
| Check | Table | Rule | Failure action |
|
||||
|---|---|---|---|
|
||||
| Completeness | `staging.events` | `event_id IS NOT NULL` — 100% of rows | Block load / Alert |
|
||||
| Uniqueness | `mart.user_daily_activity` | `(user_id, date)` must be unique | Block load |
|
||||
| Freshness | `mart.user_daily_activity` | `max(event_date) >= CURRENT_DATE - 1` | Alert |
|
||||
| Volume | `staging.events` | Row count within ±20% of 7-day average | Alert |
|
||||
| Referential integrity | `staging.events` | All `user_id` values exist in `users` table | Alert |
|
||||
|
||||
**DQ tool:** [dbt tests / Great Expectations / Monte Carlo / custom SQL assertions]
|
||||
|
||||
---
|
||||
|
||||
## 8. Error Handling & Recovery
|
||||
|
||||
**Retry policy:** [e.g. 3 retries with exponential back-off: 5 min, 20 min, 60 min]
|
||||
|
||||
**Failure modes and responses:**
|
||||
|
||||
| Failure | Detection | Response | Owner |
|
||||
|---|---|---|---|
|
||||
| Source unavailable | HTTP 5xx / connection timeout | Retry 3×, then alert and skip run | Data engineering |
|
||||
| Schema change in source | Column missing or type mismatch | Block load, alert schema owner | Data owner + engineering |
|
||||
| DQ check fails | dbt test failure / assertion error | Block load for P1 checks; alert for P2 | Data engineering |
|
||||
| Partial load | Row count < expected threshold | Alert; do not publish to consumers until resolved | Data engineering |
|
||||
|
||||
**Dead-letter queue:** [Where failed records are routed for manual inspection — e.g. `raw.dlq_events`]
|
||||
|
||||
---
|
||||
|
||||
## 9. Monitoring & Observability
|
||||
|
||||
**Metrics to track:**
|
||||
- Pipeline run duration (p50, p95)
|
||||
- Rows processed per run
|
||||
- DQ check pass rate
|
||||
- Source freshness lag
|
||||
- Error rate per source
|
||||
|
||||
**Alerting:**
|
||||
- [Slack channel: #data-alerts]
|
||||
- [PagerDuty: data-on-call escalation for P1 SLA breaches]
|
||||
- [Dashboard: [link to monitoring dashboard]]
|
||||
|
||||
**Logging:** [What gets logged and where — e.g. Airflow task logs to CloudWatch, structured JSON to data lake]
|
||||
|
||||
---
|
||||
|
||||
## 10. Dependencies & Sequencing
|
||||
|
||||
**Upstream dependencies:** [Which pipelines or data sources must succeed before this pipeline runs?]
|
||||
|
||||
**Downstream dependents:** [Which dashboards, pipelines, or models depend on this pipeline's output?]
|
||||
|
||||
```
|
||||
[upstream pipeline A] ──► THIS PIPELINE ──► [downstream dashboard B]
|
||||
└──► [downstream pipeline C]
|
||||
```
|
||||
|
||||
**Coordination mechanism:** [Airflow DAG dependency / dbt ref() / event trigger / manual gate]
|
||||
|
||||
---
|
||||
|
||||
## 11. Security & Compliance
|
||||
|
||||
- **PII fields:** [List columns containing PII — e.g. `email`, `ip_address`, `name`]
|
||||
- **Masking / Pseudonymisation:** [e.g. email hashed with SHA-256 before landing in mart layer]
|
||||
- **Access control:** [Who can query the destination tables? — e.g. Role-based access in Snowflake]
|
||||
- **Data residency:** [Which regions is data permitted to transit and rest in?]
|
||||
- **Audit trail:** [Is pipeline execution auditable for compliance purposes? Where are logs retained?]
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- [ ] Every source has an incremental key or full-refresh justification
|
||||
- [ ] Business logic rules are documented, not just the SQL
|
||||
- [ ] SLAs are agreed with consumers, not set unilaterally by engineering
|
||||
- [ ] DQ checks cover completeness, uniqueness, freshness, and volume
|
||||
- [ ] Failure modes include a documented recovery owner
|
||||
- [ ] PII fields are identified and a treatment plan is specified
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- [ ] Do not spec a pipeline without defining SLAs — "as fast as possible" is not an acceptable freshness target
|
||||
- [ ] Do not omit error handling and dead-letter queue strategy — every pipeline must specify what happens to failed records
|
||||
- [ ] Do not design idempotent loads without documenting the deduplication key — assume reruns will happen
|
||||
- [ ] Do not leave data quality rules implicit — schema validation, null checks, and referential integrity must be explicit
|
||||
- [ ] Do not ignore schema evolution — specify how upstream schema changes are detected and handled
|
||||
|
||||
## Example Trigger Phrases
|
||||
|
||||
- "Design a data pipeline for our Salesforce to Snowflake sync"
|
||||
- "Write a pipeline spec for ingesting Stripe events into our data warehouse"
|
||||
- "Build an ETL spec for our user activity data"
|
||||
- "Document our dbt pipeline from raw events to the analytics mart"
|
||||
- "Spec out the pipeline that feeds the executive dashboard"
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
description: "Build a metrics framework for any product, team, or business. Use when asked for a metrics tree, KPI framework, North Star metric, AARRR funnel, HEART framework, or OKR metrics. Produces a structured metrics hierarchy from North Star down to leading indicators, with measurement guidance."
|
||||
globs:
|
||||
alwaysApply: false
|
||||
---
|
||||
|
||||
# Metrics Framework Skill
|
||||
|
||||
This skill builds a complete metrics framework tailored to a product or business. It connects the North Star metric to actionable leading indicators, making it clear which metrics to track, which to optimise, and how they relate to each other.
|
||||
|
||||
## Required Inputs
|
||||
|
||||
Ask the user for these if not provided:
|
||||
- **Product or business description** (one paragraph is enough)
|
||||
- **Business model** (SaaS / Marketplace / E-commerce / Consumer app / B2B / Other)
|
||||
- **Stage** (Pre-PMF / Growth / Scale / Mature)
|
||||
- **Framework preference** (if they have one): North Star + Metric Tree / AARRR / HEART / OKRs / Custom
|
||||
- **Primary goal this quarter** (e.g. grow activation, reduce churn, increase revenue)
|
||||
|
||||
If no framework preference is given, recommend the best fit based on stage and business model.
|
||||
|
||||
## Output Structure
|
||||
|
||||
### 1. Framework Recommendation (if not specified)
|
||||
|
||||
Explain in 2–3 sentences why you're recommending this framework for their context.
|
||||
|
||||
---
|
||||
|
||||
### 2. North Star Metric
|
||||
|
||||
**[Metric Name]:** [Definition — exactly what is measured and how]
|
||||
|
||||
**Why this is the right North Star for this business:**
|
||||
[2–3 sentences. It should reflect customer value delivered, not just revenue or activity. Explain what behaviour it captures and why maximising it correlates with long-term business health.]
|
||||
|
||||
**How to measure it:** [Formula or data source]
|
||||
**Current baseline:** [Leave as [ADD BASELINE] for user to fill]
|
||||
**Target:** [Leave as [ADD TARGET] for user to fill]
|
||||
|
||||
---
|
||||
|
||||
### 3. Metric Tree
|
||||
|
||||
Show how supporting metrics roll up to the North Star. Format as a hierarchy:
|
||||
|
||||
```
|
||||
[North Star Metric]
|
||||
├── [Driver 1: e.g. Acquisition]
|
||||
│ ├── [L2 metric: e.g. Organic signups / week]
|
||||
│ └── [L2 metric: e.g. Paid CAC by channel]
|
||||
├── [Driver 2: e.g. Activation]
|
||||
│ ├── [L2 metric: e.g. % users completing onboarding within 7 days]
|
||||
│ └── [L2 metric: e.g. Time to first value action]
|
||||
└── [Driver 3: e.g. Retention]
|
||||
├── [L2 metric: e.g. Day 30 retention rate]
|
||||
└── [L2 metric: e.g. Feature adoption depth]
|
||||
```
|
||||
|
||||
For each L2 metric, provide:
|
||||
- **Definition:** [What exactly is measured]
|
||||
- **Why it matters:** [How it connects to the North Star]
|
||||
- **Leading or lagging?** [Leading = predictive / Lagging = outcome]
|
||||
- **How to measure:** [Data source or calculation]
|
||||
|
||||
---
|
||||
|
||||
### 4. Counter-Metrics
|
||||
|
||||
[2–3 metrics to watch that prevent optimising the North Star in ways that damage the business. E.g. "If we optimise for signups, we need to watch spam account rate. If we optimise for engagement, we need to watch support ticket volume."]
|
||||
|
||||
---
|
||||
|
||||
### 5. Dashboard Recommendation
|
||||
|
||||
Suggest a 3-tier dashboard structure:
|
||||
- **Exec view (weekly):** [3–5 metrics — outcomes only]
|
||||
- **Team view (daily):** [7–10 metrics — leading indicators + outputs]
|
||||
- **Diagnostic view (on demand):** [Metrics to drill into when something looks wrong]
|
||||
|
||||
---
|
||||
|
||||
### 6. Metric Health Check Questions
|
||||
|
||||
[5 questions the team should ask in their weekly metrics review to turn numbers into insights. e.g. "Is our activation rate improving while retention stays flat? That suggests onboarding quality issue, not a product-market fit problem."]
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- [ ] North Star reflects customer value, not just business activity
|
||||
- [ ] Metric tree has 3–4 distinct drivers (not all one category)
|
||||
- [ ] Each L2 metric is classified as leading or lagging
|
||||
- [ ] Counter-metrics are included to prevent perverse incentives
|
||||
- [ ] Dashboard tiers are tailored to the product stage
|
||||
- [ ] All metric definitions are unambiguous (formula or clear description)
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- [ ] Do not set a North Star metric that measures business activity (revenue, pageviews) rather than customer value delivered — this creates incentives misaligned with product quality
|
||||
- [ ] Do not define metrics without specifying the formula or data source — an ambiguous metric will be measured differently by different people
|
||||
- [ ] Do not skip counter-metrics — optimising any single metric without a guard rail will eventually produce perverse incentives
|
||||
- [ ] Do not include more than 4–5 metrics in a daily team view — a dashboard with 20 metrics is a dashboard nobody looks at
|
||||
- [ ] Do not classify all metrics as "leading" — be honest about which are lagging outcome metrics and which genuinely predict future outcomes
|
||||
|
||||
## Example Trigger Phrases
|
||||
|
||||
- "Build a metrics framework for [product]"
|
||||
- "What should our North Star metric be?"
|
||||
- "Create a KPI tree for [business]"
|
||||
- "Give me an AARRR breakdown for [product]"
|
||||
- "What metrics should our [team type] team track?"
|
||||
@@ -0,0 +1,144 @@
|
||||
---
|
||||
description: "Explains, optimises, writes, and documents SQL queries. Use when asked to explain a SQL query, optimise slow SQL, translate SQL to plain English for non-technical stakeholders, write a query from a natural language description, or produce query documentation. Produces plain-English explanations, annotated optimised queries, or a data dictionary covering output shape, assumptions, and known limitations. Works across PostgreSQL, MySQL, BigQuery, Snowflake, and standard SQL."
|
||||
globs:
|
||||
alwaysApply: false
|
||||
---
|
||||
|
||||
# SQL Query Explainer Skill
|
||||
|
||||
This skill explains SQL queries in plain language, identifies optimisation opportunities, and helps communicate data logic to non-technical stakeholders. It also writes and documents new queries from natural language descriptions.
|
||||
|
||||
## Modes
|
||||
|
||||
Detect which mode the user needs based on their request:
|
||||
|
||||
1. **Explain** — Translate existing SQL into plain English
|
||||
2. **Optimise** — Review SQL for performance issues and suggest improvements
|
||||
3. **Write** — Generate SQL from a natural language description
|
||||
4. **Document** — Produce a data dictionary or query documentation
|
||||
|
||||
---
|
||||
|
||||
## Mode 1: Explain
|
||||
|
||||
When given a SQL query, produce:
|
||||
|
||||
### Plain English Summary
|
||||
[1–3 sentences. What does this query do? What data does it return? Write as if explaining to a business analyst, not a developer.]
|
||||
|
||||
### Step-by-Step Walkthrough
|
||||
|
||||
Break the query into logical sections. For each section:
|
||||
- Quote the SQL clause
|
||||
- Explain what it does in plain English
|
||||
- Flag any complexity (e.g. window functions, subqueries, CTEs)
|
||||
|
||||
### What the Result Looks Like
|
||||
|
||||
[Describe the shape of the output: "Returns one row per user, with columns for X, Y, Z. Ordered by [field] descending."]
|
||||
|
||||
### Potential Issues to Flag
|
||||
|
||||
- [Gotchas, edge cases, or implicit assumptions in this query]
|
||||
- [e.g. "This will include NULLs in the user_id column if the LEFT JOIN finds no match"]
|
||||
|
||||
---
|
||||
|
||||
## Mode 2: Optimise
|
||||
|
||||
When asked to optimise a query, produce:
|
||||
|
||||
### Performance Assessment
|
||||
|
||||
Rate overall: 🟢 Well-optimised / 🟡 Some improvements possible / 🔴 Significant issues
|
||||
|
||||
### Issues Found
|
||||
|
||||
For each issue:
|
||||
|
||||
**Issue [N]: [Short name, e.g. "Missing index on join column"]**
|
||||
- **What it is:** [Plain explanation]
|
||||
- **Why it matters:** [Performance impact — e.g. "Full table scan on a 10M row table"]
|
||||
- **Fix:**
|
||||
```sql
|
||||
-- Before
|
||||
[original snippet]
|
||||
|
||||
-- After
|
||||
[improved snippet]
|
||||
```
|
||||
- **Expected improvement:** [Estimate if possible]
|
||||
|
||||
### Optimisation Checklist
|
||||
|
||||
- [ ] SELECT * used? (Replace with specific columns)
|
||||
- [ ] Implicit type conversions on JOIN/WHERE columns?
|
||||
- [ ] Missing indexes on JOIN or WHERE columns?
|
||||
- [ ] N+1 patterns (queries inside loops)?
|
||||
- [ ] DISTINCT used where GROUP BY would be faster?
|
||||
- [ ] Window functions used where a subquery would be clearer/faster?
|
||||
- [ ] CTEs re-used or materialised unnecessarily?
|
||||
- [ ] Large IN() lists that could use a JOIN instead?
|
||||
|
||||
---
|
||||
|
||||
## Mode 3: Write
|
||||
|
||||
When given a natural language description, generate the SQL query and then explain it using Mode 1.
|
||||
|
||||
Ask the user to confirm:
|
||||
- **Database/dialect** (PostgreSQL / MySQL / BigQuery / Snowflake / SQLite / Standard SQL)
|
||||
- **Table and column names** (if known; otherwise use descriptive placeholder names like `users`, `orders`, `user_id`)
|
||||
- **Any filters, sorting, or aggregation requirements**
|
||||
|
||||
Produce:
|
||||
1. The SQL query with inline comments
|
||||
2. Plain English explanation (Mode 1 format)
|
||||
|
||||
---
|
||||
|
||||
## Mode 4: Document
|
||||
|
||||
When asked to create documentation for a query or table:
|
||||
|
||||
### Query Documentation
|
||||
|
||||
```
|
||||
Query: [Name]
|
||||
Purpose: [One sentence — what business question this answers]
|
||||
Author: [If provided]
|
||||
Last reviewed: [If provided]
|
||||
|
||||
Inputs:
|
||||
- Table: [table_name] — [what it contains]
|
||||
- Filter: [any WHERE conditions and their business meaning]
|
||||
|
||||
Output columns:
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| [name] | [type] | [plain English description] |
|
||||
|
||||
Assumptions:
|
||||
- [Any implicit assumptions the query makes]
|
||||
|
||||
Known limitations:
|
||||
- [Edge cases not handled, data quality dependencies, etc.]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quality Checks
|
||||
|
||||
- [ ] Plain English explanation avoids SQL jargon
|
||||
- [ ] Optimisation suggestions include before/after SQL
|
||||
- [ ] Written queries include inline comments
|
||||
- [ ] Output shape is described (columns, row grain, ordering)
|
||||
- [ ] Dialect-specific syntax is flagged when non-standard
|
||||
|
||||
## Example Trigger Phrases
|
||||
|
||||
- "Explain this SQL query: [paste query]"
|
||||
- "Optimise this slow query: [paste query]"
|
||||
- "Write a SQL query that [natural language description]"
|
||||
- "Document this query for my non-technical stakeholders"
|
||||
- "Why is this query returning unexpected results?"
|
||||
Reference in New Issue
Block a user