v1.0
This commit is contained in:
Pawel Huryn
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: ab-test-analysis
|
||||
description: "Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and actionable recommendations. Use when evaluating experiment results, checking if a test is significant, interpreting A/B test data, or deciding whether to ship a variant. Triggers: A/B test, AB test, experiment results, statistical significance, test analysis, split test, which variant won."
|
||||
---
|
||||
|
||||
## A/B Test Analysis
|
||||
|
||||
Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
|
||||
|
||||
### Context
|
||||
|
||||
You are analyzing A/B test results for **$ARGUMENTS**.
|
||||
|
||||
If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
|
||||
|
||||
### Instructions
|
||||
|
||||
1. **Understand the experiment**:
|
||||
- What was the hypothesis?
|
||||
- What was changed (the variant)?
|
||||
- What is the primary metric? Any guardrail metrics?
|
||||
- How long did the test run?
|
||||
- What is the traffic split?
|
||||
|
||||
2. **Validate the test setup**:
|
||||
- **Sample size**: Is the sample large enough for the expected effect size?
|
||||
- Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
|
||||
- Flag if the test is underpowered (<80% power)
|
||||
- **Duration**: Did the test run for at least 1-2 full business cycles?
|
||||
- **Randomization**: Any evidence of sample ratio mismatch (SRM)?
|
||||
- **Novelty/primacy effects**: Was there enough time to wash out initial behavior changes?
|
||||
|
||||
3. **Calculate statistical significance**:
|
||||
- **Conversion rate** for control and variant
|
||||
- **Relative lift**: (variant - control) / control × 100
|
||||
- **p-value**: Using a two-tailed z-test or chi-squared test
|
||||
- **Confidence interval**: 95% CI for the difference
|
||||
- **Statistical significance**: Is p < 0.05?
|
||||
- **Practical significance**: Is the lift meaningful for the business?
|
||||
|
||||
If the user provides raw data, generate and run a Python script to calculate these.
|
||||
|
||||
4. **Check guardrail metrics**:
|
||||
- Did any guardrail metrics (revenue, engagement, page load time) degrade?
|
||||
- A winning primary metric with degraded guardrails may not be a true win
|
||||
|
||||
5. **Interpret results**:
|
||||
|
||||
| Outcome | Recommendation |
|
||||
|---|---|
|
||||
| Significant positive lift, no guardrail issues | **Ship it** — roll out to 100% |
|
||||
| Significant positive lift, guardrail concerns | **Investigate** — understand trade-offs before shipping |
|
||||
| Not significant, positive trend | **Extend the test** — need more data or larger effect |
|
||||
| Not significant, flat | **Stop the test** — no meaningful difference detected |
|
||||
| Significant negative lift | **Don't ship** — revert to control, analyze why |
|
||||
|
||||
6. **Provide the analysis summary**:
|
||||
```
|
||||
## A/B Test Results: [Test Name]
|
||||
|
||||
**Hypothesis**: [What we expected]
|
||||
**Duration**: [X days] | **Sample**: [N control / M variant]
|
||||
|
||||
| Metric | Control | Variant | Lift | p-value | Significant? |
|
||||
|---|---|---|---|---|---|
|
||||
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
|
||||
| [Guardrail] | ... | ... | ... | ... | ... |
|
||||
|
||||
**Recommendation**: [Ship / Extend / Stop / Investigate]
|
||||
**Reasoning**: [Why]
|
||||
**Next steps**: [What to do]
|
||||
```
|
||||
|
||||
Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.
|
||||
|
||||
---
|
||||
|
||||
### Further Reading
|
||||
|
||||
- [A/B Testing 101 + Examples](https://www.productcompass.pm/p/ab-testing-101-for-pms)
|
||||
- [Testing Product Ideas: The Ultimate Validation Experiments Library](https://www.productcompass.pm/p/the-ultimate-experiments-library)
|
||||
- [Are You Tracking the Right Metrics?](https://www.productcompass.pm/p/are-you-tracking-the-right-metrics)
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
name: cohort-analysis
|
||||
description: "Perform cohort analysis on user engagement data. Identifies retention patterns, feature usage trends, and suggests qualitative follow-up research. Use when analyzing user retention by cohort, studying feature adoption over time, or investigating engagement patterns. Triggers: cohort analysis, retention analysis, user cohorts, engagement trends, cohort data."
|
||||
---
|
||||
|
||||
# Cohort Analysis & Retention Explorer
|
||||
|
||||
## Purpose
|
||||
Analyze user engagement and retention patterns by cohort to identify trends in user behavior, feature adoption, and long-term engagement. Combine quantitative insights with qualitative research recommendations.
|
||||
|
||||
## How It Works
|
||||
|
||||
### Step 1: Read and Validate Your Data
|
||||
- Accept CSV, Excel, or JSON data files with user cohort information
|
||||
- Verify data structure: cohort identifier, time periods, engagement metrics
|
||||
- Check for missing values and data quality issues
|
||||
- Summarize key statistics (cohort sizes, date ranges, metrics available)
|
||||
|
||||
### Step 2: Generate Quantitative Analysis
|
||||
- Calculate cohort retention rates and engagement trends
|
||||
- Identify retention curves, drop-off patterns, and anomalies
|
||||
- Compute feature adoption rates across cohorts
|
||||
- Calculate month-over-month or period-over-period changes
|
||||
- Generate Python analysis scripts using pandas and numpy if requested
|
||||
|
||||
### Step 3: Create Visualizations
|
||||
- Generate retention heatmaps (cohorts vs. time periods)
|
||||
- Create line charts showing cohort progression
|
||||
- Build comparison charts for feature adoption
|
||||
- Visualize drop-off points and engagement trends
|
||||
- Output as interactive charts or static images
|
||||
|
||||
### Step 4: Identify Insights & Patterns
|
||||
- Spot one or more significant patterns:
|
||||
- Early churn in specific cohorts
|
||||
- Late-stage engagement changes
|
||||
- Feature adoption clusters
|
||||
- Seasonal or temporal trends
|
||||
- Highlight surprising findings and deviations
|
||||
- Compare cohort performance to establish baselines
|
||||
|
||||
### Step 5: Suggest Follow-Up Research
|
||||
- Recommend qualitative research methods:
|
||||
- Targeted user interviews with churning users
|
||||
- Feature usage surveys with engaged cohorts
|
||||
- Session replays of key interaction patterns
|
||||
- Win/loss analysis for high vs. low retention cohorts
|
||||
- Design follow-up quantitative studies
|
||||
- Suggest A/B tests or feature experiments
|
||||
|
||||
## Usage Examples
|
||||
|
||||
**Example 1: Upload CSV Data**
|
||||
```
|
||||
Upload cohort_engagement.csv with columns: cohort_month, weeks_active,
|
||||
user_id, feature_x_usage, engagement_score
|
||||
|
||||
Request: "Analyze retention patterns and identify why Q4 2025 cohorts
|
||||
underperform compared to Q3"
|
||||
```
|
||||
|
||||
**Example 2: Describe Data Format**
|
||||
```
|
||||
"I have monthly user cohorts from Jan-Dec 2025. Each row shows:
|
||||
cohort date, user ID, purchase frequency, and support tickets.
|
||||
Analyze which cohorts show best long-term retention."
|
||||
```
|
||||
|
||||
**Example 3: Feature Adoption Analysis**
|
||||
```
|
||||
Upload feature_usage.xlsx with cohort adoption data.
|
||||
|
||||
Request: "Compare adoption curves for our new feature across cohorts.
|
||||
Which cohorts adopted fastest? Any patterns?"
|
||||
```
|
||||
|
||||
## Key Capabilities
|
||||
|
||||
- **Data Reading**: Import CSV, Excel, JSON, SQL query results
|
||||
- **Retention Analysis**: Calculate and visualize retention rates over time
|
||||
- **Cohort Comparison**: Compare metrics across cohort groups
|
||||
- **Anomaly Detection**: Flag unusual patterns or drop-offs
|
||||
- **Python Scripts**: Generate reusable analysis code for ongoing analysis
|
||||
- **Visualizations**: Create heatmaps, charts, and interactive dashboards
|
||||
- **Research Design**: Suggest targeted follow-up studies and interview approaches
|
||||
- **Statistical Summary**: Provide quantitative metrics and correlation analysis
|
||||
|
||||
## Tips for Best Results
|
||||
|
||||
1. **Include time dimension**: Provide data across multiple time periods
|
||||
2. **Define cohort clearly**: Make cohort grouping explicit (signup month, feature launch date, etc.)
|
||||
3. **Provide context**: Explain product changes, launches, or events during the period
|
||||
4. **Multiple metrics**: Include retention, engagement, feature usage, revenue, etc.
|
||||
5. **Sufficient data**: At least 3-4 cohorts for meaningful pattern identification
|
||||
6. **Request specific output**: Ask for visualizations, Python scripts, or research recommendations
|
||||
|
||||
## Output Format
|
||||
|
||||
You'll receive:
|
||||
- **Data Summary**: Cohort overview and data quality assessment
|
||||
- **Quantitative Findings**: Key metrics, retention rates, and trend analysis
|
||||
- **Visualizations**: Charts showing retention curves, adoption patterns
|
||||
- **Pattern Identification**: 2-3 significant insights from the data
|
||||
- **Research Recommendations**: Specific qualitative and quantitative follow-ups
|
||||
- **Analysis Scripts** (if requested): Python code for reproducible analysis
|
||||
- **Next Steps**: Prioritized actions based on findings
|
||||
|
||||
---
|
||||
|
||||
### Further Reading
|
||||
|
||||
- [Cohort Analysis 101: How to Reduce Churn and Make Better Product Decisions](https://www.productcompass.pm/p/cohort-analysis)
|
||||
- [The Product Analytics Playbook: AARRR, HEART, Cohorts & Funnels for PMs](https://www.productcompass.pm/p/the-product-analytics-playbook-aarrr)
|
||||
- [Are You Tracking the Right Metrics?](https://www.productcompass.pm/p/are-you-tracking-the-right-metrics)
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: sql-queries
|
||||
description: "Generate SQL queries from natural language descriptions. Supports BigQuery, PostgreSQL, MySQL, and other SQL dialects. Reads database schemas from uploaded diagrams or documentation. Use when writing SQL queries, analyzing databases, building reports, or exploring data. Triggers: SQL query, write SQL, database query, BigQuery, data report, generate query."
|
||||
---
|
||||
|
||||
# SQL Query Generator
|
||||
|
||||
## Purpose
|
||||
Transform natural language requirements into optimized SQL queries across multiple database platforms. This skill helps product managers, analysts, and engineers generate accurate queries without manual syntax work.
|
||||
|
||||
## How It Works
|
||||
|
||||
### Step 1: Understand Your Database Schema
|
||||
- If you provide a schema file (SQL, documentation, or diagram description), I will read and analyze it
|
||||
- Extract table names, column definitions, data types, and relationships
|
||||
- Identify primary keys, foreign keys, and indexing strategies
|
||||
|
||||
### Step 2: Process Your Request
|
||||
- Clarify the exact data you need to retrieve or analyze
|
||||
- Confirm the SQL dialect (BigQuery, PostgreSQL, MySQL, Snowflake, etc.)
|
||||
- Ask for any additional requirements (filters, aggregations, sorting)
|
||||
|
||||
### Step 3: Generate Optimized Query
|
||||
- Write efficient SQL that leverages your database structure
|
||||
- Include comments explaining complex logic
|
||||
- Add performance considerations for large datasets
|
||||
- Provide alternative approaches if applicable
|
||||
|
||||
### Step 4: Explain and Test
|
||||
- Explain the query logic in plain English
|
||||
- Suggest how to test or validate results
|
||||
- Offer tips for performance optimization
|
||||
- If you want, generate a test script or sample data
|
||||
|
||||
## Usage Examples
|
||||
|
||||
**Example 1: Query from Schema File**
|
||||
```
|
||||
Upload your database_schema.sql file and say:
|
||||
"Generate a query to find users who signed up in the last 30 days
|
||||
and had at least 5 active sessions"
|
||||
```
|
||||
|
||||
**Example 2: Query from Diagram Description**
|
||||
```
|
||||
"Here's my database: Users table (id, email, created_at), Sessions table
|
||||
(id, user_id, timestamp, duration). Generate a query for average session
|
||||
duration per user in January 2026."
|
||||
```
|
||||
|
||||
**Example 3: Complex Analysis Query**
|
||||
```
|
||||
"Create a BigQuery query to analyze our revenue by region and customer tier,
|
||||
including year-over-year growth rates."
|
||||
```
|
||||
|
||||
## Key Capabilities
|
||||
|
||||
- **Multi-Dialect Support**: Works with BigQuery, PostgreSQL, MySQL, Snowflake, SQL Server
|
||||
- **File Reading**: Reads schema files, SQL dumps, and data documentation
|
||||
- **Query Optimization**: Suggests indexes, partitioning, and performance improvements
|
||||
- **Explanation**: Breaks down queries for learning and documentation
|
||||
- **Testing**: Can generate test queries and sample data scripts
|
||||
- **Script Execution**: Create executable SQL scripts for your database
|
||||
|
||||
## Tips for Best Results
|
||||
|
||||
1. **Provide context**: Share your database schema or structure
|
||||
2. **Be specific**: Clearly describe what data you need and any filters
|
||||
3. **Mention database**: Specify which SQL dialect you're using
|
||||
4. **Include constraints**: Mention data volume, time ranges, and performance needs
|
||||
5. **Request format**: Ask for the query result format if you need specific output
|
||||
|
||||
## Output Format
|
||||
|
||||
You'll receive:
|
||||
- **SQL Query**: Production-ready SQL code with comments
|
||||
- **Explanation**: What the query does and how it works
|
||||
- **Performance Notes**: Optimization tips and considerations
|
||||
- **Test Script** (if requested): Sample data and validation queries
|
||||
|
||||
---
|
||||
|
||||
### Further Reading
|
||||
|
||||
- [The Product Analytics Playbook: AARRR, HEART, Cohorts & Funnels for PMs](https://www.productcompass.pm/p/the-product-analytics-playbook-aarrr)
|
||||
- [How to Become a Technology-Literate PM](https://www.productcompass.pm/p/how-to-become-a-technology-literate)
|
||||
Reference in New Issue
Block a user