Cohort Analysis

Purpose

Track how groups of users (cohorts) who share a common starting event behave over time, revealing whether engagement and retention sustain, decay, or improve across the pilot period. Cohort analysis is the primary method for answering A3.3's central question: “Do users keep using this, or do they try it once and leave?”

Unlike aggregate metrics (“We have 200 active users this week”), cohort analysis segments by when users joined, exposing the retention curve that aggregate numbers hide. A pilot with 200 weekly actives might consist of 50 retained Week~1 users and 150 new Week~4 users—the product retains poorly but keeps acquiring. Without cohort analysis, this failure is invisible.

When to Use

Use cohort analysis when:

• Tracking A3.3 pilot retention over 4–8 weeks (the primary use case)
• Comparing behaviour of users onboarded in different controlled rollout waves (the referenced method)
• Measuring whether A3.3 iterations improve outcomes for later cohorts vs. earlier ones
• Providing retention evidence for A3.4 evaluation (viability lens—does the business model sustain?)
• Distinguishing genuine engagement growth from new-user acquisition masking churn

Do NOT use when:

• Pilot duration <4 weeks—insufficient time for meaningful retention curves
• Pilot has <50 users—cohorts become too small for reliable patterns (e.g. 10 users per weekly cohort yields noisy data)
• Measuring one-time events (conversion, purchase) rather than sustained behaviour—use funnel analysis instead

Sample Size and Duration

Minimum pilot size: 50 users (enables meaningful weekly cohorts of 10–15+)

Ideal pilot size: 100–250 users (weekly cohorts of 25–60, stable percentages)

Minimum duration: 4 weeks (bare minimum for retention curve shape)

Ideal duration: 6–8 weeks (shows whether retention stabilises or continues declining)

Analysis effort: 4–8 hours per reporting cycle (weekly during A3.3)

Prerequisites

• User-level event tracking: Analytics platform recording per-user events with timestamps (sign-up, login, feature use, session)
• Cohort definition: Agreed grouping criterion—typically week of first activation (Week~1 cohort = users who activated in pilot Week~1)
• Retention metric defined: What counts as “active”? Login? Feature use? Minimum session duration? Define before pilot starts
• Analytics platform: Mixpanel, Amplitude, or Google Analytics configured with user identification and event tracking
• Minimum 4 weeks of data: Retention curves require ≥4 data points to show meaningful trends

Complete Procedure

Step~1: Define Cohorts and Metrics (1–2 hours)

Cohort grouping: Weekly cohorts based on activation date. For a 6-week pilot with staggered onboarding:

• Cohort~1: Users activated Week~1 (Wave~1–2 of controlled rollout)
• Cohort~2: Users activated Week~2 (Wave~3)
• Cohort~3: Users activated Week~3 (Wave~4)
• Cohorts~4–6: Users activated Weeks~4–6 (ongoing additions)

Retention metric: “Active” = at least one meaningful action in the period (not just login). Define “meaningful action” based on the product's core value (e.g. completed a checkout, sent a message, created a document).

Step~2: Build Retention Table (2–4 hours setup, then automated)

Create a cohort retention table showing the percentage of each cohort still active in subsequent weeks:

Reading the table:

• Down a column: Later cohorts retaining better than earlier ones? The product is improving (A3.3 iterations working)
• Across a row: Retention stabilising or still declining? Stabilisation (“flattening”) indicates core value; continued decline indicates novelty without sustained value
• Week~4+ retention: The critical number for A3.4 evaluation. A3.3's target is typically ≥40% Week~8 retention for consumer products, ≥60% for B2B/enterprise

Step~3: Visualise Retention Curves (1 hour)

Plot retention curves (one line per cohort, x-axis = weeks since activation, y-axis = % retained). The visual reveals patterns that tables obscure:

• Healthy pattern: Curves flatten after Week~2–3 (initial drop-off, then stable core users)
• Concerning pattern: Curves continue declining through Week~8 (no retention floor)
• Improving pattern: Later cohorts' curves are above earlier cohorts' curves at the same week (product iterations working)
• Alarming pattern: Curves approach zero by Week~6 (no sustainable engagement)

Step~4: Segment Analysis (2–4 hours)

Break cohorts into sub-segments to understand who retains and who churns:

• By user type: Do power users retain differently from casual users?
• By acquisition channel: Do referred users retain better than recruited users?
• By feature usage: Do users who engage with feature X in Week~1 retain better? (This identifies “aha moment” features)
• By onboarding completion: Do users who complete full onboarding retain better than those who skip steps?

Step~5: Report for A3.4 (2–4 hours)

Produce cohort analysis report:

1. Overall retention curve with Week~4 and Week~8 benchmarks
2. Cohort-over-cohort improvement (are iterations working?)
3. Segment analysis (who retains, who churns, why)
4. Retention forecast (if Week~8 data incomplete, project from available data with stated assumptions)
5. Pass/fail assessment against A3.3 retention targets

Quality Criteria

1. Cohort-based reporting: All retention metrics reported by cohort, not as aggregates
2. Value-action definition: “Active” defined as meaningful engagement, not mere login
3. Minimum 4 weeks: Retention curves have ≥4 weekly data points before A3.4 assessment
4. Segment analysis included: At least one segmentation (user type, feature usage, or onboarding completion) explored
5. Churn investigated: Churned users contacted for exit feedback
6. Visualised: Retention curves plotted (not just tables)—curves reveal patterns tables hide

Theoretical Foundation

Seminal references:

• : Introduced the AARRR framework (Acquisition, Activation, Retention, Revenue, Referral)—“pirate metrics”—that positioned retention as the most critical metric for product viability. Cohort analysis is the standard method for measuring retention within AARRR.
• : Applied cohort analysis to Lean Startup methodology, demonstrating that cohort-based retention curves are the most reliable predictor of product-market fit. Flat or improving retention curves indicate fit; declining curves indicate the product has a novelty problem, not a value problem.

Contemporary references:

• : Established 40% “very disappointed” threshold (from Sean Ellis survey) as a product-market fit indicator, and positioned week-over-week cohort retention as the quantitative complement: if Week~8 retention exceeds ≈40%, the product likely has market fit.
• : Demonstrated that cohort analysis prevents Simpson's paradox in A/B tests and pilot metrics—where aggregate trends reverse when data is properly segmented by cohort.

Challenges and Solutions

Challenge 1: Small Cohort Sizes

Symptoms: Weekly cohorts of 10–15 users; retention percentages swing wildly (1 user churning = 7–10% change).

Solutions: Merge into bi-weekly or monthly cohorts for more stable percentages. Report absolute numbers alongside percentages (“12 of 25 retained” is more honest than “48% retained”). Flag low-n cohorts in the report.

Challenge 2: Defining “Active”

Symptoms: Team counts logins as “active”; pilot shows 70% retention. But users log in, see nothing new, leave. Real engagement is 30%.

Solutions: Define “active” as a value action, not a visit. For e-commerce: completed a browse or purchase. For SaaS: used a core feature. For communication tools: sent a message. Define before pilot starts; do not redefine mid-pilot to improve numbers.

Challenge 3: Survivorship Bias

Symptoms: Team interviews retained users, gets positive feedback, concludes product is great. Churned users—who hold the real insight—are never contacted.

Solutions: Conduct churn interviews (the referenced method) with users who left. Report both retention and churn reasons. The churned users' feedback is often more actionable than retained users' satisfaction.

Relationship to Other Methods

Cohort Analysis receives input from:

• Controlled Rollout (the referenced method): Wave membership creates natural cohorts (Wave~1 users vs. Wave~3 users)
• Usage Analytics: Event tracking provides the raw data for cohort computation
• NPS (the referenced method NPS scores can be analysed by cohort to see if satisfaction correlates with retention