Rails Long-Tail Discovery Pipeline — Design

Date: 2026-04-16 Owner: Rick Gorman Supersedes: plans/logical-enchanting-wave.md (Rails Verticals Research v2)

1. Purpose

Build a pipeline that surfaces 1,500-5,000 long-tail Rails-using companies across ten locked verticals, so Rick can (a) pick a single niche to target for the April 2026 campaign, and (b) reach out to a curated subset within the chosen niche. The eventual commercial goal is ~100 customers for a product, not yet fully defined, that helps Rails teams level up their AI use.

“Long-tail” in this project means companies that combine two traits:

• Small headcount: under ~500 total employees (under ~200 engineers where detectable)
• Low public visibility: not in RailsConf sponsor decks, not featured on HN front-page, no high-readership engineering blog

This phase does not produce contacts (names, emails, LinkedIn URLs of specific decision-makers). Contact enrichment is a later phase scoped to the picked niche.

2. Architecture

Three sequential phases separated by a decision gate. Scripts and SQLite persist across phases so the pipeline is rerunnable.

Phase 0: Build scripts (3-4h, one-time, dispatched to Sonnet subagents in parallel)
    ↓
Phase 1: Shallow breadth-first collection across 10 verticals (3-4h)
  Scripts hit 6 free sources → raw rows into SQLite
  Sonnet batch-classifies each company (vertical, business model, long-tail fit, confidence)
  Human audits ≥5% per vertical; higher for noisy sources
    ↓
Phase 2: Niche pick (1h)
  Rick reads the vertical scorecard and selects one vertical
    ↓
Phase 3: Deep dive in chosen niche (3-5h)
  Add 2-3 vertical-specific sources
  Manual, human-only classification row by row
  Output: 500-1,500 curated long-tail rows ready for contact enrichment

Total budget: 10-14 hours across 2-3 elapsed days.

3. Sources

Shallow phase (all 10 verticals, free, scriptable)

1. GitHub code search. Match gem "rails" in public Gemfile files. Extract org. Enrich via GitHub API (public members count, description, homepage). Freshness = last commit date on any org repo within last 12 months.

2. Aggregated job boards.
   • HN “Who is Hiring” archive, last 12 months, filtered for Rails + Ruby on Rails.
   • We Work Remotely — Ruby category RSS feed.
   • RemoteOK — Ruby tag API.
   • Google dorks against Greenhouse, Lever, Ashby public boards: site:boards.greenhouse.io "Ruby on Rails", etc.

3. Rails Foundation directory. Scrape member companies, sponsor companies, and any “associated with” orgs from the public site.

4. Conference sponsor lists. Scrape sponsor pages from RailsConf, RubyKaigi, EuRuKo, and RubyConf for 2023, 2024, 2025.

5. Gem-maintainer orgs. Pull top 1,000 gems by downloads from the rubygems.org API. Map each gem to its GitHub repo and owning org. Filter to company-owned orgs (heuristic: org has a homepage domain, >3 public members, or description that looks like a company).

6. Random/exploratory web search. Hypothesis-driven Google/DuckDuckGo queries that structured sources will miss. Example patterns:
   • "built with Rails" site:*.com -github.com + industry term
   • "our Rails app" "engineering blog" + industry term
   • "Ruby on Rails" "case study" on Rails consultancy sites (thoughtbot, Planet Argon, Test Double, Crowd Favorite, Arkency)
   • Ruby/Rails podcast guest lists (Remote Ruby, Code with Jason) → guest employer
   • Dev.to / Medium Rails-tag author employer fields
   • Reddit r/rails and r/ruby threads where posters name their employer

   Scripts run a batch of these queries. Sonnet filters each result for “mentions a specific company + a Rails signal,” extracts the company name and evidence URL, and writes to signals with source='web-search' and signal_type='web_mention'. Audit rate for this source: 15-20% (noisier input).

Deep phase (chosen niche, added after Phase 2)

2-3 vertical-specific sources decided at pick time. Examples:

• If niche = healthcare vertical-SaaS → HIMSS member directory, KLAS Research vendor lists, medical-tech engineering-blog rings.
• If niche = B2B dev-tools → YC dev-tools batch alumni, StackShare dev-tools category, GitHub Marketplace listing owners.

4. Signal Quality Gate

A company remains in the pool only if it has at least one of the following:

• A Rails job post dated within the last 365 days
• An engineering blog post mentioning Rails dated 2020 or later
• Any evidence from sources 3-5 (Rails Foundation directory, conference sponsor list, gem maintainer org) — these are treated as standalone signals because the source itself implies active Rails investment

Companies failing the gate are kept in signals but their classifications.primary_vertical is NULL and they are excluded from scorecard counts.

5. Schema (SQLite)

CREATE TABLE companies (
  id              INTEGER PRIMARY KEY,
  domain          TEXT    UNIQUE NOT NULL,           -- canonical dedupe key
  name            TEXT    NOT NULL,
  headcount_tier  TEXT,                              -- 'XS' <20, 'S' 20-100, 'M' 100-500, 'L' 500-2000, 'XL' 2000+
  hq_country      TEXT,
  hq_region       TEXT,
  website_url     TEXT,
  linkedin_url    TEXT,
  github_org      TEXT,
  long_tail_fit   BOOLEAN,                           -- computed: headcount_tier IN ('XS','S','M') AND NOT in conference_sponsor set
  notes           TEXT,
  created_at      DATETIME DEFAULT CURRENT_TIMESTAMP,
  updated_at      DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE signals (
  id             INTEGER PRIMARY KEY,
  company_id     INTEGER NOT NULL REFERENCES companies(id),
  source         TEXT    NOT NULL,                   -- 'github', 'wwr', 'hn-hiring', 'railsfdn', 'conf-sponsors', 'gem-maintainer', 'web-search'
  signal_type    TEXT    NOT NULL,                   -- 'job_post', 'sponsor', 'code', 'blog', 'gem_maintainer', 'web_mention'
  signal_date    DATE,
  evidence_url   TEXT,
  raw_text       TEXT,
  captured_at    DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE classifications (
  id               INTEGER PRIMARY KEY,
  company_id       INTEGER NOT NULL UNIQUE REFERENCES companies(id),
  primary_vertical TEXT,                             -- one of the 10 verticals, or NULL if gate failed
  business_model   TEXT,                             -- 'marketplace', 'b2b_saas', 'dev_tools', 'vertical_saas', 'internal_tools', 'media', 'consumer'
  confidence       REAL,                             -- 0.0 - 1.0, reported by Sonnet
  llm_notes        TEXT,
  classified_by    TEXT,                             -- 'claude-sonnet-4-6' or 'human:rick'
  classified_at    DATETIME DEFAULT CURRENT_TIMESTAMP,
  audited          BOOLEAN DEFAULT 0,
  audit_verdict    TEXT                              -- 'agree', 'disagree', 'unclear'
);

CREATE TABLE source_runs (
  id           INTEGER PRIMARY KEY,
  source       TEXT    NOT NULL,
  started_at   DATETIME,
  finished_at  DATETIME,
  rows_added   INTEGER,
  notes        TEXT
);

Signals are append-only. Re-running a source adds newer rows without mutating history. Dedup happens at the companies layer via the unique domain constraint.

Ingestion pattern. Each scraper first issues INSERT OR IGNORE INTO companies (domain, name, ...), then looks up company_id by domain, then inserts one row into signals per hit. If a later scraper has richer data (e.g., a GitHub org description), it issues UPDATE companies SET ... WHERE domain = ? AND <field> IS NULL — existing non-null values are never overwritten automatically.

Ten locked verticals

Healthcare, BFSI (Banking / Financial / Insurance), Retail / E-commerce, IT / Telecom, Manufacturing / Industry 4.0, Media, Automotive, Cybersecurity, Education, Energy.

6. Classification (Shallow Phase Only)

• Batch size: 50 companies per Sonnet subagent call.
• Input per row: company name, domain, up to 5 most recent signals rows (job text excerpts, GitHub org description, sponsor context), plus the 10-vertical definitions and 7-value business-model taxonomy as shared prompt context.
• Output per row: primary_vertical, business_model, confidence (0-1), notes (one sentence of reasoning). The LLM does not set long_tail_fit — that column is computed deterministically by a post-classification script from headcount_tier and conference-sponsor membership.
• Parallelism: dispatch multiple Sonnet subagents concurrently. Each writes results to a scratch JSONL file; a small merge script loads them into classifications.

Audit sampling (human = Rick)

• All rows with confidence < 0.6 → flagged for review.
• Random 5% sample per vertical → flagged regardless of confidence.
• All rows with source='web-search' → 15-20% audit rate.
• Audit outcome stored in audit_verdict. disagree rows get re-classified manually.

7. Niche-Pick Gate (Phase 2)

Rick reads a scorecard built from SQL views. No formula picks the niche — Rick does, using the data.

Scorecard dimensions per vertical:

Outputs of Phase 2:

• One-page scorecard in RAILS_LONGTAIL_MAP.md
• Rick’s pick and rationale committed to the same file

8. Deep Phase (Chosen Niche)

• Add 2-3 vertical-specific sources (decided at pick time).
• Re-run shallow sources (1-5) with tighter filters scoped to the vertical (e.g., GitHub orgs whose README keywords match the niche vocabulary).
• No LLM in the classification loop. Rick reviews each row manually. This is the list he will actually reach out from; per-row judgment matters.
• Enrich website_url, linkedin_url, github_org where easy (scripted lookups, no paid services).
• Target: 500-1,500 curated long-tail rows in one vertical.

9. Outputs

• rails_longtail.sqlite — working database, committed to the project.
• exports/shallow_by_vertical.csv — per-vertical slices from shallow phase, for review.
• exports/deep_<niche>.csv — the final outreach-ready list.
• RAILS_LONGTAIL_MAP.md — methodology, shallow-phase totals, scorecard, niche choice and rationale, blind-spot disclosure, paid-source escalation list.

Paid-source escalation list

At the end of Phase 1 and Phase 3, the pipeline records any gaps where a paid source is estimated to have ≥95% probability of uniquely surfacing additional relevant companies. Examples: Apollo for private-company headcount data, BuiltWith for technographic confirmation, Sales Navigator for LinkedIn headcount filtering. These are not run now — they are documented as the next spend decision if free coverage proves insufficient.

10. Non-Goals (Out of Scope for This Phase)

• Per-company contact discovery (names, emails, LinkedIn URLs of VP/Dir Eng). Planned as a later phase after niche pick.
• Paid data sources (Apollo, Clay, ZoomInfo, BuiltWith Pro, LinkedIn Sales Navigator). Tracked in escalation list but not executed.
• Outreach mechanics (sequencing, copy, CRM setup).
• Final product definition for the JoyRudder/Forge offering.
• Reconciling this dataset with the v2 Delphi plan’s Gartner-comparability framing. This project’s deliverable is an outreach-ready opportunity map, not an analyst report.

11. Verification

• Spot-check: 10 random companies per vertical audited for current Rails usage (job post <365 days OR eng-blog post ≥2020).
• Capture-recapture estimate: compute overlap rate between source 1 (GitHub) and source 2 (job boards) per vertical as a rough coverage-bias indicator. Report in RAILS_LONGTAIL_MAP.md.
• Classification audit rate: require ≥5% sampled audit per vertical before declaring Phase 1 complete.
• Reproducibility: every source run recorded in source_runs with start/end timestamps and rows added.

12. Risks and Mitigations