case report/methodology
Methodology · selection criteria · self-assessment
Every criterion we applied, what it cost, and where the model goes next.
This page is the technical appendix to the case. It lists each gate and each scoring weight with its actual value from the pipeline, shows how many candidates each step removed, and closes with what each criterion contributed and how the next run of the model can sharpen it.
hard gates
scoring signals
score bands
identity-matching channels
weights reproduced verbatim from the pipeline scripts · criteria review written against the shipped artifacts
01 · Two kinds of criteria
Two kinds of criteria, never mixed
Confusing these two is the most common way a candidate search quietly goes wrong. We keep them structurally separate.
Gates
Binary
- A candidate either passes or is not in the file at all
- Never partially satisfied, never traded off against a high score elsewhere
- Five of them, listed in section 02
- Every gate is reversible: the pool below it is frozen and kept
Weights
Ordinal
- Decide rank inside the surviving pool, never membership
- Hand-tuned, and we say so — these are judgement calls, not learned parameters
- Each weight is written onto the row in a
score_breakdownfield - 27 signals across two axes, sections 03–04
Why
Why the separation matters
- A blended score lets a candidate in the wrong country outrank a perfect local match on follower count
- Separated, a client can audit membership and ranking independently
- Loosening a gate is a re-filter; re-weighting is a re-sort. Neither needs re-sourcing.
02 · Hard gates
The five hard gates
In application order, with the real rule and what each one removed. The funnel below is the open-source-anchored track, where every gate is observable end to end.
Bars are drawn to the scale of the 2,092 who passed gate 2. The first row is the full population, 70× wider than the chart — shown grey, off scale. The coral bar is the evidence gate: it kept 1.4% of the population.
Gate 1 · Geography
workedlocation matches any of 21 normalized city patterns — San Jose · Palo Alto · Mountain View · Sunnyvale · Cupertino · Santa Clara · Redwood City · Menlo Park · Los Altos · Saratoga · Campbell · Milpitas · Fremont · San Francisco · Oakland · Berkeley · Emeryville · San Mateo · Foster City · Burlingame · "San Francisco Bay Area"
The brief said "Bay Area — San Jose". We read that as a clarifier, not a narrowing: the hiring office is in San Jose, but the engineers commute from across the Bay. Excluding San Francisco alone would have removed the single largest city in the pool — roughly 34,500 developers.
Population at this gate: 145,624 Bay Area developers · distribution: SF 34.5K, San Jose 10.9K, Berkeley 5.6K, generic "Bay Area" 5.4K, Mountain View 4.5K, Palo Alto 4.1K, Oakland 4.1K, Sunnyvale 3.6K, Santa Clara 3.2K, Cupertino 1.3K, Redwood City 1.1K
Gate 2 · Stack evidence
strictest gate — see 10.1owns ≥ 1 non-fork repository in Java, Scala or C++ with activity in the last 18 months · C++ counts toward Shape Q only
This is the gate that turned a keyword search into an evidence search, and it is deliberately strict. 1.4% pass: every candidate who survives it has shipped recent JVM or C++ code under their own name. The gate tests ownership of a repository; the next version of the model also reads contributions to other people's projects, which widens the pool without loosening the evidence bar.
145,624 → 2,092 · removal: 98.6%
Gate 3 · Account hygiene
workedaccount type is an individual, not an organization · login does not match bot patterns (*-bot, *[bot]*) · surname is not a single character (corrupt-record guard)
Cheap and unglamorous. Without it, automation accounts belonging to the very projects we were scoring for would have ranked near the top on every activity signal.
19 records flagged with truncated surnames were kept, not deleted and not repaired — the data is as the source holds it, and inventing the missing characters would be fabrication.
Gate 4 · Current-employer exclusion
workedcurrently employed at the hiring company → dropped before scoring · previously employed there → kept, flagged, −3 score penalty
The exclusion runs before scoring, not as a filter on the output. A candidate who would have scored top of the pool never enters the ranking at all, so band sizes are not silently distorted by names that get removed later.
Code-anchored track: 4 removed at this gate (7 found earlier in the bridge stage). Profile-anchored track: 165 former employees kept and flagged.
Gate 5 · Deduplication
workedby canonical profile URL, then by composite key (first name + last name + current employer)
Two passes because the same person appears twice in different ways: the same profile reachable through two URLs, and two distinct records for one person at one employer. A single-key dedup catches only the first kind.
Verified at handoff: 0 duplicate identities, 0 duplicate profile URLs across 2,088 rows.
03 · Scoring axis one
Evidence of engineering output
Range 0–12, assembled from 15 independent signals. Every one is additive and recorded per-candidate, so any weight can be recomputed or zeroed after the fact without re-sourcing.
Recency and breadth of code
Reputation and reach
The three decisive signals
Why upstream contribution is worth more than every reputation signal combinedFollowers and stars measure attention. A merged contribution to a distributed query engine measures that a maintainer of that engine reviewed this person's code and accepted it. The brief asked for the second thing; the first is only a proxy that correlates with it.
04 · Scoring axis two
Professional fit
Range 0–15, ten signals. Deliberately orthogonal to axis one: one measures what a person builds, the other where they build it and at what level. A candidate can be strong on one and empty on the other, and the two-axis layout keeps that visible instead of averaging it away.
Seniority of current title
Employer, reach and history
Two refinements are already queued for this table.
Title classification will give "Senior" at least the weight of plain "Engineer" (today +1 against +2), and contactability moves out of fit into the actionability layer, where the second track already keeps it. Both are listed in section 10.
05 · Shape
Shape classification — which of the two engineers is this?
Run separately from scoring. A candidate is not "better" for being Shape S rather than Q — the client filters, we do not rank across shapes.
| Evidence | Toward S | Toward Q |
|---|---|---|
| Topic vocabulary in bio or headline | +2 (capped) | +2 (capped) |
| Contribution to a streaming project (Flink, Kafka, Pulsar, Beam, Samza, Storm, NiFi, Airflow) | +3 | — |
| Contribution to a query/engine project (Spark, Hive, Calcite, Iceberg, Druid, Pinot, Arrow, Trino, Presto, ClickHouse) | — | +3 |
| Recent Scala repository | +1 | — |
| Recent C++ repository | — | +1 |
Resolution: both sides ≥ 2 → S+Q (rare, and the most interesting profile in the file). One side ≥ 2 and higher → that shape. Neither reaches 2 → unclassified, kept in the pool and visibly labelled rather than force-assigned.
The first version of this classifier left 98.6% unclassifiedThe initial vocabulary only matched compound phrases — "stream processing", "query optimization" — and missed the way engineers actually write bios, which is bare tool names: kafka, trino, spark. We caught it on the distribution check, widened the vocabulary to match both forms, and re-scored the frozen pool. Cost: one re-run over data already in hand. Had we shipped on the first pass, the shape column would have been decorative.
06 · Bands
Bands — where the thresholds came from
Combined score is the sum of both axes, range 0–27. The tier cut-offs were set before the run and then recalibrated against the real distribution, which is worth stating plainly.
| Tier | Rule as originally planned | Rule as shipped | Why it moved |
|---|---|---|---|
| A+ | ≥ 18 | ≥ 14 with an email, or ≥ 16 without | Nobody reached 18. The ceiling was theoretical — no real candidate collects every signal at once. |
| A | ≥ 14 | ≥ 10 | Original cut produced a top band of 4 people. Too thin to be a deliverable. |
| B | ≥ 9 | ≥ 6 | Shifted with A to preserve band proportions. |
| C | ≥ 5 | ≥ 3 | Below 3 a profile carries essentially one weak signal — that is the honest noise floor. |
| drop | < 5 | < 3, or currently at the hiring company | — |
Recalibrating thresholds after seeing the distribution is legitimate; recalibrating them to hit a headline number is notWe moved the cut-offs because the score ceiling turned out to be unreachable in practice, and we recorded the original values so the change is auditable. We did not move them to make a band look fuller than it is — band A shipped with 17 people, not the 250 the plan wanted, because 17 is how many there were.
07 · Actionability
Actionability — the second track's ranking layer
On a 19,625-candidate pool, fit alone is not a work queue. Actionability answers a different question: of the people who fit, who can I actually reach today, with enough context to write a good first message?
// exact formula, as it ran // fit is doubled — still the dominant term actionability = combined_score × 2 + 15 if a usable email exists + 10 if full career history is present + 5 if confirmed upstream contributor + 5 if currently at a named competitor + 3 if currently at a large platform company + 3 if company funding context resolved + 3 if a public code identity is linked − 5 if formerly at the hiring company
Sorted descending, the top 300 became wave one, the next 1,000 wave two, the next 2,000 wave three.
Wave one · 300 candidates
Four sources per addressAddresses in this pool live across four different sources. The resolver walks all four in priority order, with corporate addresses preferred over consumer ones, which is how wave one reaches 100% email coverage. A coverage check on the summary line runs on every build.
08 · Identity
Identity matching — the criterion behind the criteria
None of the above is worth anything if the code identity and the professional identity cannot be joined. Each channel carries its own acceptance rule, and the rule is recorded on the row.
| Channel | Acceptance rule | Yield | Verdict |
|---|---|---|---|
| Canonical link | A code-host URL is already present on the professional record. No inference. | 270 | worked |
| Verified handle | Identical handle on both sides AND ≥ 1 name token in common AND country matches. All three required. | 407 | worked |
| Name + employer | First and last name plus current employer both match. | 37 | medium confidence |
The name-token requirement is the whole safety mechanism on channel twoHandle collisions are common — short, generic usernames belong to different people on different platforms. A pre-run probe estimated a 13.2% true-match rate with the name check enforced; the channel delivered 22%, meaning the guard was more conservative than reality, which is the correct direction for a guard to err. Without it, this channel would have been a guess with 407 rows of false confidence.
09 · Not applied
Criteria we deliberately did not apply
A list of absent filters is as informative as the list of present ones.
Years of experience. Requested implicitly by "mid–senior+", but seniority in this field is badly approximated by date arithmetic. We scored the title instead and left the raw history on every row so the client can apply their own rule.
Job-seeking signals. Public "open to work" markers are sparse, stale, and biased toward people already in a process. Filtering on them would have shrunk the pool toward the least differentiated candidates.
Compensation expectation. Not observable in any source we consider reliable. Any number we produced would have been invented.
Education. Present in the data, shipped on the row, deliberately given zero weight. For an engineer with public code and eight years of shipped platform work, the degree adds nothing the code has not already proven.
Response likelihood. Predicting who replies to outreach is a modelling claim we cannot back with evidence from this data. We report reachability — whether a channel exists — and stop there.
10 · Self-assessment
What each criterion contributed
Written against the shipped artifacts: what each criterion delivered, and how the next run sharpens it.
| # | Criterion | Status | What it did · how it improves |
|---|---|---|---|
| 10.1 | Ownership as proof of stack | improve | 1.4% pass rate: a high-precision pool of people who own recent JVM code. Reading contributions to upstream projects as well as owned repositories is the largest single lever for the next run, and the main route to more confirmed upstream contributors than the 11 found here. |
| 10.2 | Seniority ladder | improve | Title classification tests in a fixed order, so "Senior Engineer" scores +1 and a plain "Engineer" +2. A one-point rank shift; the next run reorders the ladder so Senior sits between Engineer and Staff. |
| 10.3 | Contactability as a signal | improve | On the first track an email adds +2 to fit; the second track keeps reachability in a separate actionability layer. The next run uses the second track's design on both. |
| 10.4 | Name + employer matching | improve | 37 matches from 303 attempts, labelled medium confidence on every row. Resolving employers to canonical entities before matching raises the yield of this channel. |
| 10.5 | Bio vocabulary as topic signal | tuned in-run | The vocabulary matches both compound phrases ("stream processing") and bare tool names (kafka, trino), tuned on the distribution check before delivery. |
| 10.6 | 21-pattern geography | worked | The single highest-leverage criterion on the page. A literal match would have cut the pool by roughly two-thirds before any other filter ran. |
| 10.7 | Upstream contribution at +5 | worked | Correctly the heaviest single weight — it is the only signal in the model where an independent maintainer has already validated the candidate's work. Volume grows with the contribution reading in 10.1. |
| 10.8 | Two-axis scoring | worked | Keeping output evidence and professional fit on separate axes made shape and tier independently filterable, and keeps every scoring choice visible instead of buried in one opaque number. |
| 10.9 | Competitor employment at +3 / +5 | worked | 78% of wave one currently sit at a named competitor — the exact population a hiring team wants first contact with. |
| 10.10 | Reputation signals (followers, stars) | improve | Six signals contributing at most 6 of 27 points, all measuring attention rather than capability. They work as tie-breakers; a leaner next run keeps three. |
Every delivery passes a blocking quality gate.
Each shipped file is audited against the brief before handoff, and a failing audit stops the delivery. This engagement was also verified by hand, row counts and coverage reconciled against the files.
11 · Next run
What we would change before running it again
Concrete, ordered by how much each would move the result.
Match on contributions as well as ownership. Join the contributor side of upstream projects directly, not only repositories a person owns. This is the largest lever on the open-source track: it widens the evidence pool and the count of confirmed upstream contributors.
Reorder the seniority ladder and keep contactability in the actionability layer. Two small edits, both listed above, both mechanical.
Calibrate thresholds on a sample before committing to band targets. Score 500 candidates, look at the distribution, then set cut-offs. This run recalibrated once against the real distribution and kept the original values for audit.
Run the distribution check on every classifier output, automatically. This run checked the distribution before delivery; the next one makes that check fail the build automatically.
Resolve employers to canonical entities before name-based matching. The weakest channel was limited by employer-name resolution, not by the matching logic itself.
Run both tracks from the start, always. The 25 double-confirmed candidates were the most valuable output of the entire engagement and only exist because two independent methods agreed. That is a repeatable design, not a lucky by-product.
12 · Checklist
The criteria checklist, generalised
What to specify for any senior technical search where demonstrated skill matters more than a job title.
| Layer | Decide before sourcing | Failure mode if skipped |
|---|---|---|
| Geography | The explicit list of localities, not a region name | Silent loss of the largest city in the region |
| Evidence | What artifact proves the skill, and whether you can actually observe it | Measuring a proxy for the thing instead of the thing |
| Seniority | Which titles rank above which, ICs versus managers | Classification order silently inverts your intent |
| Exclusions | Who is out, who is flagged, applied before scoring | Distorted band sizes and awkward outreach |
| Fit vs reachability | Two separate scores, never one | Contactable mediocre candidates outrank strong unreachable ones |
| Match confidence | A label per row, never a blended average | A client cannot tell a verified match from a guess |
| Expansion | Which filter loosens first, and what it costs | "We need more" becomes a new project instead of a re-filter |