How AI Tools Shift (or Don’t) Open-Source Career Luck

How AI Tools Shift (or Don’t) Open-Source Career Luck

When you look at someone’s GitHub profile, it’s tempting to treat their track record like proof of talent. But new research from the arXiv paper suggests a more nuanced story for open-source software careers—one that’s familiar from studies of success in science and the arts. The big idea: for most people, the timing of their biggest hit is close to a lottery, while how large that hit can be depends partly on a stable personal factor and partly on momentum and luck.

This new study digs into open-source in a way earlier work couldn’t. Instead of looking at citations or audiences, it uses GitHub’s public activity record at massive scale (2015–2025) and measures impact by how widely a contributor’s packages get pulled into the dependency graph. They reconstruct 102.2M works by 6.15M contributors, and focus more tightly on 908k contributors whose repositories publish packages—where impact can be measured via downstream dependencies.

Even more interesting: the authors check whether the “luck vs. persistence” pattern changes when generative AI coding tools arrive—specifically around the release of ChatGPT. Their answer is basically: no detectable break in the structure of success once you account for normal aging. In other words, generative AI may change how much code gets written, but it doesn’t clearly reshuffle who has the durable advantage, nor how much of success is luck.

Why This Matters: Your “Open-Source Résumé” Isn’t a Pure Ability Score

Open-source platforms have become a hiring and reputation engine. Recruiters and engineers often treat visible activity as a proxy for skill, reliability, and taste. But this research matters right now because we’re in the middle of a transition: AI coding assistants are likely increasing overall output. If output rises across the board, what happens to the “signal” people use to judge contributors?

The study implies a warning: track records on open platforms can reliably reflect something about people—but not in the simple way most of us assume. The timing of a contributor’s biggest impact looks almost random; the durable advantage signal (the stable factor) exists, but it explains only a minority slice of why impact persists from one work to the next. A lot is “success begets attention begets success,” and that’s not the same as pure ability.

A concrete scenario where this is immediately useful: imagine you’re hiring maintainers for a critical infrastructure library. If you compare two candidates based on who has the earlier “best release,” this research says you’re probably ranking them by timing luck more than by capability. Instead, you’d want to evaluate consistency across many works (not just the peak), and be skeptical of “this one project succeeded, so they must be the best” logic—especially if you’re evaluating people who are at different career stages.

How does this build on prior AI research? Earlier studies on coding assistants tend to measure task speed or completion rates (and often find gains that are larger for less experienced people). This paper instead asks a career-level question: after AI tools arrive, do the winners and the luck structure change? That’s a different and more practical question, and it’s why their conclusions are actionable for platform-based hiring and evaluation.

What the Researchers Measured: Packages, Dependencies, and the “Biggest Hit” Test

The authors’ central move is to translate “impact” into something you can observe repeatedly over time in open source. They focus on contributors who publish packages, because for those repositories, impact can be proxied by how many downstream packages depend on them.

The dataset: GitHub activity at industrial scale

They use GitHub Archive to reconstruct commits from January 2015 through September 2025, yielding:

• 102.2M works (each “work” = one person’s contribution to one repository)
• 6.15M contributors
• Then, for the package-releasing subset where dependency impact is measurable:
  • 2.44M works
  • 908,417 contributors
  • 1.82M repositories

Dependency info comes from two ecosystem datasets:
- Libraries.io snapshots (June 2017 to January 2020; 5 releases)
- Ecosyste.ms dumps (August 2022 onward; 8 releases)

There’s a gap (Jan 2020 to Aug 2022), which they bridge via interpolation and explicitly test for how that might affect results.

Measuring impact via downstream dependencies

Impact is defined as how many packages depend on the repository’s packages, either:
- at two hops through the dependency graph (reach2), or
- via direct count (in-degree; an alternate measure)

They evaluate impact at horizons of 24, 36, or 60 months after a work begins. To handle the fact that ecosystems differ in size/density and change over time, they log-scale impact and center it relative to other works published in the same year and ecosystem.

Also important: a huge fraction of works have zero measurable downstream dependencies. At the 60-month horizon:
- 63.0% of works reach zero downstream projects (and 56% in their primary sample).

They keep zeros in the analysis, which (as they note) tends to make it harder to detect stable “skill” effects.

Credit assignment: contributors don’t get equal credit by default

Repositories are collaborative. So the authors define how to split credit for a “work” across multiple contributors using different rules:

They report all conditions across these credit rules, and they only “headline” results if the direction doesn’t flip.

The “biggest hit” and the lottery prediction

For each contributor’s career, the authors identify the single highest-impact work (the “biggest hit”). Then they ask:

If success is mostly luck, would that biggest hit land at a random position in the person’s career?

They test a lottery-like model: in a career with N works, the biggest hit should be equally likely to occur at any work index from 1 to N. They analyze not only the average position, but also distribution fit (e.g., total variation distance, permutation tests).

This is the same conceptual logic used in classic success research (science/arts), and it’s what makes the study comparable to earlier “random-impact rule” findings.

The Luck vs. Persistence Pattern Holds for Open Source—Mostly

Let’s translate the results into plain human terms.

RQ1: The timing of the biggest hit is almost a lottery

At the authors’ primary condition, they find a small but real deviation from pure randomness:

• mean biggest-hit position: 0.548
• lottery expectation: 0.562
• so hits lean slightly earlier than the lottery predicts

This deviation is quantified in a few ways (including total variation distance and permutation p-values). The key message: it’s close to random, and it appears consistently across the 54 pre-registered conditions.

Across conditions:
- 54/54 conditions reject the exact lottery model after multiple-testing correction.
- Yet the size is small: median total variation distance is 0.028 (max 0.052).
- The held-out half (their confirmation dataset) shows similar outcomes, with mean 0.550 vs. 0.562 at the primary condition.

So contributors don’t control when their biggest impact arrives in any strong way. You can work hard, but the peak’s timing still behaves a lot like chance.

Why hits are slightly earlier: attention advantage for early work

The authors propose a mechanism that fits the pattern: early works may accumulate attention as repositories mature and ecosystems form fewer competitors later.

And they find something telling:
- the “early lean” grows as careers get longer

When they raise the minimum career length requirement (e.g., from ≥3 to ≥5 to ≥10 works), the median early shift increases—meaning the longer people have been active, the more their best hit tends to look “front-loaded” rather than purely random.

This is consistent with a platform/evaluation dynamic, not some inherent switch in skill over time.

RQ2: Impact persistence isn’t all stable talent—momentum dominates

Now to the second part of the classic success decomposition: even if timing is lottery-ish, maybe the size of hits is predictable.

Using a decomposition method similar to the QQ-model (from earlier success research), they separate persistence into:
- a stable individual (positional) factor (think: durable advantage)
- momentum (success breeding success temporarily)

Across conditions, the stable component exists, but it’s modest.

At the primary specification:
- stable share (AA), in the held-out consistent estimate: around 0.251
- persistence ratio (stable share of work-to-work correlation): about 0.189
- in plain terms: roughly one-fifth of persistence comes from the stable factor; the rest is momentum

They also apply stricter “verdict” criteria, and note that their primary condition is slightly borderline under some checks—partly due to how years are handled (more on that below). Still, the broad conclusion is stable: persistence is real, but it’s not mostly “pure ability.”

An important caveat: measurement gaps can make momentum look bigger

The paper includes a technical-but-crucial point: there’s a missing dependency snapshot coverage between Jan 2020 and Aug 2022, so some impact values are interpolated. Interpolation can create smoother trajectories that look like momentum.

When they restrict to “clean cadence” time periods (or drop measurements crossing the snapshot gap), the decomposition shifts notably—sometimes dramatically for the strict verdicts. Their lottery result doesn’t move much; the persistence decomposition gets more believable when the calendar problem is reduced.

So: treat the timing result as extremely robust, and view the persistence split as robust in direction but sensitive in magnitude.

Generative AI Didn’t Break the Luck-and-Persistence Structure (Within Limits)

Here’s the headline you probably care about: did ChatGPT change the career physics?

Their era comparison design

They compare the same people before vs. after the generative AI era begins, but they’re careful because comparing across time naturally includes aging effects.

They define:
- pre-AI era: Jan 2015–Dec 2019
- post-AI era: Jan 2023–Sep 2025
- divide point is anchored around the timeframe of ChatGPT’s release (Nov 2022), with a focus on incumbents who remain active

They use a placebo divide for people who entered earlier (where “AI shouldn’t matter”), to estimate what aging alone would do.

The timing story: still lottery-like after AI tools arrive

For the biggest hit timing:
- in the held-out primary condition, lottery deviations remain small
- real vs. placebo timing contrasts are consistent with no meaningful change
- the interaction statistic for timing covers zero with wide uncertainty

So generative AI doesn’t appear to change when people hit their peak.

The persistence story: the stable factor might rise—but it rises like aging does

For the stable share (AA), they see a rise across the real divide, but placebo cohorts show a comparable rise too.

At the primary condition (held-out):
- stable share rose in the real cohort by about +0.088
- placebo cohort rose by about +0.060
- the difference-in-differences contrast leaves DD ≈ +0.035, with a 95% CI [−0.218, +0.230]

That interval is wide enough that the study can rule out only fairly large shifts. Their conclusion is:

• there’s no detectable break in the structure
• any true era effect must be relatively small relative to their detection limits

They also show that the observed raw increase can be explained by design artifacts:
1. censoring: post-era works have less time to accumulate dependencies
2. career stage differences: even within “incumbents,” comparing by calendar time changes where they are in their personal career arc

When they correct for these (matched-censoring placebo, career-stage alignment), the estimated era shift largely disappears.

What This Means for How We Read GitHub, Hire Maintainers, and Build Teams

Let’s turn the findings into decisions.

Don’t overinterpret a single “peak” repo

Because biggest-hit timing is nearly random, a person’s most successful project is not a clean diagnostic of intrinsic ability. If you’re comparing two engineers based on “who had the earlier blockbuster,” you’re likely measuring luck + platform lifecycle effects more than true capability.

Look for durable patterns across many works

The stable factor exists, but momentum and chance are major drivers. So the best practical proxy is consistency: does someone repeatedly contribute to higher-impact dependency outcomes, across projects and over time?

Expect momentum to be real—so provide early support strategically

If momentum explains the majority of persistence, then early visibility and early adoption matter. That suggests organizations should invest in:
- onboarding maintainers
- ensuring documentation and integration paths
- reducing friction for downstream adoption

Even if AI coding changes output volume, momentum can still dominate impact trajectories.

Generative AI likely changes throughput, not the “who wins” structure (at least here)

The study’s main conclusion is that generative AI tools didn’t clearly change who gets durable advantage or how much luck governs peaks. That doesn’t mean AI is irrelevant—it may change the amount of work and possibly the absolute level of outputs—but in this dataset it doesn’t show a large structural shift in career outcomes.

Key Takeaways

• Open-source follows the same broad success pattern seen in science and the arts:
  biggest-hit timing is close to a lottery; hit size/persistence depends partly on a stable factor and partly on momentum.

• In the primary analysis, the biggest hit arrives nearly randomly, with only a small lean toward earlier-career peaks (mean biggest-hit position 0.548 vs. lottery 0.562), and this appears in held-out results too.

• Persistence is real but not mostly “pure quality.”
  The stable factor explains about ~one-fifth of why impact persists between works (persistence ratio around 0.189 in the primary condition), while the rest is momentum (success feeding on itself for a period).

• Generative AI tools (ChatGPT era) didn’t create a detectable break in the luck-vs-persistence structure for incumbents:
  the stable-share shift is small and statistically indistinguishable from what aging and measurement artifacts would produce (DD about +0.035 with wide uncertainty).

• Practical advice for hiring and evaluation today:
  Don’t treat peak timing on GitHub as a strong ability signal. Prefer assessing consistency across many contributions, and design around momentum (reduce barriers to adoption early).

• How to interpret your own career strategy:
  You may not control when your biggest hit lands, but you can influence whether your work enters the ecosystem early enough to benefit from attention and adoption dynamics—which is where momentum can kick in.

If you want, I can also turn this into a “how to read a GitHub profile” checklist (what metrics matter, what to ignore, and what questions to ask maintainers) based directly on the study’s decomposition logic.