Are Automated PRs Closing the Gap in Dependency Updates?

TL;DR Author Conclusion:

• Not as many popular projects use automated PRs as I initially thought.

• Using dependabot to keep GitHub Actions up-to-date seems much more popular than I expected.

• I’m still unsure how I feel about auto-upgrading dependencies (both GitHub Actions & software dependencies) without some form of risk calculation, specifically regarding malicious releases.

• The response rates for generated PRs were incredibly fast for the sampled projects.

If you’re reading this, you likely fall into one of these three categories: your organization

1. does not update software dependencies.

2. only updates software dependencies when a new desirable feature or a known security vulnerability exists.

3. receives automated PRs to update dependencies that get merged on a regular schedule.

This blog will focus on #3, automated PRs, and solutions like Dependabot and Renovate. Are they effective? What can we extrapolate by looking at the historical usage of these solutions on major open-source projects?

Recently, I was asked, “How many hours does it take to upgrade a package to fix the average dependency vulnerability?” And my brain pretty much melted from trying to develop a data-driven response.

This led me to discover a 2017 paper titled Can Automated Pull Requests Encourage Software Developers to Upgrade Out-of-Date Dependencies? by NC State’s Samim Mirhosseini & Chris Parnin.

Notes:

• The study included users of greenkeeper (acquired by Snyk), a bot that automates updating dependency versions, and travisbot, a continuous integration build to identify breaking changes.

• The overall lag time (in days) for these major projects to update dependencies was 85 days with automated PRs, 113 days when a badge notified maintainers of available updates, and 114 days in the control (nothing).

• 32% of PRs were merged for projects that automated CI build checks; without this, 24% of PRs were merged.

• 2% of failing builds were merged anyways 🤔

85+ days for a dependency to get upgraded, if at all, sounds astronomical. But, given that, in 2024, most organizations only upgrade dependencies if there is a known vulnerability, the data is less shocking.

This led me to investigate how popular open-source projects use or don’t use automated dependency upgrade solutions - and, if so, their general effectiveness.

Finding popular projects that use automated PRs for dependency updates; top 20 downloaded NPM packages in the past year:

This might seem like a waste of time, but here’s what I learned:

• Confirmation that top downloaded npm packages typically don’t require dependencies. They are the foundation of the npm ecosystem, often introduced as a dependency or transitive dependency.

• Dependabot can help maintain up-to-date GitHub Actions in addition to software dependencies.

• 0% of projects used Renovate. This is likely because it focuses on software dependencies and breaking changes.

Clearly, we need a different method for finding major projects using automated PRs to upgrade software dependencies. To do this, I searched GitHub for dependabot.yml files and chose four widely used and actively maintained projects.

It was at this moment, he realized how few open-source projects automate dependency updates.

electron

For electron, dependabot is configured for:

• Weekly updates on GitHub Actions

• Daily updates, with a limit of two open PRs, for npm packages in the following directories:

  • /

  • /spec

  • /npm

Focusing on the main branch, there have been 111 PRs generated by dependabot; 80% (89/111) have been merged, two are currently open, and the remaining were closed/ignored.

Only four of the last 50 PRs affected production dependencies; three were merged, and one was closed/ignored.

Overall: 80% merge rate

*75% merge rate for production dependencies in the last 50 PRs

core

For core, dependabot is configured for:

• Weekly updates on composer packages for the / directory, with a limit of 10 open PRs

• Weekly updates on npm packages for the /build directory, with a limit of 10 open PRs

Focusing on the master branch, there have been 316 PRs generated by dependabot; 56% (177/316) have been merged, seven are currently open, and the remaining were closed/ignored.

Forty of the last 50 PRs affected production dependencies; 32 were merged, and eight were closed/ignored.

Overall: 57% merge rate

*80% merge rate for production dependencies in the last 50 PRs

RSSHub

For core, dependabot is configured for:

• Daily updates on GitHub Actions for the / directory, with a limit of 100 open PRs

• Daily updates on npm packages for the / directory, with a limit of 100 open PRs

Focusing on the master branch, there have been 2,134 PRs generated by dependabot; 85% (1,808/2,134) have been merged, two are currently open, and the remaining were closed/ignored.

27 of the last 50 PRs affected production dependencies; 24 were merged, two were closed/ignored, and one is currently open.

Overall: 85% merge rate

*92% merge rate for production dependencies in the last 50 PRs

wxt

For core, dependabot is configured for:

• Monthly updates on npm packages for the / and packages/* directories

• Monthly updates for GitHub Actions on the / directory

Focusing on the main branch, there have been 107 PRs generated by dependabot; 76% (81/107) have been merged, seven are currently open, and the remaining were closed/ignored.

23 of the last 50 PRs affected production dependencies; 20 were merged, three were closed/ignored, and two are currently open.

Overall: 76% merge rate

*87% merge rate for production dependencies in the last 50 PRs

Conclusion

Although this was a limited data sample, nearly all PRs generated by Dependabot from this analysis were merged or closed within 24-48 hours, a stark contrast to the 2017 study, where dependency upgrades lagged by over 85 days. This shift could reflect a change in culture, with today’s developers more accustomed to rapid dependency management workflows. However, considering that many of these PRs are generated monthly or quarterly—and often have limits on the number created per run—the true lag time for upgrading dependencies may still exceed 85 days in some cases.

In addition, a 75% merge rate for automated PRs seems quite favorable and even higher than expected, while the 84% merge rate for production dependency upgrades is particularly impressive.

Automated updates offer clear benefits in keeping dependencies current but also raise security questions. How do we balance the convenience of automation with a strategy that mitigates potential risks, like malicious releases or unvetted changes?