Beyond the CVE: Analyzing the Depth of GitHub Security Advisories

Table of Contents

• Understanding the GitHub Security Advisory Databas …

• Project Overview and Ecosystem Support

• Evaluating Security Advisory Coverage

• The Necessity of Manual Review

• Speed of CVE Reviews

• Which Ecosystems Receive the Most Attention?

• Conclusion & Google’s osv.dev

Understanding the GitHub Security Advisory Database

Are you familiar with the GitHub Security Advisory Database? It’s a must-know tool for anyone who cares about open-source software vulnerabilities. In my opinion, it is crucial for two reasons:

1. There are over 250,000 CVEs, but only a tiny fraction impact open-source software. There is no easy way to filter the CVE database for those impacting open-source software.

2. Many open-source project maintainers prefer the control and simplicity of GitHub security advisories over the CVE submission process. >2,000 GitHub Security Advisories have no corresponding CVE.

Note: GitHub is an authorized CVE Numbering Authority (CNA). Therefore, GitHub can assign a CVE to security advisories under their scope.

Project Overview and Ecosystem Support

The open-source project is described as a “Security vulnerability database inclusive of CVEs and GitHub originated security advisories from the world of open source software.”

Currently, as per the project’s GitHub repository, the database ingests security advisories from various sources and supports the following ecosystems:

The above were counts reported by GitHub. However, adding up the number of advisories per ecosystem gives 19,557 🤔. By parsing the repository, I got a total count of 19,316.

Note: Packages often have numerous releases/versions. PyPi reports 550,000 unique packages but almost 6,000,000 releases. If we apply this ratio (10 releases per unique package), we’re talking ~60,000,000 total packages for these supported ecosystems.

Evaluating Security Advisory Coverage

How do you feel about <20,000 security advisories for ~60,000,000 total packages?

Additionally, are these numbers signs that specific ecosystems are deprived of security? Also, should these figures be considered when determining ROI for security scanning solutions?

Everyone seems to agree that open-source software vulnerabilities are essential. However, if you’re primarily writing Dart/Flutter code, is it worth deploying a tool to identify the ten known vulnerabilities?

🌶️ Truthfully, these numbers alone have made me disagree strongly with the belief that open-source software is more secure.

The Necessity of Manual Review

This project is undeniably a gift to the open-source community, and I’m grateful for all of GitHub’s efforts. But it’s not a silver bullet. Specifically, advisories require manual intervention to become “approved” security advisories, and currently, there are >220,000 unreviewed advisories in the queue, and god knows how many more aren’t.

This also means security advisories tend to stop, or at least slow down, around certain times of the year for holidays. And there isn’t a transparent distribution of efforts across the ecosystems. Depending on the week, you may see more Maven advisories versus NPM or vice versa.

Speed of CVE Reviews

The following graph shows that GitHub is reasonably quick. As shown, GitHub is up to par with NIST’s NVD; for some, they are even faster!

Similarly, in 2023, we see similar results, but with almost 3,000 data points.

Which Ecosystems Receive the Most Attention?

I don’t know why specific ecosystems get more love than others, and I don’t want to speculate. If I were in their shoes, I’d likely consider the (GitHub Advanced Security) paying customers despite being an open-source project.

What sparked this blog post idea was recognizing how many new composer (Packagist / PHP) advisories were being reviewed. Looking at the graph for 2024, it appears my observation was correct.

However, compared to 2023, the spread looks fairly identical, besides fewer reviewed Maven advisories in 2024. Comparing 2023 and 2022, they look even more similar.

Conclusion & Google’s osv.dev

Overall, this analysis supports my belief that the GitHub Security Advisory Database is an invaluable source of vulnerability data for open-source projects.

However, the database sadly does not include malicious advisories identifying known malicious projects. GitHub maintains this data, which is queriable via the web interface, revealing over 10,000 known malicious projects.

Fortunately, Google has an open-source vulnerability database (osv.dev), which ingests GitHub Security Advisory data, including malware advisories and additional sources not supported by the GitHub Security Advisory project.

The best part is that Google has this database available for free download and keeps it continuously up-to-date.

Therefore, if I had to pick one source for my vulnerability data, I would go with osv.dev. But I’d be naive to compare the two. The quality of osv.dev advisories largely relies on the work done by its sources, such as the GitHub Security Advisory project.

Until Next Time! 👋 

Hey, you made it to the bottom – thanks for sticking around!

Questions, ideas, or want to chat? Slide into my inbox! 💌

Don’t hesitate to forward if someone could benefit from this.

See you next Monday!
-Kyle

P.S. CramHacks now has a Supporter tier! You can upgrade here to support CramHacks and its free weekly content 😃.