We currently see more and more vulnerabilities being discovered across the board of open-source projects, which we all have been using in production for over a decade. Obviously, the first thought might be “ARRRG, that’s all because of this AI stuff which scans all the code and creates so much more work for us” and that might not be entirely wrong. At the same time, there might be another driver for this increased amount of CVEs in the last months. Adoption. We currently see more and more people adopting open-source software, like OpenStack, on their journey towards digital sovereignty. And more adoption almost always means more people working with the code and discovering paths that were rarely used before. Finally, this increased adoption itself might lead to just another reason for an increased number of CVE reports. More people using a software makes this software more interesting as a target for attacks. So discovering a vulnerability in OpenStack might actually be worth a lot more now, than it was 10 years ago.
But enough for the reasoning why we see so many CVEs at the moment. How can we fix it?
The process of fixing CVEs in open-source code itself is relatively straightforward: find the Bug/Gap/Loophole, create a patch (or multiple) for it, test it, merge it, build it, ship it. Done!
Or are we?
I think a large part of the work to actually fix the CVE begins after the code has been fixed and the new software build is available. It needs to be rolled out to production environments with real workloads on it and people behind them that pay for uptime and reliability.
With the previous rather slow discovery of CVEs, a regular patch interval within the usual maintenance windows of every couple of month or so was perfectly fine. From time to time, there was a bigger issue, and we had to announce an additional maintenance windows to fix this, but that was rather an exception. With the current speed, it seems, patching more than once a month, has become the new required normal. And even when this does not always require a full restart on all layers with actual customer impact, it almost always requires a certain amount of preparation and testing to not cause any significant impact in production.
How do you currently deal with this increased amount of work? Or is your existing patch management and rollout process already good enough to handle this amount of patching? How often do you inform your customers about those CVEs and patches, or how often do they ask? What do you think is needed on the orchestration, testing and notification level in the future to cope with the workload generated by AI scanning for vulnerabilities? Are there any areas where you think the concept of open operations can be employed to help here?
EDIT:
- This thread is discussing how to improve the notification part: Creating an "SCS Security Team"