CVE Scan

The osquery project has a CI job which once a day scans for CVEs that are present and not yet addressed in its third party libraries. The scan is done by a python script at tools/ci/scripts/cve/third_party_libraries_cves_scanner.py, which uses the NIST database queried via their NVD APIs; a manifest file at libraries/third_party_libraries_manifest.json contains the list of third party libraries and their metadata necessary to correctly download the CVEs.

The manifest file format is validated everytime the CVEs download happen and on every PR by the script tools/ci/scripts/cve/validate_maninfest_libraries_scanner.py; additionally the third party library versions in the manifest are verified, to ensure that they are up to date with their state in the repository.

After having downloaded the list of CVEs, the script will open issues in the osquery repository, for all the unresolved CVEs, checking against the already opened ones to prevent duplicates, and it will go back in time up to 6 months old.
The issues can be recognized because they will be opened by the github-bot author and will have the security, libraries, cve and severity-<level> labels on them.

NOTE: This product uses the NVD API but is not endorsed or certified by the NVD.

Updating a third party library to resolve a CVE

The process of updating a third party library is the usual, but in the PR updating the library the contributor MUST:

  1. Link the CVE(s) issue(s) the PR is going to close, so that when it's merged, they are automatically closed
  2. Update the manifest and specifically the version and commit fields with the information of the new library. Remember that the commit has to be the one of the submodule in the osquery repository, which might not always match the commit of the library original repository.

Failing to do step 1. only leads to having to manually close those issue and link them back to the PR for tracking purposes.

Failing to do step 2. will lead to the PR not being mergeable because the CI checks the commit field against the actual git submodule commit. Note that if the commit is updated but not the version, this will not be detected by the CI and the periodic scan will use the incorrect version to download CVEs, finding again the fixed CVE and reopening the issue. If this happens, one just needs to do another PR that updates the version correctly.

Any other situation where the version is incorrect (older than what previously was or newer than what actually is) is still not detected, and will either cause the script to open issues for already fixed CVEs or to miss CVEs, so it's very important that the PR review process double checks the new version.

Important: Do not merge this kind of PR if the CI CVE scan job is running (which happens only once a day between 23:00 and 00:00 UTC), otherwise the job could start with an old view of the repository and open new issues on already fixed CVEs. If this happens, we just need to close or even delete those issues, but it's mostly to avoid additional work or confusions.

Ignoring a CVE not affecting osquery

There are cases where the API returns CVEs that are not affecting a third party library, not directly, but they are affecting other software that uses the third party library. There might be something we can do in the future to resolve what seems a bug in the API, but for now in the manifest it's possible to list CVEs that should be ignored, so that issues for those are not opened again in the future.

Additionally we often have the case where a CVE is not affecting osquery due to how or what parts of the third party library are used, so having a way to ignore a CVE helps with that too.

The process therefore is to:

  1. Open a PR which updates the manifest and specifically updates the ignored-cves field of the library the CVE comes from.
  2. Describe in the CVE(s) issue(s) the reason why they are going to be closed
  3. Link the above issues to the PR, so that they are closed when the PR gets merged

Important: As with updating a library, one has to ensure that the CI CVE scan job is not running

Adding a new library

When a new library gets added, the manifest needs to be updated too, otherwise the CI check that verifies the manifest in the PR will fail.

Currently the JSON format for a third party library as a submodule (taking as an example libdpkg) is:

"libdpkg": {
    "product": "dpkg",
    "vendor": "debian",
    "version": "1.21.7",
    "commit": "e61f582015a9c67bbb3791cb93a864cfeb9c7151",
    "ignored-cves": []
},

The name of the library, libdpkg, and the commit field must match the name of the folder containing the submodule source code folder, and the commit at which the submodule currently is, respectively.

The product, vendor and version fields are used in the NVD APIs instead, and they must match what the NIST database uses.
This is a matter of using the CPE search at https://nvd.nist.gov/products/cpe/search, and trying to find which are the correct product and vendor using a cpe like cpe:2.3:a:*<partial vendor guess>*:*:*:*:*:*:*:*:*:* and for the product cpe:2.3:a:<vendor>:*<partial product guess>*:*:*:*:*:*:*:*:*.

Another way could be to download the full dictionary from https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz, then playing with grep and awk.
For instance if we want to see all the unique products that the amazon vendor has:

cat official-cpe-dictionary_v2.3.xml | grep -Eo "cpe:2.3:a:amazon:[^\"]*" | awk -F ":" '{ print $5 }' | sort | uniq | less

For the ignored-cves refer to Ignoring a CVE not affecting osquery; the field will likely be empty at the beginning.

Special cases

Libraries without a CPE

Some libraries do not have a CPE assigned, so no CVEs will be found in the NIST database. We track these libraries in the manifest, because the validation script will check for it, but one can provide less fields; only vendor and commit are required.

The script though also needs to know that this is a library of that kind, so the contributor has to update the list of libraries that does not have a CPE in tools/scripts/ci/cve/osquery/manifest_api.py

Libraries not imported as a submodule

Right now there's only one case and ideally, the only, but openssl is not a submodule, so there's no commit to use to check if the manifest is up to date. version is used instead, and it's parsed from the CMake file at libraries/cmake/formula/openssl/CMakeLists.txt.
If it will ever happen that osquery needs to add another library of this kind, then logic to get its version should be written in the tools/scripts/ci/cve/validate_manifest_libraries_versions.py script.

Additionally the name of the library should be added in tools/scripts/ci/cve/osquery/manifest_api.py and finally the manifest fields requirements would be the same as for a normal library, just without the commit field.

Libraries not used in the release build

Another case is when the library is only used for testing purposes; this doesn't need to be tracked in the manifest, but still needs to be ignored by the script that checks that all the necessary libraries are present and up to date in the manifest. The script to update is tools/scripts/ci/cve/validate_manifest_libraries_versions.py (the current example is googletest).