SBOM vs. SBOM: Comparing SBOMs from Different Tools and Lifecycle Stages | Endor Labs

What it is

A small Endor Labs experiment on Eclipse Steady 3.2.5 (module rest-backend, a Spring Boot app in Docker). It ran three unnamed open source SBOM generators and scored them against a Maven ground truth. The generators are anonymized as tools A and B (generic) and C (Maven plugin). The ground truth came from mvn dependency:tree -pl rest-backend: 114 compile, 2 runtime and 41 test dependencies. A component only counted as correct if its groupId, artifactId and version (GAV) all matched.

Results

• Tool A on source: it found only 20 components. It read pom.xml but resolved neither transitive dependencies nor versions. It also picked up test archives from src/test/resources.
• Tool A after package: it found the built JAR in target/ and reported 130 components (122 unique). That count includes test dependencies, which the SBOM can no longer tell apart from runtime ones.
• Tools B and C: their SBOMs did not change at all after the build, so they appear to rely only on pom.xml. Tool B gets some versions wrong and misses dependencies for reasons the page does not explain.
• Docker image: precision and recall were computed only on pkg:maven PURLs, because OS packages are not in the ground truth. Tool B still missed the Java runtime and one of its libraries, which tool A found.

Why SBOMs from different tools don't match

The same dependency, org.dom4j:dom4j:2.1.3, came out three ways:

A: pkg:maven/dom4j/dom4j@2.1.3                 (wrong groupId)
B: pkg:maven/org.dom4j/dom4j@2.1.3
C: pkg:maven/org.dom4j/dom4j@2.1.3?type=jar

B and C are both correct, but the type=jar qualifier alone breaks exact PURL matching. The same thing happens with apt PURLs for OS packages, where the tools disagree on qualifiers. Matching on name only raises overlap a lot, but the page notes that names are not enough to look up vulnerabilities on OSV or updates on Maven Central.

What to do

• Do not treat an SBOM from a generic scanner as authoritative for a Maven project. Where you can, generate it through the build tool, since the Maven-integrated plugin was the only perfect result.
• If you compare or merge SBOMs from several tools, normalize PURLs first (strip or standardize qualifiers like type=jar) or diff will show false disagreement.
• For Docker images, expect to need a generic scanner for OS-level components. Filter by PURL prefix (e.g. pkg:maven) when scoring against a language-level ground truth.
• Check whether the SBOM separates test or build dependencies from runtime ones. Tool A's output did not, so test dependencies will be treated like production ones.

Caveats

• Single project, single ecosystem (Maven/Java), and the tools are not named, so you cannot map the numbers to a product.
• Tool C could not scan Docker images, so it has no result at that stage.
• The page offers its explanation for tool C's perfect score ("supposedly" its Maven integration) as a guess. The causes of tool B's missed dependencies are unknown.
• Ground truth was only possible because Eclipse Steady is open source. The page says this check is impossible for proprietary software, and its call for a shared benchmark remains a proposal.