Skip to content

Back to the catalog

A Large-Scale Measurement of AI Bill of Materials Completeness in Hugging Face Models
arxiv.orgpaper

A Large-Scale Measurement of AI Bill of Materials Completeness in Hugging Face Models

This paper empirically studies AI Bills of Materials (AIBOMs) generated for ~97.5K public Hugging Face model repositories, measuring how completely they capture provenance, licenses, datasets, model-card details, safety-risk assessments, and environmental information. It finds that generated AIBOMs satisfy required structural/metadata schema fields but are weak or missing on AI-specific documentation such as model cards, responsible-use notes, limitations, and dataset declarations, exposing transparency and governance gaps in the ML model supply chain. The authors argue for better model-card practices, repository-level traceability, and automated AIBOM validation.

TL;DR Large-scale study of 97.5K Hugging Face AIBOMs finds required fields are complete but AI-specific documentation (model cards, datasets, limitations, safety info) is largely missing.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

Empirical study (arXiv:2607.17242, Erfan/Ryan/Rahman, University of Alabama) measuring AIBOM completeness across Hugging Face models. The concrete detail the summary omits: AIBOMs were generated with the OWASP GenAI Security Project AIBOM Generator, which takes a HF model identifier, scrapes repository metadata/model-card/license/config/external references, and emits CycloneDX JSON. The tool also produces a completeness score out of 100 across five weighted categories.

How it works

Dataset construction is reproducible: the authors snapshot 2,942,466 public HF model records and retain 97,940 with more than 100 downloads, then generate one AIBOM per model. Completeness is scored at four levels — overall score, category, individual field, and repository-characteristic. The five scoring categories are:

  • Required fields — minimal CycloneDX structure for model identification
  • Metadata — AIBOM/model purpose info
  • Component-basic — model identity and licensing
  • Model-card documentation — description, parameters, evaluation, limitations, ethical considerations
  • External references — links to source, datasets, docs, model files

Coverage is compared across task, license availability, dataset declaration, paper reference, model family, architecture, parameter size, and format.

What it establishes

Required-field coverage is complete because the generator synthesizes CycloneDX structure regardless of source content. Everything AI-specific — model-card fields, responsible-use, environmental, limitations, meaningful descriptions, and even general metadata — is weakly represented or missing. The gap is in the upstream repositories, not the schema: a valid, schema-complete AIBOM can still carry almost no governance-relevant information.

For an engineer acting on this

If you generate AIBOMs from HF models with this tooling, a passing structural score is not evidence of transparency. Validate at the field level. Prior tool ALOHA covered only 312 models; this scales the same CycloneDX approach to ~98K. Experimental data: https://figshare.com/s/185c21d7bf411fa4b952

Organisations named here

  • AIBOM

Technologies named here

  • Hugging Face
  • SBOM

Related in Supply Chain