DRILS: Dependency-based Requirement Change Impact Analysis Using LLMs

What it claims

DRILS identifies which natural-language requirements are affected by a change request. It combines an LLM multi-agent pipeline with a Requirement Dependency Graph (RDG). On three automotive systems it averages 87.1% recall, 76.3% precision and 81% F1. Against its closest baseline, ProReFiCIA (an LLM-based CIA technique), average F1 rises from 55% to 81% and precision from 41.33% to 76.3%. The authors are from the University of Ottawa and General Motors. They argue that precision is the gain that matters, because every false positive is a requirement an analyst must check by hand.

How it works

DRILS takes three inputs: the system's requirements list, an RDG of dependencies among them, and the change request. RE-RAG also uses the Software Requirements Specification (SRS) document. The pipeline has four steps.

1. RE-RAG. Three agents work here. A resolver removes referential ambiguity, such as an unstated actor. A summarizer condenses the SRS. An enricher uses that summary to add domain context to each requirement as short bullet points, which keeps noise low.
2. Clustering. DBSCAN groups the enriched requirements by semantic similarity, so large specifications are processed one cluster at a time.
3. Impact set identification. For each cluster, two generator agents work independently with the change request and the RDG. Each returns impacted requirements, a rationale, the location in the requirement to change, and a confidence score.
4. Validation. A judge agent compares the two generators' outputs and returns the final impact set, with a rationale for each requirement.

flowchart LR
  SRS[SRS document] --> E[RE-RAG enrichment]
  R[Requirements] --> E
  E --> C[DBSCAN clusters]
  C --> G1[Generator 1]
  C --> G2[Generator 2]
  CR[Change request] --> G1
  CR --> G2
  RDG[Dependency graph] --> G1
  RDG --> G2
  G1 --> J{Judge agent}
  G2 --> J
  J --> F[Impact set + rationales]

Evidence

• Baselines. DRILS is compared with four baselines, one from each of four CIA technique categories. It beats all of them on every metric, at lower cost. Its relative precision gain over the closest baseline is 41.1%, 83.4% and 149.8% on the three datasets.
• Auxiliary outputs. The generators' rationale, change location and confidence score all help. Together they raise average recall from 71% to 87%. On one dataset recall goes from 62.5% to 91.6%.
• Components. RE-RAG, clustering and the RDG each help. The RDG has the largest and most consistent effect, raising F1 from 77% to 81%, a 5.2% relative gain.

How to use it

Adopting DRILS requires an existing RDG and an SRS document, in addition to the requirements and the change request. The fetched text does not say how the RDG is built. The three manually annotated change-impact datasets are released. The excerpt gives no licence, repository link, or LLM model names.

Limits

All evaluation uses three automotive systems, which are safety-critical domains with certification constraints. Results in other domains are untested here. The introduction reports 80.8% F1 and 87% recall, while the abstract reports 81% and 87.1%, so the figures are rounded inconsistently. The page claims lower cost but the excerpt gives no cost figures. The fetched text ends in Section 2.1, so the experimental setup, the per-dataset tables and the threats-to-validity section are not covered here.