How we found 24 Android vulnerabilities using our open source AI security agent

What it is

GitHub Security Lab's seclab-taskflow-agent includes a mobile-specific audit mode built from two custom prompt files layered on top of a general taskflow framework: gather_mobile_entry_point_info.yaml (classifies entry points as mobile vs. non-mobile so the agent targets the right attack surface in mixed repos) and an edited classify_application_local.yaml (seeds the LLM with a checklist of known Android vulnerability classes per entry-point type, e.g. intent-based entry points get checked against confused-deputy and insecure-broadcast patterns).

How it works

flowchart LR
    A[gather_mobile_entry_point_info] --> B[classify entry points:\nmobile vs non-mobile]
    B --> C[classify_application_local:\napply vuln-class checklist per type]
    C --> D[repeated runs: strict checklist]
    C --> E[repeated runs: open-ended LLM creativity]
    D --> F[audit_results SQLite table]
    E --> F

Running both the strict, checklist-driven prompt and the broad, open-ended prompt across multiple runs is deliberate: the strict pass catches known bug classes reliably, the open pass lets the model surprise you with logic bugs outside the checklist.

What to do

To run it yourself (requires a GitHub Copilot license; consumes premium model requests and can be token-heavy):

./scripts/audit/run_mobile.sh myorg/myrepo

Start a codespace in the seclab-taskflows repo first; a medium-sized repo takes one to two hours. Results open in a SQLite viewer — check the audit_results table for rows with a checkmark in has_vulnerability.

Caveats (stated in the post)

• Every AI finding needs review by a researcher with mobile security knowledge — the LLM still surfaces low-severity or near-impossible-to-trigger issues even when told not to.
• Severity estimates are frequently wrong because the model misses mitigating factors (e.g., a path traversal confined to external storage is low severity, but the LLM may not account for internal storage taking priority over attacker-controlled external data, producing false positives).
• Fixing this currently requires explicitly prompting the model to build a proof-of-concept (forcing it to attempt exploitation) or giving it a debugger — GitHub frames this as a current limitation that should shrink as model context and reasoning improve, not a solved problem.
• Where the model does well: inferring safe/unsafe API behavior across languages well enough that PoCs needed little manual correction.