How we found 24 Android vulnerabilities using our open source AI security agent

What it claims

GitHub Security Lab adapted its existing audit taskflows to Android and reports 24 vulnerabilities found so far. Most are simple, such as path traversal. A handful are critical, including the OsmAnd and Wikipedia chains. The author notes that findings land where an Android researcher would expect: cross-app scripting in WebViews, exposed JavaScript bridges, exported components that trust intent extras, and suffix-based hostname checks.

How it works

The Android work builds on earlier generic audit taskflows and changes two things:

1. gather_mobile_entry_point_info.yaml (new) sorts the discovered entry points into mobile and non-mobile. This lets a monorepo that also holds web servers or desktop apps be audited against the correct attack surface.
2. classify_application_local.yaml (edited) gives the model a fixed checklist of vulnerability classes for each entry-point type. For example, an intent-based entry point is always checked for confused deputy and insecure broadcasts. The rationale is that mobile bug classes are less widely known and LLM output is non-deterministic.

Results come from combining runs:

• Strict checklist prompt, run repeatedly: keeps obvious bugs from being missed.
• Broad, open-ended prompt: leaves room for the model's creativity.

A separate "create a proof of concept" pass is used to make the model try to exploit each finding, which exposes mitigating factors.

The two bug patterns the taskflows caught generalise:

• Trusted-caller extras on an exported activity. OsmAnd's MapActivity read silent_import, replace and export_type_list_key as if they came from its AIDL service. Any app, even one with no permissions, can set those extras.
• Hostname checks done with endsWith. The Wikipedia app used endsWith(WikiSite.BASE_DOMAIN) both for deeplink routing and in SharedPreferenceCookieManager.kt:101. evil-wikipedia.org passes both checks.

How to use it

Requirements and costs:

• A GitHub Copilot licence is required.
• Runs consume premium model requests and many tool calls, so token use can be large.
• A medium-sized repo takes about an hour or two.

To run it, start a codespace on the seclab-taskflows repository, wait for it to initialise, then run:

./scripts/audit/run_mobile.sh myorg/myrepo

When the run finishes, an SQLite viewer opens. Open the audit_results table and filter for rows where has_vulnerability is checked.

Limits

• Noisy output. The model reports low-severity bugs and bugs needing near-impossible states, even when told not to. Every finding needs review by someone who knows mobile security.
• Unreliable severity. The model misses mitigating factors unless pushed through a PoC pass. Example: a path traversal confined to external storage is low severity.
• Wrong assumptions about data flow. The model may assume attacker-written external-storage data affects the app when internal storage actually takes precedence, producing a false positive. The stated fixes are giving the model a debugger to run the PoC against the original code, or prompting it explicitly to look for this.
• Where it does well. The model knows security-relevant API behaviour well (e.g. Go's path.Clean versus filepath.Clean), and generated PoCs usually needed little modification.
• No measured rates. The page gives no false-positive rate, no per-run counts and no breakdown of the 24 findings. The model behind Copilot is not named.