Measuring Users' Security and Privacy Concerns About Third-Party Community Bots on Discord

What it is

A measurement study from Kofi Ampomah and Tu Le (University of Alabama), set for NDSS 2027 (arXiv, October 2026). It analyzes user reviews of third-party Discord bots to find what security and privacy (S&P) concerns users raise. The authors also release BotLens, a proof-of-concept auditing tool.

How it works

• Data: a Playwright crawler collected 201,733 reviews and bot metadata from Top.gg. After cleaning (reviews under 5 characters dropped, non-English text translated with Argos Translate), 183,142 reviews remained. English made up 49.9% and valid non-English codes 41.0%. Another 9.0% had undetermined language.
• Taxonomy: the 20-theme Android privacy-review taxonomy of Akgul et al. was extended with 11 bot-specific themes. These include Over-Listening, Permission Scope Mismatch, Message Content Access, DM Privacy, Gateway Intent Abuse, Webhook Misuse, Server Profiling, Voice Privacy and Bot Moderation Transparency. The new themes came from open coding of 500 S&P reviews by two researchers.
• Classifier: Llama 3 8B (temperature 0.1, run locally through Ollama) labeled reviews as the teacher. A DeBERTa student was trained on those labels. Of five labeling budgets (10% to 45% of the corpus), 30% scored best, with F1 = 0.938. Every classifier-positive review was then checked manually.

Findings beyond the headline

• The pipeline yield of S&P reviews is 4.82% overall. Confession and anonymity bots exceed 50%, more than ten times that rate.
• Only 0.77% of bots reference a privacy policy in their listing.
• The authors say star ratings may hide how often S&P issues come up and whether they are praised or criticized.
• Permission count and administrator status show only weak associations with reported concerns. The authors conclude that bot function and data sensitivity are more informative.

What to do

BotLens runs as an MCP server with six tools. It combines review-derived S&P rate, sentiment, themes and requested permissions for single-bot checkups and server-wide comparison. Code is at https://github.com/ua-spectral/BotLens, with the taxonomy and pipeline archived at https://doi.org/10.5281/zenodo.23583221. The fetched text does not list the six tools or give install steps.

Caveats

• Reviewers are self-selected and may not represent Discord users generally.
• All data comes from one marketplace, Top.gg.
• The fetched text stops partway through the methodology. Classifier precision and recall, the affective-expression results and BotLens evaluation are not covered here.