Master Urlscan.io Dorking for Bug Bounty Hunting

Overview

This technical tutorial guides security researchers through utilizing Urlscan.io as an Open Source Intelligence (OSINT) and asset enumeration platform for bug bounty reconnaissance. The presenter demonstrates how Urlscan functions as a headless web sandbox that analyzes client-side page execution, network telemetry, DOM structures, and HTTP requests. Viewers are taught how to navigate scan reports, understand ElasticSearch-based search syntax, and leverage targeted "dorking" queries to identify attack surface components—including origin servers, undocumented APIs, directory indexes, cloud storage leaks, exposed credentials, and authentication portals.

Contents

0:00 Disclaimer

An introductory screen displays a legal notice:

DISCLAIMER! "This Channel DOES NOT Promote or encourage Any illegal activities, all contents provided by This Channel is meant for EDUCATIONAL PURPOSE only."

The presenter reiterates that ethical hacking requires explicit, written authorization before testing any asset, emphasizing that techniques demonstrated are strictly for educational purposes and responsible disclosure.

0:38 What is Urlscan.io?

The speaker introduces Urlscan.io as a free web service and sandbox that scans and analyzes websites. When a target is submitted:

1. Urlscan launches an automated, real browser instance (headless Chrome).
2. It browses to the specified URL or hostname.
3. It records network activity, HTTP requests, responses, DOM structures, cookies, console messages, and rendered screenshots.
4. It indexes these parameters into a searchable database with built-in dorking capabilities.

flowchart LR
    User[Target Submission] --> Browser[Automated Headless Chrome]
    Browser --> Analysis[Client-Side & Network Capture]
    Analysis --> Index[Searchable Database]
    Index --> Features[HTTP Transactions\nDOM Snapshots\nCertificates & DNS\nConsole & Cookies]

0:57 Urlscan Scanning Process & Results Walkthrough

The speaker demonstrates scanning a primary target: https://www.nasa.gov/. Entering the domain and clicking Public Scan runs the target through Urlscan's infrastructure.

Once complete, the inspection interface is split into structured modules:

1. Header & Summary Section (1:35)

• Target URL & IP Address: Shows the canonical scan URL, primary IP address (2a04:fa87:fffd:c000::426c), hosting location (Ireland), and primary autonomous system (AS 15129).
• Scan Timestamps: Exact UTC execution date and age of scan.
• Verdict: Automated classification identifying phishing or malicious content.
• Live Information: Active DNS records, registrar (get.gov), and original domain creation date (October 2, 1997).

2. Domain & IP Information (1:59)

Tabbed navigation providing granular network relationships:

• IP/ASNs: Categorizes connected IPs by their respective Autonomous System Numbers (e.g., Google, Amazon, Fastly).
• IP Detail: Reverse DNS (PTR) records, geolocation, and ASN descriptions.
• Domain Tree: Visual breakdown of primary domains and downstream requested hosts.
• Links & Certificates: Every TLS certificate encountered along with validity dates and Subject Alternative Names (SANs).
• Frames: Identifies nested iframes loaded inside the page.

3. Visual & Technical Details (2:10)

• Screenshot: Visual rendered image of the webpage.
• Page Title & Technologies Detected: Outlines identified frameworks and plugins (e.g., WordPress CMS, USWDS, DoubleClick Floodlight, Google Tag Manager, jQuery).
• Page Statistics: Overview metrics including total HTTP requests (109), HTTPS coverage (95%), IPv6 adoption (52%), domains contacted (11), subdomains (18), transfer size (12,092 kB), and cookies written (11).

2:25 HTTP Transactions, Links, and Behavior

HTTP Transactions (2:25)

The speaker emphasizes that this tab is vital for bug bounty hunters. It catalogues every individual resource requested by the page:

• Protocol: HTTP/1.1, H2, or H3.
• Method & Status Code: GET, POST, 200 OK, 204 No Content, 404 Not Found.
• Resource Details: Exact paths to loaded JavaScript libraries, cascading style sheets, media files, and API endpoints (e.g., /wp-content/plugins/nasa-blocks/).
• Initiator & Timings: Size and latency metrics for each network transaction.
• Hidden Assets: Uncovers private API endpoints, staging assets, or third-party webhooks that do not appear directly in visible source code.

Redirects & Links (2:53)

• Redirect Chains: Traces full redirect sequences (useful for discovering open redirects).
• Outgoing Links: Lists all internal and external hyperlinks discovered on the page. Helpful for discovering broken links that could lead to broken link hijacking.

Behavior Section (3:16)

Logs everything that executed within the browser runtime:

• JavaScript global window variables (126 variables indexed in the demonstration).
• Initialized client-side cookies with domains, paths, expiration dates, and values.
• Browser console messages, error exceptions, and active Security Headers (e.g., Strict-Transport-Security, X-Frame-Options).

3:31 Indicators, Similar Hits, DOM, Content, and API Output

• Indicators (3:31): A centralized security index listing contacted IP addresses, domain hashes, and threat indicators.
• Similar (3:46): Compares structural HTML, DOM hierarchy, and ASN hosting signatures to locate identical or structurally similar sites hosted on other servers.
• DOM Tree (4:07): The post-execution Document Object Model. Allows analysts to inspect dynamically rendered hidden form fields, inline developer comments, and CSRF tokens.
• Content (4:25): Raw text and form analysis extracted from the DOM.
• API Tab (4:44): Provides the full raw JSON data payload of the scan, suitable for programmatic parsing, automated CLI ingestion, or custom tooling.

5:04 External Lookups and Reporting

In the top right menu, the Go To dropdown allows researchers to pivot from Urlscan into other OSINT engines:

• VirusTotal
• SecurityTrails
• crt.sh (Certificate Transparency logs)
• Google Safe Browsing
• Censys
• Archive.org (Wayback Machine)

A Report button (5:19) provides a structured form to submit malicious scans for administrative takedown or verdict modification.

5:27 Understanding Search Operators & Query Syntax

Urlscan utilizes an ElasticSearch query string syntax. The speaker highlights that certain queries require signing into a free user account.

Query Formatting Guidelines (5:49)

• Terms & Operators: Supports boolean operators (AND, OR, NOT).
• Exact Matches: Wrap search values containing special characters in quotation marks ("...").
• Escape Characters: Forward slashes (/) within paths must be escaped using a preceding backslash (\/).
• Wildcards: Asterisk (*) acts as a multi-character wildcard; question mark (?) acts as a single-character wildcard.

6:15 Dork 1: Recently Scanned Assets

To focus only on newly discovered assets rather than historical artifacts, use relative date math combined with domain parameters.

page.ip:* AND date:>now-7d

• Explanation: Filters for any scan executed within the past seven days where an active IP address was resolved.
• Refinement: The date range can be customized (e.g., date:>now-30d, date:>now-365d).
• Bug Bounty Application: Identifying newly deployed web pages and subdomains launched within the last week.

6:58 Dork 2: URL Prefix Search

The page.url.keyword field matches the exact beginning of a URL path.

page.url.keyword:"https:\/\/www.paypal.com\/*"

page.url.keyword:"https:\/\/digitalassets.tesla.com\/tesla-contents\/image\/upload\/*"

• Explanation: Returns all URLs beginning with that specified directory prefix.
• Demonstration (7:30): Running this on Tesla's digital assets repository exposes internal documentation, standard contractual clauses, guidelines, and investor presentations (Investor-Day-2023-Keynote.pdf).

8:22 Dork 3: Third-Party Interactions

To locate websites that interact with a target brand without belonging to the target's primary apex domain:

domain:paypal.com AND NOT page.domain:paypal.com

• Mechanism: Queries scans where paypal.com was loaded (via scripts, API calls, or payment iframes), but the scanned host (page.domain) is external.
• Bug Bounty Application (9:05): Discovers merchant integrations, third-party payment workflows, CORS misconfigurations, and trust relationships across unrelated domains.

9:14 Dork 4: Fuzzy Domain Search

Fuzzy matching identifies permutations and typographical errors of a given target brand.

page.domain:/paypal.com/ AND NOT page.domain:paypal.com

page.domain:/tesla.com/ AND NOT page.domain:tesla.com

• Mechanism: Searches for domains containing closely related character strings.
• Findings: Identifies typosquatting domains, phishing infrastructure, spoofed login portals, and unapproved partner portals mimicking the target brand.

9:57 Dork 5: Regular Expression Domain Search

Regular expressions provide precise control over structural domain pattern matching.

page.domain:/payp[.]/ AND NOT paypal.com

page.domain:/tesla[.]/ AND NOT tesla.com

• Explanation: Captures domains starting strictly with payp. or tesla., followed by various top-level domains or internal staging suffixes.
• Findings (10:35): Uncovers internal infrastructure like tesla-control-panel.pages.dev, tesla-battery-experts.com, and tesla-service.in/us/.

10:44 Dork 6: IP and Date Range Filtering

Isolates historical records of assets hosted on a specific server within a bounded timeframe.

page.ip:148.251.45.170 AND date:[2018 TO 2019]

page.ip:49.12.22.100 AND date:[2018 TO 2026]

• Application: Used when mapping virtual host colocation or tracking historical web server infrastructure changes over defined years.

11:26 Dork 7: ASN and Organization Searches

Autonomous System Numbers (ASNs) aggregate all IP ranges routed by an entity.

page.asn:AS24940 OR page.asnname:hetzner

page.asnname:tesla

Reconnaissance Expansion Workflow (11:46)

1. Query bgp.tools for the organization name (e.g., "Tesla Motors, Inc." resolves to AS394161).
2. Search Urlscan using: ``text page.asn:AS394161 ``
3. Pivot through mxtoolbox.com or asnmap to map announced CIDR prefixes.

12:51 CIDR Subnet Range Dorking

To query an entire network block owned by an enterprise:

page.ip:209.133.79.0\/24

• Note: The forward slash must be escaped (\/).
• Findings (13:05): Maps all services hosted across that specific /24 block, exposing backend portals, custom VPN endpoints, and remote execution nodes.

13:14 Dork 8: WordPress Uploads

Targets publicly exposed WordPress upload directories across indexed domains.

page.url:"wp-content/uploads" OR filename:"wp-content/uploads"

• Findings: Returns uploaded PDFs, database backups, zip archives, spreadsheets, and leaked configuration files stored within poorly secured media directories.

13:46 Dork 9: SHA-256 and Favicon Hash Searching

Matches identical files using cryptographic checksums (SHA-256).

Favicon Hash Tracking (13:56)

1. Generate the hash of a target favicon using online tools (kristalz.io/favicon-hash).
2. Example hash for Apple favicon: ``text 5493c614cf25c73a1d46cd9b7d0a693fee578a74f41442938 ``
3. Query Urlscan: ``text hash:5493c614cf25c73a1d46cd9b7d0a693fee578a74f41442938 ``
4. Result (14:08): Identifies all domains serving the exact same favicon, surfacing phishing sites and staging servers.

Library / Script Hashing (14:13)

• Download a specific target JavaScript library via wget.
• Compute its SHA-256 checksum: sha256sum <file.js>.
• Search hash:<sha256> in Urlscan to locate every other asset running the identical JavaScript file or library version affected by a known CVE.

14:54 Wildcard and Negation Filtering

• Wildcard Subdomain Search (14:57): ``text tesla* `` Returns all domains, subdomains, and hostnames beginning with "tesla".
• Excluding Keywords (15:16): ``text tesla* -refund ` `text tesla* -refund -tesla.com -tesla.no ` Appends minus (-`) operators to strip irrelevant or noisy third-party marketing domains.

15:32 Origin IP Discovery via Cloud/VPS Filtering

A practical method to discover a website's real origin IP hidden behind a Web Application Firewall (WAF) or Cloudflare CDN:

page.domain:zomato.com AND (page.asnname:"Hetzner" OR page.asnname:"DigitalOcean")

• Logic: When targets route production traffic through Cloudflare/Akamai, older or auxiliary endpoints may be hosted directly on VPS providers (DigitalOcean, Hetzner, AWS, Linode). Querying the target domain constrained to these ASNs surfaces direct-to-origin IP addresses.

15:51 Advanced Dorking: Admin Panels and Login Interfaces

Targets sensitive web dashboards and management portals:

page.url:"admin"

page.title:"Control Panel" OR page.title:"Administrator" OR page.title:"Sign In"

page.title:"Control Panel" AND page.domain:gov

page.title:"Control Panel" AND page.domain:mil

• Findings (25:48 - 27:00): Demonstrates hits across government and military top-level domains exposing control panels (e.g., controlpanel.fleet-tc.gov.uk, democracy.aec.gov.au, my-brave1.delta.mil.gov.ua).

16:09 Searching for Sensitive Tokens, Keys, and Passwords

API Keys & Access Tokens

page.url:"key="

page.url:"token="

page.url:"jwt"

• Findings: Locates URLs that accidentally leak bearer tokens, active session IDs, API credentials, and JWT strings in query parameters.

Password Reset Endpoints (17:01)

page.url:"password-reset="

page.url:("reset-password" OR "resetpassword" OR "forgot-password" OR "forgotpassword" OR "password-reset")

• Findings: Identifies password recovery endpoints, verification handlers, and exposed password reset workflows.

17:26 Directory Indexing and Exposed Backups

Open Directory Indexes

page.title:"Index of /"

• Refining with Targets & Categories (17:55): ``text page.title:"Index of /" AND page.url:backup ` `text page.title:"Index of /" AND page.url:uploads ` `text page.title:"Index of /" AND page.url:private ` `text page.title:"Index of /" AND page.domain:mil ` `text page.title:"Index of /" AND page.domain:edu ``
• Demonstration (18:08): Displays accessible web directories exposing raw database archives, SQL dumps, internal documents, and scripts.

18:57 Targeting Specific MIME Types & API Endpoints

JSON Responses

page.domain:tesla.com AND page.mime'type:"application/json"

• Findings (19:15): Surfaces subdomains like employeefeedback.tesla.com exposing billing package definitions (package.json) and internal API inventory routes (/v1/inventory-results).

Historical Retrieval via Wayback Machine Integration (21:28)

When an identified asset returns a 404 Not Found live, the speaker uses an extension to load historical Wayback Machine snapshots (web.archive.org/cdx/search/cdx?url=https://www.tesla.com/sites/default/files/*). This allows the recovery of decommissioned PDFs, agreements, and internal guides.

22:04 Filtering by Web Server and HTTP Response Codes

Server Header Dorking

page.domain:tesla.com AND page.server:apache

• Identifies hosts utilizing specific web servers (e.g., Apache 2.4.6 CentOS).

Status Code Filtering

page.domain:tesla.com AND page.status:403

page.domain:tesla.com AND page.status:404

page.domain:tesla.com AND page.status:503

page.domain:tesla.com AND page.status:301

• Security Value:
• 403 Forbidden assets often indicate internal administrative panels accessible only via IP whitelisting or missing authentication headers.
• 503 Service Unavailable reveals internal pricing gateways or DNS lookup failures (e.g., apigateway-pricing-gateway.tesla.com).
• 301 Moved Permanently surfaces legacy subdomains redirecting to obsolete infrastructure.

28:46 Locating Swagger UI and API Documentation

API documentation provides blueprints of backend endpoints, request parameters, and authentication methods.

page.title:"Swagger UI" OR page.url:"swagger" OR page.url:"swagger-ui"

page.title:"Swagger UI" AND page.domain:gov

page.url:"/graphql" AND page.domain:x.com

• Findings (29:28): Uncovers public Swagger documentation on government domains (e.g., api.stateentityprofile.ca.gov/swagger/ui/index) and GraphQL endpoints on x.com.

29:50 Probing for Vulnerable Files, Scripts, and Actuators

PHP XML-RPC & PHPInfo

page.url:"wp-content/uploads" page.url:"xmlrpc.php"

page.url:"xmlrpc.php"

filename:phpinfo.php

filename:phpinfo.php page.domain:gov

• Risk: xmlrpc.php is frequently targeted for brute-force amplification and DDoS attacks. phpinfo.php exposes internal server configurations, PHP compilation flags, environment variables, and module versions.

Spring Boot Actuators (31:47)

page.url:heapdump AND page.url:actuator

• Risk: Exposes raw memory heap dumps (/actuator/heapdump), often containing plaintext database credentials, encryption keys, and active user sessions.

30:08 Searching for Sensitive Services & Database Portals

Surfacing public monitoring, log ingestion, and database management dashboards:

page.title:"Grafana" OR page.title:"Kibana" OR page.title:"Prometheus"

page.title:"phpMyAdmin" OR page.title:"Adminer" OR page.title:"Mongo Express"

• Findings (30:25): Returns unauthenticated or exposed database management login screens (phpMyAdmin, Adminer) directly connected to production databases.

30:31 Cloud Storage Bucket Enumeration (S3, GCP, Azure)

Finds sensitive files hosted on major cloud object storage providers:

page.url:"s3.amazonaws.com" OR page.url:"blob.core.windows.net" OR page.url:"storage.googleapis.com"

• Refining with File Extensions & Organization Names (37:23): ``text page.mime'type:"application/zip" AND "tesla" ` `text page.mime'type:"application/zip" AND "apple" ` `text page.mime'type:"application/pdf" AND page.url:confidential ` `text page.mime'type:"application/pdf" AND page.url:internal ``
• Demonstration (38:20): Downloads leaked zip archives and sensitive PDF guidelines directly from unprotected storage buckets.

33:34 Searching for Sensitive Paths & LFI Artifacts

Identifies scans where Local File Inclusion (LFI) payloads or system files were requested:

page.url:etc/passwd

page.url:etc/shadow

page.url:shell

page.url:callback

• Findings (34:07): Shows URLs containing traversal sequences (e.g., ../../../../etc/passwd) captured during automated or manual penetration testing scans previously run on Urlscan.

34:53 Target Environment Subdomains & PTR Records

Environment Subdomain Discovery

page.domain:tesla.com AND page.url:stage

page.domain:tesla.com AND page.url:prod

page.domain:tesla.com AND page.url:dev

page.domain:tesla.com AND page.url:test

page.domain:tesla.com AND page.url:auth

• Findings: Efficiently isolates environment-specific subdomains such as stage.atom.tesla.com, auth.tesla.com, and sso-dev-origin.tesla.com.

Reverse DNS (PTR) Record Enumeration (39:57)

page.ptr:tesla.com

• Mechanism: Queries reverse DNS records where tesla.com appears in the PTR hostname mapping.
• Findings (40:04): Surfaces marketing, events, and communication platforms (e.g., events.tesla.com, marketing.tesla.com).

40:09 Conclusion & Outro

The speaker concludes by advising researchers to combine these manual dorks into automated workflows. Viewers are encouraged to subscribe, follow the presenter's Medium channel for additional bug bounty dorking cheat sheets, and continuously experiment with operator combinations.

Data & Visuals Index

Key Takeaways

• Automated Client-Side Telemetry: Urlscan.io operates a headless browser sandbox that records post-render DOM structures, network transactions, executed JavaScript variables, and dynamic HTTP requests that standard crawlers miss.
• Precision Substring Matching: Leveraging page.url.keyword enables researchers to enumerate unlinked files, documents, and uploads under known directory prefixes.
• Mapping Hidden Trust Boundaries: Dorks using domain:<target> AND NOT page.domain:<target> identify external sites embedding the target's scripts, forms, and payment gateways, uncovering third-party attack vectors.
• Origin Server Discovery: Filtering a target domain by cloud and VPS providers (such as Hetzner, DigitalOcean, or Linode) can identify origin servers bypassing Cloudflare and WAF protections.
• Network Block Mapping: Converting organizational names to ASNs via BGP tools and querying escaped CIDR blocks (page.ip:<subnet>\/24) reveals forgotten infrastructure across entire enterprise ranges.
• Hash Tracking for Component Exposure: SHA-256 hash queries identify identical favicons, reused scripts, and vulnerable third-party JavaScript libraries across the internet.
• Error Code & State Discovery: Filtering target domains by HTTP status codes (e.g., page.status:403, 503, or 404) surfaces restricted administrative panels, decommissioned staging endpoints, and broken backend gateways.
• Direct Asset Leaks: Searching by page.mime'type:"application/json", application/pdf, and application/zip paired with organizational keywords routinely exposes unprotected backups, internal documentation, and configuration files.