gau: pulling archived URLs from public datasets

gau (getallurls) · MIT

gau fetches known URLs for a domain from public archives — the Wayback Machine, Common Crawl, URLScan, Open Threat Exchange. It sends nothing to the target. What it returns is a historical record of URLs those services observed, which routinely includes endpoints the current application no longer links to.

Install

go install -v github.com/lc/gau/v2/cmd/gau@latest

Core commands

# everything the archives know
gau example.com

# include subdomains
gau --subs example.com

# filter to interesting extensions
gau --subs example.com --blacklist png,jpg,gif,css,woff,svg,ico

# constrain by date and provider
gau --from 202401 --to 202608 --providers wayback,commoncrawl example.com

# JSON
gau --json --subs example.com > gau.jsonl

Without a blacklist you will drown in static assets. Always filter.

Output

One URL per line by default:

https://example.com/api/v1/user?id=1
https://example.com/admin/backup.sql
https://example.com/old/login.php

Where it sits in Tandera

gau runs in the enrich phase of recon_web_lite and recon_web_full, alongside waybackurls. Running both is intentional — they query overlapping but not identical providers, and the union is larger than either.

Its output joins katana’s live crawl to form the URL corpus that unfurl, arjun and the parameter-analysis steps consume.

Using it in a pentest

Historical URLs are the point. /old/login.php returning 404 today is uninteresting. /old/login.php returning 200 is a forgotten endpoint outside the current release, which is exactly the kind of thing that has not been patched. Always re-probe the corpus:

gau --subs example.com --blacklist png,jpg,css,woff 
  | httpx -silent -status-code -title -mc 200,401,403,500

Grep the corpus before you probe it. Archived URLs often contain parameters with real values — tokens, emails, internal IDs — captured when the page was crawled. Search for key=, token=, password=, api_key= in the raw list.

Mind the scope boundary. --subs pulls every subdomain the archives know, including ones outside the engagement. Filter against your scope list before probing anything.