# gau: pulling archived URLs from public datasets

> How to install and run gau, what its URL corpus is good for, and how Tandera uses archived URLs to surface endpoints that no longer appear in a crawl.

`gau` fetches known URLs for a domain from public archives — the Wayback Machine, Common Crawl, URLScan, Open Threat Exchange. It sends nothing to the target. What it returns is a historical record of URLs those services observed, which routinely includes endpoints the current application no longer links to.

## Install

```bash
go install -v github.com/lc/gau/v2/cmd/gau@latest
```

## Core commands

```bash
# everything the archives know
gau example.com

# include subdomains
gau --subs example.com

# filter to interesting extensions
gau --subs example.com --blacklist png,jpg,gif,css,woff,svg,ico

# constrain by date and provider
gau --from 202401 --to 202608 --providers wayback,commoncrawl example.com

# JSON
gau --json --subs example.com > gau.jsonl
```

Without a blacklist you will drown in static assets. Always filter.

## Output

One URL per line by default:

```
https://example.com/api/v1/user?id=1
https://example.com/admin/backup.sql
https://example.com/old/login.php
```

## Where it sits in Tandera

gau runs in the **enrich** phase of `recon_web_lite` and `recon_web_full`, alongside `waybackurls`. Running both is intentional — they query overlapping but not identical providers, and the union is larger than either.

Its output joins katana's live crawl to form the URL corpus that `unfurl`, `arjun` and the parameter-analysis steps consume.

## Using it in a pentest

**Historical URLs are the point.** `/old/login.php` returning 404 today is uninteresting. `/old/login.php` returning 200 is a forgotten endpoint outside the current release, which is exactly the kind of thing that has not been patched. Always re-probe the corpus:

```bash
gau --subs example.com --blacklist png,jpg,css,woff \
  | httpx -silent -status-code -title -mc 200,401,403,500
```

**Grep the corpus before you probe it.** Archived URLs often contain parameters with real values — tokens, emails, internal IDs — captured when the page was crawled. Search for `key=`, `token=`, `password=`, `api_key=` in the raw list.

**Mind the scope boundary.** `--subs` pulls every subdomain the archives know, including ones outside the engagement. Filter against your scope list before probing anything.

---

Canonical: https://tandera.io/tools/gau
This page as markdown: https://tandera.io/tools/gau.md
Index for agents: https://tandera.io/llms.txt

Every page here is also available as markdown: append `.md` to the path (e.g. `/recon.md`, `/index.md` for this homepage, `/blog/<slug>.md`), or request the canonical path with `Accept: text/markdown`.