waybackurls: Wayback Machine URL extraction for recon

waybackurls · MIT

waybackurls does one thing: it asks the Wayback Machine’s CDX API for every URL it has recorded for a domain and prints them. No flags to speak of, no configuration. It is a Unix tool in the original sense.

Install

go install github.com/tomnomnom/waybackurls@latest

Core commands

# a single domain
echo example.com | waybackurls

# include subdomains, keep the first-seen date
cat roots.txt | waybackurls -dates

# no subdomains
echo example.com | waybackurls -no-subs

# the usual pipeline position
cat roots.txt | waybackurls | sort -u > wayback.txt

Output

One URL per line. With -dates, a timestamp is prefixed:

2024-03-11T08:14:22Z https://example.com/api/v1/export?format=csv

Where it sits in Tandera

waybackurls runs in enrich on recon_web_lite and recon_web_full, paired with gau.

Why both? gau queries several providers including Common Crawl and URLScan; waybackurls queries the Wayback CDX index directly and tends to return deeper history for it. Their outputs overlap heavily and diverge at the edges, and the edges are where the forgotten endpoints are. Tandera merges and deduplicates the two before anything downstream reads them.

Using it in a pentest

Dates are triage. -dates tells you when a URL was last observed. A path first seen in 2015 and never since is probably gone; one seen last month and absent from your crawl is a live endpoint the application no longer links to. The second kind is worth your time.

Extract parameter names, not just URLs. The corpus is a free wordlist tailored to this application:

cat roots.txt | waybackurls | unfurl -u keys | sort -u > params.txt

Treat it as leads, never as evidence. Everything here is a third-party record of what a URL looked like at some point. Nothing goes in a report until you have re-requested it yourself and captured your own response.