unfurl: decomposing URL corpora for analysis

unfurl · MIT

unfurl takes URLs on stdin and prints one component of them: the domain, the path, the parameter keys, the values. It is a text utility, not a scanner, and it is what makes a 200,000-line URL corpus usable.

Install

go install github.com/tomnomnom/unfurl@latest

Core commands

# hostnames
cat urls.txt | unfurl -u domains

# paths only
cat urls.txt | unfurl -u paths

# parameter names — a wordlist in the application's own vocabulary
cat urls.txt | unfurl -u keys

# parameter values — where leaked tokens show up
cat urls.txt | unfurl -u values

# a custom shape
cat urls.txt | unfurl format '%d%p'

-u deduplicates. The format directives are %s scheme, %d domain, %p path, %q query, %f fragment.

Output

Plain text, one component per line.

Where it sits in Tandera

unfurl runs in the analyze phase of recon_web_lite. It sits after the URL producers — katana, gau, waybackurls — and turns their combined corpus into structured inputs: hostname lists that feed back into discovery, path lists, and parameter-name lists that arjun uses.

Using it in a pentest

unfurl -u keys is the best wordlist you will get. Generic parameter wordlists guess. This one is derived from URLs that actually existed on this application, so it reflects the developers’ naming conventions.

Grep the values, not just the keys. Archived URLs captured with real query strings leak tokens, session identifiers and email addresses:

cat urls.txt | unfurl -u values | grep -iE '^(eyJ|ghp_|sk_|AKIA)'

Use unfurl -u domains as a feedback loop. URL corpora contain hostnames your subdomain enumeration missed. Feed them back into dnsx and httpx and you will occasionally find a live host nothing else surfaced.