Burp Spider

Burp Spider is a tool for mapping web applications. It automates the laborious task of cataloging an application's content and functionality, and lets you:

Work manually via your browser, by passively inspecting traffic passing through Burp Proxy and cataloging everything that this identifies.
Actively crawl the application, by automatically following links, submitting forms, and parsing responses for new content.
Browse a detailed site map of discovered content, in tree and table form.
Retain full control of all spidering actions, with fine-grained scope definition, automatic or user-guided submission of forms, and detailed configuration of the spidering engine.
Send interesting items to other Burp Suite tools with a single click.
Deal with complex applications, with automatic handling of login credentials and session cookies, and detection of custom "not found" responses.
Save all of your work, and resume working later.

When you run Burp, the Spider runs by default in passive mode, and builds up a detailed site map of your target application, by recording all of the requests that you make via Burp Proxy, and parsing all of the responses for new links and functionality. After browsing the whole application, you can use Burp's site map to review the content you have discovered. You can then use the active spidering function to map out any areas you may have missed, or you can select individual items or branches within the site map, and send these to other Burp tools for further manual or automated attacks.

Screenshots - click to enlarge

Burp Suite, the leading toolkit for web application security testing

Burp Spider