Burp Spider Help
What is Burp Spider?
Burp Spider is a tool for mapping web applications. It uses various intelligent techniques to generate a comprehensive inventory of an application's content and functionality.
Burp Spider maps a target application by following hyperlinks found within HTML and JavaScript, submitting forms, and using other clues such as directory listings, source code comments and the robots.txt file. Results are displayed in the target site map in both tree and table format, providing a clear and highly detailed view of the target application.
Burp Spider enables you to obtain a detailed understanding of how a web application works, avoiding the time-consuming and unreliable task of manually following links, submitting forms and scouring HTML source code. Potentially vulnerable application functions can be quickly identified, allowing you to check for specific vulnerabilities such as SQL injection and directory traversal.
Using Burp Spider
To use Burp Spider against an application requires two simple steps:
- With your browser configured to use Burp Proxy as its proxy server, browse to the target application. (You can turn off interception within the Proxy, to save time.)
- Go to the site map in the "target" tab, and select the host(s) and directories where the target application resides. Choose the "spider this host/branch" option from the context menu.
You can also choose "spider this item" from the context menu on any request or response within any of the Burp tools.
When you send a branch of the site map for spidering, the Spider will first check if that branch is within the currently defined spidering scope. If not, Burp will prompt you to confirm that you want to add the relevant URLs to the scope. Burp will then start spidering, and will perform the following actions:
- Request any unrequested URLs already discovered within the branch.
- Submit any discovered forms whose action URLs lie within the branch.
- Re-request any items in the branch which previously returned 304 status codes, to retrieve a fresh (uncached) copy of the application's responses.
- Parse all content retrieved to identify new URLs and forms.
- Recursively repeat these steps as new content is discovered.
- Continue spidering all in-scope areas until no new content is discovered.
Note that the Spider will follow links for any URLs that are within the currently defined spidering scope. If you have already defined a wider target scope, and you select an individual branch within this for spidering, then the Spider will follow any links into the wider target scope, and so will spider outside of the selected branch. To ensure that the Spider only requests items within a specific branch, you should first configure the spidering scope to include only this branch.
You should use Burp Spider with caution. In its default configuration, the Spider will automatically submit any forms within the spidering scope using default input values, and will request various URLs that normal users would not ordinarily request if using only a browser. If any URLs within your defined scope are used to perform sensitive actions, then these actions may actually be carried out within the application. It is normally preferable to perform some manual mapping of the application using your browser before initiating any fully automated content discovery.
Control tab
This tab is used to start and stop Burp Spider, monitor its progress, and define the spidering scope.
Spider running - This is used to start and stop the Spider. While the Spider is stopped it will not make any requests of its own, although it will continue to process responses generated via Burp Proxy, and any newly-discovered items that are within the spidering scope will be queued to be requested if the Spider is restarted.
The display also shows some metrics about the Spider's progress, enabling you to see the size of the in-scope content and the work remaining to fully spider it.
Clear queues - If you want to reprioritise your work, you can completely clear the currently queued items, so that other items can be added to the queue. Note that the cleared items may be re-queued if they remain in-scope and the Spider's parser encounters new links to the items.
Spider scope - This panel lets you define exactly what is in-scope for the Spider to request. If you have already configured the Suite-wide target scope with details of your current target, then you can normally leave the default setting, which is to use the Suite-wide scope to define the Spider's activities. If you need to define a different scope for the Spider to use, then select "use custom scope". A further configuration panel will appear which functions in the same way as the Suite-wide scope panel. If you have selected to use a custom scope and you send any out-of-scope items to the Spider, then Burp will automatically update this custom scope, rather than the Suite scope.
Options tab
This tab contains various configuration options which control the behaviour of Burp Spider, as described below. These settings can be modified after the Spider has started running, and will be applied retrospectively to prior results. For example, if the maximum link depth is increased, then links which were previously outside the maximum depth will be queued to be requested if appropriate.
check robots.txt - If checked, Burp Spider will request and process the robots.txt file from all in-scope domains. This file is used by the robots exclusion protocol to control the behaviour of spider-like agents on the Internet. Note that Burp Spider does not confirm to the robots exclusion protocol. Because Burp Spider is designed to comprehensively enumerate a target application's content, all entries in robots.txt will be requested if they are in-scope.
use cookies - If checked, Burp Spider will process Set-Cookie instructions in server responses, and will submit any received cookies in subsequent requests to the same domain. This option is normally necessary when spidering web applications which persist any kind of state in a server-side session.
detect custom "not found" responses - The HTTP protocol requires web servers to return a 404 status code if a requested resource is not found. However, many web applications return customised "not found" pages that use a different status code. If this is the case, then using this option can prevent false positives in the mapping of site content. Burp Spider detects custom "not found" responses by requesting several nonexistent resources from each domain, and compiling a fingerprint with which to diagnose "not found" responses to other requests.
ignore links to non-text content - It is often possible to deduce the MIME type of a particular resource from the HTML context in which links to it appear. For example, URLs within IMG tags will probably return images; those within SCRIPT tags will probably return JavaScript. If this option is checked, the Spider will not request items which appear, from this context, to be non-text media resources. Using this option can reduce spidering time with minimal risk of overlooking interesting content as a result.
request the root of all directories - If checked, Burp Spider will request all identified web directories within the target scope, in addition to files within those directories. This option is particularly useful if directory indexing is available on the target site.
make a non-parameterised request to each dynamic page - If checked, Burp Spider will make a non-parameterised GET request to all in-scope form action URLs. Dynamic pages usually respond differently if the expected parameters are not received, and this option may successfully detect additional site content and functionality.
maximum link depth - This is the maximum number of "hops" which Burp Spider will navigate from any seed URL. A value of zero will cause Burp Spider to request seed URLs only. If a very large number is specified, then in-scope links will be followed effectively indefinitely.
These options control the interface between Burp Proxy and Burp Spider, which allows "passive" spidering of web applications, controlled through your browser.
passively spider as you browse - If checked, Burp Spider will process all HTTP requests made through Burp Proxy, to identify links and forms on web pages visited. Using this option can enable Burp Spider to build up a detailed picture of an application's contents even when you have only browsed a subset of that content with your browser, because all content that is linked from visited content is automatically added to the Suite site map.
link depth to associate with proxy requests - This option controls the "link depth" which will be associated with web pages accessed through Burp Proxy. To prevent Burp Spider following any links in these pages (even when the Spider is running and these links are in-scope) set a higher value for this option than the "maximum link depth" option above.
Note: Earlier versions of Burp Spider contained options here to control how the Spider cookie jar was updated based on cookies in Proxy requests and responses. These configurations have now been removed, and you should use the suite-wide session handling support instead.
individuate forms - This option configures the criteria for individuating unique forms (action URL, method, fields, values). When Burp Spider processes each form, it will check these criteria to determine if the form is "new". Forms which are not new will not be queued for submission.
do not submit - If selected, Burp Spider will not submit any forms.
prompt for guidance - If selected, Burp Spider will prompt you for guidance before submitting each identified form. This allows you to enter custom data into form input fields as required, and choose which submit fields to send to the server, or whether to iterate through all submit fields.
automatically submit - If selected, Burp Spider will automatically submit any in-scope forms using the defined rules to fill out the values of text input fields. Each rule lets you specify a simple or regular expression to match on form field names, and the value to submit in fields whose names match the expression. A default value can be specified for any unmatched fields.
This option is particularly useful if you want to automatically spider through registration forms and similar functions, where applications typically require data in a valid format for each input field. Burp comes with a set of default rules that have proven successful when automatically submitting form data to a wide range of applications. Of course, you can modify these or add your own rules if you encounter form field names that you want to submit specific values in. You should use this option with caution, as submitting bogus values in forms may sometimes result in undesirable actions.
Many forms contain multiple SUBMIT elements, which result in different actions within the application, and the discovery of different content. You can configure the Spider to iterate through the values of all submit elements within forms, submitting each form multiple times up to a configurable maximum.
Login forms play a particular role within applications, and you will often want Burp to handle these in a different way than ordinary forms. Using this configuration, you can tell the Spider to perform one of four different actions when a login form is encountered:
- Burp can ignore the login form, if you don't have credentials, or are concerned about spidering sensitive protected functionality.
- Burp can prompt you for guidance interactively, enabling you to specify credentials on a case-by-case basis. This is the default option.
- Burp can handle login forms in the same way as any other form, using the configuration and auto-fill rules you have configured for those.
- Burp can automatically submit specific credentials in every login form that is encountered.
In the last case, any time Burp encounters a form containing a password field, it will submit your configured password in that field, and will submit your configured username in the text input field whose name most looks like a username field. If you have credentials for an application, and want to let the Spider handle the login for you, then this is normally the best option.
These options let you fine-tune the spidering engine, depending on the performance impact on the application, and on your own processing power and bandwidth. If you find that the Spider is running slowly, but the application is performing well and your own CPU utilisation is low, you can increase the number of scan threads to make your scans proceed faster. If you find that connection errors are occurring, that the application is slowing down, or that your own computer is locking up, you should reduce the thread count, and maybe increase the number of retries on network failure and the pause between retries.
Request headers - This section allows customised HTTP headers to be configured which will be used in all requests. This may be useful to meet specific requirements of individual applications - e.g. to emulate an expected user agent when testing applications designed for mobile devices.
use Referer header - If checked, Burp Spider will submit the relevant Referer header when requesting any item that was linked to from another page.
Spider results
All of the content discovered during spidering is added to the target site map that is shared between Suite components. This map shows tree and table views of the content discovered via the Spider and Proxy. It lets you filter from view any items you are not interested in, and perform numerous other actions such as initiating vulnerability scans and further spidering, and sending individual requests to other Burp tools to perform customised attacks. Please consult the site map help for further details.