Spider Help

About Spider

Spider is a web application which can recursively visit the pages of a web site. It is customizable to perform arbitrary processing on the web content it visits and is especially useful in performing quality assurance tasks on web content. Spider is supported only by Gecko-based browsers. To learn more about Spider, visit its home page or subscribe to http://bclary.com/home.rss where announcements regarding new versions are made.

This software is a modified version of the CSpider application hosted at http://devedge-temp.mozilla.org/toolbox/examples/2003/CSpider/.

Developers can learn more about the internal operation of the CSpider JavaScript Class which provides the basic framework for this application from CSpider JavaScript Class manpage

Special Considerations

Security Privileges

Spider is capable of capturing error messages and spidering sites from other domains through the use of extended security privileges.

If you are running Spider as a XUL application from Firefox Chrome, the application automatically has the necessary security privileges. However if you are running Spider as an HTML application or a remote XUL application, you will need to grant the application the necessary privileges.

In order to configure Firefox to enable the use of these security privileges, you must set the preference signed.applets.codebase_principal_support to true. You can do this either by using about:config to add the boolean preference, or modifying the user.js preferences file in your Firefox profile directory to contain the line

user_pref("signed.applets.codebase_principal_support", true);

For more information, please see Bypassing Security Restrictions and Signing Code by Arun Ranganathan.

Preferences

Required Preferences

signed.applets.codebase_principal_support is required in order to allow Spider access to internal Firefox APIs as well as cross-domain access. browser.dom.window.dump.enabled is required in order to allow Spider to send messages to STDOUT. Without dump, Spider can only report messages to the JavaScript Console.

user_pref("signed.applets.codebase_principal_support", true);
user_pref("browser.dom.window.dump.enabled", true);
    

Useful Preferences

javascript.options.strict enables Firefox to report on JavaScript strict warnings which are very useful in determining potential problem areas in scripts. javascript.options.showInConsole can be used to report internal browser errors to the JavaScript console and is useful for Firefox developers.

user_pref("javascript.options.strict", true);
user_pref("javascript.options.showInConsole", true);

Spider can be halted through alerts which either Firefox sends or which are created by the web pages being visited. If you wish to run Spider unattended, you have several choices.

New in Spider 0.0.1.21 You can register a WindowWatcher observer in your userhook functions to automatically close dialogs after 10 seconds. Simply call registerDialogCloser() in userOnBeforePage() and unregisterDialogCloser() in userOnAfterPage().

You can also disable as many alert messages as possible through the user of preferences. In addition, by using browser.xul.error_pages.enabled to replace network error alerts with error pages, Spider can easily detect and report on network errors.

user_pref("browser.xul.error_pages.enabled", true);
user_pref("dom.disable_open_during_load", true);
user_pref("dom.disable_window_flip", true);
user_pref("dom.disable_move_resize", true);
user_pref("dom.disable_window_open_feature.status", true);
user_pref("security.warn_entering_secure", false);
user_pref("security.warn_entering_weak", false);
user_pref("security.warn_submit_insecure", false);
user_pref("security.warn_leaving_secure", false);
user_pref("security.warn_viewing_mixed", false);
user_pref("capability.policy.default.Window.alert", "noAccess");
user_pref("capability.policy.default.Window.confirm", "noAccess");
user_pref("capability.policy.default.Window.prompt", "noAccess");

If you wish to enable alerts on specific sites, you can enable them on a per site basis using a policy and a site list for which you will allow alerts.

user_pref("capability.policy.policynames", "trusted");
user_pref("capability.policy.trusted.sites", "...");
user_pref("capability.policy.trusted.Window.alert", "sameOrigin");
user_pref("capability.policy.trusted.Window.confirm", "sameOrigin");
user_pref("capability.policy.trusted.Window.prompt", "sameOrigin");

You can download user.js which contains these preferences and save it to your Firefox profile directory.

If you need to Spider sites which require authentication, the following preferences can be useful to automatically negotiate authentication credentials for specific sites and domains.

 // basic authentication
 user_pref("network.negotiate-auth.trusted-uris", site-list);
 user_pref("network.negotiate-auth.delegation-uris", site-list);

 // ntlm authentication
  user_pref("network.automatic-ntlm-auth.allow-proxies", true);
  user_pref("network.automatic-ntlm-auth.trusted-uris", site-list);

 // Confirm user intent whenever URL of the form 
 // http://user:pass@my.site.com is accessed?
 user_pref("network.http.defensive-auth-prompting", false);

where site-list is a string containing a comma-delimited list of protocols, sites or domains. Be careful not to unintentially broaden the scope of sites where authentication negotiation is used since it might expose your credentials to crackers.

In order for automatic negotiation to work, you will need to be logged into your local machine using the same credentials that are used on the remote site. You can find more information about these preferences in Nigel McFarlane's Firefox Hacks (online).

Starting Spider

Spider can be run as either an HTML web application, a remote XUL web application or as a chrome XUL application. When run as an HTML application, it uses an IFRAME to load pages which sends a null HTTP referer. When run as a XUL application, Spider uses a xul:browser element along with the method loadURI to send the url of the page containing a link as its referer.

Spider sends logging output to both the JavaScript console and to STDOUT if the dump function is enabled. Since the JavaScript console is limited in the number of messages it may contain, it is important to also send the logging messages to STDOUT. This allows the results of a Spider run to be collected in a text file for later review and processing. In order to see this output you should start Firefox from the command line and redirect STDOUT to a text file.

The following command will start Firefox, automatically select the profile test, load Spider and direct all output to the file test.log.

HTML

firefox -P test http://bclary.com/2004/07/10/spider/chrome/content/spider/spider.html > test.log 2>&1

Remote XUL

firefox -P test http://bclary.com/2004/07/10/spider/chrome/content/spider/spider.xul > test.log 2>&1

Chrome XUL Application

via command line with chrome url

Spider can be run directly as in:

firefox -P test -chrome chrome://spider/content > test.log 2>&1

However, this can have an undesired side-effect of creating a window which can not be resized in recent Firefox trunk (Deer Park alpha) builds based on Gecko 1.8.

To create a resizable window, invoke open.xul as in:

firefox -P test -chrome chrome://spider/content/open.xul > test.log 2>&1

open.xul will create a resizable window using javascript, then load spider.xul while passing any querystring along to spider.xul.

Using this approach you can also specify the initial values of Spider's parameters however you must properly encode the url in order to start Spider this way. To create the proper query string, you can start Spider, fill in the parameters, then click the button Generate Spider URL. For example, the following will start spider, set the URL to http://bclary.com/ and automatically spider the site to a depth of 2.

firefox -P test -chrome "chrome://spider/content/spider.xul?url%3Dhttp%253A%252F%252Fbclary.com%252F%26domain%3Dbclary.com%26depth%3D2%26timeout%3D120%26waittime%3D5%26autostart%3Don%26restrict%3Don"

via command line with command line arguments - new in 0.0.2.0

-spider              Start Spider (required)
-url <url>           Spider site at <url>
-uri <url>           Spider site at <uri>
-domain <domain>     Restrict Spider to urls matching <domain>
-depth <depth>       Spider to depth of <depth>
-timeout <timeout>   Time out Spider if page takes more than <timeout>
                     seconds
-wait <wait>         Pause Spider for <wait> seconds after each page
-hook <hookscript>   Execute Spider <hookscript>
-start               Automatically start Spider
-quit                Automatically quit when finished
-robot               Obey robots.txt
-fileurls            Allow file:// urls
-debug               Debug Spider
-jserrors            Display JavaScript errors
-jswarnings          Display JavaScript warnings
-chromeerrors        Display chrome errors
-xblerrors           Display XBL errors
-csserrors           Display CSS errors
-httpresponses       Display HTTP responses
-invisible           Hide loaded page

The previous example can be rewritten as

firefox -P test -spider -url http://bclary.com/ -domain bclary.com -depth 2 -start

Note that when run as a Chrome application, Spider can prevent frame-busting code from replacing the top level browser window however when run as either an HTML or Remote XUL application, Spider can be stopped by frame-busting code.

Data

URL

URL is the location which you wish to spider. This can be a fully qualified domain such as http://www.example.com/ or a partial domain such as example.com.

Domain

Domain is used to restrict Spider to follow links on a specific domain. If this value is not specified, it will be generated from the URL by removing the protocol and any leading www.

Domain is useful when you wish to start at a given URL but do not wish to restrict Spider to URLs which contain the initial URL.

Depth

Depth is the number of links to follow during the spider. 0 will load only the initial page. 1 will load the initial page plus all pages linked from the initial page.

A specified Depth will reach the same set of pages that an imaginary visitor would reach using the same number of mouse clicks.

Page timeout

The number of seconds that Spider will wait for an external page to load before firing CSpider.mOnPageTimeout() and entering the paused state.

Page wait time

The number of seconds that Spider will wait after a page has displayed before beginning to load the next page. This can be use to allow the user time to visually inspect the page.

Wait for User Hook

Instead of waiting a specified number of seconds before loading the next page, "Wait for User Hook" will cause the spider to wait until the global variable gPageCompleted is set to true.

"Wait for User Hook" is useful for situations where processing of the loaded page's content may take an arbitrary amount of time and where the User Hook functions need to control page transitions.

Autostart

If Spider is invoked with a query string containing autostart=on, Spider will automatically begin executing with the data values specified in the query string.

To generate a link containing the desired parameters, first set the data inputs, check the Autostart check box, then click Generate Spider URL.

Autoquit

If Spider is invoked with a query string containing autoquit=on, Spider will automatically exit when it has completed its run, i.e. reached the Stopped state.

To generate a link containing the desired parameters, first set the data inputs, check the Autoquit check box, then click Generate Spider URL.

Restrict URLs

If Restrict URLs is checked, then the spider will only follow links which contain the Domain. For example if you enter http://www.example.com/ as the initial URL, the spider will follow links of the form http://www.example.com/help/ but not http://www.foo.example.com/.

If you wish to restrict the spider to a domain, simply enter the partial domain such as example.com which will follow all links which contain example.com.

If you wish the spider to follow any link regardless of site or domain, uncheck Restrict URLs.

Respect robots.txt

If Respect robots.txt is checked, Spider will obey the rules specified in a site's robots.txt file and refuse to load blocked urls.

Allow file urls

If Allow file urls is checked, Spider will follow file:/// urls. Note that this can have security implications since it allows Spider to follow file:// links from arbitrary pages on the web and is not enabled by default.

Starting in Spider 0.0.4.0, Spider will allow you to load a file url as the starting url, but will not follow file links unless you have specified Allow file urls.

This change has been made primarily to allow Spider to be used with local file based test cases and it not intended for use when spidering content on the wild wild web.

Debug spider

If Debug spider is checked, Spider will output debugging messages to the JavaScript Console and STDOUT. Only useful to someone hacking Spider.

Error Output

JavaScript
Errors

Select this option to send JavaScript Errors and Exceptions to stdout.

Warnings

Select this option to send JavaScript Warnings to stdout.

Chrome

Select this option to send Chrome Errors and Warnings to stdout.

XBL

Select this option to send XBL Errors and Warnings to stdout.

CSS Errors

Select this option to send CSS Errors and Warnings to stdout.

User Hooks

Script URL

Script URL is the location of an optional external JavaScript file which can be used to customize the operation of Spider through the use of any or all of the following functions. Note that the global object gSpider exposes the same interface as CSpider.

WARNING: These functions will operate in the chrome security context of the browser.

function userOnStart()
{
  // add custom code here
  // to be called by the Spider's mOnStart handler
}

function userOnBeforePage()
{
  // add custom code here
  // to be called by the Spider's mOnBeforePage handler
}

function userOnAfterPage()
{
  // add custom code here
  // to be called by the Spider's mOnAfterPage handler
  // this function is especially useful for performing
  // tests upon the DOM of a loaded web page.
  //
  // If "Wait for User Hook" is checked, then userOnAfterPage()
  // is responsible for setting the global variable gPageCompleted
  // in order to load the next page.
}

function userOnStop()
{
  // add custom code here
  // to be called by the Spider's mOnStop handler
}

function userOnPause()
{
  // add custom code here
  // to be called by the Spider's mOnPause handler
}

function userOnRestart()
{
  // add custom code here
  // to be called by the Spider's mOnRestart handler
}

function userOnPageTimeout()
{
  // add custom code here
  // to be called by the Spider's mOnPageTimeout handler
}

gConsoleListener.onConsoleMessage = 
function userOnConsoleMessage(s)
{
  // add custom code here to handle
  // the message which was sent to the
  // JavaScript Console and STDOUT.
  // You can use this function to
  // store messages in databases etc.
};

Spider 0.1.18 introduced a function loadScript(aScriptUrl[, aScope]) which can be used by user hook functions to load additional utility scripts. This allows the modularization and reuse of user hook code.

Controls

Run

Run will begin spidering the specified site.

Pause

Pause will cause the spider to enter the Paused state after it finishes loading the current page.

Restart

If the spider is Paused either because you have clicked Pause or the spider has timed out, you can press Restart to continue.

Stop

Stop will stop the spider.

Generate Spider URL

Generate Spider URL will open a new window with a link which can be used to open Spider and populate the Data inputs and optionally automatically Run Spider if Autostart is checked.

Reset

Reset will reset the Data inputs to their default values. Note that the URL containing any pre-existing query string value is not changed via Reset.

License

This software is licensed under the MPL, GPL and LGPL licenses. View source to see the license agreement and read mozilla.org's Mozilla & Netscape Public Licenses for more details.

spider