Category: Spider

Spider 0.0.5.0 is now available. It is marked as compatible for Firefox 18, removes the attempts to use netscape capability features in the html version and changes the status bar to display the build id rather then the Gecko build date which is no longer indicative of the actual build date.

Spider 0.0.4.0 is now available. It is marked as compatible for Firefox 15, contains a minor patch to FormPersist.js to fix a mistyped ; after conditional and the addition of a new option -fileurls to enable the ability to spider file:// urls. This later is useful when attempting to reproduce crashes using locally saved versions of web pages but is not recommended for use on the wild wild web due to possibility of web based content linking to your local file system.

I’ve just pushed the most recent update of Spider to version 0.0.3.8.

Spider 0.0.3.8 is a simple change which adds compatibility for Firefox 11 Nightly builds.

Spider 0.0.3.7 was an update which I failed to push to the web site but which was available from hg.mozilla.org/automation/sisyphus/spider. Sorry that I wasn’t more timely in announcing the update. Spider 0.0.3.7 included significant changes:

• Load handlers where changed from using useCapture true to false. This is I believe a better approach than what I have used in the past, but if you see any issues with pages failing to fire the load event please contact me.
• CSpider no longer uses direct CallWrappers to detect page time outs. Instead, time out detection is now handled by the CPageLoader.
• The CPageLoader’s initialization has been changed to include explicit references to:
  • the content element used to load the web page;
  • the callback function used to handle the load event;
  • the callback function used to handle page time outs;
  • the time out interval;
  • and the http responses observer.

The change to remove the singleton nature of the CHTTPResponseObserver was to allow the use of additional CHTTPResponseObservers which could be used in userhook functions. I found this very helpful when performing investigations of how the effect of a 2 digit rv: value in Firefox 10 might affect web site’s user agent detection of Firefox.

Please let me know if you have any issues by contacting me at bob @ this domain. Thanks for using Spider, and I hope you continue to find it useful.

Spider 0.0.3.3 includes changes to support Firefox 4’s changes in
XPCOM’s component registration and removal of xpnativewrapper and improvements to onload detection.
It also includes a new command line parameter -invisible to hide the content being loaded.