Comments on: YASU – Spider 0.0.2.20

By: bc

bc — Fri, 11 Sep 2009 20:24:14 +0000

I’ll look into why the resizing is occurring and see what I can do. I’ll also think about adding a new command line argument to pass name/value pairs to the Spider.

By: Mohamed

Mohamed — Fri, 11 Sep 2009 19:07:20 +0000

Thanks. That fixes the resizing issue. The drawback with this method is that I could not find a way to pass any parameter to the user hook function.

With the chrome application, I could pass parameters as querystring via open.xul.

By: bc

bc — Fri, 11 Sep 2009 06:41:01 +0000

try the command line arguments as described in help.

firefox -P test -spider -url http://bclary.com/ -domain bclary.com -depth 2 -timeout 120 -wait 5 -start

By: Mohamed

Mohamed — Fri, 11 Sep 2009 04:12:18 +0000

When I invoke the spider from the command line in chrome mode passing parameters, the window resizes itself to a smaller dimension. If you don’t pass any parameter, this does not happen.

For example, if you issue the command (the example you have provided in your help page)
firefox -P test -chrome “chrome://spider/content/spider.xul?url%3Dhttp%253A%252F%252Fbclary.com%252F%26domain%3Dbclary.com%26depth%3D2%26timeout%3D120%26waittime%3D5%26autostart%3Don%26restrict%3Don”

you can see the window resizes itself. If you just call “firefox -P test -chrome chrome://spider/content/open.xul” no resizing occurs.

When used in automated environments, this resizing causes problems (Some web pages with new rich media content, for example, don’t load properly when the window size is smaller).

Is there a remedy for this?

Thanks
Mohamed

By: bc

bc — Sun, 06 Sep 2009 13:36:37 +0000

I think the problem is that the original url is redirected, but Spider is seeing the original response where there are no links available. I’ll have to handle redirected pages better I think. Sorry for the delay in responding, I missed the comments in my feeds somehow.