An idea for a spider emulator Firefox plugin

[wige]

I have an idea for a plugin for Firefox that will render the page in a way similar to how a spider might see that page - for example, highlighting links and different types of text, suppressing client-side scripting, etc. I know there are tools and web sites that already allow similar functionality, but personally, I prefer being able to analyze pages of a site directly in my browser, rather than having to go to an outside program/site to do the test.

Would you be interested in using such an add-on? Also, would you be interested in assisting with the development or testing of the addon?

[Webnauts]

Originally Posted by wige

I have an idea for a plugin for Firefox that will render the page in a way similar to how a spider might see that page - for example, highlighting links and different types of text, suppressing client-side scripting, etc. I know there are tools and web sites that already allow similar functionality, but personally, I prefer being able to analyze pages of a site directly in my browser, rather than having to go to an outside program/site to do the test.

Would you be interested in using such an add-on? Also, would you be interested in assisting with the development or testing of the addon?

I am for sure interested!

[incrediblehelp]

I will help test, but that is all I have time for now.

[DaveSawers]

Don't take this the wrong way, but I find it hard to understand what use such a tool would be.

Search engine spiders each see different things. Search engine providers prefer to keep the details of their algorithms secret and change them regularly.

Whilst I can see that it is useful to calculate things such as keyword density, etc. what else would this tool tell me and what benefit would I get from it?

[wige]

The current concept:

The first iteration would simply remove scripting, stylistic tags, and CSS content, convert images to text, and display expanded information about links on the page (showing the alt text etc.) to give a quick overview of the page so someone can spot the more obvious (and likely) issues such as javascript-based navigation that the bots can't follow.

The next version would include the ability to hide and/or highlight sections of the page that the spiders may see as suspicious, for example marking tiny text or hidden text.

This is of course subject to change as I get feedback and develop the concept more.

[activeco]

Originally Posted by DaveSawers

Search engine spiders each see different things. Search engine providers prefer to keep the details of their algorithms secret and change them regularly.

No, they all see the same thing (well, most of them). It has nothing to do with algorithms, but rather a simple spider view, just a bit above the code level.
Something like Google's text cache: Activeminds Software Development & Website Design

With such a view it is much easier to spot problems which are usually hard to see when fully rendered.

I personally use Lynx for the purpose, but I'm pretty sure a lot of Windows users would appreciate the plugin.

[incrediblehelp]

Doesnt the Web Developer plugin for FF do all of this?

[DaveSawers]

Originally Posted by activeco

No, they all see the same thing (well, most of them). It has nothing to do with algorithms, but rather a simple spider view, just a bit above the code level.
Something like Google's text cache: Activeminds Software Development & Website Design

With such a view it is much easier to spot problems which are usually hard to see when fully rendered.

I don't pretend to be an SEO specialist, so apologies if this is old hat. Just trying to learn a little more...

OK, so the text view is mildly interesting, but what can I learn from it?

And while I'm asking questions, why don't the spiders look at Javascript? It's not like it hasn't been around for years and it's not as if it doesn't contain interesting links and information relevant to search engines. I used to think it was just laziness on the part of spider developers but I hope there's a better reason. With more and more AJAX applications around, not probing the Javascript is going to make the search engines increasingly out of touch with the very content they should be indexing.

[activeco]

Originally Posted by DaveSawers

And while I'm asking questions, why don't the spiders look at Javascript?

Simply because of "what they see is what they get". As you know it is a client side scripting language, so they need an interpreter too, which would require huge resources. The same for flash, etc.

Actually Google often look for the links inside the js code and they try harder in the last time: "We're getting better at parsing some JavaScript, but we still don't get it all 100%."

[wige]

The main purpose I had in mind was to make it so the user can spot navigation that doesn't work. But I would also want to expand on it to show possible problem areas.

As far as Javascript, I have heard a few reasons why the spiders don't use it. The main thing is that implementations of Javascript vary from client to client, so what standard should the bot follow in interpreting the script? Should it process it the way IE would, the way Firefox would? Should it use the latest engine, or an old engine, or some combination? What about users who don't use Javascript, would they have a different experience, or be taken to a page that would not be accessible to them normally?

Processing Javascript would also require the spider to run the page through a fairly complex parser to process each page, which would increase the resources required. In addition, it could open up an avenue to spam the engines, as pages could be coded so the search engines would see the content, but the script would break in most browsers.

Javascript is also left alone because it gives a way to hide content from the search engines. If there are links that you want to ensure most spiders and bots won't follow, the surest way is to obscure them in javascript. This will stop normal search spiders and many spam bots.

Search engines do, however, take at least a cursory look at Javascripts, but this is done more as a quality control measure. The spider will look for redirects and content that appears spammy, without fully processing the script. Some discovery of links is also done, but this tends to be unreliable.

[DaveSawers]

Originally Posted by activeco

As you know it is a client side scripting language, so they need an interpreter too, which would require huge resources. The same for flash, etc.

Javascript isn't that resource hungry so I doubt the required resources would be "huge".

SWF files can now be opened up for examination althoguh I don't know how far that goes. If they just extract text then that is of limited use as lots more interesting stuff is likely to be buried in the ActionScript.

[DaveSawers]

Originally Posted by wige

As far as Javascript, I have heard a few reasons why the spiders don't use it...

None of which I find particularly compelling.

I'm particularly interested in this aspect because it seems that the emphasis on web site development is becoming more and more directed towards search engine placement and less on providing compelling information to end users. That's the tail wagging the dog. If search engines can't be bothered to properly index ALL content maybe it's time for a revised paradigm.

Of course I realize that if excellent content can't be easily found it might as well not be there so there's no doubt search engines are important. They just seem to me to be doing a poor job.

Sorry for hijacking your thread.

[Webnauts]

Wige I think if is done in a way a page may be viewed in Lynx, it will be excellent.

[imvain2]

This maybe be done initially with greasemonkey.