robots file for spiders

[mantawebsolutions]

I came accross this URL (http://www.searchengineworld.com/cgi-bin/sim_spider.cgi) to test what spiders see when accessing a site, so i tested my URL

The first part of the results came back fine but further down i saw something that worries me.

It picked up the URL's fine but for most it showed two variants
example:
http://www.divesouthafrica.co.za/aboutsouthafrica.html
and
http://aboutsouthafrica.html

why does it pick up the url without the domain ???
If this is a problem in my robots.txt file and, if so, how do i correct it ?

[flood6]

I'd say it's a design flaw in the spider; it doesn't seem to like relative links. I tried one of my sites with a "base href" tag and got the same problems with relative links.

I imagine you could search webmasterworld (searchengine world's sister-site) and maybe find some discussion about it.

I don't think you have anything to worry about.

[mantawebsolutions]

thanks flood, much appreciated

[ebonage]

Originally Posted by flood6

I'd say it's a design flaw in the spider; it doesn't seem to like relative links. I tried one of my sites with a "base href" tag and got the same problems with relative links.

I imagine you could search webmasterworld (searchengine world's sister-site) and maybe find some discussion about it.

I don't think you have anything to worry about.

Cool! Thanks for the tip! Here is another site that covers robots.txt configurating robots.txt that site also references searchengineworld.com.

Regards,
ebonage