Page Can't Be Spidered??

cz · #1 (**permalink**) 09-24-2008, 04:11 PM

Hi,

I have been checking GG Analytics and used a couple of utilities and they are saying that this page can't be accessed due to "Robot Text". I never put any of that in the page and the rest of the items within the same section are fine.

Can someone take a look at this page and clue me in about how to find this robot text that is blocking the page from GG and maybe other engines?

CN, CS, Tear Gas, Information, Effects, Reports this is the page that Analytics is flagging as being inaccessible.

Thanks!

**wige** · #2 (**permalink**) 09-24-2008, 06:06 PM

In robots.txt you have the following directive:
Disallow: /cs
The page URL is:
/csteargasfaq.html

You also have a few issues in the code structure that may cause issues with some browsers and spiders (javascript between <html> and <head>, for example) that you may want to check and correct.

cz · #3 (**permalink**) 09-24-2008, 06:20 PM

Thanks wige,

I never even knew there was robots.txt in my site? The site is a Yahoo! store as you probably noticed. They insert code in the craziest places trying to implement features and there's nothing that I can access to correct it. No Yahoo! Store that I'm aware of unless created and uploaded to the hosting side of their platform, will even come anywhere close to validating.

I've spent hours trying to correct errors found by utilities - only to discover that I have no access to change 98% of the errors - that are built in to their product.

I'll try to find that but if I did for instance, I'd be afraid of deleting it or my site might go "haywire"!

Thanks again!

SemAdvance · #4 (**permalink**) 09-25-2008, 06:51 PM

I would not remove anything from your robots.txt as it is used to operate your site properly.

If you look at the file there are a few other directories as well which are blocked.

You might want to prepare to migrate to a new host and cart in the event Yahoo goes bye bye and you are forced to move out.

Peace!

cz · #5 (**permalink**) 09-25-2008, 08:16 PM

Since that page can't be spidered, I could copy the material to a page that has no "domain/cs" and that would open it up again? Somehow it has managed to have PR1.

SemAdavance, is there something I should know about?

There are solutions like storehost that will transfer a Y! store over in one day.

Thanks!

**wige** · #6 (**permalink**) 09-26-2008, 11:51 AM

If you have control over your robots.txt file, you might want to change the directories from /cz to /cz/, for example. Otherwise, if you are able to do redirects you could do a 301 redirect to a new URL. The only problem is that you will lose any benefit of links to the current page, since the search engines can't crawl the redirecting page.

dharrison · #7 (**permalink**) 09-26-2008, 01:52 PM

Hi cz

Make a backup of the original robots.txt, then if it does go haywire then you can easily revert back to the original.

I don't reckon it will go haywire though. I find it hard to believe that even Yahoo are that thick that they create an unspiderable page (?)

tnt7 · #8 (**permalink**) 10-05-2008, 11:33 AM

I Hate to be the bearer of bad news, but Yahoo! Store owners have no control over the robot.txt file.

cz · #9 (**permalink**) 10-05-2008, 05:58 PM

tnt7

what are your recommendations? the page, domain.com/cs... has PR (only PR1) do you think forget about it or change the page url and dump the old one? It just keeps sitting there in GG analytics like I should do somehting about it. Funny thing is it never showed up that way in all of the link checkers or in analytics until a few weeks ago?? Now it shows up in different site shceking tools as robot txt disallowed??

Thanks

tnt7 · #10 (**permalink**) 10-09-2008, 09:36 PM

Quote:

Originally Posted by cz

tnt7

what are your recommendations? the page, domain.com/cs... has PR (only PR1) do you think forget about it or change the page url and dump the old one? It just keeps sitting there in GG analytics like I should do somehting about it. Funny thing is it never showed up that way in all of the link checkers or in analytics until a few weeks ago?? Now it shows up in different site shceking tools as robot txt disallowed??

Thanks

I personally would do one of two things:
1. Call Yahoo! and complain and see if they fix it - Wait.
2. Change Url - You should get pr 1 back with your internal linking, and still call Yahoo! and complain.

They probably changed the global htacess with one of their recent updates, and that is why it has just become an issue.

They really have no excuse for this issue. I Wonder how many pages in the thousands of stores that are dissallowed??

Good Luck

cz · #11 (**permalink**) 10-09-2008, 10:38 PM

Quote:

I Wonder how many pages in the thousands of stores that are dissallowed??

Thanks for the advice, I think I'll do just that. That's a very interesting thought you mentioned as well!

Thanks

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Need code to NOT pass page rank or let a link get spidered.	MeanSEO	Search Engine Optimization Forum	16	07-26-2006 08:44 PM
2 domains - one spidered, second - only 1 page;	sandis.viksna	Yahoo! Discussion Forum	0	12-26-2005 08:12 PM
Why I am not being spidered???	pcm535	Search Engine Optimization Forum	4	11-29-2005 04:52 AM
Amount of words spidered per page?	ergobob	Search Engine Optimization Forum	2	08-13-2004 08:37 PM
only first page is spidered	ireneherz	Google Discussion Forum	3	04-09-2004 04:17 PM