View Full Version : Slurp and 404s w/ meta refresh redirect
I'm hoping someone may have some insight on this. I changed this website for the Great Lakes Cisco Users Group [www.glcug.com] from your standard HTML pages (which were in the root directory) to dynamic content. I did this about 3 or 4 years ago. It was a good three months ago when I finally deleted all the pages in the root directory that were of the old website and changed most of the dynamic content pages (which happened to be .php extensions) to static .html pages using mod_rewrite to allow better indexing for SEs. ..Which googlebot handled and continues to index beautifully, by the way.
My question is this, how in the heck can I get Inktomi Slurp to drop all the old pages and index the new website? I've submitted it about 5 or 6 times in the last three months and the bot is scanning my site about once or twice a week, but the search results only contain all the old pages. There's probably about 2 pages of listings on Yahoo! that are dead links which have not been in use for 3 years; and have been taken off the site completely as of 3 months ago...
http://search.yahoo.com/search?_adv_prop=web&x=
op&ei=UTF-8&prev_vm=p&fr=fp-top&va=&va_vt=any&vp=&
vp_vt=any&vo=cisco&vo_vt=any&ve=&ve_vt=any&vd=all&
vst=on&vs=glcug.com&vf=all&vm=i&vc=&fl=0&n=10
The way the site stands now, if you click on any of the links on yahoo!, it will return a custom 404 which will meta refresh a re-direct to the correct website homepage after 5 seconds. It also contains a hyperlink that says "click here" if you are not automatically redirected.
I can't for the life of me think of anything else I should do to get this corrected so any opinions or feedback would be greatly appreciated.
Great Lakes Cisco Users Group:
http://www.glcug.com/
bhartzer
11-03-2004, 12:02 PM
Probably the best way to deal with this is to follow the Yahoo! guidelines for removing pages from their index.
You can find more information from them here (http://help.yahoo.com/help/us/ysearch/deletions/deletions-03.html).
Thanks bhartzer, guess there's not much else I can do except wait. I just didn't know if custom 404s directing to the index page was was the best solution or not... I was thinking that maybe someone knew if using Temporary or Permanent redirects for each of the pages listed in Yahoo! would work better than 404 meta refresh redirects. Just trying to iron out the best way to get Inktomi to pick up on the change... Any other food for thought?
sfowler
11-04-2004, 10:19 AM
If they are crawling the site, the problem is Yahoo! and the legend of indexing. They do update the index every year on te 30. February, I have heard. There is nothing more you can do. I am not sure if Google likes automatic redirects, though. This is a typical porn site trick. I would just use a clickable link.
mbhmirc
11-05-2004, 07:40 AM
Thanks bhartzer, guess there's not much else I can do except wait. I just didn't know if custom 404s directing to the index page was was the best solution or not... I was thinking that maybe someone knew if using Temporary or Permanent redirects for each of the pages listed in Yahoo! would work better than 404 meta refresh redirects. Just trying to iron out the best way to get Inktomi to pick up on the change... Any other food for thought?
googleguy said 301 perm moved header in a php/html file should do the trick :)
sfowler
11-05-2004, 09:03 AM
But Yahoo! has a problem with 301 redirects.
incrediblehelp
11-08-2004, 11:33 AM
But Yahoo! has a problem with 301 redirects.
301 redirects on Apache or IIS or both?
webdumby
11-08-2004, 08:07 PM
hello, this is off the subject but as my name states I'm new to this. I read your post and the mod_rewrite that allows better indexing for SEs. caught my eye. I'm working on a PHP site http://www.competitionmotorsports.us and was wondering if you could give me the name of the program your using. Thanks
Thanks for all the feedback on this. I subscribed to this topic, but the link must have got broken because I didn't get anymore notifications that you guys replied. Anyway....
If they are crawling the site, the problem is Yahoo! and the legend of indexing. They do update the index every year on te 30. February, I have heard. There is nothing more you can do. I am not sure if Google likes automatic redirects, though. This is a typical porn site trick. I would just use a clickable link.
Yeah, tough pill to swallow with the Yahoo! problem. That's too bad, I have always held Yahoo! in such high regard but it seems to be very difficult over a very long period of time to accomplish absolutly nothing. Oh well. Thanks for the input though. As for the re-directs, I figured 404s would do the trick which refresh using the meta refresh tag which I have set to like 10 seconds or so. Not sure if the time matters or if I should take the refresh out and just use the link that's on the page or not. Would that be better practice? I also thought about using window.location JS instead of the meta refresh tag. I've seen it get handled better when it comes to indexing pages that the command redirects to. Alternativly, what I was hoping to accomplish with meta refresh is that the SEs wouldn't like it and remove the page they had in the index that referenced that location. Not sure, food for thought?
googleguy said 301 perm moved header in a php/html file should do the trick :)
But Yahoo! has a problem with 301 redirects.
Thanks guys, I was hoping to get some feedback on this. According to the other topics posted in this form, it seems that this has been a topic of heated debate for some time so I was hoping to get something here that was more concrete. Some say it is good, some say it isn't, so I guess its still a matter of debate... So I think I'll just stick to the 404s unless someone really knows about which would be better and has implimented such to success. Thanks though, it was what I was hoping to iron out.
hello, this is off the subject but as my name states I'm new to this. I read your post and the mod_rewrite that allows better indexing for SEs. caught my eye. I'm working on a PHP site http://www.competitionmotorsports.us and was wondering if you could give me the name of the program your using. Thanks
You'll have to read up on the mod_rewrite function to understand how it works so you can determine the best way to impliment it in your current environment. Check out the following page:
http://www.modrewrite.com/
It has great links to information and resources available to learn and communate in forums. Sorry, I can't be of more help, I hope this at least gets you started.
Stephen
Great Lakes Cisco Users Group
www.glcug.com
Just thought I'd followup and let everyone know that the 404s and delayed meta refresh seemed to do the trick ... it seems to be totally fine now in Yahoo! so maybe this could be useful to someone else later on...
Thanks for the suggestions.