Desperate woman drowning in world of seo

[bobbyeg]

I had an "outdated" website replaced by a new, "more professional" website and have plummeted in the search engines since the new site was submitted. Historically had enjoyed good placement but now am hard to find anywhere. Any thoughts or suggestions would be more appreciated than you can imagine. My website is www.RollingThunderRiverCo.com

[minstrel]

Originally Posted by bobbyeg

I had an "outdated" website replaced by a new, "more professional" website and have plummeted in the search engines since the new site was submitted. Historically had enjoyed good placement but now am hard to find anywhere.

Need more information...

- Did anything else change besides the site design? That is, did your domain name change? did the names of any individual pages on your site change? if so, it may be that the new pages haven't yet been indexed...

- how long ago did this change to your website take place? when did you "plummet"? what keywords did you plummet for? (i.e., what were your best keywords before?) There is a new and ongoing update happening at Google which is producing somewhat unusual results currently - could it be part of this rather than anything to do with your site?

Looking at your home page... the title, description, and keywords tags all look okay but then I see this:

<META NAME="Revisit" CONTENT="30 days">

You are telling any spiders that pay attention to this tag not to bother returning to look at your site for a month -- is this really what you want?

<META NAME="classification" CONTENT="outdoors, recreation, sports, adventure travel, vacations, health, fitness, culture, camping">

This tag isn't necessary and may be limiting how some search engines or directories categorize you.

<META NAME="Rating" CONTENT="General">
<META NAME="Copyright" CONTENT="Adventure Sports Online / OutdoorMind LLC http://www.adventuresports.com - HTML Coding Copyright © by OutdoorMind LLC, All Rights Reserved.">
<META NAME="Author" CONTENT="Adventure Sports Online / OutdoorMind LLC http://www.outdoormind.net HTML Coding Copyright © by OutdoorMind LLC, All Rights Reserved.">
<META NAME="Hosting" CONTENT="This site Hosted by Adventure Sports Online / OutdoorMind LLC http://www.outdoormind.net web hosting">

These tags are also unnecessary - probably doing nothing except taking up space...

In general though, I don't see anything blatantly amiss on the home page at least and that page has text links for the bots to follow to your main subpages...

You are in a popular category - a Google search for "white water rafting" yields 612,000 results...

See questions above - if the changes were recent and involved creation of new page names, you may need to wait a bit longer to get re-indexed, and removing that "revisit after" tag might help speed that up.

If the "plummet" happened in the past week to 10 days, it may just be the Google updates...

[bobbyeg]

Thanks for your comments. New site was submitted to search engines in June, 2003. We seemed to just disappear. It did not involve a change in domain name. Keywords Ocoee River, Ocoee River rafting, Nantahala River and Nantahala River rafting were good keywords. Not sure what you mean by page names changing. Our title tags were changed. I don't even know where I am showing up in the search engines because we are so far down the list. Except for some reason in Teoma (which I didn't even know existed until recently).

[minstrel]

By "page names changing" I mean the actual file names of the pages on your website.

For example:

home.htm
nantahala.htm
ocoee.htm
toccoa.htm
groups.htm
lodging.htm
contact.htm
faqs.htm
...
etc.

did these pages exist under those filenames before or are they new?

Looking at the keyphrases you mention, I see you still have a lot of competition... I would suggest looking at those sites that rank high in the search results especially in terms of what they highlight in the content of their pages that might match your keywords... for example, on your home page I see this:

"Nantahala River in western North Carolina, the exciting Olympic class Ocoee River in east Tennessee and the gentle Toccoa River in north Georgia"

no "rafting" anywhere... just the river or state names... you would do better to find ways to rewrite that text so that whole phrases you want to target occur on the page, and more than once if you can do so in a way which isn't simple keyphrase repetition...

[bobbyeg]

minstrel, some page names stayed the same and some changed, will work on rewriting copy now. many thanks.

[Mel]

Hi Bobbyeg
The reason your pages are not ranking well is simple - they are not being indexed by the search engines.

Google knows about links to 33 of your pages but has indexed none.

MSN knows about one link to your pages but has indexed none

Alltheweb (feeds Lycos too) knows about one link to your pages but has indexed none.

ALtavista has indexed only one of your pages.

Thats the problem but why? Well because in your robots.txt file you have told all robots not to index your site:
User-agent: *
Disallow: /

Rewrite that robots.txt file like this:
User-agent: *
Disallow:

In looking over your site there are many other small things which, taken together, might also give you problems in the future:


1. Your site seems to spider ok, but there are lots of broken links on your links page:
http://www.rollingthunderriverco.com/links.htm
http://www.mountainvistainn.com/
\_____ error code: 12007 (no such host)
http://www.kaylormtnhomes.com/
\_____ error code: 12017 (cancelled / timeout)
http://www.premiercabinrentals.com/
\_____ error code: 12007 (no such host)
http://www.acorncabinrentals.com/
\_____ error code: 12007 (no such host)
http://www.mountaintopcabinrental.com/
\_____ error code: 12007 (no such host)
http://www.lakeviewlodgerentals.com/info.htm
\_____ error code: 12007 (no such host)
http://www.wildnernessviewcabins.com/
\_____ error code: 12007 (no such host)
http://www.mtngetawaycabins.com/
\_____ error code: 12017 (cancelled / timeout)
http://www.blueridgega.com/DaysInn
\_____ error code: 404 (not found)
http://www.southernregion.fs.fed/us/cherokee
\_____ error code: 12007 (no such host)
http://www.adventuretrailrides.com/
\_____ error code: 12017 (cancelled / timeout)
http://www.georgie-atclub.com/
\_____ error code: 12007 (no such host)
http://www.southernregion.fs.fed/us/ocoee
\_____ error code: 12007 (no such host)
http://ngeorgia.com/adventure/mountain_biking.htm
\_____ error code: 404 (not found)

http://www.rollingthunderriverco.com/maps.htm
http://www.rollingthunderriverco.com...irlinks_04.gif
\_____ error code: 404 (not found)

2. Your DTD is a customised DTD (written by your designer?) but there is no url to where this document can be found. I would suggest that you change this to a standard DTD.

3. You have a number of unnecessary meta tags in your header including <META NAME="Revisit" CONTENT="30 days"> which asks the spiders not to visit your site for another 30 days.

[minstrel]

Originally Posted by Mel

The reason your pages are not ranking well is simple - they are not being indexed by the search engines. Google knows about links to 33 of your pages but has indexed none.

??? I'm not sure what you're seeing Mel but the home page IS indexed in Google - when I search for the domain name, it finds it along with sveral backlinks, including Yahoo...

1. Your site seems to spider ok, but there are lots of broken links on your links page:
http://www.rollingthunderriverco.com/links.htm/

Any broken links need to be fixed, certainly, but since these are outgoing links and largely to other sites, that shouldn't affect the indexing of her site...

2. Your DTD is a customised DTD (written by your designer?) but there is no url to where this document can be found. I would suggest that you change this to a standard DTD.

Again, a good recommendation, but again this shouldn't affect spidering or indexing...

[Mel]

HI Minstrel
I have since found the problem and edited my post above to indicate that the robots.txt file is at fault.

What you are seeing in Google is URLs with no title or description, which indicates that google is aware of the link but has not yet indexed it.

[minstrel]

Holy mackerel! Nice detective work, Mel!

That would certainly explain a lot....

Also thanks for the explanation re: the URL and "awareness"/indexing - I thought you were saying that the URL wasn't in the database...

[Mel]

Bad robots.txt files happen more often than you might imagine Minstrel, I should have checked there right off when I saw that the pages were not being indexed, but Altavista having indexed a page threw me off.

Most search engines follow the same practice of including only a URL without a title or description if they only have a link to the page (I suppose in their crawler index) but it has not been indexed.

You can see it in action more clearly if you enter the url in the Alltheweb search bar and when you get back the URL without title or description, click the option to find all web pages indexed under that URL which will then take you to a page which says that no pages are indexed.


I should also mention that Bobbyeg should re-submit her site to places like Altavista and other crawler based search engines in order to speed up the indexing process.

Also interesting to note that the Altavista spider ignored the robots.txt directive and indexed one page, and from what bobbyeg is saying looks likethe Teoma spider is also ignoring the robots.txt protocol. Good thing to remember.

Now would be a good time for everyone to review thier robots.txt files and make sure they are uptodate and work the way you want them to.[/b]

[jackson992]

Mel: What if we have no robots.txt file?

I think I only have it on one of my 5 sites, only because I wanted no pictures indexed on that site

[Mel]

Hi Jack
You can do without a robots.txt but on almost every site there are directories we normally want kept private like:

/cgi
/mail
/stats...

[jackson992]

I've never used those directories lol.

Anything I need to know about?

[Mel]

FTP to your site and look at the directory structure your host has put up and you'll probably find directories like that or something similiar.

In addition to keeping, say, your site stats your own business, this also can save spiders time allowing them to spider more of the pages you want then to.

[jackson992]

yeah I've seen the directories there on my ftp.

Are you saying I should add a robots.txt file then and if so what do I put in it lol

[minstrel]

It should look something like this, Jack:

User-agent: *
Disallow: file1.htm
Disallow: /misc/file2.htm
Disallow: /cgi-bin/
Disallow: /stats/

where file1.htm and file2.htm are individual files you don't want indexed and cgi-bin and stats are entire directroies you don't want indexed.

Anything you don't "disallow" is allowed, meaning it will be spidered.

[bobbyeg]

can I sum up this discussion by saying that my site has NOT been indexed by major search engines? The company I have been paying to handle the submissions and optimization have made an error which has cost me a place in the "sun".

[minstrel]

Originally Posted by bobbyeg

can I sum up this discussion by saying that my site has NOT been indexed by major search engines? The company I have been paying to handle the submissions and optimization have made an error which has cost me a place in the "sun".

Well, unless you created the robots.txt file yourself, I would guess that the SEO company is responsible - although the problem is easily reversed, it may still have cost you money for the time that robots.txt file has been on your site and for however long it takes you to get your site re-indexed.

I suspect that what may have happened is that whoever worked on your site added that "disallow" while the new site was in development and forgot to "un-disallow" when the job was finished. Either that, or they simply don't know what they're doing...

Either way, I think you may have a case for a refund of some of your money, if not damages...

[bobbyeg]

Minstrel, a followup question please. The company which designed and "submitted" my new website says they indexed it when the new site was finished in May, 2003 and then after I complained and complained about no results, they say they reindexed it the week of December 8, 2003. Would this mistake most likely have been a part of this situation from the very beginning? Or would it have just happened perhaps when they "reindexed" it in December? Since I haven't seen any results since the very beginning, I am inclined to believe that the mistake has been there since the beginning. But I know too little to start "assuming" anything. You people have been a real blessing for me. Thanks.

[minstrel]

Originally Posted by bobbyeg

The company which designed and "submitted" my new website says they indexed it when the new site was finished in May, 2003 and then after I complained and complained about no results, they say they reindexed it the week of December 8, 2003. Would this mistake most likely have been a part of this situation from the very beginning? Or would it have just happened perhaps when they "reindexed" it in December? Since I haven't seen any results since the very beginning, I am inclined to believe that the mistake has been there since the beginning.

You said your new site was submitted in May/June and disappeared at that time so there almost certainly was a problem from that time. As for the company claiming they "indexed" your site in May and "reindexed" it in December, I don't know what they mean by that - I would suspect they mean they "submitted" it, but that wouldn't have helped since when the spiders visited they would have run into the robots.txt file saying "go away - do not spider this site". If they mean that your site was "spidered" in May and December (which they could determine from your site logs), again that would only mean that your site was visited by the spiders, who then found the instruction in the robots.txt file to "go away".

When you change the robots.txt file (or have it changed), don't forget about the other issues noted in your meta tags, especially the "revisit after" tag telling spiders not to return for a minimum of a month - that should be deleted. When these changes have been made, you should resubmit your site to all the major search engines.

[Mel]

I agree Minstrel and have seen this kind of mistake on a new site more than once as designers will often put up a no index robots.txt while in the process of design and sometimes forget to change it when they go live.

In this case its hard to know who made the mistake originally, but since Bobblyeq has been paying someone to optimize and submit her pages, they should be taken to task for not finding and correcting the problem.

There is a software company that uses the slogan "because submitting is not enough" and this is really true for SEOs, who should as a matter of course be telling their clients how many pages are indexed in each search engine and what the rankings are for the chosen keywords. If it were my site I would be having a serious discussion with a company who I paid to optimize my pages and submit them.

Asking questions like:

Did you check the robots.txt file (answering this question either yes or no is is an admission of not doing their job right)

Did you check to see if the pages you submitted were being indexed?

Did you check to see what the rankings of the pages you optimized and submitted were?

Do you expect to keep my money after doing an incompetent job?

[jackson992]

Thanks guys:

I have made a default robots.txt file for all my sites

[eyetest]

Hi everybody! Mick here.
First time caller, long time webmaster.
Everybody is mentioning some of the meta tags as being the problem, and they're right. But, another major problem IS the title, description and keywords also. The combination of these is disastrous. For starters go to
Scrub The Web's meta check tool to verify that your title is WAY too long. The title being one of the most important factors in search engines, should be under 60 characters and have your main keywords in it only. The description is also too long and the keywords should be broken down into smaller phrases. eg "whitewater rafting adventures, whitewater rafting, rafting adventures" should all be used to give each other relevance. When using the full phrase whitewater rafting adventures it counts as three keywords. This can be done with all keyword phrases. Just gotta keep'em seperated, as always though.
Doing this along with the other suggestions should put you up in the searches again.
Whew! That's alot, but you gave the search engines quite a few reasons to drop the site.

[spiderbait]

I think I can answer your question about "when" this happened with your robots.txt file:

Originally Posted by bobbyeg

Minstrel, a followup question please. The company which designed and "submitted" my new website says they indexed it when the new site was finished in May, 2003 and then after I complained and complained about no results, they say they reindexed it the week of December 8, 2003. Would this mistake most likely have been a part of this situation from the very beginning? Or would it have just happened perhaps when they "reindexed" it in December? Since I haven't seen any results since the very beginning, I am inclined to believe that the mistake has been there since the beginning. But I know too little to start "assuming" anything. You people have been a real blessing for me. Thanks.

If you check the last modified date of the robots.txt you'll find the following:
Last-Modified 38 weeks 4 days ago (Thu, 08 May 2003 21:12:16 GMT)

So that gives you the date on which someone uploaded the current robots.txt file. Using that date you can determine more accurately who may be responsible.

Hope this helps.

[icb01co2]

Mel - i noticed before when you stated:

Your site seems to spider ok, but there are lots of broken links on your links page:

that it followed by lots of spider information, what tool did you use to get this info?

Thanks, Chris.

[spiderbait]

Due to some private messaging on this topic I'm posting the source of the above mentioned last modified date.

You can see the cacheability of the file which includes the "last modified" date here.
http://www.ircache.net/cgi-bin/cache...com/robots.txt

you can use the cacheability checker for other urls here:
http://www.ircache.net/cgi-bin/cacheability.py

and if you look around the above site you can see that they also have a handy tool you can install that gives a right-click contextual menu choice for checking the cacheability of any object on any site.

[Bark Bakery]

Minstrel,
I need your help. I have being reading over your messages for the past few weeks from the newsletter and you seem to know what you are talking about. I have spent $700 + in the last 2 weeks on 2 programers and I can not spend any more money.
I am learning alot about SEO & Links. I will start working with people (within the Dog Industry) to share links but will you look at my HTML and make sure there is nothing in there that should not be in there?? Not sure what to ask you for but I need the "Spiders to look at my site."
Can you help me? I will try to check this section to see if you have written me back but not sure if I can find my way back here. Maybe send me a quick e-mail as to where you will post my answer. How do you get into my HTML? From my V-Deck??
amy@barkbakery.com oh..... The website is www.barkbakery.com
Thank you so very much,
Amy

[minstrel]

Originally Posted by Bark Bakery

I have spent $700 + in the last 2 weeks on 2 programers and I can not spend any more money. I am learning alot about SEO & Links. I will start working with people (within the Dog Industry) to share links but will you look at my HTML and make sure there is nothing in there that should not be in there?? Not sure what to ask you for but I need the "Spiders to look at my site."

Starting with your home page, the people who worked on it didn't give the spiders much to index. First, make a list of kwywords and keyphrases that people are likely to enter into a search box to find the products you sell - maybe something like "dog treats". "home made dog treats", "dog food", etc. - you know the business better than I do so I'm sure you can do better. Go look at your competitors sites and see what words they are using in the text of their pages and in their mata tags.

Using those words, work them into the text (i.e., the content) of your home page - this is the main page people and spiders will hit when they type in www.barkbakery.com, so you want it to highlight what your site is about. Describe for your visitors as well as for the spiders what it is you have to offer...

Next, the titles of your sub-pages aren't helping you (page1.html, page2.html, etc.). Rename these to something meaningful, like "letters.html", "menus.html", that sort of thing, or make them even more specific than that if you can.

Now, your HTML - again, there is very little for the spiders to index here:

<title>Bark Bakery</title>

That's it - a title which says nothing - and nothing else.
Change the title to something the spiders can index, including words like the key phrases I mentioned above. Having the title of your site there is okay but move it to the end - put the more important search words up front.

Add a description tag with similar key phrases:
<META name="description"
content="home made dog treats, dog cookies, specialty dog foods, etc">

Then add a keywords tag, again with similar phrases:
<META NAME="keywords"
CONTENT="specialty dog treats, homemade dog food, dog cookies, etc">

Next, you have a whole bunch of images, again with very unimaginative and unhelpful file names, and none of them have ALT tags, which again is content the spiders can index but your page has nothing:

[img]img0.gif[/img]
[img]img1.gif[/img]

For each image tag, add at the end an ALT tag, like this:

[img]img0.gif[/img]
[img]img1.gif[/img]

Create a sitemap.html page which includes text links to all your subpages. Then at the bottom of your home page, add a text link to sitemap.html (down where it has all those other text links in a row.

Now, go to work and make the same kinds of changes on the other pages on your site - I just had a quick look and there seems to be a bit more content but again it's not highlighting words and phrases that people are likely to type into Google to find you, so the content neds work - and the problems with the title tag, the meta tags, and the image alt tags also need to be fixed on those other pages.

How do you get into my HTML? From my V-Deck??

I don't understand your question... what's a V-Deck? You can go to your website and save any one of your pages using File | Save as... in Internet Explorer, and then open it in notepad or in Publisher (that seems to be the program used to create your site) or in any HTML editor...

[Mel]

HI Chris
I use the Xenu link spider, which is a free download (just search for the name in Google or MSN). There is one other nice feature of this tool that is often not appreciated, and that is the option to generate a site map based on the spidering of the site, and using the page titles of each page as anchor text to link to the pages.

[bobbyeg]

Originally Posted by Mel

HI Minstrel

What you are seeing in Google is URLs with no title or description, which indicates that google is aware of the link but has not yet indexed it.

Mel, a followup question please. you say that google has not yet indexed my site. google emailed me today in response to my query that my "site is currently in their index." I'm getting confused. But I know you can clear it up for me.

[minstrel]

Originally Posted by Mel

I use the Xenu link spider, which is a free download (just search for the name in Google or MSN).

Yikes! I'm agreeing with Mel again!

I also use Xenu - a great freeware tool - fast and configurable to ignore certain types of files or links.

Get it here:

Xenu Link Checker

[minstrel]

Originally Posted by bobbyeg

Mel, a followup question please. You say that google has not yet indexed my site. Google emailed me today in response to my query that my "site is currently in their index." I'm getting confused. But I know you can clear it up for me.

Yes, Google is aware that your site exists and they have the URL listed in their database. However, they have no other information about the site other than that it exists - no content or keywords, etc. - because your robots.txt file has instructed Google and everyone else NOT to index your website.

[Mel]

Hi Bobbyeg
Ok do a search like this
site:rollingthunderriverco.com +"the" which searches your site for any pages which have the word the on them. You will get back a page which shows four results were found but which do not have either a title or description like normal listings do.

Then search for your URL in Google -again on epage found but no title or description.

Now search for allinurl:rollingthunderriverco.com and you will see that it returns 33 pages none of which have a title or description when normally they would return the page title as a description and a text snippet taken from somewhere on the page.

So when Google says your site is in their index, thats sort of correct, the url is there but there is no detailed information nor are there any rankings. You will also note that there is no cache of any of the pages. Compare your listings with normal results to see what I mean.

When google run across a situation like yours they spider the home page, noting all the links, but they do not follow the links nor do they index the contents of the pages. Google calls this partial indexing, and indicates that it is aware of the page but that for some reason has not spidered it fully and indexed the detailed results.

But writing to Google is not going to help your situation much. Once your robots.txt file is changed (I note that it still is reading:
User-agent: *
Disallow: /

why???? )

You have to get that robots.txt file changed if you want the search engines to spider you You can go to this page for confirmation of what you should have in it:
http://www.robotstxt.org/wc/exclusion-admin.html

Google will find you from one of the 150 links you have to your site (Nice links mostly right on topic, though I wonder who got you listed in the Indian, HonkKong and Canadian versions of Yahoo?) spider all your pages and start ranking you again. Its interesting to note that although you don't have any of your pages listed in Google fully, You do have PR which is coming from your inbound links.

I do not see you listed in the DMOZ directory, which will also get you listed in the google directory You should sbumit your site to recreation>>Boating>>Paddling>White water rafting category and also to the appropriate regional directory.

[icb01co2]

Thanks, Mel. Likeing The 'Xenu link spider' tool alot.

Does anyone know a good tutorial for writing a robot.txt file? And where exactly do i place it on my server?



-Chris

[bobbyeg]

You are so good at answering my questions in ways that I can actually understand. You people are so good at this that you are never going to get rid of me. I am probably going to pester you over and over. I wish I had known earlier that this forum existed. WOW!

[Mel]

Originally Posted by icb01co2

Thanks, Mel. Likeing The 'Xenu link spider' tool alot.

Does anyone know a good tutorial for writing a robot.txt file? And where exactly do i place it on my server?



-Chris

Take a look here:
http://www.webproworld.com/viewtopic.php?t=13110

[Brittany]

Originally Posted by bobbyeg

You are so good at answering my questions in ways that I can actually understand. You people are so good at this that you are never going to get rid of me. I am probably going to pester you over and over. I wish I had known earlier that this forum existed. WOW!

Thanks for the compliments! It's great to hear that you are enjoying your WebProWorld membership, and I look forward to seeing more of you around the forums! Please keep asking those questions -- that's what we're here for!

Best wishes,

Brittany
Forum Host