Sitemap's URL structure

[Ricardo Zea]

Hello all,

I tried looking for the answer in other posts but couldn't find it.

Here's my problem.

I have submitted my sitemap to Google and I see that my results' description are all messed up when I do a "site:www.aidmydebt.com" search in Google.

If you do a search right now the description of the pages look like this one:

"Index of /debt-settlement/HI-Hawaii/Lihue. Name Last modified Size Description. [DIR] Parent Directory 11-May-2007 12:51 - [ ] debt-settlement-Lihu. ..."

My sitemap XML file looks like this for that specific page:

" <url> <loc>http://www.aidmydebt.com/debt-settlement/HI-Hawaii/Lihue/debt-settlement-Lihue.php</loc>
</url> "

For some reason Google is not 'reading' the file "debt-settlement-Lihue.php" in the "/Lihue" folder, and of course, if Google doesn't see the .php file it will stay in the folder.

I have a lot of pages and all my sitemaps' URL structure looks like the one above, the only differences are the states' and cities' names.

Take into account the ending of the URL: ".php"

My question is: Does the sitemap URL structure HAS to end in a "/"?

I have read around the web and seen sitemap.xml examples and in some of them some URLs end in "/" and some others end in a simple ".html"... so I assumed that ending in ".php" would be ok too. Then again, seeing the description of my results in Google makes me want to wonder if I'm doing it wrong (which is probably the case, yet it doesn't make sense) or if it's a Google's issue.

I need to fix this fast.

Thank you for any help you can provide me with.

[ukresident]

"For some reason Google is not 'reading' the file "debt-settlement-Lihue.php" in the "/Lihue" folder, and of course, if Google doesn't see the .php file it will stay in the folder"

how do you know this?

but to answer your question the url can end in any extension as long as the url is valid

[ukresident]

sorry and I also meant to say that it can also end with a / as long as that is how you specified your url structure in htaccess

[incrediblehelp]

I guess I am confused. What does your sitemap have to do with your description? Google is going to choose to list a description from various sources:

DMOZ description or content from your page or from the tag itself. Seeing Google use the description from your description tag is rare to say the least. So if your expecting Google use that you would be wrong.

[Ricardo Zea]

ukresident and incrediblehelp, thanks for your answers.

Now to clarify.

"how do you know this?"

I know this because if you do a site: search in Google, the description shows the content of a folder without an index.* file: "Index of /debt-settlement/HI-Hawaii/Lihue. Name Last modified Size Description. [DIR] Parent Directory 11-May-2007 12:51 - [ ] debt-settlement-Lihu. ..."

Just go to Google and type: site:www.aidmydebt.com, you'll see the second result is my example here (in fact, ALL the results, except the first one, looks like my example above corresponding to each folder).

"What does your sitemap have to do with your description?"
Well, that's exactly what I'm trying to find out, if maybe Google is using my sitemap to look for a specific file ending in .php and for some unknown reason it's not seeing .php file and it only goes as far a the /.

My great concern here is: How is it that Google is indexing this URL: www.aidmydebt.com/debt-settlement/HI-Hawaii/Lihue/ if there's nothing on my website poiting to that folder?

The complete route using that path would be: www.aidmydebt.com/debt-settlement/HI-Hawaii/Lihue/debt-settlement-Lihue.php

So that's why I'm speculating that Google is using my sitemap to find that incomplete URL (AND the rest of them, which there are a lot).

--

As rare as it is, Google IS describing my website www.aidmydebt.com from my Description tag.

Of course, the only description that Google is going to put on the 'incomplete' URLs (www.aidmydebt.com/debt-settlement/HI-Hawaii/Lihue/) is the only text that comes up wich is the little text that's there when you visit that incomplete URL.

--

Taking into account what incrediblehelp said, let me rephrase the question then:

How is it that Google is indexing those 'incomplete' URLs? ie: www.aidmydebt.com/debt-settlement/HI-Hawaii/Lihue/
• I have my sitemaps well typed and structured.
• All my links in my website are correct.
• Nothing on my website points to an incomplete URL like the example above.

Thanks for your help.

[sam-adam]

I tryed a sitemap structure for some of my sites i coudn't see any differecnce in indexing the pages or receiving more hits.

So i'm not convince about usefullnes of that

[Ricardo Zea]

I'm just speculating that my issue is sitemap related, not that I know this for sure.

Again, Does anyone have any idea why Google is indexing incomplete URLs on my website? ie: Index of /debt-settlement/HI-Hawaii/Lihue

Thank you.

sam-adam, thanks for your answer too.

[incrediblehelp]

Originally Posted by Ricardo Zea

I'm just speculating that my issue is sitemap related, not that I know this for sure.

Again, Does anyone have any idea why Google is indexing incomplete URLs on my website? ie: Index of /debt-settlement/HI-Hawaii/Lihue

Thank you.

sam-adam, thanks for your answer too.

If these URLs are not on the sitemap then they are not getting them from there.

Could these URLs be available on your website somewhere? They could get them from there

Could these be available on other websites? Scrapper/spammers? Of course it could. I see many broken or unnecessary URLs index in Google and have no idea where they came when cheking index URLs for client websites. When this happens it is of no concern to me. Most of the time it is from scrapper websites.

[Ricardo Zea]

Good point incrediblehelp, scrapper websites (new term for me too )

Now, I've been talking to a friend of mine, seems like the problem is the webserver itself and not my sitemaps or anything from my end, yet we are not 100% sure. We posted a ticket with our hosting company, I will let you know as soon as I know something.

Thanks again guys.

[Ricardo Zea]

Ok, problem resolved, now it's time to wait for Google to eliminate those URLs from its index.

The problem was the server as I mentioned, the server was handing all the directories to the bots that didn't have an index.* file in them, so we disabled the directory indexes, this means that now if a folder does not contain an index.* file it can not be 'read' by anybody and this includes search engine's bots.

Now if anyone tries to visit one of the broken or 'incomplete' URLs they will get an ERROR 403, which means FORBIDDEN ACCESS.

I also edited my .htaccess file and added this line: ErrorDocument 403 http://www.aidmydebt.com/debt-settlement-quote.php. This line redirects all the visitors, humans and bots, to my form (you can choose where to redirect).

I will have to wait a few weeks to see all those incomplete URLs gone from Google's index.

Now everybody is happy

Thanks to all of you for your help.

[incrediblehelp]

Just a reminder Ricardo. Google has a URL removal tool in their Google Webmaster Tool Set.

[Ricardo Zea]

Yes! You are right incredihelp... my issue is that I'd have to type in more than 50 URLs, maybe I'll do that anyway, thanks a lot for your help