Strange URLs in Google Webmasters Tools

[ashishdabas]

Hi Everybody!

I am using Google Webmaster Tools for quite a while now and am very happy with the information it provides for a site at one place.

But a week back when I was checking one of my site in Google Webmaster Tools, it showed 258 URLs Not Found??? I was just amazed to see this! When I checked the URLs i couldn't understand what the URL is?

Let me explain you clearly. I am sorry I will not post my website here. For example, let us say I have a website http://www.mysite.com . In GWT (Google Webmaster Tools) the URL shown is http://www.mysite.com/chinese.php?u=...www.mysite.com (this is for homepage) and in the same way for all inner pages like http://www.mysite.com/chinese.php?u=...e.com/abc.html

My concern is, what is "chinese.php?u=" coming in these URLs. It also shows "spanish.php?u=" "french.php?u=" and "italian.php?u=" in URLs.

I am just puzzled. I did not upload anything like this on my server or created any pages like mentioned above ever.

Please help me in coming out of this maze. If anyone has any queries please ask.

Thanks a lot in advance.

[meinking22]

Sounds like you are using dynamic pages. Google has this to say about dynamic pages in its guidelines:

"If you decide to use dynamic pages (i.e., the URL contains a "?" character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few."

So right now, there are no guarantees for dynamic page indexing, although I have read in blogs that it is improving.

Steve

[incrediblehelp]

Don't be scared about using dynamic URLs. The SE's index them fine as long as you stay under 2-3 variables.

Now for the strange URLs it could be simply a scrapper website linking to a page on your website that doesn't exists.

If I go to my blog and create a link: http://www.youtsite.com/iamacrazyperson.php. The spiders will crawl and it try to go to it.

Your raw logs will register a 404 for this page. Now what is the scrapper is doing this a whole bunch of times (who knows why) then you will get a bunch of errors for pages that don't exist. Google Webmaster Console will also register this.

[Fitz]

Originally Posted by ashishdabas

My concern is, what is "chinese.php?u=" coming in these URLs. It also shows "spanish.php?u=" "french.php?u=" and "italian.php?u=" in URLs.

I am just puzzled.

It couldn't be Google's "Translate this page" generating on-the-fly versions of your pages for - in your examples - chinese, then spanish, then french, then italian surfers, could it? In other words, a Chinese person finds your results in Google.ch (is it ch?) and hits their equivalent of the Translate this page link - if you see what I mean? Might make sense.

Just a thought!

[krisidious]

sounds like maybe your site map has some incorrect entries?

[bj]

The best way to get around this is to get a sitemap tool that generates a sitemap. The best ones will see the same things that the search engines do. They also have filters you can set, so you can filter out urls such as you're seeing.

I had a similar problem on one of my sites with a directory script that gave people the option of sorting the listings alphabetically or by hits. This had the potential for a dupe content hit since the url looked different even though the content was the same. Generating a sitemap with filters in place will tell google what URLS on your dynamically generated site are real. I also have mine set up to generate both an html version and a yahoo version, so that there are less issues with this sort of thing.

You'll then have to submit your sitemaps. This should help, though it may take a few days to show up properly.

[marketing-helpersdotcom]

Originally Posted by krisidious

sounds like maybe your site map has some incorrect entries?

The two most common issues are the ones krisidious and incrediblehelp mentioned. I think its more likely the case incrediblehelp mentioned.

They may have linked to the wrong domain. There is a domain that links to my site on every page of their site and I can tell its a mistake because its a domain I don't advertise, and has a 301 redirect on it. Google has me down for over 400 IBLs to my site from that site, crazy.

[seoimage]

Yes, incrediblehelp may have nailed this one.

One statement in the webmaster console is that:

Googlebot found these pages either in your Sitemap or by following links from other pages during a discovery crawl

So some scrapers may be linking to you as an error. We see this a lot with the /undefined variable.

Unfortunately the Google tool does NOT display where the link is coming from so you cannot trace it backwords and try to fix it yourself.

[jtracking]

Originally Posted by meinking22

Sounds like you are using dynamic pages...

...So right now, there are no guarantees for dynamic page indexing, although I have read in blogs that it is improving.

As the wiki says: Google began as a research project in January 1996 by Larry Page and Sergey Brin, two Ph.D. students at Stanford University, California

I think Google has got a good handle on variables.

Just my opinion!

[swirt]

Google has given you the page it was looking for. Now look through your server logs and see what page was referring to that url. Then you can probably sort out who or what is generating the link that google might be following.

[ashishdabas]

Hi,

Thanks a lot to everybody for replies.

When I checked with Google Webmasters Tools today morning the list of not found URLs increased to 323 now with "portuguese.php?u=" added!!!

Sounds like you are using dynamic pages.

Hi, Meinking, I am not using any dynamic URLs anywhere in my site.

Now for the strange URLs it could be simply a scrapper website linking to a page on your website that doesn't exists.

If I go to my blog and create a link: http://www.youtsite.com/iamacrazyperson.php. The spiders will crawl and it try to go to it.

Hmm, that is possible, Mr.incrediblehelp...But we can't stop anyone linking to our site, we can only stop the people we know. So, Jaan is it possible to find out that scrapper website which is linking to my website? I didn't find any way!!!

a Chinese person finds your results in Google.ch (is it ch?) and hits their equivalent of the Translate this page link - if you see what I mean? Might make sense.

No Fitz, not possible this way.

sounds like maybe your site map has some incorrect entries?

The best way to get around this is to get a sitemap tool that generates a sitemap.

Hi Krisidious and Bj, I have both HTML and XML sitemaps on my site and have checked them properly twice, but didn't find any such strange URL written or any other errors in any version of my sitemaps and I keep updating my xml sitempas every day or two.

So some scrapers may be linking to you as an error. We see this a lot with the /undefined variable.

Unfortunately the Google tool does NOT display where the link is coming from so you cannot trace it backwords and try to fix it yourself.

Yes, I also think so! And the problem it we can't even trace these websites... So bad...

Google has given you the page it was looking for. Now look through your server logs and see what page was referring to that url. Then you can probably sort out who or what is generating the link that google might be following.

OK. I will check my logs also if I find anything there.

By the way, thanks a lot again and hoping for more replies and hope the problem will sort out this time.

[dbramley]

Do you use www.xml-sitemaps.com to create your sitemaps?

Its just that I searched for "portuguese.php?u=" and noticed it appeared in a lot of sitemaps created by: xml-sitemaps.com - not sure if they offer a page translation feature as well?

[ashishdabas]

Do you use www.xml-sitemaps.com to create your sitemaps?

Its just that I searched for "portuguese.php?u=" and noticed it appeared in a lot of sitemaps created by: xml-sitemaps.com - not sure if they offer a page translation feature as well?

Yes dbramley, I use xml-sitemaps.com to create xml sitemaps for my site. Are these strange URLs being created from this site? If yes, how to resolve it? Should I manually create my xml sitemaps as a solution?

Waiting for reply!!! Thanks.

[sande145]

Maybe of use to some of you: a good stand-alone site crawler, and it is free too!
http://gsitecrawler.com/
You can generate Google and Yahoo sitemaps, test the "crawlability" of your site, use filters etc. etc. Good German quality :-) I am using it for about half a year now and I am very pleased with it!

[OneMoreBite]

I am sorry I will not post my website here

That makes no sense. You're trying to figure out Google yet you don't want anyone to know your site's address?

Just curious.

[Clint1]

"ashishdabas", this is common problem with G. The G forums are loaded with similar posts. No one knows from where the URL's are generated. In my case (and as with other people), it's showing URL's under the 404 or "not found" area that never existed, and URL's with strange characters in them like ?, =, % and the like. It may be entries like domain.com/valid-URL.html=blah% where it's adding garbage after the .html

What I do is have to create 301's from these bogus URL's to the real page because I don't want to give G any more "excuses" to screw things up for me.

[swirt]

Originally Posted by clint1

What I do is have to create 301's from these bogus URL's to the real page because I don't want to give G any more "excuses" to screw things up for me.

Not to bring up an old discussion, but returning a code of "301 - permanently moved" for a page that never existed is creating more problems than it solves. The proper response for a page that does not exist is "404 - file not found". Google has no problems with 404 errors. The only time it is held as a bad thing against a site is if the links that generate the 404's come from your own site (and that's only if there are lots of them) If they are coming from other sites/scrapers, then they cause no harm. If 404's could cause harm then all your competition would have to do is point 1000 links to pages of your site that never existed. 301-ing them all would be a mistake and a huge waste of your time.

[Clint1]

Please explain how this can create more problems. There are DOZENS of these, and I have heard that the Gbot returning dozens of 'not found' pages at your site is a bad thing.
Thanks.

[swirt]

Originally Posted by clint1

There are DOZENS of these, and I have heard that the Gbot returning dozens of 'not found' pages at your site is a bad thing.

I'm answering this one first because it is the most important thing to come away with. Your info is wrong, as I already said. Google has no problems with 404's as long as they are not coming from a large number of links on your site (suggests the site is in disrepair). Once you have made sure that your site and your sitemap are not the cause of these phantom pages, then you should let it go and get on with other webmaster stuff.

Originally Posted by clint1

Please explain how this can create more problems.

This is only a few of what could be a very long list:
a)waste of webmaster time (see above) who could be doing something more beneficial to the site like creating content and link building.

b) By 301-ing them you generate no errors which means you have decreased your ability to track them back to their source and figure out where these links to phantom pages are coming from (to rule out that they aren't coming from your site).

c) Redirecting dozens of pages that Google has no record of can leave a footprint that might give the impression you are up to something devious like trying to increase your page count or funnel PR from non-existent pages. (it looks just like a blackhat approach at getting multiple links from a site without them looking like sitewide links)

c) MSN indexes 301 in a bad way so if there are actually links out there to your non-existent pages and you are 301-ing them. You are creating actual problems for other engines to eliminate something that was not a problem to begin with.

d) Server load... too many 301s in htaccess.

A safer way would be to block the phantom pages in robots.txt, but even that would be a waste of time.

[incrediblehelp]

Just wondering, if these links don't exist on your website, why does it matter? Google will stop crawling them eventually.

[Clint1]

Originally Posted by incrediblehelp

Just wondering, if these links don't exist on your website, why does it matter? Google will stop crawling them eventually.

Well I'm not positive it does matter, that's why I'm asking and trying to find out more about it. It stands to reason that any bot getting 404's at your site can't be a good thing. With G especially, they'll try and find ANY reason to penalize someone.

[Clint1]

Ok I understand what you're saying.

Originally Posted by swirt

b) By 301-ing them you generate no errors which means you have decreased your ability to track them back to their source and figure out where these links to phantom pages are coming from (to rule out that they aren't coming from your site).

They cannot be tracked back to the source anyway. G never gives any indication from where the bogus URL originate. I KNOW they are never coming from my site because like I said, they have properties in them I have NEVER used.

c) Redirecting dozens of pages that Google has no record of can leave a footprint that might give the impression you are up to something devious like trying to increase your page count or funnel PR from non-existent pages. (it looks just like a blackhat approach at getting multiple links from a site without them looking like sitewide links)

In my case G does have a record of the pages. For example, if I see mydomain.com/index.html showing up (and I DID), I did a 301 from that to mydomain.com . Or, if I see mydomain.com/valid-page-in-G.html[but with bogus crap here on the end], I redirect that to the real page mydomain.com/valid-page-in-G.html .

c) MSN indexes 301 in a bad way so if there are actually links out there to your non-existent pages and you are 301-ing them. You are creating actual problems for other engines to eliminate something that was not a problem to begin with.

I have yet to see MSN hit on a non-existing page.

d) Server load... too many 301s in htaccess.

Not an issue in my case.

What I think I may start doing is remove the older 301 redirects, to see if anything positive happens from it.

[swirt]

Originally Posted by clint1

It stands to reason that any bot getting 404's at your site can't be a good thing. With G especially, they'll try and find ANY reason to penalize someone.

No that does not stand to reason.

They cannot be tracked back to the source anyway. G never gives any indication from where the bogus URL originate. I KNOW they are never coming from my site because like I said, they have properties in them I have NEVER used.

Google doesn't give the source of the link ... but your server logs would and they are easier to spot if they throw a proper 404 error.

In my case G does have a record of the pages. For example, if I see mydomain.com/index.html showing up (and I DID), I did a 301 from that to mydomain.com . Or, if I see mydomain.com/valid-page-in-G.html[but with bogus crap here on the end], I redirect that to the real page mydomain.com/valid-page-in-G.html .

There is a difference between google indexing a page and trying to crawl a non-existent page.

I have yet to see MSN hit on a non-existing page.

How would you? MSN has no webmaster tools where they report crawl errors. MSN has an issue with 301's they index both the page that is 301'd and the recipient of the 301. Two pages in the index, same content. Bad news.

What I think I may start doing is remove the older 301 redirects, to see if anything positive happens from it.

That's a good start. However, absence of results is not proof that what you are doing now is the solution to a non-existing problem.

[Clarrie]

Originally Posted by sande145

Maybe of use to some of you: a good stand-alone site crawler, and it is free too!
http://gsitecrawler.com/
You can generate Google and Yahoo sitemaps, test the "crawlability" of your site, use filters etc. etc. Good German quality :-) I am using it for about half a year now and I am very pleased with it!

I use gsitecrawler as well, and it works very well. Just be a bit careful about using it (or guess any package that crawls a site to generate the site map) on big sites, like e-commerce sites with 000s of products... You can guess what happens to your bandwidth!

[incrediblehelp]

Google penalizing for 404 errors? I think not. Pages and URLs die all of the time, especially at very large websites.

[Conficio]

Originally Posted by sande145

Maybe of use to some of you: a good stand-alone site crawler, and it is free too!
http://gsitecrawler.com/
You can generate Google and Yahoo sitemaps, test the "crawlability" of your site, use filters etc. etc. Good German quality :-) I am using it for about half a year now and I am very pleased with it!

I can confirm that. It is a good tool and solid for a medium site (<5,000 pages).

K<o>

[MarkGatESS]

All of these applications that you've all mentioned are apps that are placed on the webserver itself to generate the sitemaps.

Are there any applications you can install on your local machine that will go over your internet connection to "crawl" and generate your maps, thus not having the need for you to install and run it on the web server? Something that will hit the server like a surfer using a browser?

A "free" application would be nice, but not necessary ;-)

BTW - my website has over 500 pages because it uses Early Impact's ProductCart. I suppose there's probably NO free generation application that will do that many pages.

[sande145]

Originally Posted by MarkGatESS

Are there any applications you can install on your local machine that will go over your internet connection to "crawl" and generate your maps, thus not having the need for you to install and run it on the web server? Something that will hit the server like a surfer using a browser?

A "free" application would be nice, but not necessary ;-)

BTW - my website has over 500 pages because it uses Early Impact's ProductCart. I suppose there's probably NO free generation application that will do that many pages.

The program mentioned actually works from your own (Windows) computer and crawls the site. So it's a good tool to check your internal links to. By means of filters you can exclude, or include, for instance dynamic pages to reduce the number of result if you wish. I used it for a 600 page site and it works well. I don't know how it handles really large sites. You probably need some patience then... The program automatically loads up the sitemap if you want. It is a really well thought out program!

[MarkGatESS]

Originally Posted by sande145

The program mentioned actually works from your own (Windows) computer and crawls the site.

Really?! On their website, they list in their system requirements different windows O/S's, but they list "server" with them.

So, I can load it onto my Win XP Pro workstation and it'll work??

[sande145]

Yep, it runs on your own computer and crawls the site from there!
http://gsitecrawler.com/