search bots and Javascript

[pdstein]

I've read many times that the general concensous is that search bots can't read Javascript. So, I modified some of the code on my site in the hopes that bots would see a direct link while humans who click would go through a click tracking script:

 
<a href="http:/www.otherSite.com" onClick="window.open('http://www.mysite.com/clickTrackingScript.php?bannerID=1234')">
Text
</a>

I'm finding, however, that some bots still open the tracking script. So, is it a myth that bots can't read Javascript, or does that really mean bots can't read text/HTML generated by a Javascript?

Is there a better way to do what I'm trying do?

[wige]

In my experience, at least the Google Bot is able to follow links in Javascript, despite their claims to the contrary. I ran into issues when Google started crawling pages from a AJAX system I was testing, where the only possible way to find the link was through the javascript. I am not sure how thorough this method is, however. You might be able to scrample the address in a javascript function to prevent the bots from crawling it.

[seiretto]

Is there a better way to do what I'm trying do?

Use AJAX. And here is how, add this:

onClick="track_the_click('1234')"

to the relevant links and add the JavaScript below to each page:

<script language="JavaScript"><!--
// Free with compliments from Seiretto ;-)
var url="http://www.mysite.com/clickTrackingScript.php?bannerID=";
function getHTTPObject()
{
 if (typeof XMLHttpRequest != 'undefined')
  {
  return new XMLHttpRequest();
  }
  try
  {
  return new ActiveXObject("Msxml2.XMLHTTP");
  } catch (e) {
  try
  {
   return new ActiveXObject("Microsoft.XMLHTTP");
   } catch (e) {}
   } return false;
}
function track_the_click(id)
{
 var http = getHTTPObject();
 http.onreadystatechange = function()
 {
 if (http.readyState == 4) // 4 is complete
 {
  //alert(http.responseText); // uncomment to see the response!  
 }
 }
 http.open("GET", url+id, true);
 http.send(null);
}
// -->
</script>

If you do not want any bots following your links in your scripts change the above line
FROM:

var url="http://www.mysite.com/clickTrackingScript.php?bannerID=";

TO:

var url1="http://www.mysite";
 url2=".com/clickTrackingScript.php?bannerID=";
url=url1+url2;
// most bots are too stupid to put the two together so will not index it (at least currently).

Hope it helps.

Dave.

[imvain2]

seiretto's javascript method is a good one to use. I actually came across the exact same problem, my simple solution was to place the tracking script into a seperate directory and disallow that directory in my robots.txt.

Of course my solution assumes the bots follow the robots.txt rules, which the legit ones do.

[pdstein]

Dave, thanks for the suggestion. I'll give it a try.

- Paul

[Conficio]

In my understanding, spiders (bots) do not interpret JavaScript per see. Because to be meaningful they'd need to build a DOM and that is way to expensive in terms of computation.

However, because many webmasters to use JavaScript to actually map links in the navigation, the spiders do a cursory scan for patterns of URL's in scripting. So they find your complete tracking URL, while not finding it when the URL is composed out of two string constants.

I think imvain2's solution is the best to your problem. And if you want to can also block folks that do not follow the robots.txt rules, using a bit of Redirect magic, involving the referrer URL (which needs to be a page and not blank like most spiders do).

K<o>