View Single Post
  #2 (permalink)  
Old 04-05-2004, 07:55 PM
Mark Carey's Avatar
Mark Carey Mark Carey is offline
WebProWorld New Member
 
Join Date: Apr 2004
Posts: 15
Mark Carey RepRank 0
Default

While the 101K limit has been known for some time, there is a debate about whether Google crawls beyond the 101K mark on page. For example suppose a page is 150K in size consisting of mostly links. Will Google simply stop crawling the page after 101K, thus not following the links at the bottom of the page? Or, does Google index only the first 101K, but continue to follow the remainder of the links on the page? I have read claims on both sides of the debate, but never tried to test it myself. The answer can have a significant impact on large sitemaps. Nobody cares if the entire 200K sitemap in indexed, but we certainly care that all of the links are crawled.
Reply With Quote