Quote:
|
Originally Posted by travisbickle100
I went to uptimebot and entered my site and found that Google had only indexed only a couple hundred of my pages. The web site is a database of private schools and I have almost 30,000 pages on the site. The site has been up for almost 5 years. Can anyone tell me why the entire contents have not been indexed after so long?
|
I think it can be possible by increasing the crawl time by writing the code in the robots.txt
User-agent: googlebot
Crawl-delay: 120
I dont know how much time i,e 120 or other to crawl the 30,000 pages but the robots.txt works if proper navigation is there in your site increase the time 120 if you are not crawled well.