Quote:
|
Originally Posted by nagarjuna55334
I think it can be possible by increasing the crawl time by writing the code in the robots.txt
User-agent: googlebot
Crawl-delay: 120
I dont know how much time i,e 120 or other to crawl the 30,000 pages but the robots.txt works if proper navigation is there in your site increase the time 120 if you are not crawled well.
|
Surely you're not advising to make it say 120. Wait 2 minutes in between each page requast? That'd take forever to do 30,000 pages. Leaving off the crawl delay would be much better advice.
Brian.