Quote:
|
Originally Posted by nagarjuna55334
I think it can be possible by increasing the crawl time by writing the code in the robots.txt
User-agent: googlebot
Crawl-delay: 120
I dont know how much time i,e 120 or other to crawl the 30,000 pages but the robots.txt works if proper navigation is there in your site increase the time 120 if you are not crawled well.
|
You seem to misunderstand the concept of "Crawl-delay". It is used to increase the time between spidering one page and the next by introducing a delay - this is done to reduce load on the server:
http://www.ilovejackdaniels.com/deve...ts-txt-file/3/
It has nothing to do with the time a bot takes to spider your pages.
faglork