How do I prepare a site 'Under Construction' to Spiders?

minkley · #1 (**permalink**) 07-17-2007, 07:17 PM

I am new to the whole website and search engine world. After browsing some of the posts o n this forum, it seems to me, that one needs to be careful not to do things which upset search engine spiders.

Is there anything I should be doing to mark a site as 'Under Construction' so as not to be penalized when the site goes live. What can I do to politely inform search engines that the sit is under construction and not to waste any time there.

Any advice or pointers would be appreciated.
thanks

ArthurKay · #2 (**permalink**) 07-18-2007, 10:05 AM

You could use the proper redirects (404, 301, etc) for pages which have been moved. You can also make use of your robots.txt - just tell the spiders which folders are off-limits.

Conficio · #3 (**permalink**) 07-19-2007, 05:47 AM

Use the robots.txt file to inform the spiders that you don't want them to crawl the whole site

Code:

User-Agent: *
Disallow: /

However, once you are done with the site, aggressively do tell the SE that you are ready for spidering. Youcan do this by adding a sitemap entry to the robots.txt (and remove the disallow).

Code:

Sitemap: http://www.example.com/sitemap.xml

Hope this helps!

K<o>

**incrediblehelp** · #4 (**permalink**) 07-19-2007, 12:31 PM

To be honest I would never block the spider from spidering a website. Why not throw up a splash page or not go live with the domain until ready?

minkley · #5 (**permalink**) 07-20-2007, 07:14 PM

Thanks for all the great advice. I think I may just stick with the splash page, which is already up. I have heard of robots.txt and sitemap.xml but have never created one, so I will need to read up on them.

braver55b · #6 (**permalink**) 08-01-2007, 09:53 AM

When you are ready, you can create a free sitemap good for yahoo and google at

Google Sitemap & Yahoo Sitemap Generator