Tag: content crawling

Software to help you create a robots.txt file

Robogen is a software for Windows 95, Windows 98 and Windows NT based computers which can help you create robots.txt…