Submit Your Article Forum Rules

Results 1 to 10 of 14

Thread: Best Robots.txt wordpress

Threaded View

  1. 07-19-2011, 07:26 PM #11
    deepsand
    deepsand is offline
    Senior Member deepsand's Avatar
    Join Date
    May 2004
    Location
    State College, PA
    Posts
    16,482
    You need to first identify which directories/files you do and do not want indexed.

    Bear in mind that directories such as those that are used for segregating non-contextual files, such as CSS and JS, need not be blocked, as they've no effect on the ability of an SE's indexing engine to discover and have a crawler fetch copies of those files which you do want to be indexed.
    Hound Dog Tours: Home of the Penn State Ticket Man & the world's largest privately controlled inventory of tickets for Penn State Nittany Lions Football
    Reply With Quote Reply With Quote
  2. The following user agrees with deepsand:
    vikashony
« Previous Thread | Next Thread »

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Forum Rules