Create New Topic

robots.txt files for primary and add-on domains

Home Forums Web + Tech Answers robots.txt files for primary and add-on domains

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
  • #3052

    Hi Bryan,

    I was just reading through the idea that you had about a robots file within WordPress. (Thanks for that by the way.)

    As I was sorting through what I wanted to do, I realized that I wasn’t sure how to handle the root folder.

    All my clients are in child folders under public_html. I have my own site’s wordpress in it’s own folder, but it’s index.php is at the root.

    I think I should have a robots.txt in each client’s folder (these are add-on domains).

    I think I should have a robots.txt for my site, at the root, and that it should have

    Disallow: /client1/

    Disallow: /client2/

    Disallow: /client3/

    And that I should have no robots.txt in /mysitewordpressfolder/

    Is that correct?

    Bryan Hadaway

    This is actually pretty tricky at first glance.

    I brought this very same question to the Google forums to get some second opinions:!topic/webmasters/7PitgGRXacI
    (sorry, the # is throwing off this link – you’ll have to copy and paste the whole URL)

    The most important thing you need to know is that the root robots.txt can effect addon domains in certain scenarios depending on how the URLs are used.

    Regardless, it’s best to give the root (as in your site) its own robots.txt and then every addon domain (as in a completely separate sites) their own robots.txt.

    Basically, just treat them all as completely separate websites and give them all their own separate robots.txt.

    The reason for this is because the Google bot doesn’t know the difference, it reads them all separately without acknowledging any connection.

    So, if it crawls it’ll be looking for robots.txt at:

    And if it’s looking at it’ll be looking for robots.txt at:

    Even if the robots.txt doesn’t variate at all, you still need to put one copy in the root and one in each addon domain folder.

    Thanks, Bryan


    But should I disallow the addon directories in my primary robots.txt file? Not the domains, but the folders. Such that when the spider crawls I keep it out of the folder,

    Bryan Hadaway

    I think we skipped into the specifics before really establishing what it is you want to do. Forget the domain stuff, what exactly is the goal in really basic terms as far as each website?

    I’m starting to think we should be talking about .htaccess redirects, but I’ll let you explain further.

    Bryan Hadaway

    Clear (ignore this reply)

Viewing 5 posts - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.