Create New Topic

robots.txt files for primary and add-on domains

Home Forums Web + Tech Answers robots.txt files for primary and add-on domains

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #3052

    soulcode
    Participant

    Hi Bryan,

    I was just reading through the idea that you had about a robots file within WordPress. (Thanks for that by the way.)

    As I was sorting through what I wanted to do, I realized that I wasn’t sure how to handle the root folder.

    All my clients are in child folders under public_html. I have my own site’s wordpress in it’s own folder, but it’s index.php is at the root.

    I think I should have a robots.txt in each client’s folder (these are add-on domains).

    I think I should have a robots.txt for my site, at the root, and that it should have

    Disallow: /client1/

    Disallow: /client2/

    Disallow: /client3/

    And that I should have no robots.txt in /mysitewordpressfolder/

    Is that correct?

    #3109

    Bryan Hadaway
    Keymaster

    This is actually pretty tricky at first glance.

    I brought this very same question to the Google forums to get some second opinions:

    https://productforums.google.com/forum/?fromgroups=#!topic/webmasters/7PitgGRXacI
    (sorry, the # is throwing off this link – you’ll have to copy and paste the whole URL)

    The most important thing you need to know is that the root robots.txt can effect addon domains in certain scenarios depending on how the URLs are used.

    Regardless, it’s best to give the root (as in your site) its own robots.txt and then every addon domain (as in a completely separate sites) their own robots.txt.

    Basically, just treat them all as completely separate websites and give them all their own separate robots.txt.

    The reason for this is because the Google bot doesn’t know the difference, it reads them all separately without acknowledging any connection.

    So, if it crawls primary.com it’ll be looking for robots.txt at:

    primary.com/robots.txt

    And if it’s looking at addon.com it’ll be looking for robots.txt at:

    addon.com/robots.txt

    Even if the robots.txt doesn’t variate at all, you still need to put one copy in the root and one in each addon domain folder.

    Thanks, Bryan

    #3110

    soulcode
    Participant

    But should I disallow the addon directories in my primary robots.txt file? Not the domains, but the folders. Such that when the spider crawls primarydomain.com I keep it out of the folder, primarydomain.com/otherdomain/?

    #3111

    Bryan Hadaway
    Keymaster

    I think we skipped into the specifics before really establishing what it is you want to do. Forget the domain stuff, what exactly is the goal in really basic terms as far as each website?

    I’m starting to think we should be talking about .htaccess redirects, but I’ll let you explain further.

    #3417

    Bryan Hadaway
    Keymaster

    Clear (ignore this reply)

Viewing 5 posts - 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.