- This topic has 4 replies, 2 voices, and was last updated 10 years, 11 months ago by
.
-
Posts
-
Hi Bryan,
I was just reading through the idea that you had about a robots file within WordPress. (Thanks for that by the way.)
As I was sorting through what I wanted to do, I realized that I wasn’t sure how to handle the root folder.
All my clients are in child folders under public_html. I have my own site’s wordpress in it’s own folder, but it’s index.php is at the root.
I think I should have a robots.txt in each client’s folder (these are add-on domains).
I think I should have a robots.txt for my site, at the root, and that it should have
Disallow: /client1/
Disallow: /client2/
Disallow: /client3/
And that I should have no robots.txt in /mysitewordpressfolder/
Is that correct?
This is actually pretty tricky at first glance.
I brought this very same question to the Google forums to get some second opinions:
https://productforums.google.com/forum/?fromgroups=#!topic/webmasters/7PitgGRXacI
(sorry, the # is throwing off this link – you’ll have to copy and paste the whole URL)The most important thing you need to know is that the root robots.txt can effect addon domains in certain scenarios depending on how the URLs are used.
Regardless, it’s best to give the root (as in your site) its own robots.txt and then every addon domain (as in a completely separate sites) their own robots.txt.
Basically, just treat them all as completely separate websites and give them all their own separate robots.txt.
The reason for this is because the Google bot doesn’t know the difference, it reads them all separately without acknowledging any connection.
So, if it crawls primary.com it’ll be looking for robots.txt at:
primary.com/robots.txt
And if it’s looking at addon.com it’ll be looking for robots.txt at:
addon.com/robots.txt
Even if the robots.txt doesn’t variate at all, you still need to put one copy in the root and one in each addon domain folder.
Thanks, Bryan
But should I disallow the addon directories in my primary robots.txt file? Not the domains, but the folders. Such that when the spider crawls primarydomain.com I keep it out of the folder, primarydomain.com/otherdomain/?
I think we skipped into the specifics before really establishing what it is you want to do. Forget the domain stuff, what exactly is the goal in really basic terms as far as each website?
I’m starting to think we should be talking about .htaccess redirects, but I’ll let you explain further.
- You must be logged in to reply to this topic.