November 20th, 2012 at 10:23 PM #3052soulcodeParticipant
I was just reading through the idea that you had about a robots file within WordPress. (Thanks for that by the way.)
As I was sorting through what I wanted to do, I realized that I wasn’t sure how to handle the root folder.
All my clients are in child folders under public_html. I have my own site’s wordpress in it’s own folder, but it’s index.php is at the root.
I think I should have a robots.txt in each client’s folder (these are add-on domains).
I think I should have a robots.txt for my site, at the root, and that it should have
And that I should have no robots.txt in /mysitewordpressfolder/
Is that correct?November 21st, 2012 at 9:32 AM #3109
This is actually pretty tricky at first glance.
I brought this very same question to the Google forums to get some second opinions:
(sorry, the # is throwing off this link – you’ll have to copy and paste the whole URL)
The most important thing you need to know is that the root robots.txt can effect addon domains in certain scenarios depending on how the URLs are used.
Regardless, it’s best to give the root (as in your site) its own robots.txt and then every addon domain (as in a completely separate sites) their own robots.txt.
Basically, just treat them all as completely separate websites and give them all their own separate robots.txt.
The reason for this is because the Google bot doesn’t know the difference, it reads them all separately without acknowledging any connection.
So, if it crawls primary.com it’ll be looking for robots.txt at:
And if it’s looking at addon.com it’ll be looking for robots.txt at:
Even if the robots.txt doesn’t variate at all, you still need to put one copy in the root and one in each addon domain folder.
Thanks, BryanNovember 23rd, 2012 at 7:06 PM #3110soulcodeParticipant
But should I disallow the addon directories in my primary robots.txt file? Not the domains, but the folders. Such that when the spider crawls primarydomain.com I keep it out of the folder, primarydomain.com/otherdomain/?November 24th, 2012 at 1:50 AM #3111
I think we skipped into the specifics before really establishing what it is you want to do. Forget the domain stuff, what exactly is the goal in really basic terms as far as each website?
I’m starting to think we should be talking about .htaccess redirects, but I’ll let you explain further.November 8th, 2013 at 3:33 AM #3417
Clear (ignore this reply)
- You must be logged in to reply to this topic.