Search

XenForo: How to noindex Member & Attachment Pages

Phoenix1

Member
Joined
May 7, 2021
Messages
135
Reaction Score
0
Points
23
I have my member and attachment pages currently blocked in the robots.txt file and Google is showing me that there are thousands of pages that are indexed, yet blocked in that file. I don't want these pages indexed at all because I believe they are behind my website's huge ranking drop because of Google Panda. I am not sure what the best way to deindex these pages is though. Should I unblock them and update the template code to include the meta noindex or is there a better way to go about this? I want these pages out of the index and gone forever.
 

LukeLewis

Member
Joined
May 7, 2021
Messages
134
Reaction Score
0
Points
21
I agree. The worst thing to do in Google's eyes is to block a large number of pages that are linked to on a website in the robots.txt file. I've had a big site tank in the rankings when Panda rolled out this year (2020). I think it was because I had the member and attachment pages blocked. I also had about 500 tag pages, so it may be that as well. They both probably played a role.

There is no reason for Google to crawl these pages at all. Your goal is to get these out of the index, you need to remember that. Your goal isn't to have Google going on a crawling binge of bad pages when it visits your site. So basically, you have three options to deal with your problem. First is that you can leave the pages blocked in the robots.txt file and unlink them for guests. You can accomplish this in the template code. There are also plugins that will do this too, but they'll cost you $35 per year. If you go this route, it'll take a while, but Google will drop all of these indexed but blocked pages from the index. That may take years though.

Your next option is to keep the pages linked, but use the meta noindex tag on them. You actually can't accomplish this for the attachments and it's a bad idea for the member pages, so I wouldn't recommend this.

The last option is to go into the unregistered/guest permissions and make it so these pages can't be seen by visitors, which Google is considered. When Google visits the page, it'll see a 403 Forbidden status code and will deindex the page so it drops from the index. That's exactly what you want. Just be sure to remove /members/ and /attachments/ from your robots.txt file so Google can actually crawl these pages. Once they're all out of the index, you should unlink these pages. That way, Google won't ever crawl any new members or attachments and will crawl the existing ones a lot less. This will help you with crawl budget.

Really, these pages should never have been seen by the search engines in the first place, but since they were, you now have to deal with it. Let me know how it goes.

One more thing - the reason you can't put noindex on the attachment pages is because they're not really pages at all. They're part of the lightbox image view and that's sort of confusing. There's no meta information on these "pages." You really wouldn't want Google or the other search engines crawling these things anyway. But don't worry because even if you block viewing for nonregistered guests, your images will still be indexed by the search engines. The attachment pages are in the /attachments/ directory and the images are held in the /data/attachments/ directory.
 

JGaulard

Administrator
Staff member
Site Supporter
Sr. Site Supporter
Power User
Joined
May 5, 2021
Messages
319
Reaction Score
2
Points
18
Top