What I’m going to explain today is hopefully going to spare you a lot of heartache in the future. I’d like to discuss a very important SEO aspect of XenForo forums that a lot of site owners completely overlook. It has to do with the member and attachment links that are all over many forum sites. Actually, these links come with XenForo’s standard template. You can find them on the forum list page (homepage), forum pages, and thread pages, among others. The attachment links are generally located on the thread pages, but the member links are everywhere. They’re on virtually all pages.
The problem with these links is that they should all be blocked (or not even crawled). Because the attachment links, if crawled, return soft 404 responses from Google, it’s best to hide these altogether. And because so many member accounts are empty or hold very little content, their links should be blocked or hidden too. The question is, what’s the best way to handle them?
You have a few choices here. You can keep the links live on your website and allow Google to crawl all of them and eventually inflict the Panda penalty on you, you can keep the links live, but block them in your robots.txt file, you can keep the links live, but have them return 403 status codes because they’re set to “no” in the permission system, or you can set their permissions to “no” and remove the links from the templates altogether. I’m here to suggest that the last option is the best and I’ll tell you why.
Over the past few years, I’ve been experimenting with approximately seven forums, from quite popular to websites that get hardly any traffic. To make a long story short, I’ll say that whenever I kept these links live, no matter if they were blocked in the robots.txt file or by the permission system, my Valid Submitted and Indexed pages in the Google Search Console would drop and the Crawled – Currently Not Indexed pages would increase. It’s the strangest thing. It’s almost as if, when Google crawls these bad pages, even though they’re blocked one way or another, it includes them someplace in their index and pushed out the good pages. At the beginning of January 2021, I removed the links in question from the templates of my websites. Unregistered users (including Googlebot) couldn’t access those pages. I had the permissions for these pages set to “no” so they’d return a 403 header response code and fall out of the index after a while. A few weeks after I did this, I surprisingly noticed that the valid pages began climbing and the crawled but not index pages began falling. I was very pleased with this, but I didn’t know why it was occurring. Because, for some strange reason, I decided to include the links back on the site, this happened:
Do you see a relationship there? Yeah, me too. And the best part is, about a week and a half ago, I decided to remove those links from the templates again. Check out the most recent crawl dates on the above charts. See how the trends are being reversed? That’s more than a coincidence. So basically, what I’m saying is that, for some reason (perhaps too many links per page, response code, blocked URLS – whatever), when these links are live on forums, Google doesn’t allow many valid pages to be indexed, surely causing ranking issues overall.
I’ve done some searching around the XenForo community and have found a few others who have experienced ranking issues. They ask, “Why isn’t Google indexing my pages?” and “Why are my crawled but not indexed pages so high?” There are actually quite a few people who suffer from this type of thing and most likely many more who haven’t been vocal about the issue. I’m assuming it’s a thing. Personally, I’ve decided to keep the links off the site so unregistered guests (and Googlebot) can’t see them. The moment someone registers and logs in, the site will reveal all the links the proper way.
The most important thing is to remember to disallow unregistered guests from being able to access the links, so I keep the settings for members and attachments set to “no” in the user permissions for those people. The next most important thing is to remove the links from the templates. Below, I’ll show you how to do that.
I’m going to show you some template code below. In the code snippet, I’ll explain which template gets edited and on which line. If you decided to edit your own templates with the following code, be sure to only add the comments and what’s between the comments. The other text is just there for your benefit.
Code:
-------------------------------------------------
TEMPLATE: "ANDY_SIMILARTHREADS" (IF YOU HAVE ANDY'S SIMILAR THREADS ADD-ON INSTALLED)
(remove thread starter member link)
(comments not allowed in this template)
line 22
<!-- REMOVE THREAD STARTER MEMBER LINK -->
CHANGE
<xf:avatar user="{$thread.User}" size="xs" defaultname="{$thread.username}" />
TO
<xf:avatar user="{$thread.User}" size="xs" defaultname="{$thread.username}" href="" />
<!-- REMOVE THREAD STARTER MEMBER LINK -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "MESSAGE_MACROS" (THREAD PAGE TEXT LINK UNDER AVATAR)
line 15
<!-- REMOVE USERNAME LINK FOR GUESTS-->
<xf:if is="$xf.visitor.user_id">
<h4 class="message-name"><xf:username user="$user" rich="true" defaultname="{$fallbackName}" itemprop="name" /></h4>
<xf:else />
<h4 class="message-name"><xf:username user="$user" rich="true" defaultname="{$fallbackName}" itemprop="name" href=""/></h4>
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS-->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "MESSAGE_MACROS" (THREAD PAGE AVATAR)
line 8
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<xf:avatar user="$user" size="m" defaultname="{$fallbackName}" itemprop="image" />
<xf:else />
<xf:avatar user="$user" size="m" defaultname="{$fallbackName}" itemprop="image" href=""/>
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "ATTACHMENT_MACROS" (THREAD PAGE THUMBNAIL SIZE ATTACHMENTS)
line 7
<!-- REMOVE THUMBNAIL ATTACHMENT LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<a class="file-preview {{ $canView ? 'js-lbImage' : '' }}" href="{$attachment.direct_url}" target="_blank">
<img src="{$attachment.thumbnail_url}" alt="{$attachment.filename}"
width="{$attachment.thumbnail_width}" height="{$attachment.thumbnail_height}" loading="lazy" />
</a>
<xf:else />
<span class="file-preview">
<img src="{$attachment.thumbnail_url}" alt="{$attachment.filename}"
width="{$attachment.thumbnail_width}" height="{$attachment.thumbnail_height}" loading="lazy" />
</span>
</xf:if>
<!-- REMOVE THUMBNAIL ATTACHMENT LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "BB_CODE_TAG_ATTACH" (THREAD PAGE FULL SIZE ATTACHMENTS)
line 55
<!-- REMOVE FULL SIZE ATTACHMENT LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<a href="{{ link('full:attachments', $attachment, {'hash': $attachment.temp_hash}) }}"
target="_blank"><img src="{$attachment.thumbnail_url_full}"
class="bbImage {$alignClass}"
style="{$styleAttr}"
alt="{$alt}"
title="{$alt}"
width="{$attachment.thumbnail_width}" height="{$attachment.thumbnail_height}" loading="lazy" /></a>
<xf:else />
<img src="{$attachment.thumbnail_url_full}"
class="bbImage {$alignClass}"
style="{$styleAttr}"
alt="{$alt}"
title="{$alt}"
width="{$attachment.thumbnail_width}" height="{$attachment.thumbnail_height}" loading="lazy" />
</xf:if>
<!-- REMOVE FULL SIZE ATTACHMENT LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE: "THREAD_VIEW" (THREAD PAGE - TOP OF PAGE TEXT LINK)
line 14
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<xf:username user="{$thread.User}" defaultname="{$thread.username}" class="u-concealed" />
<xf:else />
{$thread.username}
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "THREAD_LIST_MACROS"
line 18
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<xf:avatar user="$thread.User" size="s" defaultname="{$thread.username}" />
<xf:else />
<xf:avatar user="$thread.User" size="s" defaultname="{$thread.username}" href=""/>
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "THREAD_LIST_MACROS"
line 181
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<li><xf:username user="$thread.User" defaultname="{$thread.username}" /></li>
<xf:else />
<li>{$thread.username}</li>
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "THREAD_LIST_MACROS"
line 223
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<xf:username user="{$thread.LastPoster}" defaultname="{$thread.last_post_username}" />
<xf:else />
{$thread.last_post_username}
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "THREAD_LIST_MACROS"
line 236
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<xf:avatar user="{$thread.LastPoster}" defaultname="{$thread.last_post_username}" size="xxs" />
<xf:else />
<xf:avatar user="{$thread.LastPoster}" defaultname="{$thread.last_post_username}" size="xxs" href=""/>
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE "THREAD_LIST_MACROS"
line 273
<!-- REMOVE NEW THREADS WIDGET USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<xf:avatar user="$thread.User" size="xxs" defaultname="{$thread.username}" />
<xf:else />
<xf:avatar user="$thread.User" size="xxs" defaultname="{$thread.username}" href=""/>
</xf:if>
<!-- REMOVE NEW THREADS WIDGET USERNAME LINK FOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE: "NODE_LIST_FORUM"
line 121
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<xf:avatar user="{$extras.LastPostUser}" defaultname="{$extras.last_post_username}" size="xs" />
<xf:else />
<xf:avatar user="{$extras.LastPostUser}" defaultname="{$extras.last_post_username}" size="xs" href=""/>
</xf:if>
<!-- REMOVE USERNAME LINKFOR GUESTS -->
-------------------------------------------------Code:
-------------------------------------------------
TEMPLATE: "NODE_LIST_FORUM"
line 137
<!-- REMOVE USERNAME LINK FOR GUESTS -->
<xf:if is="$xf.visitor.user_id">
<li class="node-extra-user"><xf:username user="{$extras.LastPostUser}" defaultname="{$extras.last_post_username}" /></li>
<xf:else />
<li class="node-extra-user">{$extras.last_post_username}</li>
</xf:if>
<!-- REMOVE USERNAME LINK FOR GUESTS -->
-------------------------------------------------I’m pretty sure that’s all of them. If you have any questions, please let me know down below. You’ll also want to remove any redirect links as well, but I’ll save that for another post. And on the thread pages, you’ll want to remove the reaction row links and the goto links. That’s very important. Basically, you want to hide as many non-crawlable links from Google as possible so they’re removed from the index.
Leave a Reply