Indexed Pages Decreasing in XenForo Forum

[JodyBuchanan]

I want you to take a look at something. I'll tell you the backstory down below these graphics.

Crawled - Currently Not Indexed​

Valid Pages​

The graphs are for a XenForo website that I operate. Back in January of 2021, I decided to make a few changes to some templates and remove the many 301 redirects that are included by default. Since I installed the XenForo software in 2018, I haven't had many of the fairly decent pages that exist on the website indexed by Google. I always wondered why and I've long thought that the 301 redirects were causing a problem. Time and time again, I've checked the Google Search Console to see that a thread page wasn't indexed, but it's 301 redirected counterpart was. I thought this was odd, especially since everyone out there in the SEO world swears that when Google crawls a redirect, it treats it as a hard directive. Redirects shouldn't be an issue, right?

Ever since I removed those redirected URLs from the website in question, the crawled but not indexed number trended downward and the valid indexed pages trended upward. That was a good thing. You can see this from the left side of both graphs. Well, some time in mid-March of 2021, I got antsy and reset the templates so the default redirects were in place once again. Take a look at what happened. Almost immediately the valid page number began dropping and the crawled but not indexed number began rising. Check out how many valid pages were once indexed. Almost 600 of them and the number was rising. It would have risen to over 700 (maybe even up to 1,000) if I hadn't touched anything. And the crawled but not indexed number was falling. It would have been nice if that hit zero.

Is Google treating these 301 redirects as duplicate content? Why are the redirected pages being included in the index while the proper thread pages aren't? Why are valid pages falling out of Google's index?

I run a few more of these websites and the same thing is happening to all of them. So what I'm doing is running a few experiments. I've removed all of the redirects once again, but for a few of the websites, I'm blocking those redirects in the robots.txt file and for some, I'm not. I'm allowing them to redirect properly behind the scenes. I've always wanted to simply block these redirects, but I've never had the nerve. My rankings always drop when I block things. I know it's a temporary drop, but still...

Take a look at how I block the offending URLs:

Disallow: /threads/*/latest Disallow: /threads/*/post

We'll see what happens.

There is another aspect of my websites that's under suspicion. Around the same time I messed with the templates, I also blocked the members and attachments directories like this:

Disallow: /members/ Disallow: /attachments/

I'm not sure if doing this had any effect, but perhaps my experiment will shed some light on that too.

I read a post on the XenForo community forum here:

https://xenforo.com/community/threads/seo-problems.161362/

A fellow complained of the same thing I'm referring to here. Everyone in the forum, like usual, blamed his issues on how Google just does things. They said he just needed to wait it out. I'm sorry, but I know for a fact that my own hand caused these problems and after running a few experiments, I can uncause them. And that's why I'm writing this post - to offer updates, with graphs, every so often, to document my progress.

This sort of thing is frustrating, but can be rewarding if things turn out well. I'll keep you updated.

 

[JodyBuchanan]

I had an epiphany last night. For some time now, I've been thinking that the 301 redirects (some with nofollow) and member and attachment pages that return 403 header status codes were somehow affecting the indexing of my website's pages. I guess in a sense they are. The strange thing is, some of these deindexing problems only began when I returned these URLs to the forum pages. On one of my sites, I resisted returning those links to the forum pages and the indexation remained fine. It was actually growing in the right direction. I had these types of links live on the thread pages, but not the forum ones.

Just as luck would have it, right after I returned these links to the forum pages, the indexing of the website I just mentioned began to fall. To me, it seemed like Google didn't like 301s, 403s, or nofollows. I mean, of course search engines don't like those types of links, but I've never heard of valid pages dropping out of the index because of them.

Last night, as I was thinking about this issue, something struck me. Check this out:

Right now, I've got 25 threads linked to from each forum page. So basically, that should be 25 links, plus a few extra navigation links, linking out from each of these types of pages. Each of those threads gets some pagerank assigned to it from Google. Now, if I were to add 25 member links to the same forum page, that pagerank for the member pages would be cut in half. Then if I were to add another member link, a nofollowed date posted link, another nofollowed date latest post link, another member link, and then another member link after that, I'd have a whole bunch of links. Only 25 of which matter, mind you. So instead of the pagerank flow being divided between 25 links on any given forum page, it's going to be divided by 175 links. Throw in some sub-forums that are linked to along with their associated two member links and a 301 link that gets redirected to the most recent post in a thread, there can be hundreds of links per page.

What struck me is the possibility that the pagerank on my forum pages was being so dissipated, or thinned out, that the thread pages on the website had essentially no pagerank flowing to them at all. What happens when a page earns very little to no pagerank? It either never makes it into the index or it falls out. I suspect that's what's happening here.

To test this theory, I once again removed all of the unnecessary links from the forum pages, but left the member and attachment links on the thread pages. If, in a week or so, the number of valid pages begins to rise, I'll know I'm onto something. I'll keep this thread updated with my findings.

To make this simple - I think I may have too many links per page on my websites and that's why so many pages aren't being indexed and are being removed from the Google index. Perhaps this is why so many people out there, who are in similar situations, are saying that Google hates forums these days. Perhaps it's not the thin or old content they're always blaming. Maybe it's something purely technical, such as way too many links per page.