Drop in Organic Google Traffic

  • 6
  • 16
  • Problem
  • Updated 4 months ago
  • Solved
Our community has existed for 5+ years. In the past several weeks, we've noted a marked decrease in our organic traffic (Google) which had steadily grown over time.

Has anyone else noticed a similar issue in their communities? If so, how are you responding?
Photo of Daniel Luebke

Daniel Luebke

  • 236 Points 100 badge 2x thumb
  • left out

Posted 1 year ago

  • 6
  • 16
Photo of Tashina

Tashina, Community & Education Manager

  • 24,212 Points 20k badge 2x thumb
Hi everyone,

We are currently testing out some things to resolve this issue and this is a high priority for us right now. As you likely know, anything with Google can take a little time to see any changes or results. We will share more details when we have an idea of the impact.
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hey guys, I just wanted to say that we’ve been hearing your concerns and this issue is a top priority for us, so thank you for continuing to chime in. In the last few weeks we have been doing research and finding only small things to tweak or fix - but unfortunately no silver bullet, and no a-ha moments. This is mainly why we haven’t been posting many updates - we haven’t had anything interesting to report unfortunately. But we are continuing to investigate and try things. 
Photo of Justin Veenema

Justin Veenema

  • 740 Points 500 badge 2x thumb
Thanks for the updates Stephen & Tashina. 

Just a note, I think it would be worthwhile to provide timely updates regardless of whether or not there's been any progress made. Silence often looks a lot like indifference, and I'm sure that's not what y'all are trying to communicate. 

From our perspective, we're having an incredibly hard time explaining the stats-drop to higher-ups in our company, so it's frustrating to be left in the dark. 

Hopefully there's some resolution soon. 
Photo of Tashina

Tashina, Community & Education Manager

  • 23,830 Points 20k badge 2x thumb
We completely understand how it appears that way. I think that, because this is a project so many team members have been heads down on, it wasn't realized that it had been a while since we communicated about it. We are definitely going to improve that going forward.
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, just wanted to give an update that this continues to be a top priorit, we're continuing to try tweaking things as we find them but nothing is making a change so far. We have a few more things on deck to try next week. 

Have a great weekend. 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone - time for a weekly update. Our tests from this week (and we've included Extreme Drew in one of them) aren't working out, which is stumping us both. We've been doing more research and have come up with a few more ideas, but they aren't going to be that quick to implement. I just wanted to reaffirm that this continues to be a top priority in the meantime. 
Photo of Ryan Mathews

Ryan Mathews

  • 576 Points 500 badge 2x thumb
Thank you Stephen.  Really appreciate the on-going updates and efforts to sort this out.  Hope you get a break in the case soon!
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi all, time for another weekly update. We investigated some avenues that we thought had promise but didn't end up panning out. At this point based on some more research we're thinking general performance and response times may be playing a factor, and are taking a close look at every way we can improve that, along with continuing to tweak the markup.
Photo of Vera

Vera

  • 814 Points 500 badge 2x thumb
Hi Stephen,

Can we do anything to help figure out what's going on? We really appreciate the effort your team is putting into this but we're also pretty alarmed by our numbers:


Our monthly sessions have dropped from 17175 in April to 1875 in July.


Please let us know if we can test or share anything that might help your research.

Thanks!
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, in addition to the performance work we've been working on, we've found another avenue to pursue with regards to Google indexing that we'll be trying soon. 

Separately, I was wondering if anyone has checked how your traffic has been doing on other search engines such as Bing or Yahoo? 
Photo of Drew C.

Drew C., Champion

  • 11,652 Points 10k badge 2x thumb
Neither really show up very high in our analytics, but are present.

Searching for "5ghz channel planning guide" in Bing and Yahoo gives our community thread as the first result, however.  Google... not so much.
Photo of Vera

Vera

  • 814 Points 500 badge 2x thumb
Hi Stephen,

I can also confirm what Drew found. Our GetSat topics seem to rank much higher on Bing and Yahoo search than Google.
Photo of Roy

Roy

  • 17,446 Points 10k badge 2x thumb
I wonder if the plethora of stuff under the GetSatisfaction banner, but with all the Communities effectively silo'ed from one another, is being seen by Google as an anomalous pattern of links, so GS is regarded by them rather like one of those aggregating sites - even though that isn't what it is - and thus gets penalised by some change that Google made to its anti-SEO-gaming algorithm earlier this year?

But I bet Stephen et al have thought of that.....
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Thanks Drew and Vera. By now we've run our app through a few different SEO scanning tools and there are no obvious flags. The fact that Bing and Yahoo are fine seem to indicate that the markup is probably okay too. So the main path we're pursuing now is a number of ways to improve performance, especially when paginating through on the homepage (an inherently slow process) to find the links to all the topics. We're lining up a few changes in this regard for a deploy next week - stay tuned! 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, just wanted to let you know we haven't forgotten about this. We pushed out some frontend optimizations last week to help speed things up but that didn't seem to have much effect. Today we made some more markup changes, and later tonight our bigger change that we've been working on for a few weeks will go out. Fingers crossed that this will start to have an effect. 
Photo of Rav

Rav, Champion

  • 960 Points 500 badge 2x thumb
Thanks Stephen, looking forward to it! :) 
Photo of Ryan Mathews

Ryan Mathews

  • 576 Points 500 badge 2x thumb
Thanks for the update Stephen.  

We're definitely still feeling the sting of losing our Community's "Google" effect.  Hope this next change will turn that tide.
Photo of Anna Svensson

Anna Svensson

  • 3,226 Points 3k badge 2x thumb
Hi,
Any update? Just tried a google search and can ́t find anything from us anymore, we used to be in top 3.

/Anna
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi Anna, we released the change yesterday but like you haven't yet seen any effects. I think it may take some time for Google to come back and re-crawl... We're also going to push out another small markup change today to see if that helps. 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, last week we released some major performance improvements that we're serving to Google in hopes that faster load times will help things. We've also added a number of other markup changes throughout the week as we continue to dig deep and find any other recommendations and examples. Unfortunately this hasn't yet resulted in the dramatic uptick in indexing we've been hoping for... This continues to be a top priority. 
Photo of Mike

Mike

  • 3,786 Points 3k badge 2x thumb
I also did several Google searches for various topics on our Community and none of them are on the 1st page of google any longer.  I'm seeing a similar drastic reduction in traffic also. 




We've almost double the registered users in the last year, but the traffic has fallen almost in half.  

This is a pretty huge concern as the whole purpose of our Community is to deflect calls coming onto our support center. 

Our internal SEO team has been looking at this and there is some concern about what Google is likely seeing as duplicate content. We are wondering if this can be fixed by the use of canonical tags?   Here are some examples. 





This is what the head looks like for one of the pages:



When looking at a topic thread, this is what the head looks like.  Note the “link rel=canonical href....” Information.


Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Thanks for that info Mike. I was looking into this and what I think you're saying is that on a category listing page, there are many variations of parameters to say paginate and filter the list down, and all of those are being counted as duplicate content at the moment. And then you're saying that on a topic thread for pages beyond the first one we're setting a rel=canonical tag to point to the base version of that page.

So we actually did have the rel=canonical on the category lists (as well as the topic list on the home page) until pretty recently, and we just removed it for a few reasons:

One is that as you mentioned in your later reply, we're noticing that Google is indexing fewer pages and so we are trying to "open things up" to have them at least crawl more pages, especially the topic lists, to find more links to index.

The second is that we also did a bunch of research and found a decent amount of evidence saying that Google only loosely respects the canonical tag, but mostly rely on themselves to determine what's the best content.

So we decided to try removing that from the lists. We may add it back if we determine that things are worse without it, but that's not clear at this point. 
Photo of Mike

Mike

  • 3,786 Points 3k badge 2x thumb
I'm beginning to wonder if Google is dropping pages from being indexed?  If used several online tools to check for the number of indexed pages and the results are very bad.  I realize these are not 100% accurate but out of all the sites I used, the highest number of indexed pages I could get was only 79. 

Anybody else check this? 

Mike 
Photo of Drew C.

Drew C., Champion

  • 11,672 Points 10k badge 2x thumb
I believe that's the main problem - pages are not being kept in the index.
Google Search Console tells me there are only 117 pages in the index of ~5,000.  The sharp rise is when we first set up search console for our community.
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Ditto from us, and we've been continuing to try making markup changes and have been re-submitting reindexing requests to see if we can get that to improve. So far no luck. We're still researching and trying things, and one notable change aside from markup that will be coming soon will be to improve the robots.txt file and begin to auto-generate and serve a sitemap for each community. 
Photo of Mike

Mike

  • 3,786 Points 3k badge 2x thumb
Thanks for the updates Stephen!  Any ETA on the sitemaps? 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
I think around this time next week. Due to volume on some bigger communities it's been a bit tricky to implement in a performant way. 
Photo of Mike

Mike

  • 3,786 Points 3k badge 2x thumb
OK.. thanks!
Photo of Mike

Mike

  • 3,786 Points 3k badge 2x thumb
Hi Stephen Lee,,   Just curious if you have any update on the sitemaps?  

Mike 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hey Mike! Yes, as a matter of fact I do. Will post as a reply. 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, we've pushed out a change now such that if you are using your own CNAME with us, we're now serving an XML sitemap that you can ask Google to index. To find the URL, visit /robots.txt under your CNAME and you'll see it linked. 
Photo of Mike

Mike

  • 3,776 Points 3k badge 2x thumb
Great news. Thanks!!!  
Photo of Mike

Mike

  • 3,776 Points 3k badge 2x thumb
Stephen Lee, 

Any more advise.  Looks like the like only 17 pages have been indexed according to our Google Webmaster account. 

Also, how frequently will GetSat update the sitemap? 

Mike 

Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi Mike, I'm following numbers for a few communities and after that robots/sitemap change it looks like a few other communities are seeing a slight uptick (~10% in the last few days) in indexed pages, but not yours or ours. 

The sitemap itself should be fairly up to date; usually no more than a few minutes to hours old, or a day at worst. 

This week we're planning to actually move where it's served to be a little more friendly for use with the Google Search Console, as well as some other SEO-related changes. 
Photo of Mike

Mike

  • 3,776 Points 3k badge 2x thumb
Stephen,  Do you have any new information on this indexing issue.  Looks like we are still struggling to get our traffic up to what use to be a normal level.   

When I look in our Google Console it looks like there' are very few pages being indexed.  

If I go to chrome and type in:   site:community.ringcentral.com it shows over 300 results which is up from <100 a few weeks ago, but still very poor compared to where it once was.  

Also another thing I find very strange are the sub headings that Google chooses to display on the search results.  I think  if Google were indexing the URLs in the sitemap it would prioritize other links above these, especially the photo of one of the community members.  
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, so about a week ago we released a few changes that seem to finally be helping things out, albeit slowly and gradually. The notable changes in that round were serving a sitemap for our cname customers, handling errors like 404s better, and removing a crawl delay from our robots.txt file.

We have more small markup changes on the way and are also investigating things like 3rd party linking issues affecting domain reputations. 

Based on Mike's feedback above, we can also look at re-implementing some schema.org markup. 
Photo of Mike

Mike

  • 3,786 Points 3k badge 2x thumb
Thanks for the update Stephen,  

In one of your previous posts above, you mentioned that the sitemaps should be updated very frequently.  I'm not seeing this happen.  I'm not sure it's updating at all. We've taken a close look at the URLs and are seeing some returning 404 errors because the posts were removed several weeks ago. 

Also, I'm seeing several URL in the sitemap that have been merged into other topics. 

Can someone assist to see why the sitemap isn't updating. 

I'm guessing there are enough errors in the sitemap it may be deterring Google from indexing more pages. 

Mike 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Ah thanks, didn't realize that was happening. We'll take a look. 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi Mike - we found and fixed the bug related to the sitemaps not being updated, so that should be working again. Thanks for reporting it. 
Photo of Alexander

Alexander

  • 458 Points 250 badge 2x thumb
Hi guys,

We're one of the customers hit by the downturn. Short story: We used to have ~35 000 URLs indexed by Google, nowadays about 3 000. After submitting the newly generated sitemap via Google Search Console, we're seeing only ~2 800 out of 17 500 URLs indexed. We also note that our Bing search traffic has been unaffected all along.

Our analysis has suggested two possible source to the problem we're all seeing and I'd like to share this, in case it might be of any use: 

1. Domain confusion

It's clear Google has a problem understanding the domain setup of both GetSatisfaction's own community and our customer community, while Bing doesn't. Bing seems to index our forum fully:

26 700 results



vs. 3 160 results in Google



The same is true for GetSatisfaction's own domain in Bing:

69 300 results





vs. the bloated index by Google, 131 000 results



Both GetSatisfaction's own domain and our forum subdomain are subject to surprising redirects, probably confusing Google. Example:

https://getsatisfaction.com/cumulus 301-redirects to https://community.cumulusnetworks.com/cumulus



Yet https://getsatisfaction.com/cumulus yields 11 results in Google:



The actual forum URL path yields 99 results, but last time I investigated this in September it found only 7 results, reflecting that the forum is doing better but isn't indexed properly by Google:



Compare this with Bing who realizes the 301-redirect is in place:




And therefore returns a more complete indexation of the actual domain, 341 results:




We can see how confused Google is by searching for our own subdomain, showcasing other customer forums that in fact are 301-redirected:



2. Server performance issues
In September when we attempted to crawl our forum subdomain as Googlebot, the server returned ~40% DNS failure:



We then suspected the server had in place some kind of aggressive anti-spam rule that potentially hit Googlebot during crawling. Since then, we've seen no DNS failures during crawling, but as soon as we run two crawl processes at the same time from the same IP, both crawls fail. To what extent this is the product of normal anti-spam response or if it could affect the crawling of our forums, we're not really sure.

In late September, when we ran a page speed test on our forum, it displayed a response time from the server at 1,1+ seconds, in total about 4 seconds of page load time. Even though the authentication process today for the forum is a big slowdown, we're not seeing that server performance issue today (Pingdom Tools):

3,77 seconds load time



The authentication process takes 1,3 seconds:



Google's PageSpeed Insights tells us there's at least one redirect chain involving redirection from getsatisfaction.com/visma_spcs to forum.vismaspcs.se/visma_spcs–we're back to the 301-redirects that have to be processed for every forum URL the search bot encounters:



Basically what I want to highlight here is how crucial technical server performance is to successful crawling. If the server too aggressively blocks the Googlebot when it tries to index our forums, combined with slow server response time, the bot will simply abandon the site during indexation. Since we've all submitted URLs via sitemaps and this still did not work, it feels like a performance issue could be a part of the problem. How do we know for sure? It's fairly easy to find out :)

I recommend the GetSatisfaction dev team, if they haven't already, to check for server logs on their web server and use a tool like Screaming Frog Log Analysis to look at how the Google and Bing bot interact with your server. There you can follow time stamps and see exactly which URLs the bots try to visit and what the result is. If there is a performance issue at the crawling level, this is where you'll most likely discover the source to the problem. Here's a screenshot of how a website is indexed by different bots–notice how active Bing is compared to Google :-P



It'd be interesting to see what the GS dev team can do in terms of improving the domain setup, but also what may come up after a log file analysis. Whatever the problem is, it seems to boil down to issues at HTTP request or server level.
(Edited)
Photo of Mike

Mike

  • 3,786 Points 3k badge 2x thumb
My issue also seems to be just with Google. It's frustrating they are so secretive... I've been beating my head against the wall and our internal SEO team is still stumped why they aren't indexing more pages. 

I'm beginning to wonder if it's a combination of things. Another thing I've been looking at is keyword density.  I wonder if I (maybe all of us) are being penalized for duplicate/similar irrelevant keywords.  Just a thought. 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Thanks for all the info Alexander. 

It is really weird that searching for Visma with the "site:" prefix it shows many other communities on the Get Satisfaction platform. I don't see the same behavior when searching other customers' cnames in the same manner, and I wonder if it's because you were around longer and maybe those URLs got mistakenly indexed once upon a time and Google is refusing to remove them for some reason, even though they all return 301s. 

Regarding performance and indexing, we've put a few changes in place in the last few months - caching whole pages for faster load times, ensuring that we're not rate limiting Google, and removing a crawl delay directive that was in robots.txt from a long time ago. After making these changes we've seen a small uptick in indexing but obviously not as much as everyone's looking for. (Aside: for the moment we've had to turn off the use of this cache for a separate reason but will be bringing it back soon - in case you test right now and notice that page loads are a bit slower again).

Thanks for the suggestion of using Screaming Frog log analysis tool. We have our own log tools and have addressed any forms of rate limiting that we were able to find, but we'll give that one a try as well to see if it can uncover anything else. 

Stephen
Photo of Alexander

Alexander

  • 458 Points 250 badge 2x thumb
Thanks for the feedback Stephen!

I noticed not all of your forum customers share our problem, but there's something fishy either surrounding the domain management or how Google interprets your domain management.

A site search for your own domain returns 9 customer forums in Google's top 10 results, all of them displayed under your domain even though they carry their own respective domains and are redirected. A more relevant first search page would display the most important pages of your domain, especially since your domain primarily serves your own forum content: https://getsatisfaction.com/getsatisfaction (a site search for this URL path returns only 252 results here).

At the same time, I don't see getsatisfaction.com really cannibalizing customer forum content in Google's search results. Also, if that were the case, the search traffic would merely reroute to your domain and not ours–I'm assuming you haven't seen any such shift taking place at the time when this problem began appearing.

Worth double-checking:
If the domain management is fine, Google's probably unable/too lazy to follow all redirects. I don't see any technical errors here, so this could perhaps again boil down to a DNS or server performance problem. Since customer forums aren't fully crawled at all, this suggests in my mind we're looking at a server performance problem, where the redirect issue is only secondary.

I crawled your getisatisfaction.com domain (not as Googlebot) and of the 37 536 URL:s I crawled, there's a 28% client error response rate. The HTTP response was indeed in most cases 429 - too many requests. The avg. crawl rate was 5,2 URL:s/second. If almost every third URL is rejected during crawling of your domain, this might be happening for all customers. I will make a new crawl on our domain with max 1 thread and a crawl rate of 0,1 max URL:s, to see how this may affect the success rate.



You'd be able to see the HTTP responses during a log file analysis. I'd especially check:
  • Which URL paths receive the most Googlebot hits during a week? Pay close attention to Google's mobile bot, since they're nowadays prioritizing mobile crawling over desktop crawling.

  • How many of the URLs hit by Google are actual redirects–and of these, how many redirect to a customer domain?

  • Do you serve HTTP 429, 500 or similar status responses during crawls for your own domain?
(Edited)
Photo of Alexander

Alexander

  • 458 Points 250 badge 2x thumb
I just ran a second crawl session as Googlebot on getsatisfaction.com, finished 25 540 URLs (out of ~37-38 000 URLs), this time with an average crawl rate of 1,85 URLs/second only. And this time I received no 429/too many requests responses at all (any 4xx responses were regular 404/lost pages):



So obviously the crawl success is directly affected by the crawl rate and there's a server policy in place restricting the crawl rate. A forum-oriented website should be able to handle several URL requests per second, since we're talking about vast amounts of content. To take a close example, the forum Stackexchange experienced indexation problems with Google. Among other things they focused on server performance optimization and later achieved a crawl rate of 10 URLs/second:

http://meta.stackexchange.com/questions/84695/eeeeek-where-is-my-slow-performance/84701#84701

It's actually possible to control the crawl rate inside Google Search Console: https://support.google.com/webmasters/answer/48620?hl=en

The downside: It takes 90 days for any changes to take effect. This can be done by us customers on our subdomains inside GSC.

I would strongly advise you to once again look over the server performance, loosen any crawl restrictions for your server, so that we at least can see a crawl rate of 10 URLs/second works. This will be very easy to test afterwards.
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Wow, a lot of excellent points and linked posts. We'll be looking into all of this soon and will update here when we have anything. Thanks so much!
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, just wanted to give a quick update.

Over the past month we've released a number of changes all geared towards serving more content to Google more quickly. We have also been seeing a slight uptick in indexing in the last few weeks, so we feel like we're on the right track, though we know there's still a long way to go. But overall it does seem like the problem is mainly performance related.

We've brainstormed and come up with a number of new paths to pursue in order to serve more content to Google (and users) more quickly, but unfortunately these are kind of big architectural changes, are not easy, and come with some risk so we don't want to rush them. 

I just wanted to let you know that we are still working on SEO as a top priority, and we appreciate your patience in the meantime. 
Photo of Rav

Rav, Champion

  • 960 Points 500 badge 2x thumb
Thank you Stephen!
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Quick update everyone - we've put in a number of performance enhancements over the last two weeks and have started noticing a steady increase in site indexing, especially for our CNAME customers. Are you guys seeing improvements on your end? 
Photo of Daniel Foster

Daniel Foster

  • 4,508 Points 4k badge 2x thumb

That's moving in the right direction! We're a CNAME customer: feedback.techsmith.com
Photo of Drew C.

Drew C., Champion

  • 11,652 Points 10k badge 2x thumb
We're definitely seeing an improvement!  Thanks Stephen.
Photo of Glenn Dobson

Glenn Dobson

  • 1,906 Points 1k badge 2x thumb
Trending up for us as well. Thanks for continuing to work on this.
Photo of Rav

Rav, Champion

  • 960 Points 500 badge 2x thumb
Forgive my ignorance, but what is a CNAME customer and how do I tell if our Community is one?

Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi Rav, a CNAME customer is one that's on a URL other than getsatisfaction.com. You are such a customer. Thanks for sharing your chart. 
Photo of Mike

Mike

  • 3,776 Points 3k badge 2x thumb
HI Stephen,  I'm seeing a noticeable improvement, but still a ways to go. Looks like about 1/3 of our active topics are indexed according to Google Search Console. 
Photo of Alexander

Alexander

  • 458 Points 250 badge 2x thumb
Same here, we're seeing a noticeable traffic improvement from organic search. The number of indexed URLs is slowly increasing, right now a 10th of what we had in May. Stephen, out of sheer professional curiosity, what kind of performance enhancements have you guys done?

Update: Been running a crawl today of our forum at 4,36 URLs/second as average crawl rate. No 429 or other error responses and I'm 71% through all of our content. This gives me great hope the performance improvements will continue to improve Google's indexation :)
(Edited)
Photo of Karen

Karen

  • 360 Points 250 badge 2x thumb
Definitely seeing an improvement but still have a ways to go.
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Great to hear everyone, thanks for the updates on what you've been seeing.

Alexander - performance-wise, we have been doing what we can to reduce the Time To First Byte, which involved utilizing more caching in between some layers of our stack, and moving certain pieces of pages that aren't as important for indexing, to asynchronous loading.

We have a few more related changes in the pipeline, but otherwise we're expecting that the indexing numbers will continue upwards linearly over time, as opposed to a sudden jump back up. But that's just a guess. 
Photo of Sanj

Sanj

  • 866 Points 500 badge 2x thumb
Also seeing an improvement, we'll monitor to see how it goes :)
Photo of Alexander

Alexander

  • 458 Points 250 badge 2x thumb
We're currently back to our previous indexation levels, even surpassing those. We've also regained most of our organic search traffic, close to the levels in May. Of course, since Google now is re-indexing our content, one should expect that rankings will fluctuate in the beginning, and for some content one might not reach the same search visibility as before the problems began.

Regardless, great work from you and your team Stephen, you guys have earned a pre-Xmas beer to celebrate this :)
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Glad to hear it Alexander! Will pass this on to the team :) 
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
Hi everyone, it's been a few weeks since our last update and from what we can tell, everything is looking good to go so I'm going to mark this as solved. Thanks so much for all your input throughout this process! Let us know if you experience further issues. 
Photo of Ryan Mathews

Ryan Mathews

  • 576 Points 500 badge 2x thumb
Hey Stephen, 

Just wanted to thank you for the follow up(s) and persistence resolving this issue.  

Here at Extreme Networks our Community is again seeing the power of Google as referrals are finally returning to levels last seen in March 2016.  This has had a positive impact on the health of the Community and our service experience.

There's nothing easier than a Google search to kick off a self-service session and it's great to see the GetSat platform showing up on the first page again!
Photo of Mike

Mike

  • 3,776 Points 3k badge 2x thumb
Agree!!! :)
Photo of Stephen Lee

Stephen Lee, Engineering Manager

  • 20,158 Points 20k badge 2x thumb
So glad to hear it, thanks guys! 
Photo of Alexander

Alexander

  • 458 Points 250 badge 2x thumb
Hi guys,

Just wanted to share with you this pretty baffling indexation curve. It speaks volumes!



Notice how we before the idexations problems began reached levels of about 40 000 URLs. Pretty neat, we thought. Roll forward to the beginning of 2017 and we've more than doubled our indexation levels!

I knew server performance and TTLB affected organic search visibility somewhat, but that it could have this great impact surprised me. I wonder how many large forums and ecommerce sites out there actually struggle with suboptimal indexation due to technical performance issues.

Also, it's fascinating to note how Bing seems to pull way more resources into its indexation routines: