[Resolved] Selection notifications slow to send and/or not updating

  • 1
  • Problem
  • Updated 4 weeks ago
  • (Edited)
Hi all –

Just to let you know, a few of our clubs' selection notifications are very slow to send at the moment (a very small minority aren't updating the notification status at all). We've picked up the issue, and we're working on it now.

I'll let you know once we've fixed this.

Thanks for bearing with us

Cellan
Photo of Cellan

Cellan, ClubBuzz

  • 48 Posts
  • 8 Reply Likes

Posted 4 weeks ago

  • 1
Photo of Cellan

Cellan, ClubBuzz

  • 48 Posts
  • 8 Reply Likes
Update – almost all off the notifications have gone now. We’re keeping an eye on this and we’ll apply a fix in the meantime.

Thanks again
Photo of Cellan

Cellan, ClubBuzz

  • 48 Posts
  • 8 Reply Likes
Evening all,

All of the selection notifications have gone out. The fix is complete, we're just rolling it out now.

Again, very sorry about this. In the meantime, for those of you who might be interested in why it happened, rather than what happened, here's a brief summary.

What happened?

To the end user, it appeared like selection notifications weren't sending. In fact, they were going out very slowly, some taking as long as 8 hours to go. What didn't help was that the selection page can't update until the notification has gone out, so it looked like the notifications just weren't going out at all.

What's happening in the background?

Every action in ClubBuzz, whether that's writing match fees to statements, converting applicants to members, or selecting players for matches, get put into one big queue. For ease, we call all of these things 'jobs'. 

Like any normal queue, each 'job' gets worked through one at a time. This is all well and good, but if the person at the front of the queue isn't moving, and the rest of the people can't get around the person at the front, that's an issue...

Back to selections

In this case, the issue was the scheduling system as a whole. Statements for all ClubBuzz members are generated and sent on 15th of each month (yesterday). Our scheduling system wasn't behaving how it should, which meant that jobs were very slow to progress through the queue. 

Some jobs within ClubBuzz, especially the ones in the background that you don't notice, don't necessarily need to be actioned that moment, whereas more urgent ones, like selections, should.

To combat this, we've changed how we prioritise jobs within ClubBuzz. This means that selections will take priority, so this removes the bottleneck which we've experienced.

We've also made changes to our monitoring system. If selection jobs are failing altogether, which is extremely rare, we're notified by email and updates to a Slack channel we have for this purpose. (If you've never used Slack before, take a look here, it's really good). If jobs are moving slowly, which hopefully they now won't be, we'll also be notified immediately. 


Once again, thanks for bearing with us on this one, and sorry for those who had issues with slow selection emails. Any sort of function not working is rare, but we don't want to see it at all. 

If you weren't affected, we'll be keeping an eye on the notifications anyway, and I hope you found this at least semi-interesting.


All the best

Cellan