Help get this topic noticed by sharing it on Twitter, Facebook, or email.

I can't access ActiveInbox. Is anyone else currently experiencing this issue?

ActiveInbox won't load in my inbox. I get an error saying the server is playing hard to get. Refresh your screen or contact support.
12 people have
this question
+1
Reply
  • Yes, I am currently not able to log-in... Seems to happen once a quarter...
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • Andy Mitchell (CEO - ActiveInbox Founder) November 19, 2019 23:34
    Hi Barbara & Andrew,

    It was an outage. Everything is back up now (if you refresh Gmail).

    A quick post mortem of what happened...

    The Servers Are Being Upgraded

    For the last month I've been completely rewriting our servers. The code is ~12 years old, and was steadily becoming harder and harder to maintain. The new servers are much simpler, easier to audit to prove security, and will be running on a platform that auto manages scaling. In fact it auto manages nearly everything - so I'm hoping any chance of outages will become much less likely.

    But 'Starting Over' Is *Always* Risky, So I Took Major Precautions

    Mistakes made over the years have taught me to be ultra - going beyond what I think is practical - cautious... For the last week I've been testing the new servers by running them both side by side. ActiveInbox talks to the old servers (like it always has), and they relay/mimic the request to the new servers, so I can track any loss of data integrity - and see how the new servers handle the load. Crucially, the new servers weren't relied upon... they were entirely separate (and don't return data yet).

    But Obviously It Still Went Down

    Embarrassingly, I still overlooked one source of problems. The new servers did indeed get overloaded and couldn't handle the traffic, which I expected to be fine (as they weren't relied upon). Except, when they timed out, it caused our old servers to keep the connections open. Slowly these dud connections built up until the overwhelmed our old servers as well.

    I've Turned The New Servers Off

    And I rebooted the old servers. The new ones clearly aren't ready for prime time yet. I'm going to look at the telemetry to see how the bottlenecks can be optimised tomorrow. (I've already had a peek and got a pretty big hunch where the main problem lies: I over-engineered the security, and I think the encryption routines ground everything to a halt)

    When I resume testing, I'm also correct the mistake of letting the old servers die if the new ones do, so they should be truly disconnected.

    An Apology

    The above precautions came from knowing things haven't always gone smoothly in the past and thus really not wanting to repeat mistakes of old, but bluntly, I did. I'm tremendously sorry about that.

    On the bright side, I'm optimistic that this new combination of server simplification and a more managed hosting service means that the path is being set up for a much smoother ride as we grow. I hope that gives the words of humility something of a meatier meaning.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • This reply was created from a merged topic originally titled "Activeinbox couldn't load, because the server is paying hard to get....

    I keep getting the error: ActiveInbox couldn't load, because the server is playing hard to get. Please try refreshing Gmail in a few mins.

    Is there something I can do besides wait?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • Andy Mitchell (CEO - ActiveInbox Founder) November 20, 2019 16:26
    Hi all -

    It looks like I might have been wrong above (and the dud connections weren't coming from the new API server after all), as the old servers have just had another overload.

    I'm investigating as fast as I can, and will report back here.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • Andy Mitchell (CEO - ActiveInbox Founder) November 20, 2019 17:06
    It looks like everything is back under control (25 minutes and counting).

    I'm not 100% what caused this, but I have two hunches.

    1. As part of setting up the new servers, I've re-architected the database a little. One new query was really slow, for which I've added a performance-booster index.

    2. It coincided with the West Coast of the US waking up, which sends a surge of new requests. Obviously that's the same each day, and it should be manageable, but combined with #1 it might have overloaded it.

    I'm monitoring closely for the rest of the day.

    I'm also working as fast as I can to bring the new servers online, as despite the initial teething problems, they should be much more optimisable than our old ones.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • Hi Andy,

    Thank you for responding to the situation and keeping us updated!
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • I’m frustrated
    This reply was created from a merged topic originally titled ActiveInbox couldn't load, because the server is playing hard to get..

    Guys,

    keep getting same message all morning:

    ActiveInbox couldn't load, because the server is playing hard to get. Please try refreshing Gmail in a few mins, or check @activeinbox or contact support
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned happy, confident, thankful, excited kidding, amused, unsure, silly sad, anxious, confused, frustrated

  • Andy Mitchell (CEO - ActiveInbox Founder) November 21, 2019 14:33
    For Julio, Andrew and Carina - I see Lisa just merged the topic you were discussing this on.

    I hope you get this notification, and can see the explanations and remedies above (and my apology).
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited indifferent, undecided, unconcerned kidding, amused, unsure, silly sad, anxious, confused, frustrated