ecobee servers' outage?

Are there any known ecobee servers' issues or upgrades in progress? Recently, I've been having difficulties to renew my refresh token. Today, it was about 2 hours ago, and I had a similar issue about 4 days ago.

It would be useful to have a status page where ecobee components would be posted with a status (such as portal, HomeIQ, servers, network, etc.), so that developers know exactly if everything is working fine on the ecobee side.

As an example, here is the SmartThings' status page:

http://status.smartthings.com/

The first issue in my logs was at the following timeframe:

2015-05-16 11:00:08.040 AM UTC (2015-05-16T11:00:08.040Z)

Regards.
3 people have
this question
+1
Reply
  • Hello again,

    Here are the error messages some users who have installed MyEcobeeDevice in SmartThings get from time to time in recent days (although I always make sure to renew the refresh token every 55 minutes or so).

  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • ScottF (Official Rep) May 19, 2015 13:46
    Hi Yves,
    Firstly thanks for the suggestion. Because you and others have requested such a status page, I can tell you that it is something which is currently in development and we hope to have available soon.

    Now, to answer your question:
    No, there was no downtime on our servers recently where the issue you are seeing would have occurred.

    I checked our internal logs and it the problem seems to be your application is consistently sending 2 requests to refresh the token (to the /home/token end point). Both requests are sent at the same time. The first one succeeds but the second one fails (as it is no longer valid) and that is the exception you are seeing.

    Can you please double check your source code and application logic to ensure this double POST request no longer happens. After you resolve that issue I believe the problem should be resolved.

    I hope this helps,
    Scott.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Hello ScottF, my code usually repeats the last request if the 1st request does not go through.

    Please refer to the following log (34 minutes ago):


    As you can see the 1st request (from the bottom) to refresh the token didn't go through (exception groovyx.net.http.HttpResponseException: Unauthorized).

    My code went ahead and tried to do a request and got the same exception. After a second try, the refresh_tokens() was successful and the token was renewed.



    2015-05-19 3:00:05.451 PM UTC
    34 minutes ago DEVICE verboseTrace refresh_tokens>expire in 59 m... 2015-05-19 3:00:05.438 PM UTC
    34 minutes ago COMMAND refreshChildTokens
    2015-05-19 3:00:05.225 PM UTC
    34 minutes ago APP_COMMAND refreshChildTokens My ecobee Init sent refreshChildTokens command to My ecobee Kitchen true
    2015-05-19 3:00:03.237 PM UTC
    34 minutes ago DEVICE verboseTrace doRequest>exception groovyx.n... My ecobee Kitchen verbose trace is doRequest>exception groovyx.net.http.HttpResponseException: Internal Server Error, Internal... true
    2015-05-19 3:00:01.703 PM UTC
    34 minutes ago DEVICE verboseTrace doRequest>about to get with u... My ecobee Kitchen verbose trace is doRequest>about to get with uri https://api.ecobee.com/1/thermostat?f...... true
    2015-05-19 3:00:01.636 PM UTC
    34 minutes ago DEVICE verboseTrace api> about to call doRequest ... My ecobee Kitchen verbose trace is api> about to call doRequest with (unencoded) args = {"selection":{"selectionType":"thermos... true
    2015-05-19 3:00:00.818 PM UTC
    34 minutes ago DEVICE verboseTrace refresh_tokens>exception groo... My ecobee Kitchen verbose trace is refresh_tokens>exception groovyx.net.http.HttpResponseException: Unauthorized, Unauthorized... true


    But, most of the time, refresh_tokens() is called only once such as 2 hours ago


    2015-05-19 1:00:01.741 PM UTC
    2 hours ago DEVICE verboseTrace refresh_tokens>expire in 59 m... My ecobee Kitchen verbose trace is refresh_tokens>expire in 59 minutes true
    2015-05-19 1:00:01.670 PM UTC
    2 hours ago COMMAND refreshChildTokens 2015-05-19 1:00:01.373 PM UTC
    2 hours ago APP_COMMAND refreshChildTokens My ecobee Init sent refreshChildTokens command to My ecobee Kitchen true


    Here is SmartThings community thread where Smart (a member) reports the ecobee portal site being down 2 days ago in New Jersey (pleases scroll at the end):


    http://community.smartthings.com/t/ec...

    Regards.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • @ScottF, you're probably referring to this event here at 2:00 PM UTC Today:

    2015-05-19 2:00:02.866 PM UTC
    2 hours ago DEVICE verboseTrace refresh_tokens>expire in 59 m... My ecobee Kitchen verbose trace is refresh_tokens>expire in 59 minutes true
    2015-05-19 2:00:02.864 PM UTC
    2 hours ago DEVICE verboseTrace refresh_tokens>expire in 59 m... My ecobee Kitchen verbose trace is refresh_tokens>expire in 59 minutes true
    2015-05-19 2:00:02.835 PM UTC
    2 hours ago COMMAND refreshChildTokens 2015-05-19 2:00:02.830 PM UTC
    2 hours ago COMMAND refreshChildTokens 2015-05-19 2:00:02.747 PM UTC
    2 hours ago APP_COMMAND refreshChildTokens My ecobee Init sent refreshChildTokens command to My ecobee Kitchen true
    2015-05-19 2:00:02.739 PM UTC
    2 hours ago APP_COMMAND refreshChildTokens My ecobee Init sent refreshChildTokens command to My ecobee Kitchen true

    Which is quite strange, I wonder if the SmartThings platform is not firing my code twice...

    I need to investigate with SmarThings, but in my logs, I can see that the same refresh token is returned by ecobee. Is there a problem to call you twice as it looks that you're returing the same token anyway?

    Regards.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Hi ScottF,

    Here are other log trace which clearly demonstrates that most of the time, the refresh_tokens() method is called once:

    2015-05-19 6:58:41.977 PM UTC
    29 minutes ago DEVICE verboseTrace refresh_tokens>expire in 59 m... My ecobee Kitchen verbose trace is refresh_tokens>expire in 59 minutes true
    2015-05-19 6:58:41.967 PM UTC
    29 minutes ago COMMAND refreshChildTokens 2015-05-19 6:58:41.888 PM UTC
    29 minutes ago APP_COMMAND refreshChildTokens My
    ecobee Init sent refreshChildTokens command to My ecobee Kitchen true

    2015-05-19 5:00:04.318 PM UTC
    3 hours ago DEVICE verboseTrace refresh_tokens>expire in 59 m... My ecobee Kitchen verbose trace is refresh_tokens>expire in 59 minutes true
    2015-05-19 5:00:04.280 PM UTC
    3 hours ago COMMAND refreshChildTokens 2015-05-19 5:00:04.037 PM UTC
    3 hours ago APP_COMMAND refreshChildTokens My ecobee Init sent refreshChildTokens command to My ecobee Kitchen true

    My problem seems to be similar to another problem submitted in this forum

    http://developer.ecobee.com/api/topic...

    Bye for now.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Also, I can see that after the refresh_tokens(), sometimes, there is an Internet Server Exception when sending a request.

    It works well for a while, as seen in bold

    Let me know.


    2015-05-19 5:00:05.309 PM UTC
    3 hours ago DEVICE verboseTrace doRequest>exception groovyx.n... My ecobee Kitchen verbose trace is doRequest>exception groovyx.net.http.HttpResponseException: Internal Server Error, Internal... true
    2015-05-19 5:00:05.252 PM UTC


    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,event's fanMinOnTime=30 true
    2015-05-19 5:00:05.188 PM UTC
    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,event's fan mode=auto true
    2015-05-19 5:00:05.157 PM UTC
    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,event's heatHoldTemp=57.2 true
    2015-05-19 5:00:05.096 PM UTC
    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,event's coolHoldTemp=82.4 true
    2015-05-19 5:00:04.969 PM UTC
    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,event type=hold true
    2015-05-19 5:00:04.922 PM UTC
    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,event name=auto true
    2015-05-19 5:00:04.885 PM UTC
    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,indiceEvent=0 true
    2015-05-19 5:00:04.843 PM UTC
    3 hours ago DEVICE verboseTrace poll>thermostatId = 100483116... My ecobee Kitchen verbose trace is poll>thermostatId = 100483116561,Current Climate Ref=away,currentClimateTemplate=Away true
    2015-05-19 5:00:04.741 PM UTC

    3 hours ago DEVICE verboseTrace getTstatInfo> thermostatId=10... My ecobee Kitchen verbose trace is getTstatInfo> thermostatId=100483116561,name=Kitchen,hvacMode=cool,fan=auto,fanMinOnTime=30... true


    2015-05-19 5:00:04.430 PM UTC
    3 hours ago DEVICE verboseTrace doRequest>about to get with u... My ecobee Kitchen verbose trace is doRequest>about to get with uri https://api.ecobee.com/1/thermostat?f...... true
    2015-05-19 5:00:04.368 PM UTC
    3 hours ago DEVICE verboseTrace api> about to call doRequest ... My ecobee Kitchen verbose trace is api> about to call doRequest with (unencoded) args = {"selection":{"selectionType":"thermos... true
    2015-05-19 5:00:04.318 PM UTC
    3 hours ago DEVICE verboseTrace refresh_tokens>expire in 59 m... My ecobee Kitchen verbose trace is refresh_tokens>expire in 59 minutes true
    2015-05-19 5:00:04.280 PM UTC
    3 hours ago COMMAND refreshChildTokens 2015-05-19 5:00:04.037 PM UTC
    3 hours ago APP_COMMAND refreshChildTokens My ecobee Init sent refreshChildTokens command to My ecobee Kitchen true
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • ScottF (Official Rep) May 19, 2015 23:24
    Thanks for the updates Yves, we are investigating and will report back to you here with our findings, or if we require more information from you.

    Scott
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Yves,

    We have confirmed there have been no outages on the ecobee side and that this is a bug with the SmartThings integration. What is happening is that for every instance of these errors, SmartThings sends two refresh token requests simultaneously. It is not clear why SmartThings does this, and it does not do this for all token refreshes which explains why you see this intermittently. When it does send the two simultaneous requests, it causes us to generate two different refresh tokens to be used. The second one replaces the first. SmartThings is retaining the first response but not the second. It subsequently uses the refresh tokens from the first response to perform later token refreshes and this fails because that first token set was invalidated by the second set generated. 

    As Scott indicated earlier, the root of this problem is the duplicate token refresh requests that we are seeing on our end. You may not be seeing this from your client side as the requests are coming directly from SmartThings' cloud services (Hosted on Amazon EC2).
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • I’m happy
    @Hesham Fhamy and ScottF,

    Today, there were no double refresh_tokens() execution on the SmartThings side, and everything looks clean in the logs again.

    I've submitted a support ticket to ST and waiting for their response.

    In the meantime, I've implemented a workaround in refresh_tokens() to avoid this kind of exceptions.

    Hopefully, this will solve the issue once for all.

    Thnx for your support!
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Today between 3pm and 6pm (Pacific Daylight Time) several users of my app, including myself, could not refresh the tokens. The requests timed out. Was there any issue with the ecobee service?

    Having an API to call to get status of the servers, as mentioned in other threads, would be useful (of course, the API would be in different servers).

    Regards
    -B
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Hello, I must also add that today at exactly 8h30 pm (EST), there were some issues with requests to ecobee servers at my location (Montreal) and also in MA (USA) as reported by some of my Ecobee device users.

    Please let us know when the status page will be available....

    Regards.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • MarkK (API Architect) May 27, 2015 15:32
    Hi, Yes, we did have some issues last night between 8pm and 9pm EST. The issues would have impacted any API calls being made.

    I cannot provide a date when the status page will be available.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Thnx for the update, please let us know about a target date for a status page (or a related API call) as soon as possible.

    Regards.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Hello,

    This morning (at around 12h20 AM EST) there was another issue for all my requests to the ecobee servers:

    date 2015-06-06 10:00:02.575 PM PDT (2015-06-07T05:00:02.575Z)
    description My ecobee Kitchen verbose trace is doRequest>exception groovyx.net.http.HttpResponseException: Internal Server Error, Internal Server Error for null

    Just checking with you if you can see it in your logs.

    P.S. I had to re-authorize my app to make it work again.

    Regards.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Hello, again starting at around 12h30 AM EST, there was another issue when sending requests to ecobee servers, see logs below:

    Name Value
    date 2015-06-19 5:00:08.308 AM UTC (2015-06-19T05:00:08.308Z)
    description My ecobee Kitchen verbose trace is doRequest>exception groovyx.net.http.HttpResponseException: Internal Server Error for null
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Yves, we have not had any outages at all in both the incidents you reported above. Please check your actual payloads and log the full HTTP request data and HTTP response data.

    It is likely that you are sending incorrect requests that are causing the errors you are seeing.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Hi @Hesham,

    Actually, it's the same requests that I send over and over again, so I don't understand why the majority of the time, they just work fine and at some specific day & time (which is at the beginning of the day), they don't work.

    If you look at the 2 log instances below (details above in this thread), it's actually the same timeframe within few seconds:


    2015-06-07T05:00:02.575Z
    2015-06-19T05:00:08.308Z


    So, I'm thinking this is more than a coincidence.... There must be some kind of outage either on the SmartThings or on the ecobee side...
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • The service has been having issues the last few days, so that would be a great use of the API, so it can inform the app (and other users) of certain issues, and thus help reduce support calls/contacts from users.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • We have a feature on our roadmap to add this to our API and our website where users can query the status of all ecobee systems and discover any outages or service degradation. We'll update this forum when the feature is close to completion.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Hello again,

    I'd like to know how far are you in the development of the status page and related status APIs?

    The reason I'm asking is that a lot of SmartThings (ST) users are experiencing issues with their ST-Ecobee connections in the past 10 days.

    We are wondering if any server maintenance or connection pool management is done at nights (EST) as our connections between ST and ecobee are not stable...

    There are a lot of timeouts with our http requests (10 sec is the timeout by default) and we lose our authorization token very often (especially in the past 2 days).

    Any update is appreciated.

    Thnx.

    See the following thread at the bottom for more details:

    https://community.smartthings.com/t/i...

    Regards.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Hi Yves,

    We unfortunately don't have an ETA yet for this feature. We've been busy working on a lot of new features in the product that everyone will see soon :)

    We have not had any major outages in the time frame you mention above, nor in the past several months. The last exchanges we had on this forum regarding to this issue ended up uncovering an issue on the ST end where there were duplicate token refresh requests coming in simultaneously that were invalidating each other. In reading the thread you posted above, it seems that the issue has been confirmed to be something on the ST end.

    For future reference, ecobee's own mobile apps and web portal use the exact same API. So if there is a fundamental issue/outage with things like token refreshing then all our interfaces will suffer the same issue. So if you are seeing our mobile app working fine but not yours then it is highly likely that the issue is from your end.

    Hope this helps.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited kidding, amused, unsure, silly indifferent, undecided, unconcerned sad, anxious, confused, frustrated

  • Is ecobee down now? (2:11am PST, Jan 19, 2016) can't access the ecobee site, nor use the app.

    Having the status API would help inform users if the service is down.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Yes, we were down for scheduled server maintenance from 5am to 6am EST. Services are back.

    Thermostat functionality was not effected, however as you observed access to the site and app was not available during that time.

    A status API is something currently in development. I can't give a timeline on the release of that functionality yet though.

    Scott.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Scott, thanks for the answer. I know how difficult it is to create an API that works when the rest of the service is down, that has accurate and timely information, that does not trigger automatically when there are minor service disruptions, etc.

    One idea is to have a web page hosted in a separate hosting service with the status of the service (either semaphore by service, or plain text that for example shows 'planned maintenance going on'), so in case there are API failures, we can point users to read that web page to confirm if there is any issue in the service.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Ecobee.com is having issues, and also my app can't authenticate or renew the token. Are there issues with the service? Is there any way for us developers to find out when there are outages? (5/10/2016, noon, Seattle time)

    Regards,
    Bernardo
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • ScottF (Official Rep) May 10, 2016 20:03
    Yes Bernardo, you are correct.
    We had a brief period of unscheduled downtime at that time. Our services are currently returning to normal and should be fully up very shortly.

    Thank you for your patience.

    Scott
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • 1
    Hi,

    Any updates on a status page or querying the ecobee APIs for for ecobee status?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned

  • Finally, a status page is available since beginning of this year (but not very publicized)...

    https://status.ecobee.com/
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. sad, anxious, confused, frustrated kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned