Requests to Ecobee getting Timeout

At Muzzley we are having some issues connecting with the Ecobee cloud.
Since a few days ago we noticed that some requests we're making take too long to get a response from the Ecobee cloud, which then trigger a timeout on our end.

We can reproduce with this cULR:
(It doesn't have the bearer intentionally, we were expecting to receive an error "Authentication failed. Token is required." but we don't receive any response.)

curl --connect-timeout 300 --max-time 300 -v -k "https://api.ecobee.com/1/thermostat?j..."

And this is the log generated:
* About to connect() to api.ecobee.com port 443 (#0)
* Trying 216.220.61.235... Connection timed out
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

The public IP on the machine we are making the request is "137.135.36.18". Could you somehow be blocking requests made from this IP? Maybe some rate-limit is being applied?
2 people have
this problem
+1
Reply
  • MarkK (API Architect) April 12, 2016 14:46
    We are not blocking the IP you indicated. Can you provide some additional information?

    1) Can you ping api.ecobee.com?

    2) Provide the traceroute/tracepath from your system to api.ecobee.com

    3) Have your administrators blocked outbound access at your company to api.ecobee.com?

    Thanks,
    Mark.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hello,

    Here is the additional info:

    1) Yes we can :)
    [PROD]XXX@YYYY:~# ping api.ecobee.com
    PING api.ecobee.com (216.220.61.235) 56(84) bytes of data.

    2)
    [PROD]XXX@YYYY:~# tracepath api.ecobee.com
    1: muzpearls-ca01 0.086ms pmtu 1500
    1: no reply
    2: no reply
    3: no reply
    4: no reply
    5: no reply
    6: no reply
    7: no reply
    8: no reply
    9: no reply
    10: no reply
    11: no reply
    12: no reply
    13: no reply
    14: no reply
    15: no reply
    16: no reply
    17: no reply
    18: no reply
    19: no reply
    20: no reply
    21: no reply
    22: no reply
    23: no reply
    24: no reply
    25: no reply
    26: no reply
    27: no reply
    28: no reply
    29: no reply
    30: no reply
    31: no reply
    Too many hops: pmtu 1500
    Resume: pmtu 1500

    3)
    IPTables info to assure that there is no admin block

    [PROD]XXX@YYYY:~# iptables -L OUTPUT
    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination

    [PROD]XXX@YYYY:~# iptables -L INPUT
    Chain INPUT (policy ACCEPT)
    target prot opt source destination

    [PROD]XXX@YYYY:~# iptables-save
    # Generated by iptables-save v1.4.12 on Tue Apr 12 17:07:44 2016
    *filter
    :INPUT ACCEPT [143180482:38159535145]
    :FORWARD ACCEPT [0:0]
    :OUTPUT ACCEPT [153271821:25055618296]

    Do you need any other info?

    Thanks,
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • MarkK (API Architect) April 12, 2016 17:18
    Doesn't look like the packets leave your local subnet. It may also be that your router does not permit ICMP. You'll have to troubleshoot your network.

    Please traceroute from your firewall to confirm where the problem lies: within or outside your network (possibly us).

    [PROD]XXX@YYYY:~# tracepath api.ecobee.com
    1: muzpearls-ca01 0.086ms pmtu 1500
    1: no reply
    2: no reply
    3: no reply
    4: no reply
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hi again,

    I couldn't manage to make the tracepath/traceroute but here is the output from both "nslookup" and "hping3" command from our server to "api.ecobee.com".

    I also include the same commands to google and facebook to use as comparison

    We are getting a lot of "DUP" which usually translate to a failure on the reply.

    Do you know what could be happening?

    [PROD]XXX@YYYY:~# nslookup api.ecobee.com
    Server: 168.63.129.16
    Address: 168.63.129.16#53

    Non-authoritative answer:
    Name: api.ecobee.com
    Address: 216.220.61.235

    [PROD]XXX@YYYY:~# hping3 -S api.ecobee.com -p 80
    HPING api.ecobee.com (eth0 216.220.61.235): S set, 40 headers + 0 data bytes
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=0 win=14600 rtt=67.0 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=1 win=14600 rtt=67.2 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=0 win=14600 rtt=1466.9 ms
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=2 win=14600 rtt=66.9 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=1 win=14600 rtt=1261.4 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=3 win=14600 rtt=67.0 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=0 win=14600 rtt=3466.8 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=2 win=14600 rtt=1453.2 ms
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=4 win=14600 rtt=66.8 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=1 win=14600 rtt=3261.3 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=3 win=14600 rtt=1444.9 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=5 win=14600 rtt=67.1 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=2 win=14600 rtt=3454.1 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=4 win=14600 rtt=1639.5 ms
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=6 win=14600 rtt=67.2 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=5 win=14600 rtt=1234.8 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=3 win=14600 rtt=3445.8 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=7 win=14600 rtt=66.8 ms

    [PROD]XXX@YYYY:~# hping3 -S api.ecobee.com -p 443
    HPING api.ecobee.com (eth0 216.220.61.235): S set, 40 headers + 0 data bytes
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=66.9 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=1 win=14600 rtt=66.7 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=1210.3 ms
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=2 win=14600 rtt=67.0 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=1 win=14600 rtt=1606.4 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=3 win=14600 rtt=67.3 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=3210.5 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=2 win=14600 rtt=1602.1 ms
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=4 win=14600 rtt=68.3 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=3 win=14600 rtt=1398.0 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=1 win=14600 rtt=3606.3 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=5 win=14600 rtt=66.9 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=4 win=14600 rtt=1393.8 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=2 win=14600 rtt=3602.1 ms
    len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=6 win=14600 rtt=66.8 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=5 win=14600 rtt=1389.6 ms
    DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=3 win=14600 rtt=3398.1 ms
    len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=7 win=14600 rtt=71.6 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=7210.7 ms
    DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=6 win=14600 rtt=1185.7 ms

    # Google example

    [PROD]XXX@YYYY:~# hping3 -S google.pt -p 80
    HPING google.pt (eth0 172.217.0.67): S set, 40 headers + 0 data bytes
    len=46 ip=172.217.0.67 ttl=54 id=25966 sport=80 flags=SA seq=0 win=42900 rtt=1.8 ms
    len=46 ip=172.217.0.67 ttl=54 id=4942 sport=80 flags=SA seq=1 win=42900 rtt=1.8 ms
    len=46 ip=172.217.0.67 ttl=54 id=12338 sport=80 flags=SA seq=2 win=42900 rtt=1.7 ms
    len=46 ip=172.217.0.67 ttl=54 id=40024 sport=80 flags=SA seq=3 win=42900 rtt=1.7 ms

    # FB example

    [PROD]XXX@YYYY:~# hping3 -S facebook.com -p 80
    HPING facebook.com (eth0 31.13.76.68): S set, 40 headers + 0 data bytes
    len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=0 win=14100 rtt=18.7 ms
    len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=1 win=14100 rtt=18.6 ms
    len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=2 win=14100 rtt=18.5 ms
    len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=3 win=14100 rtt=18.4 ms
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hi Pedro,

    I can confirm that I don't see your IP or netblock in any block lists on our side.

    It looks like you ran a 'tracepath' which does PMTU discovery instead of a 'traceroute'. Can you run 'traceroute' instead? Also, can you confirm if you're able to ping api.ecobee.com or not? You should be able to ping us, if not then there may be a network connectivity issue between us. Finally, you may want to try 'traceroute' with TCP SYN's, do a 'traceroute -T -p 443 api.ecobee.com'. It would also be helpful if you can include tcpdump output while running the ping and traceroute commands.

    Alternatively, if your traceroute doesn't support the '-T' option then you may want to look at tcptraceroute.

    Also, is the problem intermittent? I show that you last connected to us on the 11th at approx. 18:00 UTC-4. Is that when the problem began for you? Does the problem occur from any other hosts or networks that you are using?

    Thanks,

    Jari
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hello,

    Sorry for the long delay, we had to talk with our infrastructure provider about this issue (Azure).

    It seems that we are waiting for the final ACK from Ecobee but it isn't coming. So the connection isn't closing and after a while we reach the max number of open connections.

    Attached I send you an example of the packages reaching our server.
    10.0.0.0/24 (Muzzley server)
    216.220.61.236 (Ecobee server)

    After we receive the FIN, ACK we send it back to Ecobee (to confirm we received it) and we are expecting the final ACK to close the connection. However we are not receiving it.

    Do you have any ideas what is causing this?

    Attached
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hi Pedro, thanks for the info. I'm currently investigating this. Can you tell me what user agent your client is sending?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hello Jari, thanks!

    Our user agent:

    > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Thanks Pedro. I was hoping this might be related to a special case for MSIE where the socket keep alive and shutdown behaviour has been altered. However, that doesn't seem to be the case here. Fortunately, I've managed to reproduce the problem and have a hunch as to what may be causing it. I might need a few more days to come up with a solution.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hello Jari,

    Do you have any status update on this question?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hi Pedro, I found the source of the problem yesterday. We should have a fix in place tomorrow morning. I'll update this post at that time.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Hi Pedro, I'd like to clarify that I don't see how this fix will solve the original issue that you reported. However, this does fix the FIN-ACK issue that you mentioned in your update on the 12th. Do you still have issues with the initial SYN/connect to www.ecobee.com timing out? That may be a separate problem.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • The thing is that since we didn't get the FIN-ACK the connection wasn't closed and after some requests, we were having too many connections open and any additional requests was blocked.

    I hope this fixes that too, but we'll see. I'll give you some feedback on this after is fixed
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • 1
    Hi Pedro, the connection shutdown behaviour should be fixed now. Can you let us know if you're still having problems?
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated

  • Great!

    It's fixed yes! Problem solved, thank you very much!
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. indifferent, undecided, unconcerned kidding, amused, unsure, silly happy, confident, thankful, excited sad, anxious, confused, frustrated