Requests to Ecobee getting Timeout

Edit Subject

We are not blocking the IP you indicated. Can you provide some additional information?

1) Can you ping api.ecobee.com?

2) Provide the traceroute/tracepath from your system to api.ecobee.com

3) Have your administrators blocked outbound access at your company to api.ecobee.com?

Thanks,
Mark.

Hello,

Here is the additional info:

1) Yes we can :)
[PROD]XXX@YYYY:~# ping api.ecobee.com
PING api.ecobee.com (216.220.61.235) 56(84) bytes of data.

2)
[PROD]XXX@YYYY:~# tracepath api.ecobee.com
1: muzpearls-ca01 0.086ms pmtu 1500
1: no reply
2: no reply
3: no reply
4: no reply
5: no reply
6: no reply
7: no reply
8: no reply
9: no reply
10: no reply
11: no reply
12: no reply
13: no reply
14: no reply
15: no reply
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
31: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500

3)
IPTables info to assure that there is no admin block

[PROD]XXX@YYYY:~# iptables -L OUTPUT
Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[PROD]XXX@YYYY:~# iptables -L INPUT
Chain INPUT (policy ACCEPT)
target prot opt source destination

[PROD]XXX@YYYY:~# iptables-save
# Generated by iptables-save v1.4.12 on Tue Apr 12 17:07:44 2016
*filter
:INPUT ACCEPT [143180482:38159535145]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [153271821:25055618296]

Do you need any other info?

Thanks,

Doesn't look like the packets leave your local subnet. It may also be that your router does not permit ICMP. You'll have to troubleshoot your network.

Please traceroute from your firewall to confirm where the problem lies: within or outside your network (possibly us).

[PROD]XXX@YYYY:~# tracepath api.ecobee.com
1: muzpearls-ca01 0.086ms pmtu 1500
1: no reply
2: no reply
3: no reply
4: no reply

Hi again,

I couldn't manage to make the tracepath/traceroute but here is the output from both "nslookup" and "hping3" command from our server to "api.ecobee.com".

I also include the same commands to google and facebook to use as comparison

We are getting a lot of "DUP" which usually translate to a failure on the reply.

Do you know what could be happening?

[PROD]XXX@YYYY:~# nslookup api.ecobee.com
Server: 168.63.129.16
Address: 168.63.129.16#53

Non-authoritative answer:
Name: api.ecobee.com
Address: 216.220.61.235

[PROD]XXX@YYYY:~# hping3 -S api.ecobee.com -p 80
HPING api.ecobee.com (eth0 216.220.61.235): S set, 40 headers + 0 data bytes
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=0 win=14600 rtt=67.0 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=1 win=14600 rtt=67.2 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=0 win=14600 rtt=1466.9 ms
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=2 win=14600 rtt=66.9 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=1 win=14600 rtt=1261.4 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=3 win=14600 rtt=67.0 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=0 win=14600 rtt=3466.8 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=2 win=14600 rtt=1453.2 ms
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=4 win=14600 rtt=66.8 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=1 win=14600 rtt=3261.3 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=3 win=14600 rtt=1444.9 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=5 win=14600 rtt=67.1 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=2 win=14600 rtt=3454.1 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=4 win=14600 rtt=1639.5 ms
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=80 flags=SA seq=6 win=14600 rtt=67.2 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=5 win=14600 rtt=1234.8 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=3 win=14600 rtt=3445.8 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=80 flags=SA seq=7 win=14600 rtt=66.8 ms

[PROD]XXX@YYYY:~# hping3 -S api.ecobee.com -p 443
HPING api.ecobee.com (eth0 216.220.61.235): S set, 40 headers + 0 data bytes
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=66.9 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=1 win=14600 rtt=66.7 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=1210.3 ms
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=2 win=14600 rtt=67.0 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=1 win=14600 rtt=1606.4 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=3 win=14600 rtt=67.3 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=3210.5 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=2 win=14600 rtt=1602.1 ms
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=4 win=14600 rtt=68.3 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=3 win=14600 rtt=1398.0 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=1 win=14600 rtt=3606.3 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=5 win=14600 rtt=66.9 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=4 win=14600 rtt=1393.8 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=2 win=14600 rtt=3602.1 ms
len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=6 win=14600 rtt=66.8 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=5 win=14600 rtt=1389.6 ms
DUP! len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=3 win=14600 rtt=3398.1 ms
len=46 ip=216.220.61.235 ttl=44 DF id=0 sport=443 flags=SA seq=7 win=14600 rtt=71.6 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=0 win=14600 rtt=7210.7 ms
DUP! len=46 ip=216.220.61.235 ttl=43 DF id=0 sport=443 flags=SA seq=6 win=14600 rtt=1185.7 ms

# Google example

[PROD]XXX@YYYY:~# hping3 -S google.pt -p 80
HPING google.pt (eth0 172.217.0.67): S set, 40 headers + 0 data bytes
len=46 ip=172.217.0.67 ttl=54 id=25966 sport=80 flags=SA seq=0 win=42900 rtt=1.8 ms
len=46 ip=172.217.0.67 ttl=54 id=4942 sport=80 flags=SA seq=1 win=42900 rtt=1.8 ms
len=46 ip=172.217.0.67 ttl=54 id=12338 sport=80 flags=SA seq=2 win=42900 rtt=1.7 ms
len=46 ip=172.217.0.67 ttl=54 id=40024 sport=80 flags=SA seq=3 win=42900 rtt=1.7 ms

# FB example

[PROD]XXX@YYYY:~# hping3 -S facebook.com -p 80
HPING facebook.com (eth0 31.13.76.68): S set, 40 headers + 0 data bytes
len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=0 win=14100 rtt=18.7 ms
len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=1 win=14100 rtt=18.6 ms
len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=2 win=14100 rtt=18.5 ms
len=46 ip=31.13.76.68 ttl=83 DF id=0 sport=80 flags=SA seq=3 win=14100 rtt=18.4 ms

Hi Pedro,

I can confirm that I don't see your IP or netblock in any block lists on our side.

It looks like you ran a 'tracepath' which does PMTU discovery instead of a 'traceroute'. Can you run 'traceroute' instead? Also, can you confirm if you're able to ping api.ecobee.com or not? You should be able to ping us, if not then there may be a network connectivity issue between us. Finally, you may want to try 'traceroute' with TCP SYN's, do a 'traceroute -T -p 443 api.ecobee.com'. It would also be helpful if you can include tcpdump output while running the ping and traceroute commands.

Alternatively, if your traceroute doesn't support the '-T' option then you may want to look at tcptraceroute.

Also, is the problem intermittent? I show that you last connected to us on the 11th at approx. 18:00 UTC-4. Is that when the problem began for you? Does the problem occur from any other hosts or networks that you are using?

Thanks,

Jari

Hello,

Sorry for the long delay, we had to talk with our infrastructure provider about this issue (Azure).

It seems that we are waiting for the final ACK from Ecobee but it isn't coming. So the connection isn't closing and after a while we reach the max number of open connections.

Attached I send you an example of the packages reaching our server.
10.0.0.0/24 (Muzzley server)
216.220.61.236 (Ecobee server)

After we receive the FIN, ACK we send it back to Ecobee (to confirm we received it) and we are expecting the final ACK to close the connection. However we are not receiving it.

Do you have any ideas what is causing this?

Attached

Hi Pedro, thanks for the info. I'm currently investigating this. Can you tell me what user agent your client is sending?

Hello Jari, thanks!

Our user agent:

> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3

Thanks Pedro. I was hoping this might be related to a special case for MSIE where the socket keep alive and shutdown behaviour has been altered. However, that doesn't seem to be the case here. Fortunately, I've managed to reproduce the problem and have a hunch as to what may be causing it. I might need a few more days to come up with a solution.

Hello Jari,

Do you have any status update on this question?

Hi Pedro, I found the source of the problem yesterday. We should have a fix in place tomorrow morning. I'll update this post at that time.

Hi Pedro, I'd like to clarify that I don't see how this fix will solve the original issue that you reported. However, this does fix the FIN-ACK issue that you mentioned in your update on the 12th. Do you still have issues with the initial SYN/connect to www.ecobee.com timing out? That may be a separate problem.

The thing is that since we didn't get the FIN-ACK the connection wasn't closed and after some requests, we were having too many connections open and any additional requests was blocked.

I hope this fixes that too, but we'll see. I'll give you some feedback on this after is fixed

Hi Pedro, the connection shutdown behaviour should be fixed now. Can you let us know if you're still having problems?