Spotify only connects to the official server?

Spotify stops playing every now and then, probably for buffering since it starts again after a while. The only connection spotify has open is to host 78.31.8.18 on port 4070 which I assume is the official server. Is it so slow because this is the only connection it manages to start? Any ideas on what to do? It should be able to atleast create outbound connections to other peers...

I'm on a shared 50/50 connection, so this really shouldn't be the problem. We're far away from using the whole bandwith.