One possible cause we just discovered was a misconfiguration of the cluster nodes, 3 of the 4 nodes were on an incorrect subnet causing their network traffic to go through a much slower switch, so it could have just been network congestion / packet loss causing the issue which would fit with what I was seeing (high latency with no apparent cause on either end)