I have a Couchbase server cluster with 5 nodes with similar configuration. We found that network traffic from one of the node is 5 times than others node, but CPU is same (all around 40-50%), is it normal?
e.g.
node1 traffic : 200M
node2 traffic : 200M
node3 traffic : 200M
node4 traffic : 1000M
node5 traffic : 200M
I tried remove node4, rebalance, then add node4 and rebalance again, but same situation, how to check and fix?
We are using Couchbase .NET client 2.7.x., Couchbase server is 4.5.1
Are you using Community or Enterprise? Are you running the same services on each node?
I can’t say for sure this is related, but 4.5.1 is quite an old version now, with 7 now in beta. Is upgrading a possibility?
This may be normal depending on how your app uses data. For example, we’ve seen some apps which frequently request a “configuration” that is shared among the application logic nodes. Since they have to request it frequently, and it’s a single document, it comes off of a single node.
It might be fine. In extreme situations, you may want to move the cache closer to the use of the data (e.g., cache a bit in the app side).
Have a look at the “top keys” in the UI to see if there is something that helps you identify the imbalance.
Couchbase’s scheme to auto-shard the data was selected, and has proven to be, nice and gaussian , assuming your app’s workload is evenly distributed across the docs. It’s really a function of your application logic.
Community version, for upgrade, will consider later on., but need to fix existing issue first.
is there any way to share the network loading to other nodes? Coz I think this is quite unhealthy if most of the traffic from 1 node only.