goxdcr CPU usage on clusters are permanent ~30-90%
We have very simple installation 4.0.0 community version (after 3.0.1 upgrade).
Cluster A - 1 node, 3 bucket
Cluster B - 1 node, 3 bucket
Both-directional XDCR Cluster A [3 bucket] <=> Cluster B [3 bucket]
centos6(64) (no any virtualization)
workflow is very low:
all count of items is less then 3k
count of set/delete is less then 0.1k per day
There is problem with 1 bucket:
in WEB on ClusterA and ClusterB we can see every minute Number of mutations to be replicated to other clusters(measured from replication_changes_left) ~25k
and Incoming XDCR total ops/sec. ~0.4k
file goxdcr.log on Cluster A has errors:
,“errMsg”:“dcp_a753ef60019bfd8ce8c4555a8003656b/freeswitchconf/freeswitchconf_10.2.1.201:11210_0:Dcp is stuck for dcp nozzle dcp_a753ef60019bfd8ce8c4555a8003656b/freeswitchconf/freeswitchconf_10.2.1.201:11210_0”
file goxdcr.log on Cluster B has errors:
"errMsg":"CheckpointMgr:Target bucket’s topology has changed"
and
"errMsg":"dcp_6ddca8d8e0311b2cb2b634cc8e5e57a6/freeswitchconf/freeswitchconf_10.2.1.202:11210_0:Dcp is stuck for dcp nozzle dcp_6ddca8d8e0311b2cb2b634cc8e5e57a6/freeswitchconf/freeswitchconf_10.2.1.202:11210_0"
network link is good ( ~1gb)
Keys on clusters are synchronized, but CPU load and errors are not meaningful
Seems it is about this issue Loading...
After reduce nozzles to 2, CPU usage was reduced too (to 5-15%).
But this error is still repeatable in goxdcr.log file on both clusters (with ~10sec period)
"errMsg":"dcp_6ddca8d8e0311b2cb2b634cc8e5e57a6/freeswitchconf/freeswitchconf_10.2.1.202:11210_0:Dcp is stuck for dcp nozzle dcp_6ddca8d8e0311b2cb2b634cc8e5e57a6/freeswitchconf/freeswitchconf_10.2.1.202:11210_0"