Hello,
I’m using YCSB and CB 4.6.2. Ping between the servers is 150ms.
Single doc replication with no traffic takes aprox. 180ms.
recordcount=1000000
operationcount=2000000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0.5
updateproportion=0.1
scanproportion=0
insertproportion=0.4
fieldlength=700
requestdistribution=zipfian
YCSB params: -target 1400 -threads 10
During load, in our tests, documents greater than 7k are taking much more time to be replicated than 4k docs. If we mix the docs with 7k and 4k, 7k will affect those with 4k.
Our tests show that while 4k docs take aprox. 400ms to be GR-replicated, 7k docs can vary between 10sec to 30sec. Looking the OUTBOUND XDCR OPERATIONS TO BUCKET…, I can see that mutations keep a low record when 4k, but high(and increasing) when 7k. The numbers are:
4k between 900 and 1.300, not higher than this.
7k growing and varying, aprox. 90.000 at the end.
I would like to know how to identify where is the gap. I tried to increase XDCR Nozzles, batch count and any other XDCR config, but I keep getting the same numbers. Disk read and write are also fine, memory utilization(30%) and CPU(40% idle) ok.
I tried to increase the memory from 16G to 32G, to see if I get better numbers. I increased buckets memory allocation, but I have the same throughput.
top - 11:01:09 up 18 min, 1 user, load average: 5.12, 4.14, 2.44
Tasks: 173 total, 2 running, 171 sleeping, 0 stopped, 0 zombie
%Cpu0 : 3.8 us, 1.4 sy, 0.0 ni, 62.5 id, 31.6 wa, 0.0 hi, 0.3 si, 0.3 st
%Cpu1 : 3.8 us, 1.4 sy, 0.0 ni, 44.5 id, 50.3 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 4.8 us, 1.4 sy, 0.0 ni, 58.0 id, 35.5 wa, 0.0 hi, 0.0 si, 0.3 st
%Cpu3 : 6.1 us, 3.4 sy, 0.0 ni, 76.5 id, 9.9 wa, 0.0 hi, 3.7 si, 0.3 st
%Cpu4 : 4.1 us, 2.7 sy, 0.0 ni, 36.4 id, 56.4 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu5 : 5.2 us, 3.1 sy, 0.0 ni, 51.2 id, 40.5 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 5.8 us, 1.4 sy, 0.0 ni, 59.9 id, 32.9 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 7.2 us, 1.4 sy, 0.0 ni, 65.9 id, 25.5 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32781464 total, 20617468 free, 7709416 used, 4454580 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 24627012 avail Mem
Connection speed between servers, measurement using iperf:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 85.2 MBytes 71.0 Mbits/sec
Any pro tip? 
thx!