Couchbase server version: 6.0.1 Community Edition (build-2037)
Couchbase Java SDK version: 2.7.4
I have setup a Couchabse cluster consist of 4 nodes.
Cluster 1 - (Server 1,2,3,4):
Total Processor: Output from nproc is 8 with 1 core each.
Total RAM: ~100 GB
No of Buckets: 2
RAM/Quota Usage: 27.5 GB
I am running a performance test on couchbase server with 10 users with the throughput of 4k Txns/sec. Exactly after 2.30 hours i see a drop in throughput which leads to only 2700 Txns/sec. This happens everytime i rerun the test.
In the same case, when I run the test with older version of Couchbase (4.x). I didn’t experience this type of issue and also the throughput is very consistent (~3500 txn/sec) all over the run.
Does anybody has idea on how to troubleshoot the root cause of the degradation after 2.30 hours.? It would be great if someone shed some lights on where to look at ?
There is not enough detail for us to answer why the performance dropped at the exact time in your test consistently.
But here’s my observation based on the info that you provided.
For the 1st 2.5 hours or 150 mins of the test,
on Couchbase 6.0.1, you experienced of a throughput of 4K/sec for the 1st 150 mins or a total of 36M Txns.
on Couchbase 4.x, you reported that about 3.5K/sec over this period or 31.5M Txns.
During this period (or 150 mins), it looks like Couchbase 6.0.1 actually performed faster/better than our 4.x release by 14% in your test.
The question is why the performance dropped from 4K to 2.7K after 2.5 hrs. Did the test run on a fixed number of transactions? If so, this may explain why the transaction throughput dropped after a certain point in time in your test as some users may already complete the run at the 2.5 hr mark when it’s on Couchbase 6.0.1 since it’s faster but not on 4.x.