Constant 3.0-4.0 CPU usage on Couchbase 7.0.0 Beta with miniscule data and zero load after 20-30 days

We deploy Couchbase 7.0.0 Beta nodes on top of several Kubernetes clusters, where the K8S version, OS version, and the underlying hardware is slightly different.

We use the latest CoreOS, Kubernetes version from 1.17 to 1.18 on Intel HNS2600 nodes in general.

Fresh deployments of 1 to 4 nodes clusters usually startup with 1 bucket and then tests may make Couchbase ingest a few 10s or 100s of documents. The number of buckets/scopes/collections or data volume seems to be irrelevant, in - about a handful of days of running (20-30 days) - the CPU load goes up to 3.0-4.0 CPU. This means, for all of our CB .7.0.0 Beta clusters running, irrespective of data size and server load, we pay +25% upkeep cost in terms of CPU resources, even if there are 0 operations on all of them.

We suspect that there is a resource leakage in the long run.

Hi Zoltan,

Couple of quick question to get a better idea of your setup. Are you using the Couchbase Autonomous Operator to deploy the cluster? What services are enable in the cluster and how many nodes have the services enbaled? For example, is the cluster comprised of 3 KV and Indexer nodes and a single N1QL node? Does the CPU increase stay around indefinitely after it kicks in after 20-30 days, or does the CPU go back to baseline eventually? How much RAM is being allocated to each service in your clusters? If Indexer or Analytics service is enabled, are there any indexes created?

Thanks,
Korry

Hi Korry,

We are deploying with Helm charts: In Couchbase 7.0.0 Beta Community Edition can not add bucket with replicas - #3 by zoltan.zvara

  • All services are enabled for each node, but indexer/query/data services are used only.
  • All services have at least 100% more memory than they require to be 100% resident.

There is one bucket, no one is touching it:

image

We sometimes see HTTP requests go up a bit (100-200) (tests are poking), but mostly around 0. CPU usage never drops when requests go down to 0. CPU is around 3-5 cores when the cluster gets/or does not get traffic per node until nodes are restarted. The CPU slowly crawls up in a few days.

There are about 100 indexes.

Largest indexes:

Let me know if you need more!

Thanks,
Zoltán