In Couchbase 7.0.0 Beta, indexes may stuck in building 100% state

We are adaptively creating indexes for new collections based on JVM type specifications.

We then check whether an index gets online or not. We noticed that indexes can get stuck indefinitely in “building 100%” state.

We have a total of 2 scopes and around 60 indexes total in 1 bucket, 150,000 documents, 2 nodes, RAM quota is not used up for data or index.

@zoltan.zvara,

This could be a bug in indexer code. Can you share all the indexer logs from /opt/couchbase/var/lib/couchbase/logs directory

Thanks,
Varun

@varun.velamuri thanks for helping out, are these logs contain sensitive data? Is there any way to provide your this data in private?

Our CI flushed the bucket, I will be on the lookout for these stuck building states and if I find any I will make an attempt to extract the logs.

@zoltan.zvara,

Apologies for the delay. You can collect the logs by enabling log redaction that will filter out all sensitive information. Please refer to https://docs.couchbase.com/server/7.0/manage/manage-logging/manage-logging.html for more information.

You can DM me the link to logs in the forums.

Thanks,
Varun

1 Like

@varun.velamuri thanks, let me get back to you with the logs , it may take a while!

Indexes may be stuck in a building state as follows:

While the scope G7aa0vxt also got deleted meanwhile. These indexes probably got created in parallel and got duplicated.

In this case, the indexing service stops working and the cluster requires a restart since all queries start to fail. (After restarting the cluster, the indexes with the non-existent scope are gone.)

@zoltan.zvara Would be great if you can share the logs requested by Varun?

@jeelan.poola I have already shared the logs with Varun and I got a confirmation that it was downloaded and opened successfully.

Let me know if you need anything else.

@jeelan.poola @zoltan.zvara, Apologies for the confusion. I have the logs.

@zoltan.zvara,

Apologies for the long delay. The index build has been stuck due to MB-43117 - which has been fixed after the beta release. The workaround in this case is to restart indexer.

Thanks,
Varun

1 Like

@varun.velamuri thansk, do you plan to publish couchbase-server-7.0.0-4131 or later to Docker repository?

@zoltan.zvara, We will soon be having a beta-refresh which will have this issue fixed. Now, I don’t know the exact dates as when this release can happen. Will let you know once I have any official update on this.

Thanks,
Varun