If I try to create indices (even deferred indices) at the same time from different threads, I am getting
com.couchbase.client.core.CouchbaseException: Error creating secondary index my_index_1: [{“msg”:“GSI CreateIndex() - cause: Create index cannot proceed due to rebalance in progress, another concurrent create index request, network partition, node failover, or indexer failure.”,“code”:5000}]
Is this the intended behaviour? At the moment I am creating indices whenever new document type is created and if two document types are created at the same time, the deferred index creation fails. It used to work in couchbase 5.1.1., does not work in couchbase 5.5. and 6.0.0.
Sample code to simulate the problem:
public class Main {
static final AtomicInteger counter = new AtomicInteger(0);
public static void main(String[] args) {
final CouchbaseEnvironment environment = DefaultCouchbaseEnvironment.builder()
.connectTimeout(10000)
.build();
final Cluster cluster = CouchbaseCluster.create(environment, "localhost");
final Bucket bucket = cluster.openBucket("bucket", "password");
final Runnable task1 = new IndexTask(bucket, "type", "Account");
final Runnable task2 = new IndexTask(bucket, "iban", "Account");
final Runnable task3 = new IndexTask(bucket, "client", "Account");
new Thread(task1).start();
new Thread(task2).start();
new Thread(task3).start();
}
static class IndexTask implements Runnable {
private Bucket bucket;
private String attributeName;
private String entityType;
IndexTask(Bucket bucket, String attributeName, String entityType) {
this.bucket = bucket;
this.attributeName = attributeName;
this.entityType = entityType;
}
@Override
public void run() {
final BucketManager bucketManager = bucket.bucketManager();
bucketManager.createN1qlIndex(
"my_index_" + counter.getAndIncrement(),
Collections.singletonList(attributeName),
Expression.x(Constants.EntityFields.ENTITY_TYPE).eq(entityType),
true,
true
);
}
}
Index creation cannot happen concurrently under any circumstance. If an index is currently being created by the indexer service, all other create indexes requests will result in this message. It is best to continuously retry the create index request until it goes through. This is the intended behaviour. Not sure why this worked in other versions. Perhaps the index creation requests where actually completing before any subsequent create index requests made it to the indexer services.
Adding some more details here. Starting 5.5, create index mechanism uses a much more advanced placement policy(than the round robin used in earlier versions) and tries to find the most optimal placement for the new index by checking the resource usage of all index service nodes in the cluster. As allowing concurrent indexes to be created could change the resource profile of the index service nodes, only one create index is allowed at a time.
We generally do not expect to see a lot of concurrent create index in the system as the total number of indexes are fairly limited in any given setup. You could either serialize the index creation in your application or retry if you get this failure.
Hi,
This isn’t so rare case, we have micro services system and run first installation in parallel, so we get into this corner since all micro service are trying to create indexes in parallel (more or less). Serialize the index creation is ineffective since it ruin the independent of micro service and retry has its limitations (retry until when?).
Thanks
To understand the use case a little more, the application is creating deferred indexes in parallel. When does it build the indexes? Is it done in batches or one by one?
The indexes are created during install/upgrade/rollback of our micro services. We have one or two indexes in each micro service and they created one by one.
Thanks @assaf.katz, I assume all micro services are using a single bucket. When you say “create”, does that mean create index in defer mode? I am trying to understand if your application builds the indexes one at a time while waiting for previous index build to finish.
Hi,
Each micro service do the following steps (with checking of existence before each step) in parallel to other micro services:
Create the bucket
Create user with permission on bucket
Create index on bucket
This means that application builds the indexes without waiting for previous index build to finish. If the micro service has more than one bucket (for cache), it runs the steps above in loop but still in parallel to other micro services.
We don’t use defer mode but it won’t really help us too much anyway, since we want to start the micro service right after the creation of index. This means that we will need to get into waiting loop in beginning of each micro service.