Concurrent creation of deferred indexes fails

index

#1

If I try to create indices (even deferred indices) at the same time from different threads, I am getting

com.couchbase.client.core.CouchbaseException: Error creating secondary index my_index_1: [{“msg”:“GSI CreateIndex() - cause: Create index cannot proceed due to rebalance in progress, another concurrent create index request, network partition, node failover, or indexer failure.”,“code”:5000}]

Is this the intended behaviour? At the moment I am creating indices whenever new document type is created and if two document types are created at the same time, the deferred index creation fails. It used to work in couchbase 5.1.1., does not work in couchbase 5.5. and 6.0.0.

Sample code to simulate the problem:
public class Main {

static final AtomicInteger counter = new AtomicInteger(0);

public static void main(String[] args) {
    final CouchbaseEnvironment environment = DefaultCouchbaseEnvironment.builder()
            .connectTimeout(10000)
            .build();

    final Cluster cluster = CouchbaseCluster.create(environment, "localhost");
    final Bucket bucket = cluster.openBucket("bucket", "password");


    final Runnable task1 = new IndexTask(bucket, "type", "Account");
    final Runnable task2 = new IndexTask(bucket, "iban", "Account");
    final Runnable task3 = new IndexTask(bucket, "client", "Account");

    new Thread(task1).start();
    new Thread(task2).start();
    new Thread(task3).start();
}


static class IndexTask implements Runnable {

    private Bucket bucket;

    private String attributeName;

    private String entityType;

    IndexTask(Bucket bucket, String attributeName, String entityType) {
        this.bucket = bucket;
        this.attributeName = attributeName;
        this.entityType = entityType;
    }

    @Override
    public void run() {
        final BucketManager bucketManager = bucket.bucketManager();
        bucketManager.createN1qlIndex(
                "my_index_" + counter.getAndIncrement(),
                Collections.singletonList(attributeName),
                Expression.x(Constants.EntityFields.ENTITY_TYPE).eq(entityType),
                true,
                true
        );
    }
}

}


#3

Hi @yansplichal,

Index creation cannot happen concurrently under any circumstance. If an index is currently being created by the indexer service, all other create indexes requests will result in this message. It is best to continuously retry the create index request until it goes through. This is the intended behaviour. Not sure why this worked in other versions. Perhaps the index creation requests where actually completing before any subsequent create index requests made it to the indexer services.

–Korry


#4

Adding some more details here. Starting 5.5, create index mechanism uses a much more advanced placement policy(than the round robin used in earlier versions) and tries to find the most optimal placement for the new index by checking the resource usage of all index service nodes in the cluster. As allowing concurrent indexes to be created could change the resource profile of the index service nodes, only one create index is allowed at a time.

We generally do not expect to see a lot of concurrent create index in the system as the total number of indexes are fairly limited in any given setup. You could either serialize the index creation in your application or retry if you get this failure.