Creating deferred Indices Concurrently

SQL++
, ,
1

We wrote a little tool that creates all indices we need before one of our microservice starts. We use Java SDK 2.7.

We first create all indices with defer_build.

CREATE INDEX <name> ON <bucket>(<fields>) WHERE <condition> USING GSI WITH {"defer_build":true,"nodes":[<index nodes>]}]

Only after all indices are created we run the actual BUILD INDEX ... and wait for completion.

As far as I learned, this is the recommended appoarch. Nonetheless, we get a lot of errors in the CREATE INDEX step saying:

Create index cannot proceed due to rebalance in progress, another concurrent create index request, network partition, node failover, or indexer failure.

We got it working by adding a random delay and retrying the operation (sometimes up to 30 times).

Any recommendation on how to create multiple (in the order of 40) indices at the same time?

1 Like
2

That’s strange I have the same issue.

1 Like
3

Hi @johannesjasper and @gersseba. Pls see Concurrent creation of deferred indexes fails.
We have a product improvement being worked upon to address this limitation - https://issues.couchbase.com/browse/MB-32238

4

Hey @mihir.kamdar, thank you for your reply. Yes I saw the other post but I was not sure whether the ticket was with regard to deferred index creation. Thank you for revivng it.

5

Just as a follow up: we removed any kind of concurrency from our index creation process and we experience no further errors. Running the index creation in parallel and retrying on error, however, was not slower, sometimes even faster.