Create index via application logic

[wai.kwang.mak]

We are looking at creating index during operating hours via application logic. Currently we have a working system so far, but I have a few concerns in which I seek advice from the community.

We have a custom web server that implements XDCR protocol (much like a connector). It replicates data from a specific bucket (bucket A) that contains information of any newly introduced schema and create indexes on the fly. So far this model serves us well as it addresses both our needs of indexing based on application logic and cross region index creation. In our case, we are sure that there are no existing documents that matches the index in the bucket (bucket B) where the newly index reside. Documents that match this newly created index will only take place AFTER insertion of the document into bucket A.

Questions

1. what will happen if there are 2 of such servers attempting to create index at any 1 instance? Will one fail while the other succeed? Performance implication when this happens

2. assuming that there are million of existing documents in bucket B, that does NOT match the index criteria, what is the cost and time taken of creating index in this bucket?

3. assuming that the replication across regions does not happen in order for different buckets (any action executed between A and B will not be in order of replication, though action within 1 bucket will be). There bounds to be scenario where data is replicated in A before it happens in B. Meaning there will be data that matches the criteria of the newly created index before the application logic kicks in to create the index.

Also note that this creation of index does NOT happen often, (say once in a few days).

[keshav_m]

@wai.kwang.mak

1. what will happen if there are 2 of such servers attempting to create index at any 1 instance? Will one fail while the other succeed? Performance implication when this happens

Answer: In each couchbase instance, index name has to be unique PER BUCKET. If two of your servers try to create two indexes with the same name, one of them will get an error.
However, if two servers create indexes with different name, but on the same keys, both will get created. And Couchbase will use them both.

@wai.kwang.mak

1. assuming that there are million of existing documents in bucket B, that does NOT match the index criteria, what is the cost and time taken of creating index in this bucket?

2. assuming that the replication across regions does not happen in order for different buckets (any action executed between A and B will not be in order of replication, though action within 1 bucket will be). There bounds to be scenario where data is replicated in A before it happens in B. Meaning there will be data that matches the criteria of the newly created index before the application logic kicks in to create the index.

@jliang: Can you please answer this?

[wai.kwang.mak]

@keshav_m Thank you. Indeed we will definitely be creating them using the same name.

Can someone help me on the rest of the matters? Thank you.

[deepkaran.salooja]

@wai.kwang.mak

assuming that there are million of existing documents in bucket B, that does NOT match the index criteria, what is the cost and time taken of creating index in this bucket?

For documents that do not match the index criteria, there is a small overhead to evaluate and ignore those. But it should be significantly lower compared to a document that gets indexed.

@wai.kwang.mak

assuming that the replication across regions does not happen in order for different buckets (any action executed between A and B will not be in order of replication, though action within 1 bucket will be). There bounds to be scenario where data is replicated in A before it happens in B. Meaning there will be data that matches the criteria of the newly created index before the application logic kicks in to create the index.

This is fine. When an index is created, all the documents already in the bucket get evaluated and indexed.

[wai.kwang.mak]

@deepkaran.salooja Thank you