Indexing by flavor (nested documents) very slow

Couchbase Server
1

Hi,
We have been trying to index around 100M documents formatted with a flavor, see below. We found it very very slow, both on cb 4.1.1 and 4.5.0.

{
 "flavorType1": {
     "field1": "data1", 
     "field2": "data2",
     "field3": "data3",
     "filed4": "data4",
     "field5": "data5",
  }
}

CREATE INDEX flavorType1-index-1 ON default(flavorType1.field3);

Is there any known problem with this flavor-based format? Should we use the “type:” attribute instead?

Thanks and regards.

2

Hi jfrodriguez,
Pls clarify some more details. What is the document size? how long it takes? Have you tried memory optimized indexes option in 4.5? How many nodes in the cluster (data, index)? Each node memory/cpu etc configuration??

thanks,
-Prasad

3

Hi Prasad,
thanks for your reply!

Details about our environment:

  • 100M documents
  • 1.5K per item
  • 35-40 fields per item
  • 4 nodes 8cpu/56GB (data, query, index)
  • CB 4.5 using GSI

Thanks,
JF

4

how long is it taking? And how many nodes (data & index) in the cluster? thanks.

5

4 nodes: 8cpu + 56GB ram each one
It takes about 18 hours with low drain rates (1k per second)

Thanks!