I am using couchbase community edition 7.0.1 build 6102 version and trying to do a performance test on 3 node cluster.
Memory quota are allocated as follows
Data Service : 8851 MB
Index Service : 4096 MB
Search Service : 512 MB
I started execution to insert 300 million records in DB but after 123 million record insertion , I am getting below error and not able to insert any record further.
Exception in thread "main" com.couchbase.client.core.error.AmbiguousTimeoutException: InsertRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0x32ae907d00000001","idempotent":false,"lastChannelId":"32AE907D00000001/000000008EBB1962","lastDispatchedFrom":"10.236.172.147:59321","lastDispatchedTo":"xxxxxxxxxxxxxxxxxx:11210","reason":"TIMEOUT","requestId":112,"requestType":"InsertRequest","retried":12,"retryReasons":["KV_TEMPORARY_FAILURE","COLLECTION_MAP_REFRESH_IN_PROGRESS"],"service":{"bucket":"jcache","collection":"RecordDup","documentId":"volte#1632975655638#10000000000","opaque":"0x30","scope":"Mediation","type":"kv"},"timeoutMs":2500,"timings":{"dispatchMicros":265217,"encodingMicros":9456,"totalDispatchMicros":799452,"totalMicros":2505195}}
at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51)
at com.couchbase.client.java.Collection.insert(Collection.java:404)
I am attaching screen shot of cluster cpu, ram and disk size.
Please suggest what is the root cause of this error. I am expecting that couchbase should evict data from memory as it keeps on getting insert request.