MAYDAY! "Lost data" after changing RAM quota


#1

Hey guys!

I changed the “Index RAM Quota” from Couchbase Server 4.5.1-2845 Community Edition settings and now all the queries don’t find any data…
When I run “SELECT id FROM myBucket” I get 4 ids… The bucket contains thousands of documents, not just 4.

If I click on “Data Buckets” > “MyBucket” > “Documents” I see all the documents, but the queries seem broken.

Any ideas?!

Thank you


#2

Changing “Index RAM Quota” would cause index service to restart. You can check in the UI -> Index tab if index service has recovered after restart and the status of all indexes is Active(and not Bootstrap).


#3

This was the first thing to check! All the indexes are Active, but have no documents inside them, and the queries seem to not “view” the documents in the database…

I created a new database, restored in it the latest backup of the original database, created all the indexes for the new db and the system seems to be working. The old database though doesn’t work properly!

But we have to find out what happened so that we avoid or bypass such incidents… And so far, we have found nothing that was to blame! :frowning:


#4

You can check from the UI stats if the number of documents in each index match with what you expect. Also check indexer.log for any errors.


#5

I have checked them already. On the primary index there are 4 documents. I tried to create a new primary index, it has all the documents. All the other indexes have 0 documents inside them! And the queries don’t work yet!

The index.log, only for today, has already 9000 lines…! Most of them are:

2018-04-18T09:48:05.358+03:00 [Warn] StorageMgr::handleCreateSnapshot Skipped Creating New Snapshot for Index blahblahblah PartitionId 0 SliceId 0. No New Mutations. IsDirty false

and

2018-04-18T09:48:05.368+03:00 [Error] Flusher::processUpsert Error indexing Key: id1 id2 etc


#6

Could you please paste the full log message related to the above error?


#7

First of all, thank you for trying to help me!

I set up a testing environment, an exact copy of the original system. So, the database contained the 2 buckets in it (the one that failed and the second that was working).

In the test env, I deleted the bucket that failed, and now the second bucket that was working is showing the same behaviour as previously… All the queries see about 29 documents from 25000+…

The indexer.log shows something like this…

2018-04-19T11:50:57.474+03:00 [Info] Indexer::ReadMemstats Time Taken 5.9995ms
2018-04-19T11:51:00.048+03:00 [Info] [Queryport “:9101”] connection “127.0.0.1:50102” doReceive() …
2018-04-19T11:51:07.478+03:00 [Info] Indexer::ReadMemstats Time Taken 999.5µs
2018-04-19T11:51:17.479+03:00 [Info] Indexer::ReadMemstats Time Taken 998.7µs
2018-04-19T11:51:27.480+03:00 [Info] Indexer::ReadMemstats Time Taken 0
2018-04-19T11:51:37.265+03:00 [Info] memstats {“Alloc”:67447952, “TotalAlloc”:1858600496, “Sys”:139360504, “Lookups”:3946, “Mallocs”:14948915,“Frees”:14888395, “HeapAlloc”:67447952, “HeapSys”:129957888, “HeapIdle”:57376768, “HeapInuse”:72581120,“HeapReleased”:9469952, “HeapObjects”:60520,“GCSys”:4286464, “LastGC”:1524127884442987100,“PauseTotalNs”:272153800, “PauseNs”:[3001300, 74943400, 1005400], “NumGC”:36}
2018-04-19T11:51:37.482+03:00 [Info] Indexer::ReadMemstats Time Taken 996.2µs
2018-04-19T11:51:46.893+03:00 [Info] connected with 1 indexers
2018-04-19T11:51:46.893+03:00 [Info] index 17659286877591686124 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 973919861808772053 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 4043332725832211836 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 6618941672646985231 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 18154954271506823711 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 4910397343005207096 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 9826060717523150543 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 10614370033160656617 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] index 10915181323075896552 has 1 replicas
2018-04-19T11:51:46.893+03:00 [Info] client load stats {}
2018-04-19T11:51:47.485+03:00 [Info] Indexer::ReadMemstats Time Taken 999.9µs
2018-04-19T11:51:49.203+03:00 [Info] PeriodicStats = {“index_not_found_errcount”:0,“indexer_state”:“Active”,“memory_quota”:1073741824,“memory_used”:83855360,“memory_used_queue”:0,“memory_used_storage”:12443648,“needs_restart”:false,“num_connections”:2,“storage_mode”:“forestdb”,“myBucket:#primary:avg_scan_latency”:0,“myBucket:#primary:avg_scan_request_alloc_latency”:0,“myBucket:#primary:avg_scan_request_init_latency”:0,“myBucket:#primary:avg_scan_request_latency”:0,“myBucket:#primary:avg_scan_wait_latency”:0,“myBucket:#primary:avg_ts_interval”:0,“myBucket:#primary:avg_ts_items_count”:0,“myBucket:#primary:build_progress”:100,“myBucket:#primary:client_cancel_errcount”:0,“myBucket:#primary:data_size”:0,“myBucket:#primary:delete_bytes”:0,“myBucket:#primary:disk_load_duration”:0,“myBucket:#primary:disk_size”:16384,“myBucket:#primary:disk_store_duration”:0,“myBucket:#primary:flush_queue_size”:0,“myBucket:#primary:frag_percent”:0,“myBucket:#primary:get_bytes”:1508,“myBucket:#primary:insert_bytes”:0,“myBucket:#primary:items_count”:29,“myBucket:#primary:not_ready_errcount”:0,“myBucket:#primary:num_commits”:0,“myBucket:#primary:num_compactions”:1,“myBucket:#primary:num_completed_requests”:1,“myBucket:#primary:num_docs_indexed”:0,“myBucket:#primary:num_docs_pending”:0,“myBucket:#primary:num_docs_processed”:78411,“myBucket:#primary:num_docs_queued”:0,“myBucket:#primary:num_flush_queued”:0,“myBucket:#primary:num_items_flushed”:0,“myBucket:#primary:num_items_restored”:0,“myBucket:#primary:num_last_snapshot_reply”:0,“myBucket:#primary:num_requests”:1,“myBucket:#primary:num_rows_returned”:29,“myBucket:#primary:num_snapshot_waiters”:0,“myBucket:#primary:num_snapshots”:0,“myBucket:#primary:scan_bytes_read”:754,“myBucket:#primary:scan_wait_duration”:0,“myBucket:#primary:since_last_snapshot”:0,“myBucket:#primary:timings/dcp_getseqs”:“0 0 0”,“myBucket:#primary:timings/scan_pipeline_iterate”:“1 0 0”,“myBucket:#primary:timings/storage_clone_handle”:“1 0 0”,“myBucket:#primary:timings/storage_commit”:“0 0 0”,“myBucket:#primary:timings/storage_del”:“0 0 0”,“myBucket:#primary:timings/storage_get”:“0 0 0”,“myBucket:#primary:timings/storage_info”:“1 0 0”,“myBucket:#primary:timings/storage_iterator_next”:“29 0 0”,“myBucket:#primary:timings/storage_meta_get”:“3 3960300 5874994490000”,“myBucket:#primary:timings/storage_meta_set”:“0 0 0”,“myBucket:#primary:timings/storage_new_iterator”:“1 0 0”,“myBucket:#primary:timings/storage_persist_snapshot_create”:“1 999400 998800360000”,“myBucket:#primary:timings/storage_set”:“0 0 0”,“myBucket:#primary:timings/storage_snapshot_close”:“0 0 0”,“myBucket:#primary:timings/storage_snapshot_create”:“0 0 0”,“myBucket:#primary:total_scan_duration”:0,“myBucket:#primary:total_scan_request_duration”:0,“myBucket:someIndex2:avg_scan_latency”:0,“myBucket:someIndex2:avg_scan_request_alloc_latency”:0,“myBucket:someIndex2:avg_scan_request_init_latency”:0,“myBucket:someIndex2:avg_scan_request_latency”:0,“myBucket:someIndex2:avg_scan_wait_latency”:0,“myBucket:someIndex2:avg_ts_interval”:0,“myBucket:someIndex2:avg_ts_items_count”:0,“myBucket:someIndex2:build_progress”:100,“myBucket:someIndex2:client_cancel_errcount”:0,“myBucket:someIndex2:data_size”:147628,“myBucket:someIndex2:delete_bytes”:0,“myBucket:someIndex2:disk_load_duration”:0,“myBucket:someIndex2:disk_size”:5189632,“myBucket:someIndex2:disk_store_duration”:0,“myBucket:someIndex2:flush_queue_size”:0,“myBucket:someIndex2:frag_percent”:97,“myBucket:someIndex2:get_bytes”:0,“myBucket:someIndex2:insert_bytes”:0,“myBucket:someIndex2:items_count”:0,“myBucket:someIndex2:not_ready_errcount”:0,“myBucket:someIndex2:num_commits”:0,“myBucket:someIndex2:num_compactions”:1,“myBucket:someIndex2:num_completed_requests”:0,“myBucket:someIndex2:num_docs_indexed”:0,“myBucket:someIndex2:num_docs_pending”:0,“myBucket:someIndex2:num_docs_processed”:78411,“myBucket:someIndex2:num_docs_queued”:0,“myBucket:someIndex2:num_flush_queued”:0,“myBucket:someIndex2:num_items_flushed”:0,“myBucket:someIndex2:num_items_restored”:0,“myBucket:someIndex2:num_last_snapshot_reply”:0,“myBucket:someIndex2:num_requests”:0,“myBucket:someIndex2:num_rows_returned”:0,“myBucket:someIndex2:num_snapshot_waiters”:0,“myBucket:someIndex2:num_snapshots”:0,“myBucket:someIndex2:scan_bytes_read”:0,“myBucket:someIndex2:scan_wait_duration”:0,“myBucket:someIndex2:since_last_snapshot”:0,“myBucket:someIndex2:timings/dcp_getseqs”:“0 0 0”,“myBucket:someIndex2:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:someIndex2:timings/storage_clone_handle”:“0 0 0”,“myBucket:someIndex2:timings/storage_commit”:“0 0 0”,“myBucket:someIndex2:timings/storage_del”:“0 0 0”,“myBucket:someIndex2:timings/storage_get”:“0 0 0”,“myBucket:someIndex2:timings/storage_info”:“1 0 0”,“myBucket:someIndex2:timings/storage_iterator_next”:“0 0 0”,“myBucket:someIndex2:timings/storage_meta_get”:“3 1986300 1972826690000”,“myBucket:someIndex2:timings/storage_meta_set”:“0 0 0”,“myBucket:someIndex2:timings/storage_new_iterator”:“0 0 0”,“myBucket:someIndex2:timings/storage_persist_snapshot_create”:“1 0 0”,“myBucket:someIndex2:timings/storage_set”:“0 0 0”,“myBucket:someIndex2:timings/storage_snapshot_close”:“0 0 0”,“myBucket:someIndex2:timings/storage_snapshot_create”:“0 0 0”,“myBucket:someIndex2:total_scan_duration”:0,“myBucket:someIndex2:total_scan_request_duration”:0,“myBucket:someIndex:avg_scan_latency”:0,“myBucket:someIndex:avg_scan_request_alloc_latency”:0,“myBucket:someIndex:avg_scan_request_init_latency”:0,“myBucket:someIndex:avg_scan_request_latency”:0,“myBucket:someIndex:avg_scan_wait_latency”:0,“myBucket:someIndex:avg_ts_interval”:0,“myBucket:someIndex:avg_ts_items_count”:0,“myBucket:someIndex:build_progress”:100,“myBucket:someIndex:client_cancel_errcount”:0,“myBucket:someIndex:data_size”:147608,“myBucket:someIndex:delete_bytes”:0,“myBucket:someIndex:disk_load_duration”:0,“myBucket:someIndex:disk_size”:5189632,“myBucket:someIndex:disk_store_duration”:0,“myBucket:someIndex:flush_queue_size”:0,“myBucket:someIndex:frag_percent”:97,“myBucket:someIndex:get_bytes”:0,“myBucket:someIndex:insert_bytes”:0,“myBucket:someIndex:items_count”:0,“myBucket:someIndex:not_ready_errcount”:0,“myBucket:someIndex:num_commits”:0,“myBucket:someIndex:num_compactions”:1,“myBucket:someIndex:num_completed_requests”:0,“myBucket:someIndex:num_docs_indexed”:0,“myBucket:someIndex:num_docs_pending”:0,“myBucket:someIndex:num_docs_processed”:78411,“myBucket:someIndex:num_docs_queued”:0,“myBucket:someIndex:num_flush_queued”:0,“myBucket:someIndex:num_items_flushed”:0,“myBucket:someIndex:num_items_restored”:0,“myBucket:someIndex:num_last_snapshot_reply”:0,“myBucket:someIndex:num_requests”:0,“myBucket:someIndex:num_rows_returned”:0,“myBucket:someIndex:num_snapshot_waiters”:0,“myBucket:someIndex:num_snapshots”:0,“myBucket:someIndex:scan_bytes_read”:0,“myBucket:someIndex:scan_wait_duration”:0,“myBucket:someIndex:since_last_snapshot”:0,“myBucket:someIndex:timings/dcp_getseqs”:“0 0 0”,“myBucket:someIndex:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:someIndex:timings/storage_clone_handle”:“0 0 0”,“myBucket:someIndex:timings/storage_commit”:“0 0 0”,“myBucket:someIndex:timings/storage_del”:“0 0 0”,“myBucket:someIndex:timings/storage_get”:“0 0 0”,“myBucket:someIndex:timings/storage_info”:“1 0 0”,“myBucket:someIndex:timings/storage_iterator_next”:“0 0 0”,“myBucket:someIndex:timings/storage_meta_get”:“3 3882500 5575159570000”,“myBucket:someIndex:timings/storage_meta_set”:"
0 0 0",“myBucket:someIndex:timings/storage_new_iterator”:“0 0 0”,“myBucket:someIndex:timings/storage_persist_snapshot_create”:“1 0 0”,“myBucket:someIndex:timings/storage_set”:“0 0 0”,“myBucket:someIndex:timings/storage_snapshot_close”:“0 0 0”,“myBucket:someIndex:timings/storage_snapshot_create”:“0 0 0”,“myBucket:someIndex:total_scan_duration”:0,“myBucket:someIndex:total_scan_request_duration”:0,“myBucket:someIndex3:avg_scan_latency”:0,“myBucket:someIndex3:avg_scan_request_alloc_latency”:0,“myBucket:someIndex3:avg_scan_request_init_latency”:0,“myBucket:someIndex3:avg_scan_request_latency”:0,“myBucket:someIndex3:avg_scan_wait_latency”:0,“myBucket:someIndex3:avg_ts_interval”:0,“myBucket:someIndex3:avg_ts_items_count”:0,“myBucket:someIndex3:build_progress”:100,“myBucket:someIndex3:client_cancel_errcount”:0,“myBucket:someIndex3:data_size”:0,“myBucket:someIndex3:delete_bytes”:0,“myBucket:someIndex3:disk_load_duration”:0,“myBucket:someIndex3:disk_size”:16384,“myBucket:someIndex3:disk_store_duration”:0,“myBucket:someIndex3:flush_queue_size”:0,“myBucket:someIndex3:frag_percent”:0,“myBucket:someIndex3:get_bytes”:0,“myBucket:someIndex3:insert_bytes”:0,“myBucket:someIndex3:items_count”:29,“myBucket:someIndex3:not_ready_errcount”:0,“myBucket:someIndex3:num_commits”:0,“myBucket:someIndex3:num_compactions”:1,“myBucket:someIndex3:num_completed_requests”:0,“myBucket:someIndex3:num_docs_indexed”:0,“myBucket:someIndex3:num_docs_pending”:0,“myBucket:someIndex3:num_docs_processed”:78411,“myBucket:someIndex3:num_docs_queued”:0,“myBucket:someIndex3:num_flush_queued”:0,“myBucket:someIndex3:num_items_flushed”:0,“myBucket:someIndex3:num_items_restored”:0,“myBucket:someIndex3:num_last_snapshot_reply”:0,“myBucket:someIndex3:num_requests”:0,“myBucket:someIndex3:num_rows_returned”:0,“myBucket:someIndex3:num_snapshot_waiters”:0,“myBucket:someIndex3:num_snapshots”:0,“myBucket:someIndex3:scan_bytes_read”:0,“myBucket:someIndex3:scan_wait_duration”:0,“myBucket:someIndex3:since_last_snapshot”:0,“myBucket:someIndex3:timings/dcp_getseqs”:“0 0 0”,“myBucket:someIndex3:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:someIndex3:timings/storage_clone_handle”:“0 0 0”,“myBucket:someIndex3:timings/storage_commit”:“0 0 0”,“myBucket:someIndex3:timings/storage_del”:“0 0 0”,“myBucket:someIndex3:timings/storage_get”:“0 0 0”,“myBucket:someIndex3:timings/storage_info”:“1 0 0”,“myBucket:someIndex3:timings/storage_iterator_next”:“0 0 0”,“myBucket:someIndex3:timings/storage_meta_get”:“3 49982900 2402167773050000”,“myBucket:someIndex3:timings/storage_meta_set”:“0 0 0”,“myBucket:someIndex3:timings/storage_new_iterator”:“0 0 0”,“myBucket:someIndex3:timings/storage_persist_snapshot_create”:“1 0 0”,“myBucket:someIndex3:timings/storage_set”:“0 0 0”,“myBucket:someIndex3:timings/storage_snapshot_close”:“0 0 0”,“myBucket:someIndex3:timings/storage_snapshot_create”:“0 0 0”,“myBucket:someIndex3:total_scan_duration”:0,“myBucket:someIndex3:total_scan_request_duration”:0,“myBucket:someIndex4:avg_scan_latency”:0,“myBucket:someIndex4:avg_scan_request_alloc_latency”:0,“myBucket:someIndex4:avg_scan_request_init_latency”:0,“myBucket:someIndex4:avg_scan_request_latency”:0,“myBucket:someIndex4:avg_scan_wait_latency”:0,“myBucket:someIndex4:avg_ts_interval”:0,“myBucket:someIndex4:avg_ts_items_count”:0,“myBucket:someIndex4:build_progress”:100,“myBucket:someIndex4:client_cancel_errcount”:0,“myBucket:someIndex4:data_size”:293397,“myBucket:someIndex4:delete_bytes”:0,“myBucket:someIndex4:disk_load_duration”:0,“myBucket:someIndex4:disk_size”:9691136,“myBucket:someIndex4:disk_store_duration”:0,“myBucket:someIndex4:flush_queue_size”:0,“myBucket:someIndex4:frag_percent”:96,“myBucket:someIndex4:get_bytes”:0,“myBucket:someIndex4:insert_bytes”:0,“myBucket:someIndex4:items_count”:32,“myBucket:someIndex4:not_ready_errcount”:0,“myBucket:someIndex4:num_commits”:0,“myBucket:someIndex4:num_compactions”:1,“myBucket:someIndex4:num_completed_requests”:0,“myBucket:someIndex4:num_docs_indexed”:0,“myBucket:someIndex4:num_docs_pending”:0,“myBucket:someIndex4:num_docs_processed”:78411,“myBucket:someIndex4:num_docs_queued”:0,“myBucket:someIndex4:num_flush_queued”:0,“myBucket:someIndex4:num_items_flushed”:0,“myBucket:someIndex4:num_items_restored”:0,“myBucket:someIndex4:num_last_snapshot_reply”:0,“myBucket:someIndex4:num_requests”:0,“myBucket:someIndex4:num_rows_returned”:0,“myBucket:someIndex4:num_snapshot_waiters”:0,“myBucket:someIndex4:num_snapshots”:0,“myBucket:someIndex4:scan_bytes_read”:0,“myBucket:someIndex4:scan_wait_duration”:0,“myBucket:someIndex4:since_last_snapshot”:0,“myBucket:someIndex4:timings/dcp_getseqs”:“0 0 0”,“myBucket:someIndex4:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:someIndex4:timings/storage_clone_handle”:“0 0 0”,“myBucket:someIndex4:timings/storage_commit”:“0 0 0”,“myBucket:someIndex4:timings/storage_del”:“0 0 0”,“myBucket:someIndex4:timings/storage_get”:“0 0 0”,“myBucket:someIndex4:timings/storage_info”:“1 0 0”,“myBucket:someIndex4:timings/storage_iterator_next”:“0 0 0”,“myBucket:someIndex4:timings/storage_meta_get”:“3 100018800 9802900226720000”,“myBucket:someIndex4:timings/storage_meta_set”:“0 0 0”,“myBucket:someIndex4:timings/storage_new_iterator”:“0 0 0”,“myBucket:someIndex4:timings/storage_persist_snapshot_create”:“1 0 0”,“myBucket:someIndex4:timings/storage_set”:“0 0 0”,“myBucket:someIndex4:timings/storage_snapshot_close”:“0 0 0”,“myBucket:someIndex4:timings/storage_snapshot_create”:“0 0 0”,“myBucket:someIndex4:total_scan_duration”:0,“myBucket:someIndex4:total_scan_request_duration”:0,“myBucket:idIndex:avg_scan_latency”:0,“myBucket:idIndex:avg_scan_request_alloc_latency”:0,“myBucket:idIndex:avg_scan_request_init_latency”:0,“myBucket:idIndex:avg_scan_request_latency”:0,“myBucket:idIndex:avg_scan_wait_latency”:0,“myBucket:idIndex:avg_ts_interval”:0,“myBucket:idIndex:avg_ts_items_count”:0,“myBucket:idIndex:build_progress”:100,“myBucket:idIndex:client_cancel_errcount”:0,“ticketing
2:idIndex:data_size”:0,“myBucket:idIndex:delete_bytes”:0,“myBucket:idIndex:disk_load_duration”:0,“myBucket:idIndex:disk_size”:16384,“myBucket:idIndex:disk_store_duration”:0,“myBucket:idIndex:flush_queue_size”:0,“myBucket:idIndex:frag_percent”:0,“myBucket:idIndex:get_bytes”:0,“myBucket:idIndex:insert_bytes”:0,“myBucket:idIndex:items_count”:25915,“myBucket:idIndex:not_ready_errcount”:0,“myBucket:idIndex:num_commits”:0,“myBucket:idIndex:num_compactions”:1,“myBucket:idIndex:num_completed_requests”:0,“myBucket:idIndex:num_docs_indexed”:0,“myBucket:idIndex:num_docs_pending”:0,“myBucket:idIndex:num_docs_processed”:78411,“myBucket:idIndex:num_docs_queued”:0,“myBucket:idIndex:num_flush_queued”:0,“myBucket:idIndex:num_items_flushed”:0,“myBucket:idIndex:num_items_restored”:0,“myBucket:idIndex:num_last_snapshot_reply”:0,“myBucket:idIndex:num_requests”:0,“myBucket:idIndex:num_rows_returned”:0,“myBucket:idIndex:num_snapshot_waiters”:0,“myBucket:idIndex:num_snapshots”:0,“myBucket:idIndex:scan_bytes_read”:0,“myBucket:idIndex:scan_wait_duration”:0,“myBucket:idIndex:since_last_snapshot”:0,“myBucket:idIndex:timings/dcp_getseqs”:“0 0 0”,“myBucket:idIndex:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:idIndex:timings/storage_clone_handle”:“0 0 0”,“myBucket:idIndex:timings/storage_commit”:“0 0 0”,“myBucket:idIndex:timings/storage_del”:“0 0 0”,“myBucket:idIndex:timings/storage_get”:“0 0 0”,“myBucket:idIndex:timings/storage_info”:“1 0 0”,“myBucket:idIndex:timings/storage_iterator_next”:“0 0 0”,“myBucket:idIndex:timings/storage_meta_get”:“3 122003700 14643327433490000”,“myBucket:idIndex:timings/storage_meta_set”:“0 0 0”,“myBucket:idIndex:timings/storage_new_iterator”:“0 0 0”,“myBucket:idIndex:timings/storage_persist_snapshot_create”:“1 0 0”,“myBucket:idIndex:timings/storage_set”:“0 0 0”,“myBucket:idIndex:timings/storage_snapshot_close”:“0 0 0”,“myBucket:idIndex:timings/storage_snapshot_create”:“0 0 0”,“myBucket:idIndex:total_scan_duration”:0,“myBucket:idIndex:total_scan_request_duration”:0,“myBucket:mutation_queue_size”:0,“myBucket:num_mutations_queued”:0,“myBucket:num_nonalign_ts”:0,“myBucket:someIndex5:avg_scan_latency”:0,“myBucket:someIndex5:avg_scan_request_alloc_latency”:0,“myBucket:someIndex5:avg_scan_request_init_latency”:0,“myBucket:someIndex5:avg_scan_request_latency”:0,“myBucket:someIndex5:avg_scan_wait_latency”:0,“myBucket:someIndex5:avg_ts_interval”:0,“myBucket:someIndex5:avg_ts_items_count”:0,“myBucket:someIndex5:build_progress”:100,“myBucket:someIndex5:client_cancel_errcount”:0,“myBucket:someIndex5:data_size”:2159637,“myBucket:someIndex5:delete_bytes”:0,“myBucket:someIndex5:disk_load_duration”:0,“myBucket:someIndex5:disk_size”:7204864,“myBucket:someIndex5:disk_store_duration”:0,“myBucket:someIndex5:flush_queue_size”:0,“myBucket:someIndex5:frag_percent”:70,“myBucket:someIndex5:get_bytes”:0,“myBucket:someIndex5:insert_bytes”:0,“myBucket:someIndex5:items_count”:5296,“myBucket:someIndex5:not_ready_errcount”:0,“myBucket:someIndex5:num_commits”:0,“myBucket:someIndex5:num_compactions”:1,“myBucket:someIndex5:num_completed_requests”:0,“myBucket:someIndex5:num_docs_indexed”:0,“myBucket:someIndex5:num_docs_pending”:0,“myBucket:someIndex5:num_docs_processed”:78411,“myBucket:someIndex5:num_docs_queued”:0,“myBucket:someIndex5:num_flush_queued”:0,“myBucket:someIndex5:num_items_flushed”:0,“myBucket:someIndex5:num_items_restored”:0,“myBucket:someIndex5:num_last_snapshot_reply”:0,“myBucket:someIndex5:num_requests”:0,“myBucket:someIndex5:num_rows_returned”:0,“myBucket:someIndex5:num_snapshot_waiters”:0,“myBucket:someIndex5:num_snapshots”:0,“myBucket:someIndex5:scan_bytes_read”:0,“myBucket:someIndex5:scan_wait_duration”:0,“myBucket:someIndex5:since_last_snapshot”:0,“myBucket:someIndex5:timings/dcp_getseqs”:“0 0 0”,“myBucket:someIndex5:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:someIndex5:timings/storage_clone_handle”:“0 0 0”,“myBucket:someIndex5:timings/storage_commit”:“0 0 0”,“myBucket:someIndex5:timings/storage_del”:“0 0 0”,“myBucket:someIndex5:timings/storage_get”:“0 0 0”,“myBucket:someIndex5:timings/storage_info”:“1 0 0”,“myBucket:someIndex5:timings/storage_iterator_next”:“0 0 0”,“myBucket:someIndex5:timings/storage_meta_get”:“3 158010500 24340749081090000”,“myBucket:someIndex5:timings/storage_meta_set”:“0 0 0”,“myBucket:someIndex5:timings/storage_new_iterator”:“0 0 0”,“myBucket:someIndex5:timings/storage_persist_snapshot_create”:“1 0 0”,“myBucket:someIndex5:timings/storage_set”:“0 0 0”,“myBucket:someIndex5:timings/storage_snapshot_close”:“0 0 0”,“myBucket:someIndex5:timings/storage_snapshot_create”:“0 0 0”,“myBucket:someIndex5:total_scan_duration”:0,“myBucket:someIndex5:total_scan_request_duration”:0,“myBucket:someIndex6:avg_scan_latency”:0,“myBucket:someIndex6:avg_scan_request_alloc_latency”:0,“myBucket:someIndex6:avg_scan_request_init_latency”:0,“myBucket:someIndex6:avg_scan_request_latency”:0,“myBucket:someIndex6:avg_scan_wait_latency”:0,“myBucket:someIndex6:avg_ts_interval”:0,“myBucket:someIndex6:avg_ts_items_count”:0,“myBucket:someIndex6:build_progress”:100,“myBucket:someIndex6:client_cancel_errcount”:0,“myBucket:someIndex6:data_size”:2094241,“myBucket:someIndex6:delete_bytes”:0,“myBucket:someIndex6:disk_load_duration”:0,“myBucket:someIndex6:disk_size”:19783680,“myBucket:someIndex6:disk_store_duration”:0,“myBucket:someIndex6:flush_queue_size”:0,“myBucket:someIndex6:frag_percent”:89,“myBucket:someIndex6:get_bytes”:0,“myBucket:someIndex6:insert_bytes”:0,“myBucket:someIndex6:items_count”:5133,“myBucket:someIndex6:not_ready_errcount”:0,“myBucket:someIndex6:num_commits”:0,“myBucket:someIndex6:num_compactions”:1,“myBucket:someIndex6:num_completed_requests”:0,“myBucket:someIndex6:num_docs_indexed”:0,“myBucket:someIndex6:num_docs_pending”:0,“myBucket:someIndex6:num_docs_processed”:78411,“myBucket:someIndex6:num_docs_queued”:0,“myBucket:someIndex6:num_flush_queued”:0,“myBucket:someIndex6:num_items_flushed”:0,“myBucket:someIndex6:num_items_restored”:0,“myBucket:someIndex6:num_last_snapshot_reply”:0,“myBucket:someIndex6:num_requests”:0,“myBucket:someIndex6:num_rows_returned”:0,“myBucket:someIndex6:num_snapshot_waiters”:0,“myBucket:someIndex6:num_snapshots”:0,“myBucket:someIndex6:scan_bytes_read”:0,“myBucket:someIndex6:scan_wait_duration”:0,“myBucket:someIndex6:since_last_snapshot”:0,“myBucket:someIndex6:timings/dcp_getseqs”:“0 0 0”,“myBucket:someIndex6:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:someIndex6:timings/storage_clone_handle”:“0 0 0”,“myBucket:someIndex6:timings/storage_commit”:“0 0 0”,“myBucket:someIndex6:timings/storage_del”:“0 0 0”,“myBucket:someIndex6:timings/storage_get”:“0 0 0”,“myBucket:someIndex6:timings/storage_info”:“1 0 0”,“myBucket:someIndex6:timings/storage_iterator_next”:“0 0 0”,“myBucket:someIndex6:timings/storage_meta_get”:“3 221972200 48406764081740000”,“myBucket:someIndex6:timings/storage_meta_set”:“0 0 0”,“myBucket:someIndex6:timings/storage_new_iterator”:“0 0 0”,“myBucket:someIndex6:timings/storage_persist_snapshot_create”:“1 0 0”,“myBucket:someIndex6:timings/storage_set”:“0 0 0”,“myBucket:someIndex6:timings/storage_snapshot_close”:“0 0 0”,“myBucket:someIndex6:timings/storage_snapshot_create”:“0 0 0”,“myBucket:someIndex6:total_scan_duration”:0,“myBucket:someIndex6:total_scan_request_duration”:0,“myBucket:timings/dcp_getseqs”:“134 375826400 6648888625080000”,“myBucket:ts_queue_size”:0,“myBucket:typeIndex:avg_scan_latency”:470631,“myBucket:typeIndex:avg_scan_request_alloc_latency”:0,“myBucket:typeIndex:avg_scan_request_init_latenc
y”:366857,“myBucket:typeIndex:avg_scan_request_latency”:837489,“myBucket:typeIndex:avg_scan_wait_latency”:0,“myBucket:typeIndex:avg_ts_interval”:0,“myBucket:typeIndex:avg_ts_items_count”:0,“myBucket:typeIndex:build_progress”:100,“myBucket:typeIndex:client_cancel_errcount”:0,“myBucket:typeIndex:data_size”:0,“myBucket:typeIndex:delete_bytes”:0,“myBucket:typeIndex:disk_load_duration”:0,“myBucket:typeIndex:disk_size”:53248,“myBucket:typeIndex:disk_store_duration”:0,“myBucket:typeIndex:flush_queue_size”:0,“myBucket:typeIndex:frag_percent”:0,“myBucket:typeIndex:get_bytes”:0,“myBucket:typeIndex:insert_bytes”:0,“myBucket:typeIndex:items_count”:0,“myBucket:typeIndex:not_ready_errcount”:0,“myBucket:typeIndex:num_commits”:0,“myBucket:typeIndex:num_compactions”:1,“myBucket:typeIndex:num_completed_requests”:19,“myBucket:typeIndex:num_docs_indexed”:0,“myBucket:typeIndex:num_docs_pending”:0,“myBucket:typeIndex:num_docs_processed”:78411,“myBucket:typeIndex:num_docs_queued”:0,“myBucket:typeIndex:num_flush_queued”:0,“myBucket:typeIndex:num_items_flushed”:0,“myBucket:typeIndex:num_items_restored”:0,“myBucket:typeIndex:num_last_snapshot_reply”:0,“myBucket:typeIndex:num_requests”:19,“myBucket:typeIndex:num_rows_returned”:0,“myBucket:typeIndex:num_snapshot_waiters”:0,“myBucket:typeIndex:num_snapshots”:0,“myBucket:typeIndex:scan_bytes_read”:0,“myBucket:typeIndex:scan_wait_duration”:0,“myBucket:typeIndex:since_last_snapshot”:0,“myBucket:typeIndex:timings/dcp_getseqs”:“0 0 0”,“myBucket:typeIndex:timings/scan_pipeline_iterate”:“0 0 0”,“myBucket:typeIndex:timings/storage_clone_handle”:“0 0 0”,“myBucket:typeIndex:timings/storage_commit”:“0 0 0”,“myBucket:typeIndex:timings/storage_del”:“0 0 0”,“myBucket:typeIndex:timings/storage_get”:“0 0 0”,“myBucket:typeIndex:timings/storage_info”:“1 0 0”,“myBucket:typeIndex:timings/storage_iterator_next”:“0 0 0”,“myBucket:typeIndex:timings/storage_meta_get”:“0 0 0”,“myBucket:typeIndex:timings/storage_meta_set”:“0 0 0”,“myBucket:typeIndex:timings/storage_new_iterator”:“0 0 0”,“myBucket:typeIndex:timings/storage_persist_snapshot_create”:“0 0 0”,“myBucket:typeIndex:timings/storage_set”:“0 0 0”,“myBucket:typeIndex:timings/storage_snapshot_close”:“0 0 0”,“myBucket:typeIndex:timings/storage_snapshot_create”:“0 0 0”,“myBucket:typeIndex:total_scan_duration”:8942000,“myBucket:typeIndex:total_scan_request_duration”:15912300,“timings/stats_response”:“863 2393358200 19556971015140000”,“uptime”:“12m4.7405013s”}
2018-04-19T11:51:57.486+03:00 [Info] Indexer::ReadMemstats Time Taken 999.9µs


#8

There are some weird errors in the logs:

Error status code: -1, Error in REMOVE on a database file ‘c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\ticketing2
_seatIdIndex_4910397343005207096_0.index\data.fdb.2’, errno = 32: 'The process cannot access the file because it is being used by another process.

Did this system run out of disk space before/at the time you observed the issue?

Regarding the other error:
2018-04-18T09:48:05.368+03:00 [Error] Flusher::processUpsert Error indexing Key: id1 id2 etc

This happens if your secondary key in array index is too large. You can read this post for more information:


#9

I don’t think that the system run out of disk space at that time, I’m not sure though. :frowning:

As for the max_array_seckey_size post, I will check it out! Thanks!