Hello all,
I installed couchbase 4.1.0-5005 Community Edition (build-5005). First thing I did was to create a bucket, then immediately created an index:
create PRIMARY INDEX parse-primary-index ON parse USING GSI;
I then enabled xdcr from another cluster to sync data onto it. I now have 3681135 items in my bucket.
I did a small healthcheck. Seems like the node that contain the index is having a problem.
The CPU consumption of the indexer process seems to be in constant use.
Also, I have the following errors in the indexer.log:
2016-04-18T14:31:56.966+00:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 13813219721503989217 Data:387695753, Disk:754397184, ExtraSnapshotData:0, Fragmentation:48.60853656102725%
2016-04-18T14:31:56.966+00:00 [Info] CompactionDaemon: Compacting index instance:13813219721503989217
2016-04-18T14:31:57.064+00:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 2823981. Slice Id 0, IndexInstId 13813219721503989217, IndexDefnId 13813219721503989217
2016-04-18T14:31:57.074+00:00 [Error] ForestDB Error (data.fdb.0/default) errcode: -1, msg: Error in REMOVE on a database file ‘/index/@2i/parse_parse-primary-index_13813219721503989217_0.index/data.fdb.8’, errno = 2: 'No such file or directory’
2016-04-18T14:32:12.457+00:00 [Info] PeriodicStats = {“memory_quota”:4194304000,“memory_used”:1190699008,“needs_restart”:false,“num_connections”:1,“parse:mutation_queue_size”:0,“parse:num_mutations_queued”:3681135,“parse:num_nonalign_ts”:13,“parse:parse-primary-index:avg_scan_latency”:3448521,“parse:parse-primary-index:avg_scan_wait_latency”:6993,“parse:parse-primary-index:avg_ts_interval”:203744845,“parse:parse-primary-index:avg_ts_items_count”:6915,“parse:parse-primary-index:build_progress”:100,“parse:parse-primary-index:data_size”:387695753,“parse:parse-primary-index:delete_bytes”:0,“parse:parse-primary-index:disk_size”:754405376,“parse:parse-primary-index:flush_queue_size”:0,“parse:parse-primary-index:frag_percent”:48,“parse:parse-primary-index:get_bytes”:20000,“parse:parse-primary-index:insert_bytes”:36811350,“parse:parse-primary-index:items_count”:3584991,“parse:parse-primary-index:num_commits”:19,“parse:parse-primary-index:num_compactions”:10,“parse:parse-primary-index:num_completed_requests”:2,“parse:parse-primary-index:num_docs_indexed”:3681135,“parse:parse-primary-index:num_docs_pending”:0,“parse:parse-primary-index:num_docs_queued”:0,“parse:parse-primary-index:num_flush_queued”:3681135,“parse:parse-primary-index:num_last_snapshot_reply”:0,“parse:parse-primary-index:num_requests”:2,“parse:parse-primary-index:num_rows_returned”:1000,“parse:parse-primary-index:num_snapshot_waiters”:0,“parse:parse-primary-index:num_snapshots”:165,“parse:parse-primary-index:scan_bytes_read”:10000,“parse:parse-primary-index:scan_wait_duration”:13986,“parse:parse-primary-index:since_last_snapshot”:155123262,“parse:parse-primary-index:timings/storage_clone_handle”:“2 86404 3825033808”,“parse:parse-primary-index:timings/storage_commit”:“19 1910764158 221416036046585244”,“parse:parse-primary-index:timings/storage_del”:“0 0 0”,“parse:parse-primary-index:timings/storage_get”:“0 0 0”,“parse:parse-primary-index:timings/storage_handle_open”:“19 844494 75122123626”,“parse:parse-primary-index:timings/storage_info”:“204 1591268 16183085168”,“parse:parse-primary-index:timings/storage_iterator_next”:“998 3519276 16998102674”,“parse:parse-primary-index:timings/storage_meta_get”:“28 808320407 77228947891470103”,“parse:parse-primary-index:timings/storage_meta_set”:“19 6209626 2178888395122”,“parse:parse-primary-index:timings/storage_new_iterator”:“2 104294 5504606290”,“parse:parse-primary-index:timings/storage_set”:“3681135 55939615212 1943966048641949520”,“parse:parse-primary-index:timings/storage_snapshot_close”:“145 842217590 6219497763536780”,“parse:parse-primary-index:timings/storage_snapshot_create”:“146 5339397065 261798859796828849”,“parse:parse-primary-index:total_scan_duration”:6897042,“parse:ts_queue_size”:0,“timings/stats_response”:“1246 228955076 52745997394058”}
2016-04-18T14:32:31.155+00:00 [Info] CompactionDaemon: Finished compacting index instance:13813219721503989217
2016-04-18T14:33:01.155+00:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 13813219721503989217 Data:387695753, Disk:754413568, ExtraSnapshotData:0, Fragmentation:48.60965265672422%
2016-04-18T14:33:01.155+00:00 [Info] CompactionDaemon: Compacting index instance:13813219721503989217
2016-04-18T14:33:01.259+00:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 2823981. Slice Id 0, IndexInstId 13813219721503989217, IndexDefnId 13813219721503989217
2016-04-18T14:33:01.269+00:00 [Error] ForestDB Error (data.fdb.0/default) errcode: -1, msg: Error in REMOVE on a database file ‘/index/@2i/parse_parse-primary-index_13813219721503989217_0.index/data.fdb.9’, errno = 2: ‘No such file or directory’
Here is the indexer process info if that can help:
2570 ? Sl 12:32 | _ /opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/index/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash
Is there a way to repair an index?
Thanks,
Steeve