Indexer In Warmup State. Please retry the request later. from <IP Address>:9101

SrinivasanKuppuswamy · November 29, 2017, 7:58am

Hello All,
When I tried to access the content from couch base document (indexer is used to get data) I’m getting an issue like "Indexer In Warmup State. Please retry the request later. from :9101"
The I checked indexer.log file I’m getting errors like
[Error] WatcherServer.runOnce() error : dial tcp :9100: getsockopt: connection refused
[Error] updateIndexerList(): Cannot locate cluster node.
[Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector :9999 Topic [Error] updateIndexerList(): Cannot locate cluster node. panic: runtime error: index out of range

Error logs:
2017-11-28T07:56:18.921-06:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=b350122c24c04487be604b4709ea6af6 -storageMode=plasma --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/ssl-cert-key.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/ssl-cert-key.pem]
2017-11-28T07:56:18.921-06:00 [Info] Indexer::Main Storage Mode Set plasma
2017-11-28T07:56:19.041-06:00 [Info] Indexer::NewIndexer Status Warmup
2017-11-28T07:56:19.177-06:00 [Info] Setting buffer block size to 16384 bytes
2017-11-28T07:56:19.177-06:00 [Info] Setting maxcpus = 2
2017-11-28T07:56:19.177-06:00 [Info] Setting log level to Info
2017-11-28T07:56:19.177-06:00 [Info] 127.0.0.1:8091 starting RESTful services
2017-11-28T07:56:19.181-06:00 [Info] New settings received:
{“indexer.settings.compaction.days_of_week”:“Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday”,“indexer.settings.compaction.interval”:“00:00,00:00”,“indexer.settings.compaction.compaction_mode”:“circular”,“indexer.settings.persisted_snapshot.interval”:5000,“indexer.settings.log_level”:“info”,“indexer.settings.compaction.min_frag”:30,“indexer.settings.inmemory_snapshot.interval”:200,“indexer.settings.max_cpu_percent”:0,“indexer.settings.storage_mode”:“plasma”,“indexer.settings.recovery.max_rollbacks”:5,“indexer.settings.memory_quota”:536870912,“indexer.settings.compaction.abort_exceed_interval”:false}
2017-11-28T07:56:19.181-06:00 [Info] Setting maxcpus = 2
2017-11-28T07:56:19.181-06:00 [Info] Setting log level to Info
2017-11-28T07:56:19.349-06:00 [Info] MetadataProvider.SetClusterStatus(): healthy nodes 1 failed node 0 unhealthy node 0 add node 0
2017-11-28T07:56:19.349-06:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.
2017-11-28T07:56:19.349-06:00 [Info] Refreshed Indexer List: [192.168.2.137:9100]
2017-11-28T07:56:19.349-06:00 [Info] MetadataProvider.WatchMetadata(): indexer 192.168.2.137:9100
2017-11-28T07:56:19.349-06:00 [Error] WatcherServer.runOnce() error : dial tcp 192.168.2.137:9100: getsockopt: connection refused
2017-11-28T07:56:20.349-06:00 [Info] WatchMetadata(): unable to reach indexer at 192.168.2.137:9100. Retry in background.
2017-11-28T07:56:20.350-06:00 [Error] WatcherServer.runOnce() error : dial tcp 192.168.2.137:9100: getsockopt: connection refused
2017-11-28T07:56:20.351-06:00 [Info] MetadataProvider: Updating indexer version to 2
2017-11-28T07:56:20.351-06:00 [Info] initialized currmeta 0 force true
2017-11-28T07:56:20.351-06:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024
2017-11-28T07:56:20.392-06:00 [Info] [Queryport “:9101”] started …
2017-11-28T07:56:20.394-06:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default
2017-11-28T07:56:20.396-06:00 [Info] serviceChangeNotifier: received ServiceChangeNotification
2017-11-28T07:56:20.440-06:00 [Info] serviceChangeNotifier: received PoolChangeNotification
[FDB INFO] Forestdb blockcache size 1048576 initialized in 84 us

[FDB INFO] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore
2017-11-28T07:56:20.549-06:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.
2017-11-28T07:56:20.549-06:00 [Info] Refreshed Indexer List: [192.168.2.137:9100]
2017-11-28T07:56:20.552-06:00 [Info] switched currmeta from 0 -> 0 force true
2017-11-28T07:56:20.552-06:00 [Error] updateIndexerList(): Cannot locate cluster node.
2017-11-28T07:56:22.350-06:00 [Error] WatcherServer.runOnce() error : dial tcp 192.168.2.137:9100: getsockopt: connection refused
2017-11-28T07:56:26.350-06:00 [Error] WatcherServer.runOnce() error : dial tcp 192.168.2.137:9100: getsockopt: connection refused
2017-11-28T07:56:34.351-06:00 [Error] WatcherServer.runOnce() error : dial tcp 192.168.2.137:9100: getsockopt: connection refused
2017-11-28T07:56:37.189-06:00 [Info] indexer:: SSL server started: :19102
2017-11-28T07:56:37.189-06:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId
2017-11-28T07:56:37.190-06:00 [Info] Indexer Id b350122c24c04487be604b4709ea6af6
2017-11-28T07:56:45.113-06:00 [Info] KVSender::sendShutdownTopic Projector 192.168.2.137:9999 Topic MAINT_STREAM_TOPIC_b350122c24c04487be604b4709ea6af6
2017-11-28T07:56:45.115-06:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 192.168.2.137:9999 Topic MAINT_STREAM_TOPIC_b350122c24c04487be604b4709ea6af6. Err projector.topicMissing
2017-11-28T07:56:45.115-06:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 192.168.2.137:9999
2017-11-28T07:56:45.115-06:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
2017-11-28T07:56:45.413-06:00 [Info] serviceChangeNotifier: received PoolChangeNotification
2017-11-28T07:56:45.587-06:00 [Info] KVSender::sendShutdownTopic Projector 192.168.2.137:9999 Topic INIT_STREAM_TOPIC_b350122c24c04487be604b4709ea6af6
2017-11-28T07:56:45.592-06:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 192.168.2.137:9999 Topic INIT_STREAM_TOPIC_b350122c24c04487be604b4709ea6af6. Err projector.topicMissing
2017-11-28T07:56:45.592-06:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 192.168.2.137:9999
2017-11-28T07:56:45.592-06:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success
2017-11-28T07:56:45.592-06:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning
2017-11-28T07:56:45.592-06:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken
2017-11-28T07:56:45.592-06:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken
2017-11-28T07:56:45.592-06:00 [Info] ClustMgr:handleGetGlobalTopology &{map[]}
2017-11-28T07:56:45.592-06:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
InstanceId: 2396421628691976958 Name: primary_index Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 3474165602048080121 Name: location_search Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 8649036813394999428 Name: case_search Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 2607583017890568904 Name: list_warehouses Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 16092466313960715114 Name: container_search Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 8432851494965330310 Name: parts_search_inventory_request Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 16917357918574576199 Name: order_list Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 7439250386392425200 Name: container_category_name Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 10462036818263898349 Name: category_by_name Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 13494072480934324851 Name: receiving_containers Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 11905598617777751785 Name: partDef_by_partNumber Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 13251341068406287029 Name: surgeon_search Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 7935264519036606461 Name: categories_contain_partNumber Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 13983151672004356398 Name: user_search Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 10467578882665437910 Name: preferencecard_casetype Bucket: preludesys State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
2017-11-28T07:56:45.723-06:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.
2017-11-28T07:56:45.723-06:00 [Info] Refreshed Indexer List: [192.168.2.137:9100]
2017-11-28T07:56:45.727-06:00 [Info] switched currmeta from 0 -> 0 force true
2017-11-28T07:56:45.727-06:00 [Error] updateIndexerList(): Cannot locate cluster node.
2017-11-28T07:56:49.999-06:00 [Info] {bucket,feeds} “preludesys” created for dcp_seqno cache…
2017-11-28T07:56:50.000-06:00 [Info] Indexer::initPartnInstance Initialized Partition:
Index: 13983151672004356398 Partition: PartitionId: 0 Endpoints: [:9105]
panic: runtime error: index out of range

goroutine 78 [running]:
panic(0xce8c80, 0xc4200140a0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc4200d5788 sp=0xc4200d56f8
runtime.panicindex()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:27 +0x6d fp=0xc4200d57b8 sp=0xc4200d5788
github dot com/couchbase/nitro/plasma.(*Plasma).doRecovery.func1(0x78dc7418, 0xc42424e000, 0x0, 0xbe4f, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:423 +0xdab fp=0xc4200d58c0 sp=0xc4200d57b8
github. com/couchbase/nitro/plasma.(*lsStore).visitor(0xc42403e000, 0x78dba763, 0x78e06a1c, 0xc423d45900, 0xc423d458e0, 0xd2d2e0, 0xc41fe15001)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github
dot /couchbase/nitro/plasma/lss.go:299 +0x167 fp=0xc4200d5930 sp=0xc4200d58c0
github. com/couchbase/nitro/plasma.(*lsStore).Visitor(0xc42403e000, 0xc423d45900, 0xc423d458e0, 0xc42424c000, 0x1000)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github. com/couchbase/nitro/plasma/lss.go:288 +0x9d fp=0xc4200d5980 sp=0xc4200d5930
github. com/couchbase/nitro/plasma.(*Plasma).doRecovery(0xc42376cd80, 0xc423d5e570, 0x21)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github. com/couchbase/nitro/plasma/plasma.go:492 +0x235 fp=0xc4200d5a48 sp=0xc4200d5980
github. com/couchbase/nitro/plasma.New(0x1e, 0x12c, 0x5, 0x4, 0xeb9e68, 0xeb9e38, 0xeb9e70, 0xc423cdb160, 0xc423cdb170, 0xeb9e38, …)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github. com/couchbase/nitro/plasma/plasma.go:322 +0xd66 fp=0xc4200d5e38 sp=0xc4200d5a48
github. com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func2(0xc423cdb130, 0xc423d62300, 0xc423cdb120, 0xc424038000, 0xc423cdb150)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github. com/couchbase/indexing/secondary/indexer/plasma_slice.go:232 +0x86 fp=0xc4200d5f68 sp=0xc4200d5e38
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200d5f70 sp=0xc4200d5f68
created by github. com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github. com/couchbase/indexing/secondary/indexer/plasma_slice.go:242 +0x12b2

Please help me out to resolve this blocking issue

deepkaran.salooja · November 29, 2017, 8:43pm

@SrinivasanKuppuswamy, can you share your OS details and CB version.

SrinivasanKuppuswamy · November 30, 2017, 5:02am

Hi Deepkaran,

      Please find the details below

OS: Linux, Suse 12.0
CB: Enterprise Edition 5.0.0 build 2873

deepkaran.salooja · December 1, 2017, 6:35am

@SrinivasanKuppuswamy, this looks to be the same issue as MB-25086. Did you observe this issue after a power failure?

The underlying index file has been corrupted. You’ll have to cleanup the index file manually from /opt/couchbase/var/lib/couchbase/data/@2i.

nonorval · February 8, 2019, 7:44am

Hi everyone

I seem to be getting the same error on CB CE 6.0.0 running on Ubuntu 14.04 (x64). I have an existing node with only a data service (CB CC 5.1.0) on which I now need an indexer. I have tried several ways to accomplish this (add fresh node with indexer to the same cluster; SG replicate to a node with an indexer; XDCR replicate to a new cluster with an indexer), but the indexer seems to get stuck in warmup mode every time. @deepkaran.salooja I have tried your suggestion of cleaning the index files manually and it seemed to work for a couple of minutes (indexes showed green on the console), but then it when back to warmup mode again.

Does anyone maybe have advice for what else I can try or what the actual problem might be? I’m starting to think adding an index node to an existing cluster without it is not feasible, or that my data is corrupted somehow and is breaking the indexer. The existing node with only the data service seems to be working perfectly fine though. Indexer logs below.

Any advice would be appreciated.

2019-02-08T07:06:17.041+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:06:20.567+00:00 [Info] [Queryport ":9101"] connection "127.0.0.1:41073" doReceive() ...
2019-02-08T07:06:20.568+00:00 [Info] [Queryport ":9101"] connection "127.0.0.1:41074" doReceive() ...
2019-02-08T07:06:20.577+00:00 [Info] [Queryport ":9101"] connection "127.0.0.1:41075" doReceive() ...
2019-02-08T07:06:22.060+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:06:22.061+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:22.061+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:06:23.062+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:23.062+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:06:28.083+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:06:28.083+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:28.084+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:06:29.084+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:29.084+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:06:34.108+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:06:34.109+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:34.109+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:06:35.111+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:35.111+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:06:40.138+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:06:40.138+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:40.138+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:06:41.138+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:41.139+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:06:46.164+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:06:46.165+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:46.166+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:06:47.167+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:47.167+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:06:51.914+00:00 [Info] PeriodicStats = {"cpu_utilization":1,"index_not_found_errcount":0,"indexer_state":"Warmup","memory_free":1535696896,"memory_quota":1073741824,"memory_rss":60715008,"memory_total":4145242112,"memory_total_storage":0,"memory_used":0,"memory_used_queue":0,"memory_used_storage":0,"needs_restart":false,"num_connections":0,"num_cpu_core":2,"storage_mode":"not_set","timings/stats_response":"72 10721536 1821721196460","uptime":"1m0.173117664s"}
2019-02-08T07:06:52.183+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:06:52.184+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:52.184+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:06:53.185+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:53.185+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:06:58.207+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:06:58.208+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:58.208+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:06:59.209+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:06:59.209+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:07:04.226+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:07:04.227+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:07:04.227+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:07:05.227+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:07:05.227+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.
2019-02-08T07:07:10.242+00:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700
2019-02-08T07:07:10.243+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_38c309eb680c9b6ef3176002d64db700. Err Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:07:10.244+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM  Error Received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused from 127.0.0.1:9999
2019-02-08T07:07:11.245+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM  Error from Projector Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused
2019-02-08T07:07:11.245+00:00 [Fatal] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post http://127.0.0.1:9999/adminport/shutdownTopicRequest: dial tcp 127.0.0.1:9999: getsockopt: connection refused. Retrying.

deepkaran.salooja · February 9, 2019, 10:28pm

@nonorval, the error mentioned above is related to EE. You seem to be using CE which uses a different storage engine. The error in your logs suggests that projector process is unable to come up as port 9999 is being used by some other process. You may want to check that.

nonorval · February 28, 2019, 11:51am

@deepkaran.salooja Just an update for your info: We eventually managed to fix the issue of the projector failing frequently by adding more RAM and CPUs to the server. It seems like the indexer was actually failing because of this, even though the resources did not show that they were maxing out on the server itself, so the logs actually sent us on a bit of a wild goose chase, I think.