Couchbase started behaving slowing

I have couchbase cluster and 6.0.1(enterprise) with couchbase operator 1.2

I see following error

IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 3 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
9:15:50 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 2 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
9:15:19 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 2 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
9:15:14 AM Thu Sep 26, 2019
Enabled auto-failover with timeout 120 and max count 3
auto_failover 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
9:14:54 AM Thu Sep 26, 2019
Haven’t heard from a higher priority node or a master, so I’m taking over.
mb_master 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
9:14:54 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
9:14:52 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
9:14:52 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
9:14:49 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 3 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
8:31:50 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 2 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
8:31:19 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
8:31:16 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’) (repeated 1 times)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
8:31:14 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0000.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
Enabled auto-failover with timeout 120 and max count 3
auto_failover 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
Haven’t heard from a higher priority node or a master, so I’m taking over.
mb_master 000
ns_1@couchbase-cluster-couchbase-cluster-0003.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0002.couchbase-cluster-couchbase-cluster.default.svc
8:30:58 AM Thu Sep 26, 2019
IP address seems to have changed. Unable to listen on ‘ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc’. (POSIX error code: ‘nxdomain’)
menelaus_web_alerts_srv 000
ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc
8:30:55 AM Thu Sep 26, 2019

Can you help what can be done ?

nxdomain errors are caused by your DNS server. Your cloud provider should be able to assist in determining why it is happening.

Its in our inhouse VM Ware host

Please find the zip file location . I have collected the logs from the couchbase ui

Now the status is very bad… if i do jstack on the spark thread i see its getting timed_waiting state

“cb-computations-3” #74 daemon prio=5 os_prio=0 tid=0x000055f058fb7800 nid=0x68 waiting on condition [0x00007f70214ce000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00089a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

“cb-core-3-2” #77 daemon prio=5 os_prio=0 tid=0x000055f05a410800 nid=0x6b waiting on condition [0x00007f6fc14cb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00081d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

“cb-core-3-1” #76 daemon prio=5 os_prio=0 tid=0x000055f05a85c800 nid=0x6a waiting on condition [0x00007f6fe14cc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c0008480> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.couchbase.client.deps.com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45)
at com.couchbase.client.deps.com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.couchbase.client.deps.com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)