We are having troubles communicating to the indexer process. The information might be stale

Not really seeing any additional information besides the following.

Warning: We are having troubles communicating to the indexer process. The information might be stale.

root@p-dc2-couch1-1:/opt/couchbase/var/lib/couchbase/logs# tail -10 indexer.log
2017-07-05T18:51:36.056Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60578511 3046952156432 960069187656370026”}
2017-07-05T18:52:36.057Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60578648 3046956914365 960069373089435145”}
2017-07-05T18:53:36.057Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60578786 3046961690337 960069564973527719”}
2017-07-05T18:54:36.057Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60578923 3046966224646 960069737153901870”}
2017-07-05T18:55:36.057Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60579060 3046970608824 960069887923047138”}
2017-07-05T18:56:36.058Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60579198 3046975654585 960070102469239073”}
2017-07-05T18:57:36.058Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60579334 3046980433024 960070296334759104”}
2017-07-05T18:58:36.058Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60579472 3046985124370 960070475797388192”}
2017-07-05T18:59:36.058Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60579609 3046989855004 960070657816220874”}
2017-07-05T19:00:36.059Z+00:00 [Info] PeriodicStats = {“memory_quota”:268435456,“memory_used”:143360,“needs_restart”:false,“num_connections”:0,“timings/stats_response”:“60579747 3046994399459 960070825806084841”}

[ns_server:error,2017-07-05T19:04:44.023Z,ns_1@p-dc2-couch1-1.p.movetv.com:index_status_keeper_worker<0.395.0>:index_status_keeper:process_status:228]Indexer returned unsuccessful status:
[{<<“code”>>,<<“error”>>},
{<<“error”>>,<<“Fail to retrieve cluster-wide metadata from index service”>>},
{<<“failedNodes”>>,[<<“p-dc2-couch1-5.p.movetv.com:9102”>>]},
{<<“status”>>,
[{[{<<“defnId”>>,11644086049112534939},
{<<“name”>>,<<"#primary">>},
{<<“bucket”>>,<<“cmw”>>},
{<<“isPrimary”>>,true},
{<<“status”>>,<<“Ready”>>},
{<<“definition”>>,<<“CREATE PRIMARY INDEX #primary ON cmw USING GSI”>>},
{<<“hosts”>>,[<<“p-dc2-couch1-8.p.movetv.com:8091”>>]},
{<<“completion”>>,100}]},
{[{<<“defnId”>>,13898589545123366573},
{<<“name”>>,<<"#primary">>},
{<<“bucket”>>,<<“oauth”>>},
{<<“isPrimary”>>,true},
{<<“status”>>,<<“Ready”>>},
{<<“definition”>>,<<“CREATE PRIMARY INDEX #primary ON oauth USING GSI”>>},
{<<“hosts”>>,[<<“p-dc2-couch1-2.p.movetv.com:8091”>>]},
{<<“completion”>>,100}]}]}]

Version: 4.0.0-4051 Community Edition (build-4051)
Cluster State ID: 087-070-75

root@p-dc2-couch1-1:/opt/couchbase/var/lib/couchbase/logs# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

@kyle.m.stephenson, please make sure port 9999 is open and accessible from all nodes in the cluster.
In general, it would be good to review the list here.

i have no firewall in place.

Interesting, I have also had that with version 4.6, and it was a firewall problem with the same symptoms.

@kyle.m.stephenson, is this an intermittent problem or do you always see this message?
You can collect the logs from UI and share with us. We can take a look.