Bucket shutdown when create view

Dear all,

I have an error when creating view on couchbase server 4.1.

When I create a view in development (with default script), this bucket is going shutdown, and can’t alive again. When i delete this view, this bucket alive imidiately.

Logs:

Shutting down bucket “xxx” on ‘ns_1@ipxxxx’ for server shutdown (repeated 9 times)!

I post my log error:

crasher:
initial call: misc:turn_into_gen_server/4
pid: <11499.16632.23>
registered_name: ‘capi_set_view_manager-tesst’
exception throw: {error,{error,{dcp_socket_connect_failed,nxdomain}}}
in function couch_set_view:get_group_server/2 (/home/couchbase/jenkins/workspace/sherlock-unix/couchdb/src/couch_set_view/src/couch_set_view.erl, line 437)
in call from couch_set_view:define_group/4 (/home/couchbase/jenkins/workspace/sherlock-unix/couchdb/src/couch_set_view/src/couch_set_view.erl, line 143)
in call from timer:tc/3 (timer.erl, line 194)
in call from capi_set_view_manager:’-maybe_define_group/2-fun-0-’/3 (src/capi_set_view_manager.erl, line 299)
in call from capi_set_view_manager:maybe_define_group/2 (src/capi_set_view_manager.erl, line 299)
in call from capi_set_view_manager:’-init/1-lc$^0/1-0-’/2 (src/capi_set_view_manager.erl, line 182)
in call from capi_set_view_manager:init/1 (src/capi_set_view_manager.erl, line 182)
in call from misc:turn_into_gen_server/4 (src/misc.erl, line 608)
ancestors: [<0.24994.0>,‘single_bucket_kv_sup-tesst’,ns_bucket_sup,
ns_bucket_worker_sup,ns_server_sup,ns_server_nodes_sup,
<0.153.0>,ns_server_cluster_sup,<0.88.0>]

@vmx, any idea on this one?

I see nxdomain in the error. Do you have problems with your DNS? I know nothing about nxdomain errors, but DNS turned up after a quick search.

I’m having the same nxdomain issue on View creation in both 4.1.1 and 4.5beta.
Document related functions and XDCR appear to work fine.

For me, the issue only occurs when running inside Docker on CoreOS.
Running inside Docker on Ubuntu 15/16 or RHEL7 works fine.

Can anyone provide me with some more info on where the DNS lookup is being directed and what name is being looked up? Perhaps I can manually input an entry somewhere to workaround this issue. Or it may help me identify an issue with my routes or iptables forwarding rules.

Thanks in advance.

Below is a snippet from debug.log

[error_logger:info,2016-06-09T04:28:59.027Z,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
          supervisor: {<0.2398.0>,docs_kv_sup}
             started: [{pid,<11579.24025.0>},
                       {name,capi_set_view_manager},
                       {mfargs,
                           {capi_set_view_manager,start_link_remote,
                               ['couchdb_ns_1@127.0.0.1',"default"]}},
                       {restart_type,permanent},
                       {shutdown,1000},
                       {child_type,worker}]

[error_logger:info,2016-06-09T04:28:59.032Z,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================PROGRESS REPORT=========================
          supervisor: {<0.2398.0>,docs_kv_sup}
             started: [{pid,<11579.24029.0>},
                       {name,couch_stats_reader},
                       {mfargs,
                           {couch_stats_reader,start_link_remote,
                               ['couchdb_ns_1@127.0.0.1',"default"]}},
                       {restart_type,permanent},
                       {shutdown,1000},
                       {child_type,worker}]

[error_logger:error,2016-06-09T04:28:59.044Z,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================SUPERVISOR REPORT=========================
     Supervisor: {<0.2398.0>,docs_kv_sup}
     Context:    child_terminated
     Reason:     **{error,{error,{dcp_socket_connect_failed,nxdomain}}}**
     Offender:   [{pid,<11579.24025.0>},
                  {name,capi_set_view_manager},
                  {mfargs,
                      {capi_set_view_manager,start_link_remote,
                          ['couchdb_ns_1@127.0.0.1',"default"]}},
                  {restart_type,permanent},
                  {shutdown,1000},
                  {child_type,worker}]


[error_logger:error,2016-06-09T04:28:59.052Z,ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
  crasher:
    initial call: misc:turn_into_gen_server/4
    pid: <11579.24025.0>
    registered_name: 'capi_set_view_manager-default'
    exception throw: **{error,{error,{dcp_socket_connect_failed,nxdomain}}}**
      in function  couch_set_view:get_group_server/2 (/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/couch_set_view/src/couch_set_view.erl, line 437)
      in call from couch_set_view:define_group/4 (/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/couch_set_view/src/couch_set_view.erl, line 143)
      in call from timer:tc/3 (timer.erl, line 194)
      in call from capi_set_view_manager:'-maybe_define_group/2-fun-1-'/3 (src/capi_set_view_manager.erl, line 305)
      in call from capi_set_view_manager:maybe_define_group/2 (src/capi_set_view_manager.erl, line 305)
      in call from capi_set_view_manager:'-init/1-lc$^0/1-0-'/2 (src/capi_set_view_manager.erl, line 182)
      in call from capi_set_view_manager:init/1 (src/capi_set_view_manager.erl, line 182)
      in call from misc:turn_into_gen_server/4 (src/misc.erl, line 665)
    ancestors: [<0.2398.0>,'single_bucket_kv_sup-default',ns_bucket_sup,
                  ns_bucket_worker_sup,ns_server_sup,ns_server_nodes_sup,
                  <0.154.0>,ns_server_cluster_sup,<0.88.0>]
    messages: []
    links: [<0.2398.0>,<11579.24028.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 5148
  neighbours:

Ok after some more fiddling around, it appears the issue is because ‘localhost’ does not resolve inside the Docker container.

So /etc/hosts (which is blank in the official couchbase Docker image) just needs this line:

127.0.0.1 localhost

This can be applied to the container by using a docker volume mount to replace /etc/hosts at runtime with your own:

docker run -v /path/to/hosts:/etc/hosts:ro couchbase

Hope that helps anyone else who is experiencing running Couchbase under CoreOS with Docker.

1 Like

@arungupta is this something we need to add to the docker docs? I’m not familiar with it but you are :slight_smile:

I think the image should be updated to include this /etc/hosts mapping. @ceej @anil?

It’s actually impossible to incorporate /etc/hosts into a Docker image; it’s read-only during builds so that Docker can create a custom one when using “docker run --link”, among other reasons. The correct way to add entries in a new container is with “docker run --add-host”, although mounting /etc/hosts also works as you found.

However, if your /etc/hosts in the container doesn’t contain an entry for “localhost”, then I think something else is wrong (unless CoreOS introduces some weirdness). Using Docker 1.11.2 and running couchbase:4.1.1, my /etc/hosts has “127.0.0.1 localhost” as well as several ip6-xxxx entries and an entry for the container itself “172.17.0.2 9a7f2a19ed19”.

What version of Docker are you using?