Terrible Time with CouchBase Cluster


#1

I started implementing CB CE and all things are working well, except something about the audit write failure, which I am not worried about. We decided to start the process of converting to CB EE. After reading the docs it seem pretty straightforward to upgrade the cluster from CE to EE.

I have three servers with CE so I spun up three more servers with EE. Added them to the cluster, started a rebalance, and every time the rebalance fails. I cannot seem to get these EE servers online with the cluster to remove the CE servers. I see a notification stating that rebalance stopped by janitor and a bunch of errors in the logs.

All servers are running with the same configuration:
Ubuntu 16.04
8 core cpus
4 gb memory
32 gb hard disk
CB EE 5.5.1 build 3511

Right now we don’t have a lot of data in the one bucket, but I don’t to have to remove the data and start over again. Last time I did that using backup/restore or transfer tools I did not get the same data transferred to the new cluster.

Errors in Logs:
Service ‘memcached’ exited with status 1. Restarting. Messages:
2018-09-10T10:22:07.402440Z INFO Setting SSL minimum protocol to: tlsv1
2018-09-10T10:22:07.403060Z INFO Breakpad enabled. Minidumps will be written to ‘/opt/couchbase/var/lib/couchbase/crash’
2018-09-10T10:22:07.403183Z INFO Setting SSL minimum protocol to: tlsv1
2018-09-10T10:22:07.406463Z INFO Disabling Breakpad
2018-09-10T10:22:07.406503Z INFO Breakpad enabled. Minidumps will be written to ‘/opt/couchbase/var/lib/couchbase/crash’
2018-09-10T10:22:07.563499Z WARNING Audit: error reading config: error: failed to create log directory “/Users/user/Library/Application Support/Couchbase/var/lib/couchbase/logs”: cb::io::mkdirp("/Users") failed: Permission denied
2018-09-10T10:22:07.563547Z CRITICAL FATAL: Failed to start audit daemon

Service ‘query’ exited with status 1. Restarting. Messages:
_time=2018-09-10T10:22:33.751-04:00 _level=INFO _msg= Initialization of cbauth succeeded
_time=2018-09-10T10:22:36.572-04:00 _level=ERROR _msg=Cannot connect to default pool
_time=2018-09-10T10:22:36.572-04:00 _level=ERROR _msg=Namespace not found Namespace default - cause: invalid character ‘<’ looking for beginning of value
_time=2018-09-10T10:22:36.572-04:00 _level=ERROR _msg=Shutting down.
[goport(/opt/couchbase/bin/cbq-engine)] 2018/09/10 10:22:36 child process exited with status 1

Service ‘query’ exited with status 1. Restarting. Messages:
_time=2018-09-10T10:23:14.181-04:00 _level=INFO _msg= Initialization of cbauth succeeded
_time=2018-09-10T10:23:17.018-04:00 _level=ERROR _msg=Cannot connect to default pool
_time=2018-09-10T10:23:17.018-04:00 _level=ERROR _msg=Namespace not found Namespace default - cause: invalid character ‘<’ looking for beginning of value
_time=2018-09-10T10:23:17.018-04:00 _level=ERROR _msg=Shutting down.
[goport(/opt/couchbase/bin/cbq-engine)] 2018/09/10 10:23:17 child process exited with status 1
ns_log 000
ns_1@192.168.2.111
10:23:17 AM Mon Sep 10, 2018


#2

The paths here look like the style used by macOS (/Users/user/Library/Application Support) - did you attempt to create a cluster from a hetrogenous set of operating systems? This isn’t supported.

If you need to migrate data from say macOS to Linux, you need to backup the existinmacOS data (using cbbackup) and restore to Linux using cbrestore.

See https://developer.couchbase.com/documentation/server/3.x/admin/Tasks/backup-restore-tween-opsys.html


#3

Mixed OS’s, must have been the issue. I spun up a new cluster and transferred the data and no issues.