I started implementing CB CE and all things are working well, except something about the audit write failure, which I am not worried about. We decided to start the process of converting to CB EE. After reading the docs it seem pretty straightforward to upgrade the cluster from CE to EE.
I have three servers with CE so I spun up three more servers with EE. Added them to the cluster, started a rebalance, and every time the rebalance fails. I cannot seem to get these EE servers online with the cluster to remove the CE servers. I see a notification stating that rebalance stopped by janitor and a bunch of errors in the logs.
All servers are running with the same configuration:
Ubuntu 16.04
8 core cpus
4 gb memory
32 gb hard disk
CB EE 5.5.1 build 3511
Right now we don’t have a lot of data in the one bucket, but I don’t to have to remove the data and start over again. Last time I did that using backup/restore or transfer tools I did not get the same data transferred to the new cluster.
Errors in Logs:
Service ‘memcached’ exited with status 1. Restarting. Messages:
2018-09-10T10:22:07.402440Z INFO Setting SSL minimum protocol to: tlsv1
2018-09-10T10:22:07.403060Z INFO Breakpad enabled. Minidumps will be written to ‘/opt/couchbase/var/lib/couchbase/crash’
2018-09-10T10:22:07.403183Z INFO Setting SSL minimum protocol to: tlsv1
2018-09-10T10:22:07.406463Z INFO Disabling Breakpad
2018-09-10T10:22:07.406503Z INFO Breakpad enabled. Minidumps will be written to ‘/opt/couchbase/var/lib/couchbase/crash’
2018-09-10T10:22:07.563499Z WARNING Audit: error reading config: error: failed to create log directory “/Users/user/Library/Application Support/Couchbase/var/lib/couchbase/logs”: cb::io::mkdirp("/Users") failed: Permission denied
2018-09-10T10:22:07.563547Z CRITICAL FATAL: Failed to start audit daemon
Service ‘query’ exited with status 1. Restarting. Messages:
_time=2018-09-10T10:22:33.751-04:00 _level=INFO _msg= Initialization of cbauth succeeded
_time=2018-09-10T10:22:36.572-04:00 _level=ERROR _msg=Cannot connect to default pool
_time=2018-09-10T10:22:36.572-04:00 _level=ERROR _msg=Namespace not found Namespace default - cause: invalid character ‘<’ looking for beginning of value
_time=2018-09-10T10:22:36.572-04:00 _level=ERROR _msg=Shutting down.
[goport(/opt/couchbase/bin/cbq-engine)] 2018/09/10 10:22:36 child process exited with status 1
Service ‘query’ exited with status 1. Restarting. Messages:
_time=2018-09-10T10:23:14.181-04:00 _level=INFO _msg= Initialization of cbauth succeeded
_time=2018-09-10T10:23:17.018-04:00 _level=ERROR _msg=Cannot connect to default pool
_time=2018-09-10T10:23:17.018-04:00 _level=ERROR _msg=Namespace not found Namespace default - cause: invalid character ‘<’ looking for beginning of value
_time=2018-09-10T10:23:17.018-04:00 _level=ERROR _msg=Shutting down.
[goport(/opt/couchbase/bin/cbq-engine)] 2018/09/10 10:23:17 child process exited with status 1
ns_log 000
ns_1@192.168.2.111
10:23:17 AM Mon Sep 10, 2018