Trying to link 2 nodes in a cluster

Couchbase Server
,
#1

I haven’t used Couchbase in over a year and I was tasked from my new job with installing it on 2 computers and linking them in a cluster. I am currently using the 6.60 Community Server version for both of them.
I have created a cluster on one of the computers and then added the other one using the Add Server button.
I tried testing the configuration by importing the “beer-sample” bucket, however it loaded an empty bucket and I can’t add any documents in it. Also, whenever I try to rebalance the server I get a weird error.
The errors I got are in the image below:

image
image1617×743 104 KB

I can’t figure out the problem. I have used a previous version of Couchbase back in 2019 without all these headaches and I remeber the setup procedure being very similar to what I have already tried.
Any help is greatly appreciated!

#2

Hi @Rus_Adrian,

This is interesting, I think it’s worth explaining the issue related to loading the sample bucket first.

Loading a sample bucket is a three phase process:

  1. The bucket will be created
  2. We will wait for the new bucket to report as “healthy” (allowing it to be created/warmup)
  3. We begin loading data into the new bucket

In this case, we can see that the docloader tool has timed out whilst waiting for the bucket to report as “healthy” (after 30s). This is a timeout which has been increased from 10s to 30s in the past. This is the reason why an empty bucket was created.

Regarding the rebalance issues, I see that the rebalance failed due to indexing; it’s hard to say exactly why without the full server logs.

What that in mind, please could you upload the cluster logs; these can be collected with the following steps:

  1. Navigate to the ‘Logs’ tab
  2. Navigate to ‘Collection Information’ in the top bar
  3. Toggle ‘select all nodes’
  4. Click ‘Start Collecting’ at the bottom of the page

Thanks in advance,
James

#3

@jamesl33
Unfortunately the forum considers me a “new user” since I’ve only posted one other post back in 2019, so I have to give you a link to WeTransfer for the complete set of logs: https://we.tl/t-x4L0WUbPIi. If you don’t want to sift through the entire set of logs, I can post a screenshot of a file similar to what I did in my parent post.

#4

Hi @Rus_Adrian,

Thanks for providing the logs, I’ve had a look over them. There’s a couple of cases where requests to the indexing service are timing out and it appears to have been unreachable during the rebalance.

Please could you provide the logs from both nodes as I currently only have access to the logs from 192.168.0.122. Once I’ve got both log collections I’ll open an MB (Jira Ticket) to have this investigated further by the indexing team.

Thanks again,
James

#5

@jamesl33
Here’s the link to the other node: https://we.tl/t-RgZVI1e2ld
I have tried both the beta version (V7.0) and the current release (V6.6). From my previous experience with Couchbase back in 2019, working with it was very straight-forward. I could create a bucket (still can here) and could add documents without any issue (now I can’t).
I’m very lost as to why these basic features don’t seem to work as intended on a fresh install. I can understand that setting up a new cluster can be unsuccessful due to network issues and firewalls, but basic database functions on a local level should still work.

#6

Hi @Rus_Adrian,

Thanks for the second set of logs, from those logs we can see that the indexing service on this node is failing to connect to the projector. I’ve opened MB-45281 to track the issue and have highlighted some of the interesting observations in the description.

Regarding the issue creating documents, I believe it may be a symptom/knock on issue due to the rebalance failure; we’ve seen a similar symptom with the timeout from docloader (where the bucket is not being switched to active/healthy).

Out of interest, are you using the document editor in the WebUI to insert documents, or using one of the supported SDKs?

Thanks again,
James

EDIT: Made MB link clickable

#7

@jamesl33
I am using the WebUI for now. I am trying to work on a proof of concept distributed database example to mitigate the fact that our server which has our database has proven unstable due to power outages and Windows Updates and other such factors.
Since I’ve worked with Couchbase in the past and have configured nodes back then, I proposed it to my boss.
Hopefully your team can find a solution to my problem,
Thanks for your support,
Adrian

#8

Hi @Rus_Adrian,

This issue has now been looked at by the indexing team, please see MB-45281 for the full details.

To summarize, we see that node 111 is trying and failing to contact the node with UUID 14f1c3ec9d692532da75848c53947345, however, this is the local node. So even though this is the local node, some accesses via the REST API appear to be failing. The likely cause for this will be due to a firewall rule blocking the loopback call.

Please could you ensure all the required ports (documented here) are available and try again.

Thanks in advance,
James

#9

I will inform the network administrator about the firewall issue and see if this resolves it.
Thank you very much @jamesl33 !