Getting error on installing couchbase

Atalique_Shaikh · February 9, 2021, 12:18pm

Hi team,

Getting below error wile installing couchbase.
Error from server (AlreadyExists): error when creating “couchbase-cluster.yaml.bak”: secrets “cb-example-auth” already exists
Error from server (AlreadyExists): error when creating “couchbase-cluster.yaml.bak”: couchbasebuckets.couchbase.com “default” already exists
Error from server: error when creating “couchbase-cluster.yaml.bak”: admission webhook “couchbase-operator-admission.default.svc” denied the request: validation failure list:
bucket memory allocation (4Gi) exceeds data service quota (256Mi) on cluster cb-example

Can someone please help with this?

simon.murray · February 9, 2021, 12:28pm

The AlreadyExists errors are self explanatory, either delete the existing database, or you’ll need to rename all your resources so they don’t conflict with the existing resources.

For the validation error, bucket allocations are per-node, not across the whole cluster. Read the documentation here Couchbase Memory Allocation | Couchbase Docs for a full explanation.

Atalique_Shaikh · February 9, 2021, 12:52pm

Actually I am installing this for the first time, so I have followed the document as is, the already exist error which is there, I am not sure about how they already exit . I just followed the below link
https://docs.couchbase.com/operator/current/install-kubernetes.html

CAn you please let me know the commands to delete those resources. also below is the yaml file which I am using, could you please let me know what changes I need to make to make this run.

apiVersion: v1
kind: Secret
metadata:
name: cb-example-auth
type: Opaque
data:
username: QWRtaW5pc3RyYXRvcg== # Administrator
password: cGFzc3dvcmQ= # password

apiVersion: couchbase.com/v2
kind: CouchbaseBucket
metadata:
name: default
spec:
memoryQuota: 2Gi

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
name: cb-example
spec:
image:
cluster:
dataServiceMemoryQuota: 256Mi
indexServiceMemoryQuota: 256Mi
searchServiceMemoryQuota: 256Mi
eventingServiceMemoryQuota: 256Mi
analyticsServiceMemoryQuota: 1Gi
security:
adminSecret: cb-example-auth
buckets:
managed: true
servers:

size: 3
name: all_services
services:
- data
- index
- query
- search
- eventing
- analytics

Patrick_Stephens · February 9, 2021, 1:16pm

Normally what has happened is it has been run twice, i.e. you run the YAML file once, this creates some resources but maybe you have an issue and run it again at which point some resources already exist. These need cleanup first.

The AlreadyExists error is telling you those resources already exist so you’ll need to deal with that first. The easiest thing to do is check what resources exist already prior to running it. Check if you have that secret already and remove it or change to another name.

Atalique_Shaikh · February 9, 2021, 1:19pm

Thanks, Patrick I got your point and I did this, it worked, can you help with the other error also, I mean
Error from server: error when creating “couchbase-cluster.yaml.bak”: admission webhook “couchbase-operator-admission.default.svc” denied the request: validation failure list:
bucket memory allocation (4Gi) exceeds data service quota (256Mi) on cluster cb-example

Patrick_Stephens · February 9, 2021, 1:21pm

Simon covered that above - your allocation is per-node and your bucket is too big: Couchbase Memory Allocation | Couchbase Docs

There is an example here which uses 128 MiB so half the data service quota you have above: Create a Couchbase Deployment | Couchbase Docs

TLDR; you need to increase the data service quota (currently 256MiB) or decrease the bucket size (currently 4GiB).

apiVersion: couchbase.com/v2
kind: CouchbaseBucket
metadata:
  name: default  
spec:
  memoryQuota: 128Mi

We can’t know what you need though so you need to figure out what resources you want to allocate - if you need a 4GiB bucket then you need to increase the quota.

Atalique_Shaikh · February 9, 2021, 5:41pm

Ok, Now I have use the sample yaml file and the couchbase is created I think but I am not able to see the pods.
kubectl get pods
NAME READY STATUS RESTARTS AGE
couchbase-operator-7fcd5ccd7c-5kcql 1/1 Running 0 3m42s
couchbase-operator-admission-5b547b4b79-fjfs8 1/1 Running 0 2m27s

There are only these 2 pods running.
kubectl get couchbase
NAME VERSION SIZE STATUS UUID AGE
couchbasecluster.couchbase.com/cb-example 9m47s

NAME MEMORY QUOTA REPLICAS IO PRIORITY EVICTION POLICY CONFLICT RESOLUTION AGE
couchbasebucket.couchbase.com/default 9m48s

What else I need to do here?

simon.murray · February 10, 2021, 9:04am

Some logs would be helpful, refer to View Operator Logs | Couchbase Docs and it should help you diagnose and fix the problem. It’s the first error you should fixate on, that’s the root cause, not any subsequent and consequent errors.

Atalique_Shaikh · February 10, 2021, 11:14am

Below are the logs from operator pod.

simon.murray · February 10, 2021, 1:53pm

Do you have the first error available in the logs? unexpected cluster phase: Failed doesn’t tell me much beyond “something happened a long time ago”. If the logs are too old, and have been expired, or log-rotated out, or you haven’t streamed it off to logstash, you’ll need to delete the cluster and recreate it and get fresh logs with the initial creation steps in them.

If it’s any help, using Operator 2.1+ is a much better experience because it retries, instead of “failing”, I’d recommend using that if you can.

Thanks!

Atalique_Shaikh · February 11, 2021, 5:18am

I used 2.1.0 operator image and below are the logs from operator pod.

{“level”:“info”,“ts”:1613020509.7387714,“logger”:“main”,“msg”:“couchbase-operator”,“version”:“2.1.0 (build 250)”,“revision”:“b561a46fd687d668631596b3b2588ddca6457409”}
{“level”:“info”,“ts”:1613020510.3991814,“logger”:“controller-runtime.metrics”,“msg”:“metrics server is starting to listen”,“addr”:“0.0.0.0:8383”}
I0211 05:15:10.400393 1 leaderelection.go:242] attempting to acquire leader lease default/couchbase-operator…
{“level”:“info”,“ts”:1613020510.400515,“logger”:“controller-runtime.manager”,“msg”:“starting metrics server”,“path”:"/metrics"}
I0211 05:15:10.407964 1 leaderelection.go:252] successfully acquired lease default/couchbase-operator
{“level”:“info”,“ts”:1613020510.4083579,“logger”:“controller-runtime.controller”,“msg”:“Starting EventSource”,“controller”:“couchbase-controller”,“source”:“kind source: /, Kind=”}
{“level”:“info”,“ts”:1613020510.509057,“logger”:“controller-runtime.controller”,“msg”:“Starting Controller”,“controller”:“couchbase-controller”}
{“level”:“info”,“ts”:1613020510.5091722,“logger”:“controller-runtime.controller”,“msg”:“Starting workers”,“controller”:“couchbase-controller”,“worker count”:4}
E0211 05:15:26.521412 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v2.CouchbaseAutoscaler: couchbaseautoscalers.couchbase.com is forbidden: User “system:serviceaccount:default:couchbase-operator” cannot list resource “couchbaseautoscalers” in API group “couchbase.com” in the namespace “default”
E0211 05:15:27.523957 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v2.CouchbaseAutoscaler: couchbaseautoscalers.couchbase.com is forbidden: User “system:serviceaccount:default:couchbase-operator” cannot list resource “couchbaseautoscalers” in API group “couchbase.com” in the namespace “default”
E0211 05:15:28.526506 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105: Failed to list *v2.CouchbaseAutoscaler: couchbaseautoscalers.couchbase.com is forbidden: User “system:serviceaccount:default:couchbase-operator” cannot list resource “couchbaseautoscalers” in API group “couchbase.com” in the namespace “default”

================================================================================
And couchbase-operator-admission shows below error.
2021/02/10 23:02:38 http: TLS handshake error from 192.168.0.15:43555: remote error: tls: bad certificate

2021/02/10 23:02:38 http: TLS handshake error from 192.168.0.15:65353: remote error: tls: bad certificate

2021/02/10 23:02:38 http: TLS handshake error from 192.168.0.15:35707: remote error: tls: bad certificate

2021/02/10 23:02:38 http: TLS handshake error from 192.168.0.15:65323: remote error: tls: bad certificate

2021/02/10 23:19:34 http: TLS handshake error from 192.168.0.15:10716: remote error: tls: bad certificate

2021/02/10 23:19:34 http: TLS handshake error from 192.168.0.15:8573: remote error: tls: bad certificate

2021/02/10 23:19:34 http: TLS handshake error from 192.168.0.15:45807: remote error: tls: bad certificate

2021/02/10 23:19:34 http: TLS handshake error from 192.168.0.15:44348: remote error: tls: bad certificate

2021/02/10 23:29:28 http: TLS handshake error from 192.168.0.15:53158: remote error: tls: bad certificate

2021/02/10 23:29:28 http: TLS handshake error from 192.168.0.15:55631: remote error: tls: bad certificate

2021/02/10 23:29:28 http: TLS handshake error from 192.168.0.15:44065: remote error: tls: bad certificate

2021/02/10 23:29:28 http: TLS handshake error from 192.168.0.15:12072: remote error: tls: bad certificate

2021/02/10 23:36:30 http: TLS handshake error from 192.168.0.15:27469: remote error: tls: bad certificate

2021/02/10 23:36:30 http: TLS handshake error from 192.168.0.15:46877: remote error: tls: bad certificate

2021/02/10 23:36:30 http: TLS handshake error from 192.168.0.15:50976: remote error: tls: bad certificate

2021/02/10 23:36:30 http: TLS handshake error from 192.168.0.15:50183: remote error: tls: bad certificate

2021/02/10 23:53:26 http: TLS handshake error from 192.168.0.15:45080: remote error: tls: bad certificate

2021/02/10 23:53:26 http: TLS handshake error from 192.168.0.15:48884: remote error: tls: bad certificate

2021/02/10 23:53:26 http: TLS handshake error from 192.168.0.15:44716: remote error: tls: bad certificate

2021/02/10 23:53:26 http: TLS handshake error from 192.168.0.15:35506: remote error: tls: bad certificate

2021/02/11 00:10:22 http: TLS handshake error from 192.168.0.15:45006: remote error: tls: bad certificate

2021/02/11 00:10:22 http: TLS handshake error from 192.168.0.15:23328: remote error: tls: bad certificate

2021/02/11 00:10:22 http: TLS handshake error from 192.168.0.15:53951: remote error: tls: bad certificate

2021/02/11 00:10:22 http: TLS handshake error from 192.168.0.15:15553: remote error: tls: bad certificate

2021/02/11 00:27:19 http: TLS handshake error from 192.168.0.15:61620: remote error: tls: bad certificate

2021/02/11 00:27:19 http: TLS handshake error from 192.168.0.15:4230: remote error: tls: bad certificate

2021/02/11 00:27:19 http: TLS handshake error from 192.168.0.15:27462: remote error: tls: bad certificate

2021/02/11 00:27:19 http: TLS handshake error from 192.168.0.15:37101: remote error: tls: bad certificate

2021/02/11 00:44:15 http: TLS handshake error from 192.168.0.15:26579: remote error: tls: bad certificate

2021/02/11 00:44:15 http: TLS handshake error from 192.168.0.15:26088: remote error: tls: bad certificate

2021/02/11 00:44:15 http: TLS handshake error from 192.168.0.15:51032: remote error: tls: bad certificate

2021/02/11 00:44:15 http: TLS handshake error from 192.168.0.15:43617: remote error: tls: bad certificate

2021/02/11 01:01:11 http: TLS handshake error from 192.168.0.15:61220: remote error: tls: bad certificate

2021/02/11 01:01:11 http: TLS handshake error from 192.168.0.15:32549: remote error: tls: bad certificate

2021/02/11 01:01:11 http: TLS handshake error from 192.168.0.15:59531: remote error: tls: bad certificate

2021/02/11 01:01:11 http: TLS handshake error from 192.168.0.15:54472: remote error: tls: bad certificate

2021/02/11 01:18:07 http: TLS handshake error from 192.168.0.15:1987: remote error: tls: bad certificate

2021/02/11 01:18:07 http: TLS handshake error from 192.168.0.15:31703: remote error: tls: bad certificat

simon.murray · February 11, 2021, 10:22am

Okay, the operator error means that it hasn’t upgraded fully. You need to follow these instructions when you upgrade Upgrade the Operator | Couchbase Docs changing the image will not work. Specifically in this case the CRDs need updating with the new versions.

The admission controller suggests that the CA certificate registered with the admission web hooks is not valid for the DAC certificate. Again, fully uninstall, then reinstall as per the instructions – our tooling will automatically generate valid certificates and webhooks for you.

Atalique_Shaikh · February 11, 2021, 12:51pm

Thanks @simon.murray all the pods are up and running now after reinstallation and changing the operator image to 2.1.0. Can you please let me know what next I will have to do here, as this is my first time installation and I am not sure about its working and all those thing.

simon.murray · February 11, 2021, 1:44pm

Congratulations! First thing is to check everything is okay:

$ kubectl get cbc 
NAME         VERSION   SIZE   STATUS      UUID                               AGE
cb-example   6.6.0     3      Available   52b398e8c19ae7841df2e96dc379a060   68s

You can then play with scaling your cluster, to simulate increasing capacity as demands increase:

$ kubectl patch couchbasecluster/cb-example --type=json -p '[{"op":"replace","path":"/spec/servers/0/size","value":4}]'
couchbasecluster.couchbase.com/cb-example patched
$ kubectl get cbc --watch
NAME         VERSION   SIZE   STATUS      UUID                               AGE
cb-example   6.6.0     3      Available   52b398e8c19ae7841df2e96dc379a060   5m38s
cb-example   6.6.0     4      Available   52b398e8c19ae7841df2e96dc379a060   5m43s

Cool huh?

I’d start with the documents Introduction | Couchbase Docs. The “Learn” section gives a high level overview of how you should be configuring your cluster to get the most out of it. The “Manage” section gives a load of how-to style guides, so if you want to do a specifc task, it will show you how you to do that by changing your cluster configuration.

If you just want to start developing an application, you can write it with a Couchbase SDK, package it as a container, then run it in Kubernetes along side the cluster. Here’s how to connect to your cluster for example Configure Client SDKs | Couchbase Docs.

Getting error on installing couchbase

apiVersion: v1 kind: Secret metadata: name: cb-example-auth type: Opaque data: username: QWRtaW5pc3RyYXRvcg== # Administrator password: cGFzc3dvcmQ= # password

apiVersion: couchbase.com/v2 kind: CouchbaseBucket metadata: name: default spec: memoryQuota: 2Gi

apiVersion: v1
kind: Secret
metadata:
name: cb-example-auth
type: Opaque
data:
username: QWRtaW5pc3RyYXRvcg== # Administrator
password: cGFzc3dvcmQ= # password

apiVersion: couchbase.com/v2
kind: CouchbaseBucket
metadata:
name: default
spec:
memoryQuota: 2Gi