Operator crashes when creating a cluster

ydkn · June 19, 2020, 10:52am

Hi everyone,

I ran into a problem setting up the operator on kubernetes. It crashes every time I create a new cluster and after the restart of the operator it is not trying to get the cluster up and running and (existing cluster failed during prior initialization, state unknown)
If I add the following resources to the kubernetes cluster (example from https://docs.couchbase.com/operator/2.0/howto-couchbase-create.html with added namespace and missing memory quotas):

apiVersion: v1
kind: Secret
metadata:
  name: cb-example-auth
  namespace: couchbase-instances
type: Opaque
data:
  username: QWRtaW5pc3RyYXRvcg==  # Administrator
  password: cGFzc3dvcmQ=          # password
---
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
  name: cb-example
  namespace: couchbase-instances
spec:
  image: couchbase/server:6.5.0
  cluster:
    dataServiceMemoryQuota: 1Gi
    indexServiceMemoryQuota: 1Gi
    searchServiceMemoryQuota: 1Gi
    eventingServiceMemoryQuota: 1Gi
    analyticsServiceMemoryQuota: 1Gi
    indexStorageSetting: memory_optimized
    autoFailoverTimeout: 10m
    autoFailoverMaxCount: 1
  security:
    adminSecret: cb-example-auth
  networking:
    exposeAdminConsole: true
    adminConsoleServices:
    - data
  buckets:
    managed: true
  servers:
  - size: 3
    name: all_services
    services:
    - data
    - index
    - query
    - search
    - eventing
    - analytics

The operator crashes with:

2020-06-19T10:45:47.769Z INFO cluster Watching new cluster {"cluster": "couchbase-instances/cb-example"}
2020-06-19T10:45:47.774Z INFO cluster Janitor starting {"cluster": "couchbase-instances/cb-example"}
2020-06-19T10:45:47.788Z INFO cluster Couchbase client starting {"cluster": "couchbase-instances/cb-example"}
2020-06-19T10:45:47.964Z INFO cluster UI service created {"cluster": "couchbase-instances/cb-example", "name": "cb-example-ui"}
2020-06-19T10:45:47.984Z INFO cluster Cluster does not exist so the operator is attempting to create it {"cluster": "couchbase-instances/cb-example"}
2020-06-19T10:45:48.060Z INFO cluster Creating pod {"cluster": "couchbase-instances/cb-example", "name": "cb-example-0000", "image": "couchbase/server:6.5.0"}
2020-06-19T10:46:04.218Z DEBUG client http {"method": "GET", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/pools", "status": "200 OK", "time_ms": 15.939096}
2020-06-19T10:46:04.238Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/node/controller/rename", "status": "400 Bad Request", "time_ms": 19.547497}
2020-06-19T10:46:09.234Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/node/controller/rename", "status": "400 Bad Request", "time_ms": 14.740622}
2020-06-19T10:46:14.565Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/node/controller/rename", "status": "200 OK", "time_ms": 345.861938}
2020-06-19T10:46:14.579Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/nodes/self/controller/settings", "status": "200 OK", "time_ms": 13.977016}
2020-06-19T10:46:14.609Z INFO cluster Operator added member {"cluster": "couchbase-instances/cb-example", "name": "cb-example-0000"}
2020-06-19T10:46:14.625Z INFO cluster Initial pod creating {"cluster": "couchbase-instances/cb-example"}
2020-06-19T10:46:14.653Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/node/controller/rename", "status": "200 OK", "time_ms": 27.606468}
2020-06-19T10:46:14.659Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/nodes/self/controller/settings", "status": "200 OK", "time_ms": 5.47316}
2020-06-19T10:46:24.707Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/pools/default", "status": "200 OK", "time_ms": 46.460374}
2020-06-19T10:46:24.713Z DEBUG client http {"method": "GET", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/settings/indexes", "status": "200 OK", "time_ms": 6.553759}
2020-06-19T10:46:24.722Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/settings/indexes", "status": "200 OK", "time_ms": 8.455907}
2020-06-19T10:46:24.730Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/node/controller/setupServices", "status": "200 OK", "time_ms": 8.123087}
2020-06-19T10:46:24.784Z DEBUG client http {"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/settings/web", "status": "200 OK", "time_ms": 53.658751}
E0619 10:46:24.785710 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/1.13.3/go/src/runtime/panic.go:679
/usr/local/go/1.13.3/go/src/runtime/panic.go:199
/usr/local/go/1.13.3/go/src/runtime/signal_unix.go:394
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/util/k8sutil/k8sutil.go:446
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:664
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:296
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:209
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:142
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:71
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/1.13.3/go/src/runtime/asm_amd64.s:1357
E0619 10:46:24.785710 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/1.13.3/go/src/runtime/panic.go:679
/usr/local/go/1.13.3/go/src/runtime/panic.go:199
/usr/local/go/1.13.3/go/src/runtime/signal_unix.go:394
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/util/k8sutil/k8sutil.go:446
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:664
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:296
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:209
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:142
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:71
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/1.13.3/go/src/runtime/asm_amd64.s:1357
E0619 10:46:24.785710 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/1.13.3/go/src/runtime/panic.go:679
/usr/local/go/1.13.3/go/src/runtime/panic.go:199
/usr/local/go/1.13.3/go/src/runtime/signal_unix.go:394
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/util/k8sutil/k8sutil.go:446
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:664
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:296
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:209
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:142
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:71
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/1.13.3/go/src/runtime/asm_amd64.s:1357
E0619 10:46:24.785710 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/1.13.3/go/src/runtime/panic.go:679
/usr/local/go/1.13.3/go/src/runtime/panic.go:199
/usr/local/go/1.13.3/go/src/runtime/signal_unix.go:394
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/util/k8sutil/k8sutil.go:446
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:664
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:296
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:209
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:142
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:71
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/1.13.3/go/src/runtime/asm_amd64.s:1357
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1161479]
goroutine 313 [running]:
github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x12f7700, 0x21169c0)
/usr/local/go/1.13.3/go/src/runtime/panic.go:679 +0x1b2
github.com/couchbase/couchbase-operator/pkg/util/k8sutil.Seconds(...)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/util/k8sutil/k8sutil.go:446
github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).initMember(0xc00026c540, 0xc0000d1590, 0x3, 0xc000377330, 0xc, 0xc00010c660, 0x6, 0x6, 0x0, 0x0, ...)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:664 +0x3c9
github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).create(0xc00026c540, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:296 +0x557
github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).setup(0xc00026c540, 0x151ce58, 0xc000b55d00)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:209 +0x325
github.com/couchbase/couchbase-operator/pkg/cluster.New(0x14893cd, 0x3, 0xc000ad9200, 0xc0008aa460, 0x2, 0x2)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:142 +0x714
github.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile(0xc000566db0, 0xc0005805e0, 0x13, 0xc0003771b4, 0xa, 0x5, 0x500, 0xc000259000, 0xd)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/pkg/controller/controller.go:71 +0xbc8
github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000ca960, 0x7fb254111a00)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x20a
github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1()
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36
github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0001c8d30)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e
github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0001c8d30, 0x3b9aca00, 0x0, 0x1, 0xc0003ca120)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8
github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc0001c8d30, 0x3b9aca00, 0xc0003ca120)
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/home/couchbase/jenkins/workspace/couchbase-operator-build/goproj/src/github.com/couchbase/couchbase-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x32e

It would be great if the operator would not crash but give a error message on what is actually wrong.

Does anybody has an idea what could be wrong?
Operator and instances are in different namespaces.

Thanks,
Florian

simon.murray · June 19, 2020, 11:31am

The crash is related to a missing time value (specifically spec.cluster.autofailoverOnDataDiskIssuesTimePeriod which is a required attribute as per https://docs.couchbase.com/operator/2.0/reference-couchbasecluster.html#spec-cluster-autofailoverondatadiskissuestimeperiod, although I note the CRD validation doesn’t require it, good bug!). These should all be filled in for you by the dynamic admission controller, which is why it is a required component of running the operator. Are you running this and is it working?

ydkn · June 22, 2020, 5:10pm

Hi Simon,

thank you for the fast response - and sorry for my late reponse. I just wanted to make sure I was on the right track…

I fixed the missing key of the couchbase cluster CRD but that didn’t solve the problem.

But I figured out that if I use the example operator.yml in the tarball from (https://www.couchbase.com/downloads) and change the namespace to something other than default (e.g. cochbase-operator) that it crashes with the stacktrace of the original message:

2020-06-22T16:40:23.622Z	DEBUG	client	http	{"method": "POST", "url": "http://cb-example-0000.cb-example.couchbase-instances.svc:8091/settings/web", "status": "200 OK", "time_ms": 67.417702}
E0622 16:40:23.622479       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

Couchbase instances are also in a different namespace but it only fails when the operator is not in the default namespace.
I can provide the modified operator.yml if necessary.
If I deploy the operator to the default namespace than everything works (tested on 2 different k8s clusters).

Tested it with both operator version 2.0.0 and 2.0.1.

ydkn · June 22, 2020, 9:00pm

Hi again,

I figured out what the problem was and fixed it: It is the example CA and certificate!
The strange thing is that k8s doesn’t complain and the couchbasecluster resource is accepted while the admission hooks are present so I was thinking it is ok to use it for testing. But the operator itself seems to somehow relies on it to be valid and apparently some error or state is not checked so it crashes with “invalid memory address or nil pointer dereference”.

I think this invalid state should be checked and the operator should fail gracefully instead of crashing without any hint what is actually wrong. (I would contribute but the repo/source of the operator doesn’t seem to be public)

simon.murray · June 23, 2020, 7:58am

TLS certificates are fully checked by the admission controller:

Error from server: error when creating "example/tls/couchbase-cluster.yaml": admission webhook "couchbase-operator-admission.default.svc" denied the request: validation failure list:
certificate cannot be verified for zone: x509: certificate is valid for *.cb-example.default.svc, localhost, not host.cb-example
certificate cannot be verified for zone: x509: certificate is valid for *.cb-example.default.svc, localhost, not host.cb-example.random
certificate cannot be verified for zone: x509: certificate is valid for *.cb-example.default.svc, localhost, not host.cb-example.random.svc
certificate cannot be verified for zone: x509: certificate is valid for *.cb-example.default.svc, localhost, not cb-example-srv
certificate cannot be verified for zone: x509: certificate is valid for *.cb-example.default.svc, localhost, not cb-example-srv.random
certificate cannot be verified for zone: x509: certificate is valid for *.cb-example.default.svc, localhost, not cb-example-srv.random.svc

As I say, both of these observations are indicative that it hasn’t been installed correctly or is not working.

Rg · November 18, 2020, 12:08pm

Sadly, this issue seems to be broader than expected.
I got this same error message, I couldn’t guess where it’s coming from until I noticed that I didn’t configure
my memory allocation properly.
The bottom line is, go over all your configuration files and make sure that everything is in the right place.