Connection failure for Couchbase behind Istio in EKS

Couchbase Server Kubernetes
1

Hi,

I have a Couchbase Cluster deployed in a EKS K8s cluster behind an Istio.

My main goal is to use an MSK (Kafka) Connect. It’s failing, (TimeoutException: Did not observe any item or terminal signal within 5000ms in 'source(MonoDefer)') so I was debugging using SDK Doctor.

I have in my K8s cluster the Couchbase Autonomous Operator and I defined the cluster using:

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
  name: couchbase-cluster
spec:
  image: couchbase/server:7.1.3
  security:
    adminSecret: couchbase-cluster-auth
  networking:
    dns:
      domain: couchbase-cluster.couchbase-demo.io
    networkPlatform: Istio
  buckets:
    managed: false
  servers:
  - size: 3
    name: all_services
    services:
    - data
    - index
    - query
    - search
    - eventing
    - analytics

The Istio exposes the ports 9081-9085 and 11210 as follows:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway-tool
  namespace: istio-routing
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
    - port:
        name: http
        number: 8091
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: xcdr
        number: 8092
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: query
        number: 8093
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: search
        number: 8094
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: analytics
        number: 8095
        protocol: HTTP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
    - port:
        name: kv
        number: 11210
        protocol: TCP
      hosts:
        - '*.couchbase-demo.io'
        - couchbase-demo.io
        - ./*
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: couchbase-virtual-service
spec:
  hosts:
    - couchbase-cluster.couchbase-demo.io
  gateways:
    - istio-routing/gateway-tool
  http:
    - match:
        - port: 8091
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8091
    - match:
        - port: 8092
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8092
    - match:
        - port: 8093
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8093
    - match:
        - port: 8094
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8094
    - match:
        - port: 8095
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 8095
  tcp:
    - match:
        - port: 11210
      route:
        - destination:
            host: couchbase-cluster
            port:
              number: 11210

All those ports are open in the istio-ingressgateway as TCP.
After binding the ingress load balancer with the (private) DNS couchbase-cluster.couchbase-demo.io, I am able to query the cluster. E.g:

curl -v -X GET -u $cb_usr:$cb_pwd http://couchbase-cluster.couchbase-demo.io:8091/pools/default/nodeServices 

curl -v -X POST -u $cb_usr:$cb_pwd http://couchbase-cluster.couchbase-demo.io:8093/query/service --data 'statement=select meta().id from `beer-sample`.`_default`.`_default` data order by meta().id limit 2 offset 0'

The output I get is:

[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0003.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0003.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0004.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0004.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Key Value service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:11210` (error: dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Management service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8091` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8091/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Views service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8092` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8092/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Query service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8093` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8093/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Search service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8094` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8094/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)
[ERRO] Failed to connect to Analytics service at `couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8095` (error: Get "http://couchbase-cluster-0005.couchbase-cluster.couchbase.svc:8095/": dial tcp: lookup couchbase-cluster-0005.couchbase-cluster.couchbase.svc: no such host)

When I run sdk-doctor from a pod inside the cluster, it works fine, so I think that the problem is that some networking setting is not right.

Does anyone know why it gets lost when it tries to access the couchbase-cluster-000*.couchbase-cluster.couchbase.svc ? What am I missing?

Thanks,

Manuel

3

Hey Manuel,

I’m guessing you’re wanting to connect Kafka to Couchbase server from another location. I think you will first need to expose client services via the exposedFeatures feature. Secondly, you will need to allow your istio ingress to talk to the resulting client services created for access.

This feature allows you to specify external DNS and apply those DNS entries to Couchbase server, so that a client sdk can connect to any node within the cluster. See our documentation on external dns for more information.

Thanks,

Justin Ashworth