Is it possible to access CB cluster on K8s with NodePort Service?

I’m setting up couchbase cluster using Autonomous Operator 2.0.3 on Kubernetes 1.18.
K8s cluster is on private network without public IPs .

Now I’ve created a cb-example cluster, and intra-k8s applications can access it with at automatically created cb-example-srv service (connection string = couchbase://cb-example-srv.couchbase:11210).

What I need

I want to access CB database from external machine (e.g. dev).

For simplicity I started with NodePort Networking (Generic Networking), but now I’m not sure it’s even possible.
If you know how to setup Public Networking with external-dns on LAN, that would be an even better solution!

What I read

| IP based addressing is only used when connecting a client SDK with a node-port service.
| Both Public Networking with External DNS and Generic Networking require client traffic to cross as DNAT boundary.

But I have no idea what that means.

What I tried

I’ve added this to cluster.yaml

spec:
  image: couchbase/server:6.6.0
  networking:
    exposeAdminConsole: true
    exposedFeatures: 
    - client

operator created the this two services for me: svc/cb-example-0000 and svc/cb-example-ui.
UI can access on both of their NodePorts that map to 8091.

However Client my java client (SDK 3.0) can’t access the cluster. Constantly times out.

kubectl get svc/cb-example-0000 -n couchbase

NAME              TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                                                                                                                                                                                     AGE
cb-example-0000   NodePort   10.43.221.33   <none>        8091:31888/TCP, .... ,11210:30562/TCP,11207:31959/TCP   66m

Pod cb-example-0000 is running on node with IP 172.25.18.66.
And with connection-string= couchbase://172.25.18.66:30562 the following test fails:

@Test
    public void getTestDoc() {
        var map = bucket
                .defaultCollection().reactive()
                .get(TEST_DOC_ID)
                .map(gr -> gr.contentAs(Map.class))
                .block();
        System.out.println(map);
    }
com.couchbase.client.core.error.UnambiguousTimeoutException: GetRequest, Reason: TIMEOUT
 {"cancelled":true,"completed":true,"coreId":"0x36d0df1e00000002","idempotent":true,"reason":"TIMEOUT","requestId":3,"requestType":"GetRequest","retried":15,"retryReasons":["BUCKET_OPEN_IN_PROGRESS"],"service":{"bucket":"default","collection":"_default","documentId":"test-doc","opaque":"0x12","scope":"_default","type":"kv"},"timeoutMs":2500,"timings":{"totalMicros":2511832}}
	at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:163)
	at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
	at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
	at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
	at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
	at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:832)

Thanks for any help!

UPD

as ingenthr suggested in his reply to my older post, I ran SDK-doctor:

./sdk-doctor-macos diagnose couchbase://172.25.18.66:30562/default -u Administrator -p password

|====================================================================|
|          ___ ___  _  __   ___   ___   ___ _____ ___  ___           |
|         / __|   \| |/ /__|   \ / _ \ / __|_   _/ _ \| _ \          |
|         \__ \ |) | ' <___| |) | (_) | (__  | || (_) |   /          |
|         |___/___/|_|\_\  |___/ \___/ \___| |_| \___/|_|_\          |
|                                                                    |
|====================================================================|

Note: Diagnostics can only provide accurate results when your cluster
 is in a stable state.  Active rebalancing and other cluster configuration
 changes can cause the output of the doctor to be inconsistent or in the
 worst cases, completely incorrect.

00:06:34.333 INFO ▶ Parsing connection string `couchbase://172.25.18.66:30562/default`
00:06:34.333 INFO ▶ Connection string identifies the following CCCP endpoints:
00:06:34.333 INFO ▶   1. 172.25.18.66:30562
00:06:34.333 INFO ▶ Connection string identifies the following HTTP endpoints:
00:06:34.333 INFO ▶ Connection string specifies bucket `default`
00:06:34.333 WARN ▶ Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
00:06:34.333 INFO ▶ Performing DNS lookup for host `172.25.18.66`
00:06:34.333 INFO ▶ Attempting to connect to cluster via CCCP
00:06:34.333 INFO ▶ Attempting to fetch config via cccp from `172.25.18.66:30562`
00:06:35.585 WARN ▶ Bootstrap host `172.25.18.66` is not using the canonical node hostname of `cb-example-0000.cb-example.couchbase.svc`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
00:06:35.585 INFO ▶ Selected the following network type: external
00:06:35.585 INFO ▶ Identified the following nodes:
00:06:35.585 INFO ▶   [0] 172.25.18.66
00:06:35.586 INFO ▶                       kv: 30562,                kvSSL: 31959,              mgmtSSL: 30336
00:06:35.586 INFO ▶                  capiSSL: 30222,                  fts: 30434,    eventingAdminPort: 30563
00:06:35.586 INFO ▶                     capi: 30927,              n1qlSSL: 30084,              cbasSSL: 30186
00:06:35.586 INFO ▶                     n1ql: 32052,               ftsSSL: 31518,          eventingSSL: 30970
00:06:35.586 INFO ▶                     cbas: 31732,                 mgmt: 31888
00:06:35.586 INFO ▶ Fetching config from `http://172.25.18.66:31888`
00:06:36.398 INFO ▶ Received cluster configuration, nodes list:
nodes list:
  {
    "addressFamily": "inet",
    "alternateAddresses": {
      "external": {
        "hostname": "172.25.18.66",
        "ports": {
          "capi": 30927,
          "capiSSL": 30222,
          "kv": 30562,
          "mgmt": 31888,
          "mgmtSSL": 30336
        }
      }
    },
    "clusterCompatibility": 393222,
    "clusterMembership": "active",
    "configuredHostname": "cb-example-0000.cb-example.couchbase.svc:8091",
    "couchApiBase": "http://cb-example-0000.cb-example.couchbase.svc:8092/",
    "couchApiBaseHTTPS": "https://cb-example-0000.cb-example.couchbase.svc:18092/",
    "cpuCount": 4,
    "externalListeners": [
      {
        "afamily": "inet",
        "nodeEncryption": false
      },
      {
        "afamily": "inet6",
        "nodeEncryption": false
      }
    ],
    "hostname": "cb-example-0000.cb-example.couchbase.svc:8091",
    "interestingStats": {
      "cmd_get": 0,
      "couch_docs_actual_disk_size": 12275543,
      "couch_docs_data_size": 4253741,
      "couch_spatial_data_size": 0,
      "couch_spatial_disk_size": 0,
      "couch_views_actual_disk_size": 0,
      "couch_views_data_size": 0,
      "curr_items": 1,
      "curr_items_tot": 1,
      "ep_bg_fetched": 0,
      "get_hits": 0,
      "mem_used": 42884624,
      "ops": 0,
      "vb_active_num_non_resident": 0,
      "vb_replica_curr_items": 0
    },
    "mcdMemoryAllocated": 7417,
    "mcdMemoryReserved": 7417,
    "memoryFree": 7308726272,
    "memoryTotal": 9722867712,
    "nodeEncryption": false,
    "nodeUUID": "1c08d9a294135c9f8c7195a4d303e897",
    "os": "x86_64-unknown-linux-gnu",
    "otpNode": "ns_1@cb-example-0000.cb-example.couchbase.svc",
    "ports": {
      "direct": 11210,
      "distTCP": 21100,
      "distTLS": 21150,
      "httpsCAPI": 18092,
      "httpsMgmt": 18091
    },
    "recoveryType": "none",
    "services": [
      "cbas",
      "eventing",
      "fts",
      "index",
      "kv",
      "n1ql"
    ],
    "status": "healthy",
    "systemStats": {
      "allocstall": 0,
      "cpu_cores_available": 4,
      "cpu_stolen_rate": 0,
      "cpu_utilization_rate": 10.76115485564304,
      "mem_free": 7308726272,
      "mem_limit": 9722867712,
      "mem_total": 9722867712,
      "swap_total": 4294963200,
      "swap_used": 0
    },
    "thisNode": true,
    "uptime": "27578",
    "version": "6.6.0-7909-enterprise"
  }
]
00:06:36.753 INFO ▶ Successfully connected to Key Value service at `172.25.18.66:30562`
00:06:37.022 INFO ▶ Successfully connected to Management service at `172.25.18.66:31888`
00:06:37.237 INFO ▶ Successfully connected to Views service at `172.25.18.66:30927`
00:06:37.414 INFO ▶ Successfully connected to Query service at `172.25.18.66:32052`
00:06:37.592 INFO ▶ Successfully connected to Search service at `172.25.18.66:30434`
00:06:37.780 INFO ▶ Successfully connected to Analytics service at `172.25.18.66:31732`
00:06:39.018 INFO ▶ Memd Nop Pinged `172.25.18.66:30562` 10 times, 0 errors, 87ms min, 90ms max, 88ms mean
00:06:39.018 WARN ▶ Memcached service on `172.25.18.66:30562` on average took longer than 10ms (was: 88ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
00:06:39.018 WARN ▶ Memcached service on `172.25.18.66:30562` maximally took longer than 20ms (was: 90ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
00:06:39.018 INFO ▶ Diagnostics completed

Summary:
[WARN] Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
[WARN] Bootstrap host `172.25.18.66` is not using the canonical node hostname of `cb-example-0000.cb-example.couchbase.svc`.  This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Memcached service on `172.25.18.66:30562` on average took longer than 10ms (was: 88ms) to reply.  This is usually due to network-related issues, and could significantly affect application performance.
[WARN] Memcached service on `172.25.18.66:30562` maximally took longer than 20ms (was: 90ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.

Found multiple issues, see listing above.

Hi @captain.varabei,

Please check inline.

What I need -

I want to access CB database from external machine (e.g. dev).

[Couchbase] Yes, supported with Couchbase Autonomous Operator

For simplicity I started with NodePort Networking (Generic Networking), but now I’m not sure it’s even possible.

[Couchbase] NodePort networking is the simplest and allows clients deployed in the same Kubernetes cluster as the Couchbase server instance. Check IP Based Addressing used when connecting a client SDK with a node-port service.

If you know how to setup Public Networking with external-dns on LAN, that would be an even better solution!

[Couchbase] Yes, read the networking documentation on all the possible networking topologies for connecting clients to a Couchbase cluster. In your case you will be following Public Networking with External DNS or using Exposed Features.

Check tutorials and blogs for examples on how to access Couchbase cluster deployed within Kubernetes from externally deployed Client SDKs.

Thanks!

1 Like