I’m setting up couchbase cluster using Autonomous Operator 2.0.3 on Kubernetes 1.18.
K8s cluster is on private network without public IPs .
Now I’ve created a cb-example cluster, and intra-k8s applications can access it with at automatically created cb-example-srv service (connection string = couchbase://cb-example-srv.couchbase:11210).
What I need
I want to access CB database from external machine (e.g. dev).
For simplicity I started with NodePort Networking (Generic Networking), but now I’m not sure it’s even possible.
If you know how to setup Public Networking with external-dns on LAN, that would be an even better solution!
What I read
| IP based addressing is only used when connecting a client SDK with a node-port service.
| Both Public Networking with External DNS and Generic Networking require client traffic to cross as DNAT boundary.
But I have no idea what that means.
What I tried
I’ve added this to cluster.yaml
spec:
image: couchbase/server:6.6.0
networking:
exposeAdminConsole: true
exposedFeatures:
- client
operator created the this two services for me: svc/cb-example-0000 and svc/cb-example-ui.
UI can access on both of their NodePorts that map to 8091.
However Client my java client (SDK 3.0) can’t access the cluster. Constantly times out.
kubectl get svc/cb-example-0000 -n couchbase
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cb-example-0000 NodePort 10.43.221.33 <none> 8091:31888/TCP, .... ,11210:30562/TCP,11207:31959/TCP 66m
Pod cb-example-0000 is running on node with IP 172.25.18.66.
And with connection-string= couchbase://172.25.18.66:30562 the following test fails:
@Test
public void getTestDoc() {
var map = bucket
.defaultCollection().reactive()
.get(TEST_DOC_ID)
.map(gr -> gr.contentAs(Map.class))
.block();
System.out.println(map);
}
com.couchbase.client.core.error.UnambiguousTimeoutException: GetRequest, Reason: TIMEOUT
{"cancelled":true,"completed":true,"coreId":"0x36d0df1e00000002","idempotent":true,"reason":"TIMEOUT","requestId":3,"requestType":"GetRequest","retried":15,"retryReasons":["BUCKET_OPEN_IN_PROGRESS"],"service":{"bucket":"default","collection":"_default","documentId":"test-doc","opaque":"0x12","scope":"_default","type":"kv"},"timeoutMs":2500,"timings":{"totalMicros":2511832}}
at com.couchbase.client.core.msg.BaseRequest.cancel(BaseRequest.java:163)
at com.couchbase.client.core.Timer.lambda$register$2(Timer.java:157)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747)
at com.couchbase.client.core.deps.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472)
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:832)
Thanks for any help!
UPD
as ingenthr suggested in his reply to my older post, I ran SDK-doctor:
./sdk-doctor-macos diagnose couchbase://172.25.18.66:30562/default -u Administrator -p password
|====================================================================|
| ___ ___ _ __ ___ ___ ___ _____ ___ ___ |
| / __| \| |/ /__| \ / _ \ / __|_ _/ _ \| _ \ |
| \__ \ |) | ' <___| |) | (_) | (__ | || (_) | / |
| |___/___/|_|\_\ |___/ \___/ \___| |_| \___/|_|_\ |
| |
|====================================================================|
Note: Diagnostics can only provide accurate results when your cluster
is in a stable state. Active rebalancing and other cluster configuration
changes can cause the output of the doctor to be inconsistent or in the
worst cases, completely incorrect.
00:06:34.333 INFO ▶ Parsing connection string `couchbase://172.25.18.66:30562/default`
00:06:34.333 INFO ▶ Connection string identifies the following CCCP endpoints:
00:06:34.333 INFO ▶ 1. 172.25.18.66:30562
00:06:34.333 INFO ▶ Connection string identifies the following HTTP endpoints:
00:06:34.333 INFO ▶ Connection string specifies bucket `default`
00:06:34.333 WARN ▶ Your connection string specifies only a single host. You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
00:06:34.333 INFO ▶ Performing DNS lookup for host `172.25.18.66`
00:06:34.333 INFO ▶ Attempting to connect to cluster via CCCP
00:06:34.333 INFO ▶ Attempting to fetch config via cccp from `172.25.18.66:30562`
00:06:35.585 WARN ▶ Bootstrap host `172.25.18.66` is not using the canonical node hostname of `cb-example-0000.cb-example.couchbase.svc`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
00:06:35.585 INFO ▶ Selected the following network type: external
00:06:35.585 INFO ▶ Identified the following nodes:
00:06:35.585 INFO ▶ [0] 172.25.18.66
00:06:35.586 INFO ▶ kv: 30562, kvSSL: 31959, mgmtSSL: 30336
00:06:35.586 INFO ▶ capiSSL: 30222, fts: 30434, eventingAdminPort: 30563
00:06:35.586 INFO ▶ capi: 30927, n1qlSSL: 30084, cbasSSL: 30186
00:06:35.586 INFO ▶ n1ql: 32052, ftsSSL: 31518, eventingSSL: 30970
00:06:35.586 INFO ▶ cbas: 31732, mgmt: 31888
00:06:35.586 INFO ▶ Fetching config from `http://172.25.18.66:31888`
00:06:36.398 INFO ▶ Received cluster configuration, nodes list:
nodes list:
{
"addressFamily": "inet",
"alternateAddresses": {
"external": {
"hostname": "172.25.18.66",
"ports": {
"capi": 30927,
"capiSSL": 30222,
"kv": 30562,
"mgmt": 31888,
"mgmtSSL": 30336
}
}
},
"clusterCompatibility": 393222,
"clusterMembership": "active",
"configuredHostname": "cb-example-0000.cb-example.couchbase.svc:8091",
"couchApiBase": "http://cb-example-0000.cb-example.couchbase.svc:8092/",
"couchApiBaseHTTPS": "https://cb-example-0000.cb-example.couchbase.svc:18092/",
"cpuCount": 4,
"externalListeners": [
{
"afamily": "inet",
"nodeEncryption": false
},
{
"afamily": "inet6",
"nodeEncryption": false
}
],
"hostname": "cb-example-0000.cb-example.couchbase.svc:8091",
"interestingStats": {
"cmd_get": 0,
"couch_docs_actual_disk_size": 12275543,
"couch_docs_data_size": 4253741,
"couch_spatial_data_size": 0,
"couch_spatial_disk_size": 0,
"couch_views_actual_disk_size": 0,
"couch_views_data_size": 0,
"curr_items": 1,
"curr_items_tot": 1,
"ep_bg_fetched": 0,
"get_hits": 0,
"mem_used": 42884624,
"ops": 0,
"vb_active_num_non_resident": 0,
"vb_replica_curr_items": 0
},
"mcdMemoryAllocated": 7417,
"mcdMemoryReserved": 7417,
"memoryFree": 7308726272,
"memoryTotal": 9722867712,
"nodeEncryption": false,
"nodeUUID": "1c08d9a294135c9f8c7195a4d303e897",
"os": "x86_64-unknown-linux-gnu",
"otpNode": "ns_1@cb-example-0000.cb-example.couchbase.svc",
"ports": {
"direct": 11210,
"distTCP": 21100,
"distTLS": 21150,
"httpsCAPI": 18092,
"httpsMgmt": 18091
},
"recoveryType": "none",
"services": [
"cbas",
"eventing",
"fts",
"index",
"kv",
"n1ql"
],
"status": "healthy",
"systemStats": {
"allocstall": 0,
"cpu_cores_available": 4,
"cpu_stolen_rate": 0,
"cpu_utilization_rate": 10.76115485564304,
"mem_free": 7308726272,
"mem_limit": 9722867712,
"mem_total": 9722867712,
"swap_total": 4294963200,
"swap_used": 0
},
"thisNode": true,
"uptime": "27578",
"version": "6.6.0-7909-enterprise"
}
]
00:06:36.753 INFO ▶ Successfully connected to Key Value service at `172.25.18.66:30562`
00:06:37.022 INFO ▶ Successfully connected to Management service at `172.25.18.66:31888`
00:06:37.237 INFO ▶ Successfully connected to Views service at `172.25.18.66:30927`
00:06:37.414 INFO ▶ Successfully connected to Query service at `172.25.18.66:32052`
00:06:37.592 INFO ▶ Successfully connected to Search service at `172.25.18.66:30434`
00:06:37.780 INFO ▶ Successfully connected to Analytics service at `172.25.18.66:31732`
00:06:39.018 INFO ▶ Memd Nop Pinged `172.25.18.66:30562` 10 times, 0 errors, 87ms min, 90ms max, 88ms mean
00:06:39.018 WARN ▶ Memcached service on `172.25.18.66:30562` on average took longer than 10ms (was: 88ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
00:06:39.018 WARN ▶ Memcached service on `172.25.18.66:30562` maximally took longer than 20ms (was: 90ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
00:06:39.018 INFO ▶ Diagnostics completed
Summary:
[WARN] Your connection string specifies only a single host. You should consider adding additional static nodes from your cluster to this list to improve your applications fault-tolerance
[WARN] Bootstrap host `172.25.18.66` is not using the canonical node hostname of `cb-example-0000.cb-example.couchbase.svc`. This is not neccessarily an error, but has been known to result in strange and challenging to diagnose errors when DNS entries are reconfigured.
[WARN] Memcached service on `172.25.18.66:30562` on average took longer than 10ms (was: 88ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
[WARN] Memcached service on `172.25.18.66:30562` maximally took longer than 20ms (was: 90ms) to reply. This is usually due to network-related issues, and could significantly affect application performance.
Found multiple issues, see listing above.