Xdcr replication between private clusters

Couchbase Server Kubernetes
1

I have two Kubernetes clusters, running at my LAN environment and their services are either exposed via Ingress controller or NodePort. I have a LAN loadbalancer and i can reach the services via a vserver dns record. I had created two load balancer ip adresses that are serving to the admin service.

Here is the services at the destination side:

kubectl get svc -n foo-couchbase-b
NAME                                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                                                                                                                                                                                                                                                                                               AGE
cb-foo                                         ClusterIP   None            <none>        4369/TCP,8091/TCP,8092/TCP,8093/TCP,8094/TCP,8095/TCP,8096/TCP,9100/TCP,9101/TCP,9102/TCP,9103/TCP,9104/TCP,9105/TCP,9110/TCP,9111/TCP,9112/TCP,9113/TCP,9114/TCP,9115/TCP,9116/TCP,9117/TCP,9118/TCP,9120/TCP,9121/TCP,9122/TCP,9130/TCP,9140/TCP,9999/TCP,11207/TCP,11209/TCP,11210/TCP,18091/TCP,18092/TCP,18093/TCP,18094/TCP,18095/TCP,18096/TCP,19130/TCP,21100/TCP,21150/TCP   4d9h
cb-foo-0000                                    NodePort    10.233.61.255   <none>        8091:32556/TCP,18091:30452/TCP,11210:30435/TCP,11207:31297/TCP,8092:32494/TCP,18092:31741/TCP                                                                                                                                                                                                                                                                                         4d9h
cb-foo-0001                                    NodePort    10.233.36.89    <none>        8091:31758/TCP,18091:32409/TCP,11210:32420/TCP,11207:31520/TCP,8092:32198/TCP,18092:30690/TCP                                                                                                                                                                                                                                                                                         4d9h
cb-foo-0002                                    NodePort    10.233.17.245   <none>        8091:30897/TCP,18091:31182/TCP,11210:30310/TCP,11207:31122/TCP,8092:31552/TCP,18092:32119/TCP                                                                                                                                                                                                                                                                                         4d9h
cb-foo-srv                                     ClusterIP   None            <none>        11210/TCP,11207/TCP                                                                                                                                                                                                                                                                                                                                                                   4d9h
cb-foo-xdcr                                    NodePort    10.233.10.114   <none>        8091:32020/TCP,8092:32021/TCP,11210:32022/TCP,18091:32023/TCP                                                                                                                                                                                                                                                                                                                         4d9h
foo-couchbase-couchbase-admission-controller   ClusterIP   10.233.24.133   <none>        443/TCP

The loadbalancer ip 172.18.16.215:8091 is service to the NodePort 32020. Writing 172.18.16.215 to the xdcr target is causing couchbase to convert it to cb-foo-0000.cb-foo.foo-couchbase-b.svc. Where should i install the external-dns (at target or source?) or should i add dns.domain to the helm char t values? I am a bit confused when i read the documentation.

2

Which of the following networking options are you attempting to configure?

Also, it’s good to know what CNI plugin you are using for Kubernetes networking.

3

Inter-Kubernetes Networking with Forwarded DNS, seems the one that fits to my situation. CNI is calico.

4

Great thanks. Now a cursory glance at https://docs.projectcalico.org/networking/determine-best-networking suggests there are a lot of options!

Are you using overlay networking or not? Are the pods “routable” as described in that document?

5

The pods are not routable at the current clusters. They are not being able to reach outside world, they are NATed. Overlay networking is used. Calico community version is being used.

6

Right, with that in mind you have two options:

Public DNS
Each couchbase pod must be exposed either on the public internet using a LoadBalancer service, or you can use an “internal” LoadBalancer service that uses IP addresses on the Kubernetes node network. External DNS is used to replicate the DNS names we assign to the services to a DDNS server; these advertise the load balancer IP. For either of these cases, the IP addresses assigned to the load balancers need to be routable from the XDCR source.

Generic
The XDCR source cluster needs to be able to route to the target Kubernetes node network. These use NodePort services exclusively.

7

Hello @simon.murray
if we use NodePort exclusively, we will not be able to fix the port on the pod config, port will be dynamically allocated , this is why we need to go via a LB.
the setup you described is not always available in such on-prem environments.
can you please let us know if you know why the URL is being changed to "cb-foo-0000.cb-foo.foo-couchbase-b.svc?
is such setup recommended to be used?

8

XDCR should work like a normal client, but it doesn’t sadly, so it will exhibit this strange behaviour. When you initially set the connection, you correctly use a HA load balanced address. XDCR goes off an does its own thing after that, namely, gets the cluster topology, and caches this internally (so it wiil work across restarts provided it can contact one of the nodes it knows about). One of the side effects of this is it notices that the network name you provided is not the same as any single node, so it makes the decision to replace your original suggestion with what it thinks is correct (namely one of the actual node addresses). Not a lot that the operator can do about it, and I’ve suggested to the XDCR team this isn’t the best behaviour in the past, so it may get fixed soon. For now, it will work, so I’d not worry!

9

@simon.murray, your response was helpful to know the behaviour which explained why the IP was changing.

You say, it will work, and yes it does in most scenarios. But we tested one scenario and XDCR broke. The scenario was: all pods of remote cluster died and were recreated. In this scenario, the network name that the source knew did not exist anymore and hence XDCR could not resume. This is where the HA load balanced address could have helped. No?

Thanks

10

Hello @simon.murray
what are the events that will trigger XDCR to fall back to original URL defined for example inside CBC yaml file? will network Failure or nodeports failures push XDCR to restart itself ? or we need to do delete on GUI manually?

11

@pccb that is correct, when using IP based node port addressing, there is a danger that when full DR happens all of the addresses will have changed. This is why my personal preference is to use forwarded DNS with a PVC backed cluster. In that situation, when a disaster occurs, pods will be recovered with the same DNS name and XDCR will continue to function correctly with cached state. The load balancer doesn’t help in this situation due to how XDCR works, the only reason we advertise their use is to make the interface look the same as all the other clients and make it easier to document for you. The load balancer’s service discovery powers are only really used on the initial connection.

@Omar_Hentati there is nothing that will cause it to reset it back to the original requested address. Because Couchbase server doesn’t allow us to read what we originally wrote, we have to cache that value locally to watch for changes (like usernames and passwords etc). My advice would be just to remove the XDCR remote from the couchbase cluster resource and re-add it.

12

got it.

what if we configure XDCR to use DNS to resolve statically to LB IP? also we put TTL as low as 30 send for that DNS record>?

13

Still wouldn’t work fully, if you set it to connect to foo.my-domain.com:8091 the first thing XDCR will do is cache 172.16.0.2:30452, 172.16.0.3:30123, 172.16.0.3:31234 etc (e.g. the node port IP addresses) and discard that external addressing information. It will work for intial bootstrap, sure, but not be totally HA in the event of a full cluster rebuild.

14

ok, any recommendation on a customised script that will delete replication in case of remote cluster issue ?

15

Thanks @simon.murray.

(referring your reply to Omar_Hentati) The problem with removing XDCR remote and re-adding is that XDCR will not resume; instead it will re-sync from scratch. I do understand that it will not replicate all docs again, but it will still need to verify each doc and on a cluster with multiple buckets each having, say, 50m docs, can take quite a bit of time and compute as well network bandwidth.