Multitasking using one Cluster instance

.NET SDK
1

I’m working with multiple buckets through the BucketManager and I want to run these operations in parallel. Can I reuse the Cluster for each bucket or should I have a Cluster instance per Bucket to be able to run in parallel?

At the moment I’m pinging 6 buckets and I want to do that in parallel. One cluster instance or one per bucket?

2

The SDK is specifically designed to support parallelism. You should have one cluster for the application lifetime, and then one of each bucket for the application lifetime. Don’t call Dispose on any of them until application shutdown.

3

That’s great. I’m seeing some issues when I did this. I will try again and post back here.

4

I canøt get IBucket.PingAsync to work in parallel using one cluster instance.

This is what I’m doing.

Psedo code:

cluster = Cluster.ConnectAsync(..)
cluster.PingAsync()
delay 5 sec

parallel foreach bucketName in bucketNames 
     bucket = cluster.GetBucketAsync()
     bucket.PingAsync()

The very first bucket ping succeeds but the following fails. The bucket does exists and the parallel code works if a have a cluster per bucket.

5

What is your purpose in runing these ping reports? Are you just experimenting right now, or is it for production work?

The ping reports are primarily intended for diagnostics and perhaps for health checks. I’m honestly not sure how well they handle parallelization. The ping reports have a lot of lift in them, testing multiple services and calculating latencies and such. My earlier comments about parallelization were primarily focused on the key/value and query operations of buckets, rather than the ping.

Pinging is also a pretty new API surface, so it wouldn’t surprise me if there are bugs. I think this code should work as you have written it, so it’s probably worth a bug report.

6

I’m using BucketManager to set up my buckets for testing. I need to make sure that the buckets are created, flushed and ready (healthy) before executing the automated test.

1 Like
7

Okay, seems like there may be some bugs in Ping for that use case.

However, I may be able to offer an alternative, depending on your specific automated test use case. At CenterEdge, we use Docker for our automated test runs. In our case, Couchbase is a container, the service under test is a container, and the test suite itself is a container (plus sometimes some extra containers for other services).

For Couchbase, we created a special Docker image that supports full configuration on startup (buckets, indexes, fake data, etc) all based on declarative files that can be checked into source control. Once the initialization is complete, a file is written to disk on the container. If the folder is mapped between containers using a shared Docker volume, the other containers can watch for this file to know when Couchbase is up and running.

In case this helps, the system is available here: https://github.com/brantburnett/couchbasefakeit and there are images on Docker Hub for a lot of different Couchbase versions: https://hub.docker.com/r/btburnett3/couchbasefakeit. We use this approach extensively for automated functional testing almost all of our microservices as part of our CI/CD pipeline.

2 Likes