com.couchbase.client.core.error.UnambiguousTimeoutException : GetMetaRequest

Hello All,

First of all , we are using java couchbase-client-version -> 3.0.3 // we tried also versions 3.0.4 and 3.0.5
and Server version: “6.5.1-6299-enterprise” with (3 data nodes ,1 eventing node,2 index nodes and 1 query node).

We are using only data node ips on property file. That’s all about configuration .

fun eventManagementCollection(eventManagementBucket: Bucket): Collection {
    return eventManagementBucket.defaultCollection()

(also works on Enterprise Edition 6.0.0 build 1693 )

After running method as written below it waits and throws that error com.couchbase.client.core.error.UnambiguousTimeoutException : GetMetaRequest

Note that,
When we query using cluster.query(…). it seems ok. we get documents.

By the way same exceptions we got other applications that use java client for couhbase as below:

com.couchbase.client.core.error.UnambiguousTimeoutException: GetRequest
    at com.couchbase.client.core.msg.BaseRequest.cancel(
    at com.couchbase.client.core.Timer.lambda$register$2(
    at java.base/ Source)



What is the URL being used - especially the port number? If the management port (default 8091) is being specified for the KV service (port 11210), it will be able to make a tcp connection, but the request will never complete successfully. If your server is using the default ports, it’s easiest to just not specify a port.

Hi Michael,

I have just started migrating from 2.7.11 to 3.0.5 with server version 6.0.
I too faced the similar issue and got below error. I am not specifying any specific ports. It’s all using default port and couchbase server is in my local laptop only accessing it via “localhost”

If you could please help.

config:493 [com.couchbase.config][BucketConfigRefreshFailedEvent] Reason: INDIVIDUAL_REQUEST_FAILED, Type: KV, Cause: com.couchbase.client.core.error.UnambiguousTimeoutException: CarrierBucketConfigRequest {“cancelled”:true,“completed”:true,“coreId”:“0xc90446de00000001”,“idempotent”:true,“lastChannelId”:“C90446DE00000001/00000000C8C464B4”,“lastDispatchedFrom”:“”,“lastDispatchedTo”:“localhost:11210”,“reason”:“TIMEOUT”,“requestId”:646,“requestType”:“CarrierBucketConfigRequest”,“retried”:0,“service”:{“bucket”:“bsTransactionsLocks”,“collection”:"_default",“opaque”:“0x290”,“scope”:"_default",“target”:“localhost”,“type”:“kv”},“timeoutMs”:2500} {“coreId”:“0xc90446de00000001”}

looks like boostrapping (which is happening in parallel in SDK 3) takes longer than the 2.5s timeout of KV … can you add a bucket.waitUntilReady(Duration) to the code bfore you perform any kv ops? This will help you make sure that the kv connections are established and you should not run into timeouts after startup.

Thanks so much for the quick reply Michael Nitschinger.

I tried your suggestion as below but looks like it has no impact.Not sure if I am doing anything wrong.

				.subscribe(new Consumer<ExistsResult>() {

I tried it at the bucket level and then both at cluster and bucket level but did not work.

@Harsh_Shah okay, can you share your logs so that I can take a look at what’s going on?

Hi Michael,

Initial check for “doc exist” works fine then couple of upsert , get and remove occurs. And when tries to do the “doc exist” check second time for another doc it fails.

I am using camel async routes in application and which internally tries to interact with couchbase using reactive api.
.subscribe(new Consumer() {

The same in sync mode works fine.

Following are the logs. Check all “AsyncProcessor:155” statements.

18-Jun-2020 11:12:58.669| INFO|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|core:493 [com.couchbase.core][DnsSrvLookupFailedEvent][446ms] DNS SRV lookup failed (name not found), trying to bootstrap from given hostname directly.
18-Jun-2020 11:12:59.778| INFO|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|core:493 [com.couchbase.core][CoreCreatedEvent] {“clientVersion”:“3.0.5”,“clientGitHash”:“ac204fc7”,“coreVersion”:“2.0.6”,“coreGitHash”:“ac204fc7”,“userAgent”:“couchbase-java/3.0.5 (Windows 10 10.0 amd64; OpenJDK 64-Bit Server VM 11.0.6+10)”,“maxNumRequestsInRetry”:32768,“ioEnvironment”:{“nativeIoEnabled”:true,“eventLoopThreadCount”:2,“eventLoopGroups”:[“NioEventLoopGroup”]},“ioConfig”:{“captureTraffic”:,“mutationTokensEnabled”:true,“networkResolution”:“auto”,“dnsSrvEnabled”:true,“tcpKeepAlivesEnabled”:true,“tcpKeepAliveTimeMs”:60000,“configPollIntervalMs”:2500,“kvCircuitBreakerConfig”:“disabled”,“queryCircuitBreakerConfig”:“disabled”,“viewCircuitBreakerConfig”:“disabled”,“searchCircuitBreakerConfig”:“disabled”,“analyticsCircuitBreakerConfig”:“disabled”,“managerCircuitBreakerConfig”:“disabled”,“numKvConnections”:1,“maxHttpConnections”:12,“idleHttpConnectionTimeoutMs”:30000,“configIdleRedialTimeoutMs”:300000},“compressionConfig”:{“enabled”:true,“minRatio”:0.83,“minSize”:32},“securityConfig”:{“tlsEnabled”:false,“nativeTlsEnabled”:true,“hasTrustCertificates”:false,“trustManagerFactory”:null},“timeoutConfig”:{“kvMs”:150000000,“kvDurableMs”:10000,“managementMs”:75000,“queryMs”:300000000,“viewMs”:75000000,“searchMs”:75000,“analyticsMs”:75000,“connectMs”:60000000,“disconnectMs”:30000000},“loggerConfig”:{“customLogger”:null,“fallbackToConsole”:false,“disableSlf4j”:false,“loggerName”:“CouchbaseLogger”,“diagnosticContextEnabled”:false},“orphanReporterConfig”:{“emitIntervalMs”:10000,“sampleSize”:10,“queueLength”:1024},“retryStrategy”:“BestEffortRetryStrategy”,“requestTracer”:“OwnedSupplier”} {“coreId”:“0x3de68bb000000001”,“seedNodes”:[{“address”:“localhost”}]}
18-Jun-2020 11:12:59.787| INFO|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|node:493 [com.couchbase.node][NodeConnectedEvent] Node connected {“coreId”:“0x3de68bb000000001”,“managerPort”:“8091”,“remote”:“localhost”}
18-Jun-2020 11:13:00.494|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|DocumentExistsAsyncProcessor:155 About to attempt to check if document exists (async:true)
18-Jun-2020 11:13:00.494|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|DocumentExistsAsyncProcessor:155 Document id /s4l/GemCase/-1/s4lAggregatedGraph
18-Jun-2020 11:13:01.245| INFO|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|core:493 [com.couchbase.core][BucketOpenedEvent][760ms] Opened bucket “bsTransactionsLocks” {“coreId”:“0x3de68bb000000001”}
18-Jun-2020 11:13:01.772|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:DataConsolidationRoute_DC|DocumentProductionContext:155 ppRequest before DocProd

18-Jun-2020 11:13:20.944| INFO|ID-MUM-LAP-624-1592478369652-0-386:CreateCaseServiceRoute_SUBMIT_GRAPH|CreateCaseWorkflowContextBean:231 Returning allocated business key’ 20/WS/16350/20’
18-Jun-2020 11:13:20.946| INFO|ID-MUM-LAP-624-1592478369652-0-386:AutoGenerationDocumentRoute|AutoGenerationDocumentRoute:159 No Document failure
18-Jun-2020 11:13:20.947|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:CouchbaseCacheRouteBuilderRemoveTx|RemoveDocumentAsyncProcessor:155 About to attempt to remove document (async:true)
18-Jun-2020 11:13:20.948|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:CouchbaseCacheRouteBuilderRemoveTx|RemoveDocumentAsyncProcessor:155 Document id transaction-179ae0cf329c1c08f137aa07217f8c06ae267487cb3962611f94cc3db0883918
18-Jun-2020 11:13:20.982|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:CouchbaseCacheRouteBuilderRemoveTx|RemoveDocumentAsyncProcessor:155 Async Document remove errored com.couchbase.client.core.error.DocumentNotFoundException: Document with the given id not found {“completed”:true,“coreId”:“0x3de68bb000000001”,“idempotent”:false,“lastChannelId”:“3DE68BB000000001/000000002D83212A”,“lastDispatchedFrom”:“”,“lastDispatchedTo”:“localhost:11210”,“requestId”:11,“requestType”:“RemoveRequest”,“retried”:0,“service”:{“bucket”:“bsTransactionsLocks”,“collection”:"_default",“documentId”:“transaction-179ae0cf329c1c08f137aa07217f8c06ae267487cb3962611f94cc3db0883918”,“opaque”:“0x15”,“scope”:"_default",“syncDurability”:{“empty”:true,“present”:false},“type”:“kv”},“status”:“NOT_FOUND”,“timeoutMs”:10000,“timings”:{“dispatchMicros”:664,“totalMicros”:26730}}
18-Jun-2020 11:13:20.983|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:CreateCaseServiceRoute_POST_APPLIED_LOGIC|CreateCaseServiceRoute:155 Inside post Applied Logic: com.northgateis.pole.schema.PutPoleObjectsResponseDto@e76a7c5

18-Jun-2020 11:14:01.068|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:CouchbaseCacheRouteBuilderAddTx|UpsertDocumentAsyncProcessor:155 About to attempt to upsert document (async:true)
18-Jun-2020 11:14:01.076|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:CouchbaseCacheRouteBuilderAddTx|UpsertDocumentAsyncProcessor:155 {“gpoRequests”:["{"_type":“com.northgateis.pole.schema.GetPoleObjectsRequestDto”,“header”:{“passwordHash”:“88316137-956f-434c-a5f3-d6c282a1467c”,“clientName”:"…[A BIG JSON OF 26KB size]…“lockingType”:“DEEP”}
18-Jun-2020 11:14:01.329|DEBUG|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|UpsertDocumentAsyncProcessor:155 Async Document upserted transaction-7ba0957bf263ae8e372a94d0db2a30d13555ae6542829af5db1c0bffb2e32ebf
18-Jun-2020 11:14:01.330|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute|IsLockRequired:155 Lock is required
18-Jun-2020 11:14:01.330| INFO|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContextAcquireLockProxyBean:231 Requesting Lock for Entities [GemCase 360005]
18-Jun-2020 11:14:01.330|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContext:155 Get acquire locks request called
18-Jun-2020 11:14:01.331|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContext:155 Dont error on failure to acquire lock
18-Jun-2020 11:14:01.331|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContextAcquireLockProxyBean:155

18-Jun-2020 11:14:01.332|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:PoleWebServiceStubRoute|PoleWebServiceStubRoute:155 Sending to cxf:bean:toPoleService
18-Jun-2020 11:14:01.632|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContextAcquireLockProxyBean:155

18-Jun-2020 11:14:01.632| INFO|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContextAcquireLockProxyBean:231 Lock successfully acquired for Entities [GemCase 360005]
18-Jun-2020 11:14:01.633|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContext:155 Set lock acquired called
18-Jun-2020 11:14:01.633|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:LockingContextAcquireLocksRoute_ACQUIRE_LOCK|LockingContext:155 Lock acquired
18-Jun-2020 11:14:01.637|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:CouchbaseCacheRouteBuilderAddDeepLock|UpsertDocumentAsyncProcessor:155 About to attempt to upsert document (async:true)
18-Jun-2020 11:14:01.639|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:CouchbaseCacheRouteBuilderAddDeepLock|UpsertDocumentAsyncProcessor:155 {“gpoRequests”:["{"_type":“com.northgateis.pole.schema.GetPoleObjectsRequestDto”,“header”:{“passwordHash”:“88316137-956f-434c-a5f3-d6c282a1467c”,“clientName”:"…[A BIG JSON OF 26KB size]…“lockingType”:“DEEP”}
18-Jun-2020 11:14:01.758|DEBUG|:|UpsertDocumentAsyncProcessor:155 Async Document upserted transaction-7ba0957bf263ae8e372a94d0db2a30d13555ae6542829af5db1c0bffb2e32ebf
18-Jun-2020 11:14:01.760|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:CouchbaseCacheRouteBuilderAddDeepLock|UpsertDocumentAsyncProcessor:155 About to attempt to upsert document (async:true)
18-Jun-2020 11:14:01.761|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:CouchbaseCacheRouteBuilderAddDeepLock|UpsertDocumentAsyncProcessor:155 {“transactionId”:“7ba0957bf263ae8e372a94d0db2a30d13555ae6542829af5db1c0bffb2e32ebf”,“entityKey”:"{“objectRef”:360005,“entityType”:“GemCase”}"}
18-Jun-2020 11:14:01.845|DEBUG|:|UpsertDocumentAsyncProcessor:155 Async Document upserted entityTransaction-{“objectRef”:360005,“entityType”:“GemCase”}
18-Jun-2020 11:14:01.856|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|DocumentExistsAsyncProcessor:155 About to attempt to check if document exists (async:true)
18-Jun-2020 11:14:01.856|DEBUG|ID-MUM-LAP-624-1592478369652-0-668:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|DocumentExistsAsyncProcessor:155 Document id /s4l/GemCase/360005/s4lAggregatedGraph
18-Jun-2020 11:14:04.482| INFO|ID-MUM-LAP-624-1592478369652-0-386:SaveForLaterCouchbaseRouteBuilderDoesGraphExist|config:493 [com.couchbase.config][BucketConfigRefreshFailedEvent] Reason: INDIVIDUAL_REQUEST_FAILED, Type: KV, Cause: com.couchbase.client.core.error.UnambiguousTimeoutException: CarrierBucketConfigRequest {“cancelled”:true,“completed”:true,“coreId”:“0x3de68bb000000001”,“idempotent”:true,“lastChannelId”:“3DE68BB000000001/000000002D83212A”,“lastDispatchedFrom”:“”,“lastDispatchedTo”:“localhost:11210”,“reason”:“TIMEOUT”,“requestId”:69,“requestType”:“CarrierBucketConfigRequest”,“retried”:0,“service”:{“bucket”:“bsTransactionsLocks”,“collection”:"_default",“opaque”:“0x4f”,“scope”:"_default",“target”:“localhost”,“type”:“kv”},“timeoutMs”:2500} {“coreId”:“0x3de68bb000000001”}

… same error keeps printing

Hi Michael,

I tried to post logs with brief background about the scenario. But got message that its filtered at the moment and will appear shortly here once any one of the dev reviews it.

So request to please check the same.

Following is the message.
" Hello,

Our automated spam filter, [Akismet], has temporarily hidden your post in com.couchbase.client.core.error.UnambiguousTimeoutException : GetMetaRequest for review.

A [staff member] will review your post soon, and it should appear shortly.

We apologize for the inconvenience."


The post is available now. If you can please have a look. It would be of great help.

Thank you.

Hi Michael,

Just a friendly reminder. If you can help on this. You guys must be super busy as everybody approaching you.
But as request I have shared logs above.

Also I have been able to reproduce the issue with a simple test. So if you need I can share that too.

Please reply.

Thank you very much.

Hello All,

We did not face same error after opening all related ports as listed below.
Unencrypted : 8091-8096, 9140 [1], 11210, 11211
Encrypted : 11207, 18091-18096 [2]


Any time you suspect connectivity issues, it’d be good to diagnostically check for lower level details with SDK doctor. It may not find anything, but it will validate that common connectivity problems aren’t there.

You’ll find pre-built binaries on the release page.

Usually the summary at the end is pretty easy to interpret. If you need help with interpretation, please feel free to post it or a link to it in the forums.

1 Like

@Harsh_Shah from the logs it is not clear what’s going on unfortunately. Do you think it would be possible to have a small standalone program that reproduces the problem so i can try to reproduce locally? Also if you check with wireshark while doing the traffic, when the op times out how long does it take on the wire?

Do you open/use more than one bucket?

Hi @daschl ,

Thank you for the reply. And sorry for replying late.
No I do not open multiple buckets.

Following is the simple program which uses threads to fire multiple request to couchbase.
Just format it in your ide and it will read better.

Note: The program is providing option to run in sync and async mode. Try both mode and see the diff. in timings. sync modes works faster than async mode.

Also it uses default 100 threads. If you increase the threads to 200 and 300 you will see more and more timeout error comes in async mode and also the overall time increases to a great extent.

Not sure if I am doing the wrong config of couchbase or missing any basic thing.
Please guide/help.

Thanks a lot.

import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.function.Consumer;
import com.couchbase.client.core.env.TimeoutConfig;
public class ** CouchbaseLoadTest2 ** implements Runnable {
    private ClusterEnvironment env;
    private Cluster cluster;
    private Bucket bucket;
    private boolean runAsync = true;
    public static void main(String[] args) {
        CouchbaseLoadTest2 couchbaseTest = new CouchbaseLoadTest2();
    public void doTest() {
        try {
            env = ClusterEnvironment.builder().timeoutConfig(TimeoutConfig.queryTimeout(Duration.ofMillis(300000)).kvTimeout(Duration.ofMillis(300000)).connectTimeout(Duration.ofMillis(60000)).disconnectTimeout(Duration.ofMillis(60000)).viewTimeout(Duration.ofMillis(30000))).build();
            cluster = Cluster.connect("localhost", ClusterOptions.clusterOptions("uname", "passwd").environment(env));
            bucket = cluster.bucket("bucketname");
            List < Thread > threads = new ArrayList < Thread > ();
            for (int i = 0; i < 100; i++) {
                Thread thread = new Thread(this);
            for (Thread thread: threads) {
        } catch (Exception e) {
            System.out.println("Error " + e);
    @Override public void run() {
        final long threadId = Thread.currentThread().getId();
        try {
            String critSec = new String("critSec");
            StringBuilder done = new StringBuilder();
            final long startTime = System.currentTimeMillis();
            System.out.println("Starting thread id " + threadId);
            int requestId = Math.abs(new Random(System.nanoTime()).nextInt());
            String randomId = "id" + requestId;
            if (runAsync) {
                bucket.defaultCollection().reactive().exists(randomId).subscribe(new Consumer < ExistsResult > () {
                    @Override public void accept(ExistsResult t) {}
                }, new Consumer < Throwable > () {
                    @Override public void accept(Throwable t) {
                        System.out.println("Async Document exists errored for thread id " + threadId + " with exception " + t);
                }, new Runnable() {
                    @Override public void run() {
                        System.out.println("Complete for thread id " + threadId + " after " + (System.currentTimeMillis() - startTime) + " milliseconds");
                        synchronized(critSec) {
                String localDone = "";
                while (localDone.length() == 0) {
                    synchronized(critSec) {
                        localDone = done.toString();
            } else {
                System.out.println("Complete for thread id " + threadId + " after " + (System.currentTimeMillis() - startTime) + " milliseconds");
        } catch (Exception e) {
            System.out.println("Error in thread " + e);

@Harsh_Shah there are a couple things in your code that stand out:

  • You need to take jvm warmup into account. you are starting your jvm cold, spawning 100 threads (how many cores does your machine have? you don#t want to do much more than you have cores since… who is executing them :))
  • So use less threads as well
  • In reactive code, you don’t want to use synchronization primitives, and I’m not even sure what the code should do there. Can you explain your intent?
  • you might also want to use nanoTime instead of millis, less overhead and you likely end up with microsecond response times after warmup for ops on localhost


  • to warm up, run 1000 exists calls or so before spawning the threads
  • spawn only threads with roughly the same number you have cores
  • only use the blocking code (there is 0 advantage in your case if all you do is one op)

Thank you very much for the quick reply @daschl.

  • “You need to take jvm warmup into account.”
    >> Agree to your point.

  • " (how many cores does your machine have? you don#t want to do much more than you have cores since… who is executing them :))"
    >> I have i5 with 2 cores. So I can just use 2 threads ? Is it ? It works fine in sync mode with the same no of threads.

  • “In reactive code, you don’t want to use synchronization primitives, and I’m not even sure what the code should do there. Can you explain your intent?”
    >> It was just to try out some post processing logic in a synchronise manner.

  • “you might also want to use nanoTime instead of millis, less overhead and you likely end up with microsecond response times after warmup for ops on localhost”
    >> hmm… ok

  • to warm up, run 1000 exists calls or so before spawning the threads
    >> Ok

  • spawn only threads with roughly the same number you have cores
    >> Not sure.

  • only use the blocking code (there is 0 advantage in your case if all you do is one op)
    >> I am trying to do multiple operations in parallel using diff threads. How can we do multiple operation within a thread ?

Request to please reply on the 2 highlighted points.

If you have 2 cores, 100 threads is very much overkill. Note that in your code you are synchronizing in a callback which in turn will synchronize on our IO threads which is counter productive. If you want to do parallel ops on one thread you can use reactive, but then do it the “proper” way using flatmap. See

Thanks a ton!!! for the reply @daschl .

Much appreciated. :+1: