Warning 'Requests over Threshold found' Java SDK

Hi All,

I’m trying to create REST API using quarkus that will call query to couchbase using ReactiveCluster.
I’m using couchbase Java SDK 3.1.0
During performance test, I keep getting below warning:

WARN [com.cou.tracing] (cb-events) [com.couchbase.tracing][OverThresholdRequestsRecordedEvent][10s] Requests over Threshold found: [{“top”:[{“operation_name”:“QueryRequest”,“last_local_address”:“xxx:58144”,“last_remote_address”:“xxx:8093”,“last_dispatch_us”:2697439,“last_operation_id”:“4bf1dbac-23d4-48ec-88fc-f805c7090ebf”,“total_us”:2697826},{“operation_name”:“QueryRequest”,“last_local_address”:“10.130.3.7:58146”,“last_remote_address”:“xxx:8093”,“last_dispatch_us”:2604224,“last_operation_id”:“6881a88e-7ef1-4a58-b4d0-509a50faf9a1”,“total_us”:2604594}],“service”:“query”,“count”:2}]

Is there any idea why I keep getting this warning?
How to solve this?

Thanks

Hi,

This is part of the Observability/Threshold Logging that is included in the SDK - it’s there so you are able to track slow operations that are taking longer than the set threshold, and therefore debug issues.

If you want to reduce the number of these messages then in your CouchbaseEnvironment, you could try increasing the queryThreshold time so that these slow operations do not exceed the threshold, and/or increase the emitInterval duration to reduce how often these messages are displayed. For example:

ClusterEnvironment env = ClusterEnvironment.builder()
        .thresholdRequestTracerConfig(ThresholdRequestTracerConfig.builder()
                .queryThreshold(Duration.ofSeconds(20))
                .emitInterval(Duration.ofMinutes(1)))
        .build();

And then pass this environment into your ReactiveCluster.connect(…) as usual. Details can be found here.

You could also change the log level you are using to INFO or above, however if you still want to see other WARN level messages this would not be appropriate for you.

Hope this helps,
Will

2 Likes

Nice answer @WB!

It’s also worth mentioning that the thresholds were selected to be somewhat conservative. It’s impossible for the Couchbase SDK to know what is reasonably closer to your service level objective and what is not though. Some users, for example, run on 10gigE networks with large systems and have very low latencies, other users run on small instances on cloud providers and intentionally have all data on slower disks. Those have very different performance profiles.

What we hope is that if you see these requests over the threshold, it prompts you to think about it a bit and adjust the thresholds to something that makes sense for your service level objective. Then it helps you manage to that.

Hope that helps!