Threshold log local_dispatch_us is low and total_us is high

I am new to the Couchbase and we are using the JDK client 2.7.11 and Couchbase server 6.* During our load testing we are observing the Threshold reporter logs and total_us is very high compared to the local_dispatch_us. Here is the log for one of the operation

{
“operation_name”: “get”,
“server_us”: 27,
“last_local_id”: “48D941D2D33D64C6/0000000078611C2D”,
“last_local_address”: “10.45.176.206:38186”,
“last_remote_address”: “10.148.33.4:11210”,
“last_dispatch_us”: 641,
“last_operation_id”: “0x5081f5”,
“total_us”: 2185478
}

Our applications is using spring-data-couchbase. We run five instances of our application and above mentioned issues was observed only on couple of instances. Memory and CPU is same as we deploy our application into kubernetes cluster.

Additionally I looked at java SDK client code to see how local_dispatch_us is calculated…local_dispatch_us is measurement for Network + IO to the server. It is very clear that couchbase server is performing well.

We use G1GC and there are no STW events which are greater than 5 ms. I have analyzed the heap dump and it looks good. Any help would be much appreciated on this.

Thanks