Every day 13: 51 ~ 13: 53, get_cmd response time increases


#1
  • No crontab jobs on couchbase server
  • Network Traffic
  • NW input is SAME with another time
  • NW output is increased significantly (3x)
  • Data
  • All data is on Memory
  • Data counts : 13 billion
  • Bucket Configuration
  • Cache Metadata : Value Eviction
  • Replicas : 1 replica copy
  • Disk I/O Priority : Low
  • Auto Compaction Option is default
    – Data/View Fragmentation : 30%
    – Meta Purge Interval : 3

13:50
get_cmd (38406 total)
1us - 2us : ( 0.93%) 359
2us - 4us : ( 15.75%) 5689 ######
4us - 8us : ( 58.94%) 16589 ##################
8us - 16us : ( 94.07%) 13492 ##############
16us - 32us : ( 99.96%) 2260 ##
32us - 64us : ( 99.99%) 15
64us - 128us : (100.00%) 1
256us - 512us : (100.00%) 1
Avg : ( 5us)

13:51
get_cmd (38310 total)
0 - 1us : ( 0.01%) 4
1us - 2us : ( 0.90%) 342
2us - 4us : ( 11.75%) 4156 ####
4us - 8us : ( 61.61%) 19102 ####################
8us - 16us : ( 98.03%) 13951 ###############
16us - 32us : ( 99.83%) 691
32us - 64us : ( 99.84%) 2
64us - 128us : ( 99.84%) 1
2ms - 4ms : ( 99.84%) 1
4ms - 8ms : ( 99.85%) 3
8ms - 16ms : ( 99.86%) 2
16ms - 32ms : ( 99.87%) 7
32ms - 65ms : ( 99.91%) 14
65ms - 131ms : ( 99.96%) 17
131ms - 262ms : (100.00%) 17

Avg : ( 108us)

13:52
get_cmd (40310 total)
0 - 1us : ( 0.02%) 9
1us - 2us : ( 1.84%) 731
2us - 4us : ( 13.75%) 4801 #####
4us - 8us : ( 59.03%) 18256 ###################
8us - 16us : ( 96.80%) 15224 ###############
16us - 32us : ( 99.74%) 1183 #
32us - 64us : ( 99.75%) 6
64us - 128us : ( 99.75%) 1
1ms - 2ms : ( 99.76%) 1
2ms - 4ms : ( 99.76%) 2
4ms - 8ms : ( 99.77%) 5
8ms - 16ms : ( 99.79%) 7
16ms - 32ms : ( 99.81%) 6
32ms - 65ms : ( 99.85%) 16
65ms - 131ms : ( 99.90%) 22
131ms - 262ms : (100.00%) 40

13:53
get_cmd (38362 total)
0 - 1us : ( 0.02%) 7
1us - 2us : ( 0.84%) 317
2us - 4us : ( 11.66%) 4150 ####
4us - 8us : ( 61.42%) 19087 ####################
8us - 16us : ( 98.29%) 14144 ###############
16us - 32us : ( 99.93%) 629
32us - 64us : ( 99.93%) 3
64us - 128us : ( 99.94%) 1
512us - 1ms : ( 99.94%) 1
2ms - 4ms : ( 99.94%) 1
8ms - 16ms : ( 99.95%) 1
16ms - 32ms : ( 99.95%) 1
32ms - 65ms : ( 99.95%) 1
65ms - 131ms : ( 99.97%) 7
131ms - 262ms : ( 99.99%) 10
262ms - 524ms : (100.00%) 2

Avg : ( 66us)

13:54
get_cmd (39776 total)
1us - 2us : ( 0.88%) 352
2us - 4us : ( 15.60%) 5852 ######
4us - 8us : ( 59.41%) 17426 ###################
8us - 16us : ( 93.53%) 13573 ###############
16us - 32us : ( 99.95%) 2554 ##
32us - 64us : (100.00%) 19
Avg : ( 5us)


#2

What version are you using? One process that runs once an hour by default, but shouldn’t slow this down (or it’d be an issue that needs to be filed) is the expiration pager.

Can you file an issue and point to a cbcollect_info? The logs tab in 3.0 and later will help you create a collect info.


#3

I’ve known that above 13:51 is bucket created time.

I reinstall and create the bucket at 04:00.
And heavy latency (get_cmd) occurs at 4:00.
At that time disk utils and disk awaits is over 50%.

I assume that this issue is related with installation time or bucket creation time.