Like clockwork, every 120 seconds our view performance tanks.
If I run a quick loop to send view requests over and over, it will return with 40ms to 150ms response times. Then for 10s (say from 1:02:12pm to 1:02:24pm). It will stop responding, and then return all the calls, with times taking 10000ms or more to respond.
I first noticed this while using the couchbase-client for java, but I went ahead and just wrote a quick ruby script to hit the rest-api for the view directly, using the same queries etc.
When the latency happens, the disk io climbs a bit, and the beam.smp process maxes out a core of the VM.
I have tried searching through the logs and don’t see anything that happens every 120s that is obvious to correlate to this.
Where could I look to find whatever is happening every 120s that is causing the CPU spike and most likely the view slowness?
Looks like this might be related to the stats_archiver stuff, per: Connection timeouts during statistics
There isn’t a response on that thread on how to diagnose and/or disable/tune the stats_archiver. But it sounds like something is making the server be “underpowered” even though there is barely any traffic.