Low upsert performance under stress


#1

Hi, I’m trying to replace Memcache with Couchbase as our caching layer.

I have Flask web API with connecting to cache and datastore.

Specific details are following:

  • 3 Instances of Flask API running with uwsgi (6 processes, 2 threads)
  • Persistent Connection to a clustered datastore.
  • A cluster of 3 Couchbase server (Each: 4 vCPU, RAM 8 GB)

So far I have tried using connection pool and class instances for Couchbase.
Both method perform similarly in term of performance.

Load testing using Tsung via our API results in Client-side timeout when connecting to Couchbase. This mostly happen because I have quite large upsert_multi payload (which I utilize heavily when using Memcache; Can be up to 800 keys per call).

I performed loadtest with 3 arrival phase (10, 50, 100 per second), which simply make the request to our API.
The API then cache data using 3 separate steps for each user request.

  1. Cache list of ids. Using request filter, etc (upsert).
  2. Cache metadata. Using request filter, etc (upsert).
  3. Cache multiple documents using id as its key (upsert_multi).

With Memcache, we could serve at least 1K requests per instances. With Couchbase we are seeing lots of timeout reported.

What am I missing here?


#2

You should reduce your upsert_multi payload size or increase the timeout. The _multi methods are network-level optimizations, and would depend here on the size of each key as well as the actual expected payload size for the value.

See http://developer.couchbase.com/documentation/server/4.0/developer-guide/batching-operations.html for an overview of what ‘multi’ operations do and how to best use them.


#3

Thanks cutting the _multi size by half improve the performance significantly. Now we are running into a problem with disk being full.


#4

If your disk is getting full, it might be helped with modifying your compaction settings. You should ask another question in the Couchbase Server section