CBL 2.6: Does SG need more RAM? Initially only?

Setup:
CBL 2.7 CE Android snapshot from around 1 week ago.
SG 2.6

Current RAM usage:

1 week ago:

Most users started to use the lastest app version with the CBL 2.6 today. Before it was CBL 1.4. I’m unsure what is happening at the moment. Continuous sync stopped working. One-time pull sync works fine. Restarting the SG service on both machines allows continuous sync to start again. I’ll add another machine with SG service and keep an eye on this issue.

The primary feature released in Sync Gateway 2.6 was channel cache enhancements.
I can’t answer your question easily without understanding your usage of channels, as well as your configuration.

I’d recommend reading more about the new cache features here, it may give you some clues:
https://docs.couchbase.com/sync-gateway/2.6/index.html
https://docs.couchbase.com/sync-gateway/2.6/deployment.html#channel-and-revision-cache

Thank you for the links. I read through the documentation. The channel and revision cache is for EE versions only. That is fine. I think it has nothing to do with SG 2.6. I was running this version for some time already. The only change in the last days is that an Android app update rolled out with CBL 2.6 - previously it was CBL 1.4.
Now I have 3 SG nodes and each uses between 500MB and 1000MB. So there’s plenty of RAM unallocated. Sync works. It has been all good now for the last ~3 hours.

My SG config has been:

{
  "interface": ":4984",
  "adminInterface": ":4985",
  "maxFileDescriptors": 250000,
  "logging": {
    "log_file_path": "/home/sync_gateway/logs",
    "console": {
      "log_level": "info",
      "log_keys": [
        "HTTP"
      ],
      "color_enabled": true
    },
    "error": {
      "enabled": true,
      "rotation": {
        "max_size": 100,
        "max_age": 360,
        "localtime": false
      }
    },
    "warn": {
      "enabled": true,
      "rotation": {
        "max_size": 100,
        "max_age": 180,
        "localtime": false
      }
    },
    "info": {
      "enabled": true,
      "rotation": {
        "max_size": 100,
        "max_age": 6,
        "localtime": false
      }
    },
    "debug": {
      "enabled": false,
      "rotation": {
        "max_size": 100,
        "max_age": 2,
        "localtime": false
      }
    }
  },
  "databases": {
    "my_database": {
    "server": "couchbase://10.0.0.30,10.0.0.31,10.0.0.32",
    "bucket": "my_bucket",
      "password": "pass",
      "username": "sync_gateway_user",
      "num_index_replicas": 0,
      "revs_limit": 400,
      "allow_empty_password": true,
      "users": {
        "GUEST": {
          "disabled": true,
          "admin_channels": [
            "*"
          ]
        }
      },
      "cache": {
          "channel_cache_max_length": 1000
      },
      "enable_shared_bucket_access": true,
      "import_docs": true, <-- set at one SG instance only
      "sync":
`
function sync_gateway_fn(doc, oldDoc) {

I’d like to follow up and report that Thursday (following day) and Friday all looked good. Here’s 6 hours from Friday. Click on the picture to see all 3 SG nodes and their RAM usage.

Saturday morning was different. All SG services rapidly allocated RAM and syncing between clients did not work. I needed to restart all 3 SG nodes. Here’s a screenshot, again click on the picture to see all 3 nodes.

I still hope that this issue will go away on its own once all users updated to the latest Android app version with CBL 2.x.

This is very strange behaviour that I’d like to get to the bottom of, given it was stable for a couple of days, but then suddenly ate up a bunch of memory. Maybe a client is doing something weird which is causing SG to leak memory somehow…

If possible, next time this happens - before restarting each node, can you grab a couple of profiles from each node using these URLs:

This should let us know what is using up memory inside Sync Gateway, which hopefully we can track down a fix for.

1 Like