Android couchbase lite client making 14000 _changes longpoll requests in 1 minute, all with the same since value

#1

Our android couchbase lite client is v1.0.4 and we’ve always had issues with it making too many longpoll requests, but lately it has gotten really bad. We have one client with a lot of data (~40k documents) and about 15 android devices and syncing to the devices keeps failing due to running out of memory, or it finishes but all the data isn’t there. I noticed this morning that sync gateway was using 366% CPU (all 4 cores) of the VM we have it running on, when usually it uses less than 10% CPU. I checked the logs and it showed they were filled with android client longpoll requests. I split the log file into 5MB chunks, and in one chunk I can see 14827 longpoll requests in one minute. We have the android couchbase lite client setup based on the guides and recommendations in the documentation, but is there anything we may have missed that has been known to cause issues like this? I didn’t see this mentioned in the release notes, but is this a known issue that was solved in v1.1.0?

#2

Hi @alexegli,

We fixed many CBL Android replicator issues with v1.1.0. One of fixed issues is a memory leak. v1.1.0 could solve your encountering issue. If v1.1.0 can not solve your issue, please file the ticket. https://github.com/couchbase/couchbase-lite-java-core/issues

For server, what do you use, Sync Gateway or CouchDB? Please use latest version for server side too.

and in one chunk I can see 14827 longpoll requests in one minute.

From you notes, you have 15 devices. So 1000 longpoll reqs per device in one minute. I don’t know how client can send 1k reqs per a min. How many replicator do you create per app?

I recommend one Manager instance per application, one or more databases per Manager, and one pull replicator and one push replicator per database. Could you please check your codes?
Some user made mistakes, they create Manager, Database, Replicators per Activity. this is not recommended.

Thanks,
Hideki

#3

The replicator will be sending roughly one changes-feed longpoll request per change that it receives, once it’s up and running. So assuming you’ve got 15 clients running a pull replication, the 14827 requests/min on the gateway works out to about 16 database changes per second. (Which might be caused by each client creating/changing one document per second.) Does that match what you expect?

(Wait — I just saw the title of the post, where you added the detail “all with the same since value” … that definitely sounds wrong. Perhaps that was one of the bugs fixed in 1.1 that Hideki mentioned?)

#4

Sorry I can’t go into too much detail since I’m not an Android developer, but upgrading to v1.1.0 is a big undertaking for us. Due to the Android limitations on 3rd party libraries and method calls per file we need to do a lot of work to be able to upgrade. Our developer mentioned using code obfuscation and other tricks that android apps use to get around the limitations. It would be a week or more of work. We do want to do the upgrade and plan to do it as soon as possible, just it will take a long time so I wanted to know if this problem could be solved in 1.0.4. Thanks for the tip, we’ll try v1.1.0 and hope it works better.

#5

Hi @alexegli

For upgrading issue, please refer @vladoatanasov comment. I believe this is what you are encountering. Upgrading couchbbase Lite from 1.0.4 to 1.1.0 - Android Studio

For the issue, if possible, could you please share both client and sync gateway log messages? It might help us to understand the problem.

Thanks!

#6

I don’t think the client has any log messages, but on the server side it just shows the following line (all params the same) 14000 times:

15:57:03.872839 HTTP:  #13264: GET /todos/_changes?feed=longpoll&limit=50&heartbeat=300000&style=all_docs&since=3092895  (as <android user id>)

We downgraded sync gateway back to v1.0.3 to try and solve the CPU issue and it seems to have worked. CPU spikes but doesn’t stay at 100% and we have much fewer longpoll requests. We have a lot more broken pipe errors, but that’s because we have the connection timeout set to less than 5 minutes.

#7

Hi @alexegli,

This issue was addressed in Sync Gateway v1.1.1 that we are currently preparing to release. Please refer the ticket. https://github.com/couchbase/sync_gateway/issues/916

If you are in rush, compiling sync gateway is simple. Please git clone from https://github.com/couchbase/sync_gateway/tree/release/1.1.1

Hope this address your problem.

thanks!