Couchbase Lite downloading from but not uploading to Sync Gateway

Android OS 4.4 (quad-core 1.7GHz APQ8064, 2GB RAM, 16GB EMMC flash)
Couchbase Lite v 1.4.4
Max Rev Tree Depth 1000

Continuous Sync with Remote Sync Gateway/Couchbase Server Cluster
Couchbase Server 6.0.0 Cluster (3x 8 vCPUs, 30 GB RAM, 500GB SSD)
Sync Gateway 2.1.1 (1x 8 vCPUs, 30 GB RAM, SSD)

When the error occurs the upload process stops until the next retry. The unit still downloads changes from the sync gateway but is unable to upload any changes.

As the number of changes that haven’t been uploaded grows, the database performance degrades significantly making the app unusable.

The only way we have found to restoring syncing functionality is to purge the documents which are causing the Exception in the PusherInternal.java class.

There are two primary causes of this issue (1) [which we have minimal logs about] is a crash in the jackson library and (2) [which we have detailed logs about] is a crash in the PusherInternal.java class in the Couchbase Lite Java library:

Map<String, Object> revisions = db.getRevisionHistoryDictStartingFromAnyAncestor(populatedRev, possibleAncestors);

W/Sync    (13902): replicationInternal in unexpected state: RUNNING, ignoring start()
E/RemoteRequest(13902): RemoteRequestCompletionBlock throw Exception
E/RemoteRequest(13902): java.lang.NullPointerException
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.PusherInternal$5.onCompletion(PusherInternal.java:462)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequestRetry$1.completed(RemoteRequestRetry.java:351)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequestRetry$1.onCompletion(RemoteRequestRetry.java:305)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.respondWithResult(RemoteRequest.java:312)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.executeRequest(RemoteRequest.java:300)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.execute(RemoteRequest.java:166)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.run(RemoteRequest.java:106)
E/RemoteRequest(13902): 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
E/RemoteRequest(13902): 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
E/RemoteRequest(13902): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:152)
E/RemoteRequest(13902): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:265)
E/RemoteRequest(13902): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
E/RemoteRequest(13902): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
E/RemoteRequest(13902): 	at java.lang.Thread.run(Thread.java:841)
E/RemoteRequest(13902): RemoteRequestCompletionBlock throw Exception
E/RemoteRequest(13902): java.lang.NullPointerException
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.PusherInternal$5.onCompletion(PusherInternal.java:462)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequestRetry$1.completed(RemoteRequestRetry.java:351)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequestRetry$1.onCompletion(RemoteRequestRetry.java:305)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.respondWithResult(RemoteRequest.java:312)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.executeRequest(RemoteRequest.java:300)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.execute(RemoteRequest.java:166)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.run(RemoteRequest.java:106)
E/RemoteRequest(13902): 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
E/RemoteRequest(13902): 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
E/RemoteRequest(13902): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:152)
E/RemoteRequest(13902): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:265)
E/RemoteRequest(13902): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
E/RemoteRequest(13902): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
E/RemoteRequest(13902): 	at java.lang.Thread.run(Thread.java:841)
E/RemoteRequest(13902): RemoteRequestCompletionBlock throw Exception
E/RemoteRequest(13902): java.lang.NullPointerException
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.PusherInternal$5.onCompletion(PusherInternal.java:462)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequestRetry$1.completed(RemoteRequestRetry.java:351)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequestRetry$1.onCompletion(RemoteRequestRetry.java:305)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.respondWithResult(RemoteRequest.java:312)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.executeRequest(RemoteRequest.java:300)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.execute(RemoteRequest.java:166)
E/RemoteRequest(13902): 	at com.couchbase.lite.replicator.RemoteRequest.run(RemoteRequest.java:106)
E/RemoteRequest(13902): 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
E/RemoteRequest(13902): 	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
E/RemoteRequest(13902): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:152)
E/RemoteRequest(13902): 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:265)
E/RemoteRequest(13902): 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
E/RemoteRequest(13902): 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
E/RemoteRequest(13902): 	at java.lang.Thread.run(Thread.java:841)

We have thought about automatically purging any documents which cause unhandled exceptions in the PusherInternal.java class, but that seems unwise. We have added code that logs when these exceptions occur and are manually purging those documents on the devices. Does anyone have a better approach or solution to this issue?

Best Regards,
John

The only way we have found to restoring syncing functionality is to purge the documents which are causing the Exception in the PusherInternal.java class.

What is the characteristic of the documents that cause the issue? Could you provide more info on that?

They tend to be documents with hundreds of revisions. Each document/rev is only a few dozen bytes individually. Although we have had several cases of documents with 1-3 revisions and a doc/rev size of ~30KB.

@pasin

Any thoughts on how we can resolve this?