First time sync and skipping deleted documents cblite2.5+

bpolanec · October 4, 2019, 3:13pm

In cblite 2.5+, is it possible to skip on first time sync replication of deleted documents ?

jayahari.vavachan · October 4, 2019, 6:07pm

bpolanec · October 5, 2019, 6:38am

Using replication pull filters is not efficient. Entire replication is slower if they are processed by filter as they arrive and another drawback in using filters is that first you download deleted docs and then discard them which is waste of data traffic on mobile devices. This is especially bad if you have bigger amount of deleted docs and just some “active” docs. You could be waiting for 1M deleted docs to come down just to get the 10k active.

Extract from documentation in the link you provided:

Pull replication filters are not a substitute for channels. Sync Gateway channels are designed to be scalable (documents are filtered on the server) whereas a pull replication filter is applied to a document once it has been downloaded.

priya.rajagopal · October 5, 2019, 3:02pm

If you don’t want deleted documents to be synced down to the clients , why don’t you purge them instead on the server? You could also update your sync function to move deleted documents to an archived channel that’s not accessible to the end users

bpolanec · October 6, 2019, 6:43am

When would we move or purge deleted documents on the server?
Moving deleted documents or purging them would mean that some offline devices which are not doing “clean” first time sync might not get information that some documents are deleted, and those documents would stay “active” in local db at those devices. It would be safe to do so only if server would somehow know that ALL clients are synced and up to date, which sounds complicated, and there is also a problem that some devices might not come online for a while (days, months), until all devices would confirm to server they are synced up to point X you can’t delete or move those deleted docs.

priya.rajagopal · October 6, 2019, 4:29pm

This is especially bad if you have bigger amount of deleted docs and just some “active” docs. You could be waiting for 1M deleted docs to come down just to get the 10k active.

How realistic is this that you have 1M deleted documents and just 10K of active documents? Over what period of time do you expect that you have this?

Seems like you should be purging/expiring these deleted documents on the server. If you are using shared bucket access, note that tombstoned documents will be automatically purged after metadata-purge-interval (which defaults to 3 days). You should adjust the metadata-purge-interval setting on couchbase server to a value that’s greater than the largest expected client partition window .

If you expect clients to be offline for a period greater than the metadata-purge-interval or you just don’t know how long clients will be offline, then you should just remove the documents from the user’s channels (as indicated earlier). That way when clients come back online, instead of a delete, they will get the document “removed” notification that will auto purge the documents on the clients. Then you don’t have to worry about the new clients getting these documents either since they are removed from channel.

borrrden · October 7, 2019, 11:16am

In any case if you have a completely empty database then the first pull on there should not be pulling deleted documents. If you even have one document existing locally though then this optimization will not be enabled. You should see something like activeOnly being true in the sync gateway logs.

bpolanec · October 8, 2019, 8:09am

@priya.rajagopal looks like something we would need to try. Thanks for the info.

@borrrden We did few tests and seems in old cblite lib (1.x) where POST’s are recieved by SG2.1.2 we do get “ActiveOnly:true” while in latest cblite 2.7(from sources) where sockets are used there is no active only in logs. Database was empty when replicator was started. I debugged the “internalStart()” in AbstractReplicator but did not see where such setting would get enabled. Is it supported in 2.+ versions?

borrrden · October 8, 2019, 11:16am

It’s not set via platform code, but this optimization is detected in LiteCore (here to be exact).

bpolanec · October 8, 2019, 1:57pm

Built from sources today latest lib version:
46d5ab73 core
f477ad9 ce

I’ve tried again to see if deleted documents will get pulled if local database has 0 docs.
It seems that even if document count in database is 0 it still pulls deleted documents.

Before starting replicator I’ve checked if there are any documents in database by com.couchbase.lite.Replicator .getConfig().getDatabase().getCount() ,
I’ve also seen that LiteCore hits the line in log.

Providing log if might be useful:

I/CouchbaseLite/REPLICATOR: Replicator is starting …
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: Starting
I/CouchbaseLite/DATABASE: {DB#156}==> N8litecore14SQLiteDataFileE /data/user/0/com.myProject/files/myDb.cblite2/db.sqlite3 @0x7f0c4719c0
I/CouchbaseLite/DATABASE: {DB#156} Opening database
I/CouchbaseLite/NETWORK: {N8litecore4blip10ConnectionE#157}==> N8litecore4blip10ConnectionE ->wss://devmobile.myProject.com:6986/ag-dev/_blipsync @0x7efe444310
I/CouchbaseLite/NETWORK: {N8litecore4blip10ConnectionE#157} Opening connection…
I/CouchbaseLite/REPLICATOR: {Repl#158}==> N8litecore4repl10ReplicatorE /data/user/0/com.myProject/files/myDb.cblite2/ ->wss://devmobile.myProject.com:6986/ag-dev/_blipsync @0x7f0c3031c8
I/CouchbaseLite/REPLICATOR: {Repl#158} Pull=one-shot, Options={{channels:[“apptables_mainUser110666_blb”,“appdata_mainUser110666__blb__MJSWE4TBJ5RGUZLLORSQ0000”,“appdata_mainUser110666__blb__MJSWE4TBKJSXA33SOQ000000”,“appfiles_mainUser110666__blb__MJSWE4TBKJSXA33SOQ000000_Fotos”,“appfiles_mainUser110666__blb__MJSWE4TBKJSXA33SOQ000000_AktualisiertFotos”,“launcher_entry_update_requests_273583945126833”], cookies:“SyncGatewaySession=db80055c7a58b673efabb59b03375c53b616d830”, headers:{User-Agent:“CouchbaseLite/2.7.0-SNAPSHOT (Java; Android 8.0.0; SM-G935F) CE/release, Commit/6f849621+ Core/0.0.0 (e066b373+)”}, progress:1}}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is connecting, progress 0/0, error: null
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.statusChanged, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], status: C4ReplicatorStatus{activityLevel=2, progressUnitsCompleted=0, progressUnitsTotal=0, progressDocumentCount=0, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/NETWORK: {N8litecore4repl12C4SocketImplE#159}==> N8litecore4repl12C4SocketImplE wss://devmobile.myProject.com:6986/ag-dev/_blipsync @0x7ee33c4ff0
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: status changed: (0, 0) @C4ReplicatorStatus{activityLevel=2, progressUnitsCompleted=0, progressUnitsTotal=0, progressDocumentCount=0, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/NETWORK: {N8litecore4repl12C4SocketImplE#159} Connecting…
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is connecting, progress 0/0, error: null
E/CouchbaseLite/NETWORK: CBLWebSocket.socket_open()
I/CouchbaseLite/REPLICATOR: {Repl#158} No local checkpoint ‘cp-2zwyjhE6WFpbKlbGZ6pwV+02afI=’
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=connecting: connectionState=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160}==> N8litecore4repl6PullerE ->wss://devmobile.myProject.com:6986/ag-dev/_blipsync @0x7effe2f4c8
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=0, _activeIncomingRevs=0
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=0/0
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=connecting: connectionState=1
I/CouchbaseLite/NETWORK: {N8litecore4repl12C4SocketImplE#159} Got HTTP response (status 101)
I/CouchbaseLite/NETWORK: CBLWebSocket CONNECTED!
I/CouchbaseLite/NETWORK: {N8litecore4repl12C4SocketImplE#159} Connected!
I/CouchbaseLite/NETWORK: {N8litecore4blip10ConnectionE#157} Connected!
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=connecting: connectionState=1
I/CouchbaseLite/REPLICATOR: {Repl#158} Connected!
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} now busy
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.statusChanged, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], status: C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=0, progressUnitsTotal=0, progressDocumentCount=0, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: status changed: (0, 0) @C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=0, progressUnitsTotal=0, progressDocumentCount=0, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} Starting pull from remote seq
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is busy, progress 0/0, error: null
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=1, _caughtUp=0, _pendingRevMessages=0, _activeIncomingRevs=0
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=1, _caughtUp=0, _pendingRevMessages=0, _activeIncomingRevs=0
I/CouchbaseLite/REPLICATOR: {Repl#158} No remote checkpoint
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=0, _activeIncomingRevs=0
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=0, _activeIncomingRevs=0
I/CouchbaseLite/REPLICATOR: {N8litecore4repl9RevFinderE#162}==> N8litecore4repl9RevFinderE ->wss://devmobile.myProject.com:6986/ag-dev/_blipsync @0x7ee33c5408
I/CouchbaseLite/REPLICATOR: {N8litecore4repl9RevFinderE#162} Received 200 changes (seq ‘19128359’…‘21865424’)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl9RevFinderE#162} Responded to ‘changes’ REQ#1 w/request for 200 revs
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=200, _activeIncomingRevs=0
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=0/200
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.statusChanged, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], status: C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=0, progressUnitsTotal=200, progressDocumentCount=0, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: status changed: (0, 0) @C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=0, progressUnitsTotal=200, progressDocumentCount=0, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=0/200
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is busy, progress 0/200, error: null
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=199, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=199, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=198, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=198, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=197, _activeIncomingRevs=3
I/CouchbaseLite/DATABASE: {DB#167}==> N8litecore14SQLiteDataFileE /data/user/0/com.myProject/files/myDb.cblite2/db.sqlite3 @0x7ee376d7c0
I/CouchbaseLite/DATABASE: {DB#167} Opening database
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=197, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=196, _activeIncomingRevs=4
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=196, _activeIncomingRevs=4
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=0/9390
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=196, _activeIncomingRevs=4
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=196, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=196, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {N8litecore4repl9RevFinderE#162} Received 200 changes (seq ‘21865693’…‘21901619’)
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl9RevFinderE#162} Responded to ‘changes’ REQ#6 w/request for 200 revs
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=0/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 2 revs in 72.00ms ( 28/sec) of which 99.9% was commit
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded mainUser110666__blb__MJSWE4TBJ5RGUZLLORSQ0000 26-a81db59f08760bc815eca81d63f62a72 flags=00 (0/0)
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=0/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded ac99f776-a799-4fec-9e9e-c0a925bb3adf 1-4e131715e6d9843d9c8034c9526e8c314668d77a flags=00 (0/0)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=2/9590
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 1 revs in 33.84ms ( 30/sec) of which 99.8% was commit
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4597/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded mainUser110666__blb__MJSWE4TBKJSXA33SOQ000000 61-bb866c384fe8a29720ffbe2aa745a19f flags=00 (0/0)
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.documentEnded, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], pushing: false
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: notifyDocumentEnded: %sDocumentReplication{repl=Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}],#docs=2}
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.statusChanged, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], status: C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4597, progressUnitsTotal=9590, progressDocumentCount=2, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: status changed: (0, 0) @C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4597, progressUnitsTotal=9590, progressDocumentCount=2, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is busy, progress 4597/9590, error: null
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=396, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.documentEnded, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], pushing: false
I/CouchbaseLite/REPLICATOR: notifyDocumentEnded: %sDocumentReplication{repl=Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}],#docs=1}
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.statusChanged, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], status: C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4597, progressUnitsTotal=9590, progressDocumentCount=2, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: status changed: (0, 0) @C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4597, progressUnitsTotal=9590, progressDocumentCount=2, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is busy, progress 4597/9590, error: null
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=395, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4598/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 1 revs in 35.69ms ( 28/sec) of which 99.9% was commit
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=395, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=395, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded d0dcba3c-400f-4523-b539-2fc60bf54a8c 2-383c4e9bb63b0c023522383eca1c0fb28299c916 flags=08 (0/0)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=395, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4599/9590
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=395, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=394, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=394, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=394, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=393, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=393, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 1 revs in 72.29ms ( 14/sec) of which 99.9% was commit
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded 0df4b978-8212-4af7-9a9f-91481aa46643 3-9780ed3d6529c838f10753a0e8484618 flags=01 (0/0)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=393, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4600/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded 5c543801-4c69-428b-9bef-6cbc2f7cbc25 2-cd79090c2d76e57b64f1f832585b2155 flags=01 (0/0)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 2 revs in 45.26ms ( 44/sec) of which 99.9% was commit
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 1 revs in 8.62ms ( 116/sec) of which 99.4% was commit
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.documentEnded, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], pushing: false
I/CouchbaseLite/REPLICATOR: notifyDocumentEnded: %sDocumentReplication{repl=Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}],#docs=3}
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.statusChanged, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], status: C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4600, progressUnitsTotal=9590, progressDocumentCount=5, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: status changed: (0, 0) @C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4600, progressUnitsTotal=9590, progressDocumentCount=5, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is busy, progress 4600/9590, error: null
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4600/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded 42120dcf-1f1d-45d4-8a94-9a98b0b109e1 3-7bcfb52490c6bdcd5ad94ff05ce398a6 flags=01 (0/0)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4602/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=392, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=391, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=391, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded 61d4a54d-64a6-4954-9121-812031caedd4 3-b08f07eb756860d848d80ae38d1176b4 flags=01 (0/0)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=391, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=391, _activeIncomingRevs=1
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=390, _activeIncomingRevs=2
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4603/9590
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=389, _activeIncomingRevs=3
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=388, _activeIncomingRevs=4
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=387, _activeIncomingRevs=5
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 1 revs in 14.32ms ( 70/sec) of which 99.7% was commit
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4603/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=386, _activeIncomingRevs=6
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=385, _activeIncomingRevs=7
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=385, _activeIncomingRevs=7
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=384, _activeIncomingRevs=8
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=383, _activeIncomingRevs=9
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=382, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=382, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=382, _activeIncomingRevs=9
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=381, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded 8ac15ab3-4e1d-49ef-9fa9-a6f2c9dabf6b 3-e7c1d643b96d3f8f0b7f030c00bd9438 flags=01 (0/0)
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 4 revs in 43.83ms ( 91/sec) of which 99.8% was commit
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=381, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {Repl#158} pushStatus=stopped, pullStatus=busy, progress=4604/9590
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=381, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=381, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=381, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=381, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=381, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=380, _activeIncomingRevs=11
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=380, _activeIncomingRevs=11
I/CouchbaseLite/REPLICATOR: {N8litecore4repl8InserterE#166} Inserted 5 revs in 43.05ms ( 116/sec) of which 99.8% was commit
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=380, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.documentEnded, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], pushing: false
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=380, _activeIncomingRevs=9
I/CouchbaseLite/REPLICATOR: notifyDocumentEnded: %sDocumentReplication{repl=Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}],#docs=3}
I/CouchbaseLite/REPLICATOR: C4ReplicatorListener.statusChanged, context: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}], status: C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4604, progressUnitsTotal=9590, progressDocumentCount=9, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}]: status changed: (0, 0) @C4ReplicatorStatus{activityLevel=4, progressUnitsCompleted=4604, progressUnitsTotal=9590, progressDocumentCount=9, errorDomain=0, errorCode=0, errorInternalInfo=0}
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=380, _activeIncomingRevs=9
I/CouchbaseLite/REPLICATOR: Replicator{@1f75c36,<-,Database{@ce788e7, name=‘myDb’},URLEndpoint{url=wss://devmobile.myProject.com:6986/ag-dev}] is busy, progress 4604/9590, error: null
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=380, _activeIncomingRevs=9
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=379, _activeIncomingRevs=10
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=379, _activeIncomingRevs=9
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {Repl#158} activityLevel=busy: connectionState=2
I/CouchbaseLite/REPLICATOR: {N8litecore4repl6PullerE#160} activityLevel=busy: pendingResponseCount=0, _caughtUp=0, _pendingRevMessages=379, _activeIncomingRevs=8
I/CouchbaseLite/REPLICATOR: {Repl#158} documentEnded 435193f1-d931-4ab0-bb0c-596ab4518e04 3-e96a9743e426a398c4fba00b72377449 flags=01 (0/0)

bpolanec · October 14, 2019, 8:22am

Is there some other information I can help with to check if this is a bug ?

jens · October 14, 2019, 5:13pm

I think it is a bug. I checked the LiteCore source code, and currently the flag that enables the skip-deleted-docs optimization never gets set. I’ve filed an issue. Unfortunately there’s no workaround; there is an internal replicator option you can set to enable the optimization, but the CBL 2 API has no way to set such options.