Debugging disapearing rows with current=1 from revs table

#1

Using CB Lite 1.4.1 with SG 1.4 and after some days in some documents their current revisions disappear.
I checked revs table in sqlite db and found that rev 1 is present with current flag = 0 and rev 2 was completely missing(though in backup db it was there). The updates to document were made each time only on one side (through SG API or in CB Lite). I do not use expiration for these missing documents. I have thousands of small documents.

So far only way to see what deleted the current revision(or prevent the deletion) is to add a trigger to sqlite db.
No significant errors or messages that would indicate document purging found in logs. Found some documents with 40+ revs and in that case purging works as expected.

Anyone met this behavior?

Thanks for any comment.

#2

Just to be clear - you are seeing documents getting tombstoned (deleted) but you are not explicitly deleting them and have not set TTL / expirationDate on them ?

And it seems like its happening to certain documents.

One possibility that I can think of why - certain revisions are tomb stoned is the conflict resolution process kicking in …check if these were documents with conflicts (You can check the rev tree to see if there was a branch). Although it doesn’t quite explain why the document itself getting deleted.

Can you also share the _raw version of the documents

#3


Left side is newer version, right side is from the day before. Rev 12 was present. It was on 2 and 3 February. Interesting that time saved from SG is 19 January.
Query from SG bucket:
[
{
“_sync”: {
“channels”: {

  },
  "history": {
    "channels": [
   
    ],
    "parents": [
      5,
      3,
      7,
      0,
      11,
      6,
      2,
      4,
      -1,
      10,
      1,
      8
    ],
    "revs": [
      "8-c45a3f3035d600537f6ff44e22b053b9",
      "10-23410c4f99c07eda5dd68cd0addfc0bc",
      "5-aa986aba2bf168c7b44249ae299f41de",
      "9-51740b81b034262fa0ec2fee3d18fd90",
      "3-b5c3b3793bad2ba9b5664bed7287c0fc",
      "7-c82fff2900759cb6821ec85a45dbe684",
      "6-0d26ba2c82d055d3ab04eceb2c4e61c8",
      "4-0b6134abb892cdaedfec68f97beca15d",
      "1-90f1ac1b31538c6d1ad03d15c6ff78bb",
      "12-44b90f6b257b296a4400bac045fae7f0",
      "11-959eb66a129ea577b0a43e3c7155a0bd",
      "2-dd772189e8ed9de07b5ec46ba5f47bc1"
    ]
  },
  "recent_sequences": [
    721134,
    721135,
    721423,
    721492,
    721501,
    721592,
    721593,
    721694,
    722994,
    723007,
    723008,
    726304
  ],
  "rev": "12-44b90f6b257b296a4400bac045fae7f0",
  "sequence": 726304,
  "time_saved": "2018-01-19T07:18:38.841570409Z"
},

],

}
]

#4

This seems to be from PurgeExpired()

  TryQuery(c =>
                {
                    sequences.Add(c.GetLong(0));
                    result.Add(c.GetString(1));

                    return true;
                }, "SELECT * FROM docs WHERE expiry_timestamp <= ?", now);
                    
                if (result.Count > 0) {
                    var deleteSql = String.Format(" sequence in ({0})", String.Join(", ", sequences.ToStringArray()));

It is selecting doc_id but deletes by sequence.

#5

What language is this? CBL 1.4 has different implementations for Objective-C, Java and C#, so it’s important to specify which one you’re using.

#6

It is couchbase-lite-net release 1.4.1 C#, class SqliteCouchStore, PurgeExpired()

#7

I have the exact same issue using 1.4.1 – There is only one process updating & creating documents (a c# service) and current revisions get randomly deleted overtime from the local sqlite. Old revisions (current = 0) are present but not all of them.

Only solution is every so often to delete & rebuild the local sqlite from CB where all revisions are correctly stored.

Because of the randomness I suspect it must down to a race condition in the couchbase lite lib.

#8

The problem was in method purgeexpired.

It is classic SELECT * bug, it doesnt select sequence number but doc ID and the deletes with docid value but by sequence number.

I solved this by recompiling the sqlite plugin with fix in purgeexpired.

1 Like
#9

Thank you so much!! I’ll be doing that later today… it’s been such a pain for a long time!

#10

@kokoska69 Are you referring to a bug in Couchbase Lite? If so, could you please send us a pull request, or just a diff of the source file?

#11

@jens in case it helps this is my implementation of purgeExpired - i was just testing it and seems to work ok.

I did try to fetch from git and submit a patch for 1.4.4 but I gave up after the build kept failing.

Thanks.

public IList<string> PurgeExpired()
{
	var result = new HashSet<string>();
	RunInTransaction (() => {
		var sequences = new List<long>();
		var now = DateTime.UtcNow;
		TryQuery(c =>
		{
			sequences.Add(c.GetLong(0));
			result.Add(c.GetString(1));

			return true;
		}, "SELECT r.sequence, d.docid FROM docs as d, revs as r WHERE d.doc_id = r.doc_id AND d.expiry_timestamp <= ?", now);
			
		if (result.Count > 0) {
			var deleteSql = String.Format("sequence in ({0})", String.Join(", ", sequences.ToStringArray()));
			var vals = new ContentValues(1);
			vals["expiry_timestamp"] = null;

				StorageEngine.Delete("revs", deleteSql);
				StorageEngine.ExecSQL("UPDATE docs SET expiry_timestamp=null WHERE expiry_timestamp <= ?", now);
				return true;
		}

		return true;
	});

	return result.ToList();
}
#12