Crash with attachment after upgrading from CBL 1.4 to 2.6

Android 9
CBL 2.6

App in production used CBL 1.4. Now an update rolled out to beta users with CBL 2.6 and a when parsing a document with attachment a crash occurs.

The document is:

{
  "barcodes": [],
  "category": {
    "id": "feacf1232d19",
    "name": "épicerie"
  },
  "channels": "CacoWfYv9STcbFGuCyfVjr23M1S2",
  "id": "0a6d44633aed",
  "name": "cornichons frais du Val de Loire au vinaigre d'Orleans à l'ancienne",
  "type": "item",
  "unit": {
    "id": "GENERAL",
    "name": ""
  }
}

Meta data:

{
  "meta": {
    "id": "CacoWfYv9STcbFGuCyfVjr23M1S2::item::0a6d44633aed",
    "rev": "54928-15cb6efd705700000000000000000000",
    "expiration": 0,
    "flags": 0,
    "type": "json"
  },
  "xattrs": {
    "_sync": {
      "rev": "5-6b6d1fbe10392a26ddce597e29b00145",
      "sequence": 140113591,
      "recent_sequences": [
        138365203,
        138365264,
        138365287,
        138368527,
        140113591
      ],
      "history": {
        "revs": [
          "3-b77d61ad4108dd7dad1563d413d53a0e",
          "1-e1d63ba738360ff2c1b0b50e1c40428a",
          "2-194ea30678c2fc70733912b8dd989740",
          "4-fd531c1c8f7f838e045b25db016436a1",
          "5-6b6d1fbe10392a26ddce597e29b00145"
        ],
        "parents": [
          2,
          -1,
          1,
          0,
          3
        ],
        "channels": [
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ],
          [
            "CacoWfYv9STcbFGuCyfVjr23M1S2"
          ]
        ]
      },
      "channels": {
        "CacoWfYv9STcbFGuCyfVjr23M1S2": null
      },
      "cas": "0x00005770fd6ecb15",
      "value_crc32c": "0x0d51f271",
      "attachments": {
        "1569666393740": {
          "content_type": "image/jpeg",
          "digest": "sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk=",
          "length": 217957,
          "revpos": 2,
          "stub": true
        }
      },
      "time_saved": "2019-10-07T19:55:29.856862484+02:00"
    }
  }
}

Stacktraces:

#0. Crashed: main
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3750(ObjectMapper.java:3750)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Fatal Exception: java.lang.IllegalArgumentException: Failed to obtain content from BlobStore. digest=sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk= (through reference chain: java.util.HashMap["_attachments"]->java.util.HashMap["1569666393740"]->com.couchbase.lite.Blob["content"])
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3750(ObjectMapper.java:3750)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Caused by com.fasterxml.jackson.databind.JsonMappingException: Failed to obtain content from BlobStore. digest=sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk= (through reference chain: java.util.HashMap["_attachments"]->java.util.HashMap["1569666393740"]->com.couchbase.lite.Blob["content"])
       at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow + 316(StdSerializer.java:316)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 727(BeanSerializerBase.java:727)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3728(ObjectMapper.java:3728)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Caused by java.lang.IllegalStateException: Failed to obtain content from BlobStore. digest=sha1-EwD/0Bd/vNzukTyKL8k0skwuQYk=
       at com.couchbase.lite.Blob.getBytesFromDatabase + 588(Blob.java:588)
       at com.couchbase.lite.Blob.getContent + 300(Blob.java:300)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3728(ObjectMapper.java:3728)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)
...
Caused by com.couchbase.lite.LiteCoreException: No such file or directory
       at com.couchbase.lite.internal.core.C4BlobStore.getContents(C4BlobStore.java)
       at com.couchbase.lite.internal.core.C4BlobStore.getContents + 102(C4BlobStore.java:102)
       at com.couchbase.lite.Blob.getBytesFromDatabase + 572(Blob.java:572)
       at com.couchbase.lite.Blob.getContent + 300(Blob.java:300)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField + 688(BeanPropertyWriter.java:688)
       at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields + 719(BeanSerializerBase.java:719)
       at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize + 155(BeanSerializer.java:155)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields + 722(MapSerializer.java:722)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 643(MapSerializer.java:643)
       at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize + 33(MapSerializer.java:33)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize + 480(DefaultSerializerProvider.java:480)
       at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue + 319(DefaultSerializerProvider.java:319)
       at com.fasterxml.jackson.databind.ObjectMapper._convert + 3728(ObjectMapper.java:3728)
       at com.fasterxml.jackson.databind.ObjectMapper.convertValue + 3668(ObjectMapper.java:3668)

User reports that this crash occured after the update without the user doing anything. I created a backup of the cluster, imported it on my dev cluster, signed in as the user and I’m able to parse the document and to view the attachment/blob/image.

Does anyone have an idea what might have happened. Is there a possibility that it is a bug in CBL? Is it crashing because the info of the attachment is in the meta data but the binary file of the image cannot be found on the device? What could be an immediate workaround to stop the crash from happening?
So far this happened to only one user. Once it will be rolled out to all users - not just beta users - there might be many, many more crashes.

Thanks!

User confirmed that the older release with CBL 1.4 on a different device which uses the same account to sync the user’s data does not have this issue. The image can be viewed.
I published another app update (with CBL 2.6) and will report back if all documents with an attachment would crash, or if this issue is only with a single document.

@benjamin_glatzeder: Is this happening consistently? Does it happen with every document that contains a blob, or just some?

CBL stores blobs in files, not in the DB. The DB just contains a reference to the file. Here, it looks as though the file containing the blob cannot be located.

Android 9 has, yet again, changed how file system access works. Have you, by any chance, tried this with any version other than 9?

From the first user report as much as I can remember this happened with 1 document. I believe this user had more than 1 picture. The issue only appeared for a tiny, tiny part of the userbase. But then again not everyone uses the picture feature as it’s optional. It appeared on 3 Android versions so far. Here are the last 30 days with this issue:

I believe this happens after upgrading from CBL 1.4 to CBL 2.6. So either the issue went away on its own or users deleted the app. That’s why there are spikes which bottom out quickly.