"500 Internal Error: document value was too large" when deleting documents trough sync gateway

Mobile Sync Gateway
1

When executing the following DELETE request thorugh Sync Gateway API:

https://{SGW_URL}:{SGW_PORT}/{DATABASE}/{DOC_ID}?rev={REV_ID}

it always returns 500:

{
    "error": "Internal Server Error",
    "reason": "Internal error: document value was too large"
}

Also, monitoring packages from our iOS implementation, I noticed that _bulk_docs request also fails for the same document with the following response:

[{
	"error": "500",
	"id": "{DOC_ID}",
	"reason": "Internal error: document value was too large",
	"status": 500
}]

I got the REV_ID from a GET request made to the same DOC_ID also trough Sync Gateway API

This is happening since a month now and is only happening with that document, it doesn’t receive any updates or delete requests. We already restarted our server, compacted both Server and Mobile databases and nothing seems to solve the problem. We also deleted the document from Couchbase Server Database and Sync Gateway still returns data when requesting the document.

We are running Sync Gateway version 2.1, Couchbase Server 6.0 and Couchbase Lite iOS 1.4.4 (we are on the way migrating to 2.1)

Any possible solutions to this problem?

This is very concerning as we have our production server stuck in what seems to be an irrecoverable state, where nothing we do seems to fix it, we really need to find a solution to this.

1 Like
2

A few questions:

  • Has Sync Gateway been restarted (to force a flush of its cache) since the document was deleted on Couchbase Server?
  • If so, what data is Sync Gateway returning when the document is requested, after the document has been deleted from Couchbase Server?
  • Is the document body larger than 20MB (the document size limit for Couchbase Server)?
  • Are you running with shared bucket access enabled?
1 Like
3
  1. Yes, we have restarted it
  2. We don’t know. It’s a valid revision for the document for sure but it’s not in the database.
  3. No, it has only 2KB or so.
  4. Yes, we have that flag enabled in SGW config JSON.
4

We don’t know. It’s a valid revision for the document for sure but it’s not in the database.

What call are you making to retrieve the document? Are you requesting by ID, or by ID and rev? If ID and rev, you could be retrieving a temporary revision body backup, but those should auto-expire.

Alternatively, this may be a tombstoned document, where the document’s mobile metadata is retained by the server, even though the document is deleted. Retrieving the document via Sync Gateway’s REST API _raw endpoint can provide some insight on the state of the underlying document from Sync Gateway’s perspective:

GET https://{SGW_URL}:{SGW_ADMIN_PORT}/{DATABASE}/_raw/{DOC_ID}

5

I’m making a GET request with DocID and nothing else to retrieve the document.

I called _raw the endpoint with the {DOC_ID} and I got some usefull information:

  • The entire reponse is 2.1MB
  • Our data is not more than 1.5KB
  • Under _sync.history key I got:
  1. revs: 17781 array objects (revisions ids)
  2. parents: 17781 array objects (all numbers but some of them are -1)
  3. deleted: 3709 array objects (all numbers in ASC order)
  4. bodymap: 3709 array objects (all of the values are {\"_deleted\":true})
  5. channels: 17781 array objects (which most of them are null and the others all the same value, we use only one channel for this document)
6

Any suggestions on what’s happening?

7

It looks like the underlying issue is that the document metadata is larger than the maximum allowed in an extended attribute by Couchbase Server. You shouldn’t have 17K+ revisions in the revision tree unless you’ve got unresolved conflicts throughout the document’s history, or have increased the revs_limit value in your config significantly.

If the document is no longer valid (you mention you’ve attempted to remove it directly from Couchbase Server), you can use Sync Gateway’s purge API to remove the document and its associated metadata from the server (see the section on ‘manually purging tombstones’ here: https://docs.couchbase.com/sync-gateway/2.1/managing-tombstones.html

2 Likes
8

Thanks for your response, I’m going to try using db/_purge
If conflicting revisions is the problem, would disabling allow_conflicts flag in SGW config JSON prevent this problem from happening again the future?