Query keys parameter cannot be longer than 1772 bytes


#1

I’ve noticed that when a view is queried with a keys parameter longer than 1772 bytes (when stringified) it fails to execute with the error:

 The object requested is too big to store in the server

I’m not against a limit, but this one is not documented.

I use the nodejs client, but it’s apparently a server error.

I’m pretty sure this is not due to a limited URL length, as this is the only query parameter I define, and I can use another view with a longer name, resulting in the same limit for keys.

Similar topics like ViewQuery.keys size limit and View query keys size limit seem to deal only with the clients, not the server.


#2

I precise I found this limit on my Couchbase 3.0.1 CE on Windows, but it seems it’s the same limit on CentOS on a machine which has totally different CPU and memory specs.


#3

Hi @Bloutiouf,
Can you share the array of keys you are using in for the query and perhaps the complete URI?


#4

I can’t give you the exact keys, but I can tell you that on the query on which I found this issue, there were only 36 ascii character long keys. The limit was reached on 46 keys, so I took 45 keys and added some characters in the latest key until I reached the limit too.

Moreover, on another query where the keys are about twice the size, I noticed also a limit of about half the number of keys.


#5

I just found out that on CentOS, the limit is currently lower. I’m using a view whose name is shorter than the above one. The keys parameter is an array whose elements are all 3-element arrays, with following elements:

  • 2 byte string
  • 2 byte string
  • 36 byte string

In this case, the limit is reached on 28 elements, and precisely when the stringified array length is 1394.

However I’m noticing that the characters [ and ] may be URL-encoded to %5B and %5D respectively. That would mean that my current error, when the stringified length is 1394, would be encoded in 1506 characters. The first reported error, when the length is 1772, would be encoded in 1776 characters.


#6

Hi,

For views, we have the following limits:
Key size; 4096 byte
We also have maxEmitKVSize per design document which is total of key and values emitted per design doc for all views.This default value is 1MB.
Here are the current doc: http://docs.couchbase.com/admin/admin/Misc/limits.html
This is not complete but we are working on adding more details.

We use UTF-8 to encode characters. UTF-8 uses 1 to 4 bytes to encode chars.
So if your key length is 1776 char, it may hit the size limit of key size of 4096 bytes.

Thanks,
Qi


#7

Hi Qi,

As I understand, these are the limits for the emitted key-pair values. I don’t have any problem emitting these values. My problem is on querying them with a keys filter.

In the URL you provided, is the following correct? I find the terms confusing.

  • Max key length = max document id length
  • Max value size = max document size
  • Max View Key Size = max length for the emitted key and value

#8

Hi,

I’m sorry for the confusion.
We are working on improving documentation for 4.0.
You are correct about max key/value size.
We should rename them to be max docId and max docValue.

Max view key size = max key size for a view, which is 4096 bytes.
We have another limit for max length for emitted key/value and that is 1M.

Thanks,
Qi


#9

I still don’t get Max view key size. Provided emit(key, value), is it the max of key.length? Whereby key.length + value.length is limited to 1MB? Or is it the max length of the key and keys query options?


#10

Hi,

Max view key size is the max of key.length.
Max view key/value size is the max of key.length + value.length per design doc.

Thanks,
Qi


#11

Ok. I suggest to reword max view key size into max emitted key size.

Anyway this is not the problem here, my keys are all below 50 characters, and keys + values below ~100 characters (all characters are 1 byte long).


#12

Hello,

I have the same problem, it seems that a stringify happens on the viewQuery keys as a whole, so i cannot send more than ~1770 char long arrays of keys which is quite a small limit.
I am also using the node sdk.
Any suggestions/workarounds? I have currently implemented a for{get} but it is much slower


#13

Hi Tony,

Apart from the digression about the phrasing, I had no update too.

As fallback, I batched the queries so that each batch is not longer than the limit (though this limit has been determined empirically…). Then I query the next batch while processing the current batch’s results. So it’s like a pipeline processing, and I think it could even be better than processing the whole at once. Although I haven’t measured the Nodejs processing duration vs. Couchbase responses’ duration.


#14

I’m noticing that on my second example, there are more strings and the limit is lower, and I didn’t even thought about quotes being escaped. So with escaped quotes, on my first example, the limit would be actually 1864. On my second example, the limit would be 1589. Well, still not found.