Problem with large files (>5 mb)

Java SDK
, ,
#1

I have a problem when saving large documents (on some databases > 5 MB, on others > 10 mb).

Here’s the procedure:
Front-end is sending a request.
Back-end recieves the request and tries to save it.
The save method finishes but the document ID is null.

The save procedure is actually finished okay, the document is saved and it has its own ID. It feels like the save method finishes before its actually done and the save needs some time to save a document this large.

SELECT *
FROM qms 
USE KEYS '24b49e61-2043-4b68-bc7f-a99be9f0ae1a'

This query is taking a long time to finish.
Sometimes I also get a timeout exception (Please see Image 2).
Images 3 shows the request is OK (200) but the ID is null. (File has 9.63 MB)
Image 4 shows the same requests but with a smaller file (1 MB). This time I got the id.

I’m using Spring data Couchbase (https://spring.io/projects/spring-data-couchbase#overview)
Spring boot version : 2.2.4.RELEASE
Couchbase version 6.0

I’m saving files in base64 format, as a property of a document.

I tried to set timeout time with spring.couchbase.env.timeouts.connect=10000ms

https://ibb.co/CWvPJ3g - Image 2
https://ibb.co/VqbYwWt - Image 3
https://ibb.co/bggcFJH - Image 4
https://ibb.co/QJbzpBH - Image 5
https://ibb.co/TP4S3YM - Image 6

#2

Did you tried in UI. If you already know the key use SDKs get the documents avoid 2 hops.

#3

I did, in the UI it will execute the query in ~ 200 ms, but it doesn’t render the result for 10-20 seconds, for example a file that is 7mb. When trying the same query through my app, the query will always timeout for larger files. Tried raising query timeout with no success.

#4

@jspnvk - When you are running the query in the UI, which output format are you using? JSON text? Tables? Rendering large documents in the UI can be time consuming, though usually the Tables are the most efficient.

Also, did you try running against the query service REST API? (http://:8093/query/service?statement=select …)

#5

Thank you for your reply!

Yes, usually i use JSON text.

I just tried REST API query service and i was able to get the file, but it took ~4 seconds, as opposed to the 200ms that the UI is reporting. As i understand, this is well under the default timeout interval, so the question remains as to why it’s impossible to get the document through my app, but the query executes fine through the UI and query service.

#6

Could network time explain the difference between 200ms and 4s in getting the file to the client?

Still, it should not be timing out. I don’t know enough about the Java SDK, perhaps @ingenthr can help find who would know the answer to that.

#7

Yes, there were timeouts because threads gave up before data from CB arrived. Any timeout fixes we tried failed, because they were ignored. Both Spring and CB failed to document how to configure timeouts properly.

Here’s a proper way how to do timeouts: https://www.screencast.com/t/PfY1FNTYJpLB
(at least the only way that works for sure).