Mismatch between N1QL query and response

Java SDK
1

Hi,

Recently, we found out some of our documents were deleted when this should not have been the case!
Digging deeper, we arrived to the conclusion that it was because of a mismatch between a N1QL query and its response, leading us to retrieve the wrong document id for our query, and later on deleting the “wrong” document based on the N1QL query returned data.

We added a clientContextId for every query we make and compare it with the one returned in the response and we have some time a mismatch detected!

Error querying from datastore with query [SELECT myField FROM myBucket WHERE myField LIKE ‘prefix:%’ AND myOtherField IS VALUED ORDER BY myField LIMIT 1] with context id [n1ql-7446786d-d8a8-457e-9fb5-f073f6f8d390] - Response context id [n1ql-cc3d4da1-499e-4c69-8136-10587baad8ba]

At the same time we are having similar queries happening, all returning only the myField therefore, before adding the clientContextId were not able to detect the mismatch:

SELECT myField FROM myBucket WHERE myField LIKE ‘prefix1:%’ AND myOtherField IS VALUED ORDER BY myField LIMIT 1
SELECT myField FROM myBucket WHERE myField LIKE ‘prefix2:%:suffix2’ AND myOtherField IS VALUED

We are using Couchbase 4.5 and Java SDK client 2.3.1.

We are not using prepared statement and the snipper of code below shows how we use the SDK to query:
`
public List executeSimpleQuery(String statement) {
return executeQuery(N1qlQuery.simple(statement, N1qlParams.build().consistency(ScanConsistency.REQUEST_PLUS)));
}

public List<N1qlQueryRow> executeParameterisedQuery(String statement, JsonObject placeholderValues) {
    N1qlQuery query = N1qlQuery.parameterized(statement, placeholderValues, N1qlParams.build().consistency(ScanConsistency.REQUEST_PLUS));
    return executeQuery(query);
}

private List<N1qlQueryRow> executeQuery(N1qlQuery query) {
    // Generate a client context id for sanity check on response
    String clientContextId = generateN1QLClientContextId();
    query.params().withContextId(clientContextId);

    // Do the query
    N1qlQueryResult queryResult = bucket.query(query);

    // Check response match query
    if (!clientContextId.equals(queryResult.clientContextId())) {
        throw new RuntimeException(
                String.format("Error querying from datastore with query [%s] with context id [%s]" +
                                " - Response context id [%s] does not match",
                        query.statement().toString(), clientContextId, queryResult.clientContextId()));
    }

    // Check response success
    if (!queryResult.parseSuccess() || !queryResult.finalSuccess()) {
        throw new RuntimeException(
                String.format("Error querying from datastore with query [%s] - Errors are [%s]",
                        query.statement().toString(), queryResult.errors().toString()));
    }

    return queryResult.allRows();
}`

Also to be mentioned, if I use the query tab in the UI to retrieve the list of completed_requests with select * from system:completed_requests order by Time desc, not all the queries shows the clientContextId!

Is there something we are doing wrong in our query code?
Has anyone encountered this issue of mismatch between query and response before?

Any help will be really appreciated!
Thanks.

2

Hey @lbertrand,
That’s a big issue :frowning:
From the look of it, it doesn’t seem like the N1qlQuery objects are somehow reused… We’ll have to take a deeper look into this.

Do you happen to have tuned the CouchbaseEnvironment (especially, the queryEndpoints setting)?
Also, what did you mean by this:

completed_requests order by Time desc, not all the queries shows the clientContextId!

3

Hi,

Since I posted this question, I realised that I was reusing the N1QLParams object between queries so I think in multi threaded environment, I may have created the mismatch myself…

Correct me if I am wrong but the N1QLParams associated to a N1QLQuery is not thread safe in the sense that it can be modified as it is not copied into the query… And also, I think the SDK also modify it, like adding the server timeout parameter if not present…

So I corrected my code to create a new N1QLParams for every N1QLQuery and I have not seen this issue yet! But this does not mean it is not there, as it was not necessarily something I saw before end all the time! So I am letting it run a little bit hoping to see it again, or another log I have added in trying to track the issue we have!

Regarding the completed_requests, I meant i did the N1QL query in the Couchbase console, and not all the shown requests were having the fields clientContextId in it - but this may be explained by the issue with reusing N1QLParams above!

I checked today again and all the completed requests have a clientContextId now!

I will carry on keeping an eye on it and report later again!

4

Ah that would totally explain it, it was actually the first thing I thought about when reading your issue… I’m relieved :sweat_smile:

Indeed the N1qlParam is not thread safe and can be mutated by the SDK, so it should not be reused with several N1qlQuery.