Empty exception while waiting for scan_consistency

Hi,
I’m trying to migrate my solution using Couchbase 3.0.1 SDK and still have issues related to MutationState after upsert.
I’m doing it by the book:

try
{
var upsertResult = await _collection.UpsertAsync(id, obj);
var state = MutationState.From(upsertResult);
var request = await this.Cluster.QueryAsync(“SELECT id FROM db WHERE id=’” + id + “’”, options =>{
options.ConsistentWith(state); });
}
catch (Exception ex)
{
Logger.LogError(ex.Message)
throw ex;
}

and in the catch I’m receiving empty exception with no further information what is happening.
Inside the CouchbaseException instance I see:

Couchbase.Core.Exceptions.Query.QueryErrorContext
HttpStatus: Internal Server Error
Statementry: [{“statement”:“SELECT id FROM db WHERE id=‘a9956d45-3704-4b09-9aa2-764659fe7ad0’”,“timeout”:“75000ms”,“scan_consistency”:“at_plus”,“scan_vectors”:{“db”:{“734”:[1,“61463890692985”]}},“client_context_id”:“39c499aa-8c46-4476-8804-296717a1f51f”}]

I’m running Couchbase 6.5.1 on Windows 10 machine.
NET Core 3.1, CouchbaseNet 3.0.1.
Upsert, Select, Flush work as expected.

Regards

Hi @foximoxi,

That SELECT query has the potential to take a long time to execute, which could be the reason this error is happening. Could you try the query on its own to make sure it’s using the correct index and executing as quickly as you expect?


Also, not related to your question, but I spotted a couple other issues with this code that you might want to consider changing. You might be already aware of all these things, but it’s worth pointing out for others who may come along and read this later :slight_smile:

  • When constructing a N1QL string, it’s a good idea to avoid direct concatenation of variables, as this could lead to SQL injection. For instance, instead of "id='" + id + "'", consider using "id = $id" and then using parameterization to replace that placeholder: options.Parameter("id",id)
  • It looks like you’re using a N1QL query to look up a document by its ID, but I’m not 100% sure since you aren’t using META().id. Did you mean to use META().id?
  • If you are looking up a single document by ID, it’s almost certainly going to be faster to use the Get function instead of a N1QL query. Your query is also just selecting the ID, so maybe you can just use an Exists function

Hi,
I’m using parameters everywhere and META().id so the modified code should look like:

var request = await this.Cluster.QueryAsync(“SELECT META().id FROM db WHERE id=’$id’”, options =>
{
options.ConsistentWith(state).Parameter(“id”, id);
});

But unfortunately does not change the situation. I’m still receiving an empty exception like above.
Perhaps there is a better way to get the confirmation about consistent database?
I did not find any other information than this:
https://docs.couchbase.com/dotnet-sdk/2.7/scan-consistency-examples.html

In the meantime I found that suppressing the exception works but looks silly…
Still, all my tests are green now.

while (true)
{
try
{
var request = await this.Cluster.QueryAsync(“SELECT META().id FROM " + DbName + " WHERE
id=’$id’”, options => { options.ConsistentWith(state).Parameter(“id”, id); });
break;
}
catch (Exception ex) { }
}

It fails when I’m trying to use .Exists because I’m not able to give this ConsistenWith(state) option.

Regards

In your query text, you don’t need the quotes around the $id placeholder. And if you are trying to query by the document key, try this: SELECT META().id FROM db WHERE META().id = $id

But depending on the number of documents in your database, the performance of this query might be the issue. It could just be timing out. Which indexes have you created on the bucket? How long does this query take to run when you execute it on its own (in Query Workbench, for instance)?


Side notes:

If all you are trying to do is check if the document exists, then you do not need to use ScanConsistency. ScanConsistency only applies when executing N1QL queries (because N1QL indexes are updated asynchronously).

You can just use Exists (or ExistsAsync) method of your Bucket object instead of a N1QL query (if you want).

But further, if you are just inserting a document and want to verify that it was inserted, you maybe just want to have a look at Durability Requirements (.NET SDK 2.7). (Durability options have been expanded for the latest version of Couchbase and the .NET SDK by the way). Specifying a durability requirement may help you, if all you’re doing is trying to verify that the document has been inserted and now exists.

Hi,
Thank you for the recommendations but I’m still not able run my very simple test:

  1. Flush database
  2. Insert one record
  3. Find the record using a query

So after flush I execute:

await _collection.UpsertAsync(id, obj, o => o.Durability(PersistTo.One, ReplicateTo.None));
Logger.LogInformation(await _collection.ExistsAsync(id)).Exists));//I receive true value here
string query=“SELECT email FROM db WHERE email = $email”;
var res=await Cluster.QueryAsync(query, parameters => parameters.Parameter(“email,"some@email.com”)

I would like to be sure that after Upsert I’m able to execute query and get the record.

Thank you in advance

@foximoxi -

Two things:

  1. In your initial post the error status is: HttpStatus: Internal Server Error - this is a server error, so it wouldn’t be related to your query syntax (even though there are some errors there).
  2. Can you post the entire exception and stacktrace?
  3. Can you execute queries through the mgmt console?

-Jeff

I’ve just written a little console program that goes through those steps. It seems to run okay. Could you try running it and post the exception/stacktrace (as @jmorris has asked) where it breaks down for you?

using System;
    using System.Threading.Tasks;
    using Couchbase;
    using Couchbase.KeyValue;
    using Couchbase.Query;

    namespace ForumQuestions
    {
        class Program
        {
            static async Task Main(string[] args)
            {
                var cluster = await Cluster.ConnectAsync("couchbase://localhost","Administrator", "password");

                var bucket = await cluster.BucketAsync("foxibucket");
                var collection = bucket.DefaultCollection();

                // 0. "CREATE PRIMARY INDEX on foxibucket" has already been executed

                // 1. flush
                await cluster.Buckets.FlushBucketAsync("foxibucket");

                // 2. insert one record
                var id = $"doc-{Guid.NewGuid()}";
                var obj = new { email = "matthew.groves@couchbase.com" };
                var mutationResult = await collection.UpsertAsync(id, obj, o => o.Durability(PersistTo.One, ReplicateTo.None));

                // 3a. find the record using a query
                var queryResults = await cluster.QueryAsync<dynamic>("SELECT email FROM foxibucket WHERE email = $email",
                    options =>
                    {
                        options.Parameter("email", "matthew.groves@couchbase.com");
                        options.ConsistentWith(MutationState.From(mutationResult));
                    });
                await foreach (var result in queryResults.Rows)
                {
                    Console.WriteLine(result.email);
                }

                // 3b. instead of using N1QL, you can also use Exists (which doesn't need N1QL/index/ScanConsistency/MutationState):
                var doesDocumentExist = await collection.ExistsAsync(id);
                if (doesDocumentExist.Exists)
                    Console.WriteLine($"Document with key {id} exists!");
                else
                    Console.WriteLine($"Document with key {id} not found!");

                await cluster.DisposeAsync();
            }
        }
    }

Hi,
The example program is running without an issue, but I made a small modification to replicate the exception.
It happens in the second iteration, sometimes later. Maybe it is related to same connection without disposal in the loop?
Couchbase database is running normally, I can run queries and all my test scenarios on the 2.7.17 connector.

using System;
using System.Threading.Tasks;
using Couchbase;
using Couchbase.KeyValue;
using Couchbase.Query;

namespace ForumQuestions
{
    class Program
    {
        static async Task Main(string[] args)
        {
            string bucketName = "merchant";
            var cluster = await Cluster.ConnectAsync("couchbase://localhost", "admin", "123456");

            var bucket = await cluster.BucketAsync(bucketName);
            var collection = bucket.DefaultCollection();
            await cluster.Buckets.FlushBucketAsync(bucketName);
            for (int i = 0; i < 1000; i++)
            {
                    try
                    {
                    var id = $"doc-{Guid.NewGuid()}";
                    var obj = new { id = "matthew.groves@couchbase.com" };
                    var mutationResult = await collection.UpsertAsync(id, obj, o => o.Durability(PersistTo.One, ReplicateTo.None));
                            var queryResults = await cluster.QueryAsync<dynamic>($"SELECT id FROM {bucketName} WHERE id = $id",
                    options =>
                    {
                        options.Parameter("id", "matthew.groves@couchbase.com");
                        options.ConsistentWith(MutationState.From(mutationResult));
                    });
                await foreach (var result in queryResults.Rows)
                {
                    Console.WriteLine(result.id);
                }
                var doesDocumentExist = await collection.ExistsAsync(id);
                if (doesDocumentExist.Exists)
                    Console.WriteLine($"Document with key {id} exists!");
                else
                    Console.WriteLine($"Document with key {id} not found!");
            }
            catch (Exception ex)
            {
                var msg= ex.Message;//here is the empty exception
            }
        }
        await cluster.DisposeAsync();
    }
}

}

Regards

Hi @foximoxi,

That code snippet seems to be running fine on my machine (also Windows 10, Couchbase 6.5.1, .NET Core 3.1, SDK 3.0.1). I ended up with 1000 documents and no exceptions. But, if you are still getting the exception, could you post the details of the exception and the stack trace here?

@foximoxi

Your problem might be related to FlushAsync. Unless something has changed that I’m unaware of, flushing is an async process on the cluster. The cluster returns success back to the SDK when the flush command is being processed, but doesn’t wait for the flush to actually be complete. So the flush might be happening part way through your loop, and computer speed could be a factor in behaviors.

If I write await Task.Delay(30000) after FlushAsync, I still have the exception.
But if I flush the bucket from the console and skip flushing in the code it runs without an issue.
I was able to insert 1000 records.

The stack trace of the empty exception:

at Couchbase.Query.QueryResultExtensions.ThrowExceptionOnError[T](IQueryResult`1 result, QueryErrorContext context)
at Couchbase.Query.QueryClient.ExecuteQuery[T](QueryOptions options, ITypeSerializer serializer)
at Couchbase.Query.QueryClient.QueryAsync[T](String statement, QueryOptions options)
at Couchbase.Cluster.<>c__DisplayClass27_0`1.<<QueryAsync>g__Func|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync[T](Func`1 send, IRequest request)
at Couchbase.Cluster.QueryAsync[T](String statement, QueryOptions options)
at ForumQuestions.Program.Main(String[] args) in c:\projects\temp\CouchbaseTest\Program.cs:line 27

27th line is await cluster.QueryAsync invocation.

additionally,
when I extended the delay after flush in the code to 60secs (60000 value) I received the exception on 111th iteration:
ReadResult does not contain valid MutationToken

Stack:

at Couchbase.Query.MutationState.Add(IMutationResult[] mutationResults)
at ForumQuestions.Program.<>c__DisplayClass0_0.<Main>b__1(QueryOptions options) in c:\projects\temp\CouchbaseTest\Program.cs:line 33
at Couchbase.ClusterExtensions.QueryAsync[T](ICluster cluster, String statement, Action`1 configureOptions)
at ForumQuestions.Program.Main(String[] args) in c:\projects\temp\CouchbaseTest\Program.cs:line 29

Regardsexception.zip (868 Bytes)