Flaky Upsert Performance

.NET SDK
1

Hello -

We’re getting really flaky performance from our new Couchbase 3.0 server when trying to do a basic upsert using the .NET SDK. The code is pretty simple:

        this.CheckBucketConnection();

        var itemDictionary = new Dictionary<string, T>();
        foreach (var item in items)
        {
            itemDictionary[item.DocId] = item;
        }

        var upsert = this.TenantBucket.Upsert<T>(itemDictionary);
        if (upsert.Any(or => !or.Value.Success))
        {
            var failed = from f in upsert
                         where !f.Value.Success
                         select new { Message = f.Value.Message, Ex = f.Value.Exception, Key = f.Key };

            throw new IOException(string.Format("store operation failed for {0} items: {1}", failed.Count(), failed.First().Message));
        }

The error we’re getting is from the SDK:

Error storing Visits: store operation failed for 55 items: The node 191.239.12.1
10:11210 that the key was mapped to is either down or unreachable. The SDK will
continue to try to connect every 1000 seconds. Until it can connect every operation routed to it will fail with this exception. Sleeping for 15 seconds before retry.

The server is online and healthy, and the app node (executing the C#) and the Couchbase server are all in the Azure cloud (so shouldn’t have any weird proxy or firewall issues). What’s interesting is that it seems to run fine until Couchbase begins attempting an indexing operation. As soon as the server starts building an index, the upsert craps out.

Any ideas? We feel like we’re missing a crucial detail on either A) how to configure our CB server, or B) how to use the SDK.

Thanks,

  • Jeff
2

@jeffhoward001 -

Which version of the SDK are you using? Can you enable logging and see what is logged when this happens?

-Jeff

3

Thanks for the quick reply! We’re currently using SDK ver 2.1.1.0, so looks like we’re a minor release behind. I apologize if this is a dumb question, but we’re new to Couchbase. When you say enable logging, where do I do that? At the server level via the admin console, or is there other type of logging in the SDK?

I didn’t see very many options to change logging in the Couchbase Console.

Thanks,

  • Jeff
4

@jeffhoward001 -

Without knowing the details of the issue, I suspect it may be related to Azure datacenter load balancer and TCP keep-alives. Checkout out this jira ticket as it explains the issue and how to solve it: https://issues.couchbase.com/browse/DOC-554

Here is a link that shows how to enable logging: http://docs.couchbase.com/developer/dotnet-2.1/setting-up-logging.html

Note that in a production environment, using a verbose log setting like DEBUG will degrade performance, but show the most detail.

-Jeff

5

To close the loop on this one, we decided to run mono on our linux nodes and run our C# binaries against localhost. We did enable the SDK logging but didn’t see anything much more descriptive than what was in the original post. Running against localhost we’re able to import millions of records without issue, so I believe that will be our permanent solution.

Thanks,

  • Jeff