Linux .Net Core Atomic Operations Fail using CouchbaseNetClient

I am attempting to use the CouchbaseNetClient 2.4.0-dp3 (installed from nuget) on a linux machine running .NET Core 1.1.0.

I am able to connect to my couchbase server hosted on another computer and execute operations on it (Upsert works fine for instance). However, when I attempt to use the atomic Increment or Decrement operations I receive the following error message:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Couchbase.CouchbaseBucket.GetServer(String key, IVBucket& vBucket)
   at Couchbase.CouchbaseBucket.Increment(String key, UInt64 delta, UInt64 initial, UInt32 expiration)

Is this a bug or am I missing some sort of configuration?

FYI, the exact same code works just fine on a windows machine.

Hi @sludepredr -

Interesting, I suspect a bug, but the code paths are the same Upsert, Increment, Decrement and all other binary Memcached operations, so I do not understand why it would only fail on Decrement and Increment.

Iā€™ll see if I can replicate. What flavor of linux?

-Jeff

Hello @jmorris

Iā€™m running on CentOS 7.3.

dotnet --version returns 1.0.0-preview2-1-003177 in case itā€™s something to do with that.

Also, FYI, I attempted to create a Cluster instance as well as use the ClusterHelper - both methods produced the same error.

Thanks for looking into it!

@jmorris -

Ah, now we are getting somewhere! I am not sure about the dotnet version - I am running the same on win10. However, if you cannot create a Cluster instance (and a bucket ref off of it), how can Upsert work? Are you sure that the client is correctly bootstrapping?

-Jeff

@jmorris -

Iā€™m pretty sure (and the exact same code works fine in windows). Iā€™m not getting an exception when I create the cluster instance or open the bucket. cluster.IsOpen("my_bucket") returns true for me. Is there anything I can look at to verify everything is fully bootstrapped? Or some additional dependency I need to install specifically on linux?

I created a console application, as simple as can be that causes the error:

ļ»æpublic class Program
{
    public static void Main(string[] args)
    {
        try
        {
            var cluster = new Cluster(new ClientConfiguration { Servers = new List<Uri> { new Uri("couchbase://my_server/") }});

            using (var bucket = cluster.OpenBucket("my_bucket"))
            {
                var result = bucket.Increment("my_counter", 1);
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex);
        }
    }
}

@jmorris

I did a little more digging, and it actually looks like my Upsert is not actually working as I thought it was - it just wasnā€™t throwing an exception so I guess I just assumed it was working. When I look at the result of the Upsert though, I see this in the Exception property:

Couchbase.Core.Services.ServiceNotSupportedException: A request has been made for a service that is not configured or supported by the cluster. Please check the cluster and enable or add a new node with the requested service: Data.

This message is confusing to me because Iā€™m able to connect to the same server in a windows environment with the exact same .net core dll. I can even connect to Couchbase on this same linux machine using Node.js!! I did log into the Couchbase Console Server Nodes tab and verified that ā€œDataā€ is listed under the ā€œServicesā€ column.

This makes me think that I need to connect differently or configure something differently in the linux environment specifically for .net core? I attempted to run the application as root in case there was a permission issue, but thereā€™s no difference.

@sludepredr

I noticed that youā€™re referencing Couchbase via the url couchbase://my_server/. In all of my testing, I used urls like http://my_server:8091/. Iā€™m wondering if maybe .Net Core on Linux is doing something different with the couchbase: scheme. Could you try it with http and see what happens?

Brant

@jmorris -

The error message (Couchbase.Core.Services.ServiceNotSupportedException) is pretty straight forward: it indicates that the Data service is not configured on the server. However, if indeed the Data service is configured, then I suspect something else is going on.

As @btburnett3 suggested, can you try using an URI in the following format: http://myserver:8091?

-Jeff

@jmorris, @btburnett3

Thanks for your responses!

The Data service is configured and is working correctly. Hereā€™s how I know:

  • I can connect to Couchbase using the same .NET Core DLL on a different machine (Windows) and execute all operations against the database successfully (verified documents are created, deleted, etc in the Couchbase console).
  • I can connect to Couchbase on the same machine (Linux CentOS) using Node.js and execute all operations successfully.
  • The Couchbase Console lists Data, Index, and Query as services enabled on the server node.

The problem seems to be specifically related to the CouchbaseNetClient on Linux.

I did change the server connection to the suggested format http://myserver:8091, but I receive the same error. I even tried ā€˜http://myserver:8091/poolsā€™, ā€˜http://myserver:8091/pools/defaultā€™, ā€˜http://myserver:8091/pools/default/mybucketā€™ and a bunch of other variations to no effect.

Also, when I curl ā€˜http://myserver:8091/pools/defaultā€™, it showed "couchApiBase":"http://myserver:8092/","couchApiBaseHTTPS":"https://myserver:18092/". The ports are different, so I tried these and they donā€™t work either - same error.

I also set the UseSsl configuration to true and changed the connection to https. I logged into the Couchbase Console and downloaded the self-signed certificate and added it as a trusted certificate on Linux as well. I still receive the same error.

I guess the CouchbaseNetClient is not getting the correct metadata from the server to know that the cluster has the Data service enabled???

Iā€™m at a loss of where to go from here. Any ideas of something else I can try? Has anyone been able to replicate the issue on Linux?

@sludepredr

Okay, I have another theory. It could be network related. Even if you connect via the name ā€œmyserverā€ initially, the client uses the list of servers returned by the API call for actual communication. How are the server nodes listed in your Couchbase cluster? IP or domain name? And is the way they are listed accessible from your Linux box?

Brant

@btburnett3

Thatā€™s a great point, I didnā€™t realize that was the case. Unfortunately, I donā€™t think thatā€™s the problem though.

In the Couchbase Console, my server is listed by IP address, and Iā€™ve been connecting via IP address the whole time.

When I curl the /pools/nodes API endpoint, i see that the "hostname" listed is the same IP address and port listed in Couchbase Console which matches what Iā€™m attempting to connect to.

The server is accessible just fine from the Linux box, just not through the .NET Core application.

I am running the 4.6 Developer Preview Couchbase server, so maybe something with that? Iā€™m running out of options, so I might just try downgrading to the current 4.5 server release and see if anything changes, but Iā€™m not hopeful that will resolve the problemā€¦ :frowning:

That is what I suspect, which seems odd given that cluster map is client independent - every client gets the same map.

[quote=ā€œsludepredr, post:9, topic:11181ā€]
Iā€™m at a loss of where to go from here. Any ideas of something else I can try? Has anyone been able to replicate the issue on Linux [/quote]

In the /pools/nodes API JSON, what services do you see listed for each node? For example:

services: [
    "index",
    "kv",
    "n1ql"
]

kv is the Data service.

Hang in there!

-Jeff

@jmorris

Thanks for sticking with me!

The ā€œnodesā€ section of the json contains only one node object with the following services:

"services": [
    "index",
    "kv",
    "n1ql"
],

I just tested against CB 4.6.0 dev using .NETCoreApp 1.1 and it worked as expected on ubuntu14 with CB running on localhost.

@sludepredr

Could it be a firewall issue? Maybe all of the ports arenā€™t open from the Linux box to the cluster for some reason?

https://developer.couchbase.com/documentation/server/current/install/install-ports.html

Brant

I ran through the same issue. It worked fine on windows machine but was failing for exact same reason on the linux box. Downloaded couchbase sdk from github, added console logging. Turns out the .Net httpclient (used by the sdk) was making a call to curl that was not supported by the curl version installed on my linux box. Upgraded curl from 7.29 to 7.52 and everything works now.

2 Likes

@ankuranand -

Thanks for tip; were running into some issue testing on Centos possibly related to this.

-Jeff