"CurlException: Couldn't connect to server" on QueryAsync using Docker

I’m experiencing an issue where any QueryAsync call through the .NET SDK results in the following thrown exception ONLY when the application is in a docker container. The same code run on Kestrel does not have this issue (with Couchbase still in a docker container).

[Error] System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.CurlException: Couldn't connect to server
   at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
   at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Couchbase.N1QL.QueryClient.<ExecuteQueryAsync>d__19`1.MoveNext()

Here’s a minimal sample program I’ve written that proves that it CAN connect to the server because I am able to get back a document using GetDocumentAsync.

Program.cs

using System;
using System.Collections.Generic;
using System.Linq;
using Couchbase;
using Couchbase.Configuration.Client;
using Couchbase.Management.Indexes;

namespace ConsoleApp1
{
    class Program
    {
        private static readonly Cluster Cluster = new Cluster(new ClientConfiguration()
                                                              {
                                                                  Servers = new List<Uri>
                                                                                {
                                                                                    new Uri("http://couchbase-server1:8091/pools")
                                                                                }
                                                              });

        private static void Main(string[] args)
        {
            using (var bucket = Cluster.OpenBucket("default"))
            {
                var isOpen = Cluster.IsOpen("default") ? "open" : "closed";
                Console.WriteLine($"Cluster is {isOpen}.");

                var doc = new Document<MyModel>
                          {
                              Content = new MyModel
                                        {
                                            _service = "example",
                                            _type = "myModel",
                                            Name = "Name",
                                            DisplayName = "Display name"
                                        },
                              Id = "exampleId"
                          };

                bucket.UpsertAsync<MyModel>(doc);

                var result = bucket.GetDocumentAsync<MyModel>(doc.Id).Result;
                Console.WriteLine(
                    result.Success
                        ? $"Succesfully got back document with id '{result.Id}'"
                        : $"Failed to get document with id '{doc.Id}': Exception: {result.Exception?.Message} --- INNER --- {result.Exception?.InnerException?.Message}");

                const string indexName = "Idx_Example";
                var indexDropResult = bucket.QueryAsync<IndexInfo>($"DROP INDEX `{bucket.Name}`.`{indexName}`").Result;
                Console.WriteLine(
                    !indexDropResult.Success ? $"Failed to drop index {indexName}: Error count: {indexDropResult.Errors.Count}: Exception: {indexDropResult.Exception?.Message} --- INNER --- {indexDropResult.Exception?.InnerException?.Message}"
                    : $"Succesfully dropped index {indexName}");

                // Wait for index to drop
                System.Threading.Thread.Sleep(2000);

                var indexResult = bucket.QueryAsync<IndexInfo>($"CREATE INDEX `{indexName}` ON `{bucket.Name}`(`name`) WHERE (`_service`='example' AND `_type` = 'myModel')").Result;
                Console.WriteLine(
                    !indexResult.Success ? $"Failed to create index {indexName}: Error count: {indexResult.Errors.Count}: Exception: {indexResult.Exception?.Message} --- INNER --- {indexResult.Exception?.InnerException?.Message}"
                    : $"Successfully created index {indexName}");

                // Wait for index to build
                System.Threading.Thread.Sleep(5000);

                var queryResult = bucket.QueryAsync<MyModel>($"SELECT `{bucket.Name}`.* FROM `{bucket.Name}` WHERE `name`='Name' AND `_service`='example' AND `_type`='myModel'").Result;
                Console.WriteLine(
                    !queryResult.Success
                        ? $"Failed to execute query : Error count: {indexDropResult.Errors.Count}: Exception: {indexDropResult.Exception?.Message} --- INNER --- {indexDropResult.Exception?.InnerException?.Message}"
                        : $"Result found: {queryResult.Rows.FirstOrDefault().DisplayName}");
            }
        }
    }
}

MyModel.cs

namespace ConsoleApp1
{
    public class MyModel
    {
        public string Name { get; set; }

        public string DisplayName { get; set; }

        public string _type { get; set; }

        public string _service { get; set; }
    }
}

Result of this application when run in Kestrel:
Cluster is open.
Succesfully got back document with id 'exampleId’
Succesfully dropped index Idx_Example
Successfully created index Idx_Example
Result found: Display name

Result when run in docker:
Cluster is open.
Succesfully got back document with id 'exampleId’
Failed to drop index Idx_Example: Error count: 0: Exception: An error occurred while sending the request. — INNER — Couldn’t connect to server
Failed to create index Idx_Example: Error count: 0: Exception: An error occurred while sending the request. — INNER — Couldn’t connect to server
Failed to execute query : Error count: 0: Exception: An error occurred while sending the request. — INNER — Couldn’t connect to server

Hi @shickey

Please can you share your docker file? There have been issues with certain libraries incompatibilities with .NET Core on non-Windows hosts. Particularly with security and Curl libraries which may need to be updated.

Also, looking at your code sample, I’d like to offer some pointers:

  • By wrapping the bucket in a using statement means it will be disposed of, instead it would be better for allow the cluster to manage the bucket lifetime. The Cluster object will dispose of buckets when it is closed with cluster.Close(). It’s not a big deal in this example, but for a larger application don’t dispose of the bucket.
  • You’re using .Result property to wait on the Async methods, however you could use the sync version of the methods. I guess it’s probably because you copied the code from your asp.net app,.

Thanks

Hi @MikeGoldsmith, thanks for the response!

My docker file looks like this:

FROM microsoft/dotnet
WORKDIR /app
COPY publish .
ENTRYPOINT ["dotnet", "ConsoleApp1.dll"]

Additionally, I’ve been able to bash into the docker container and use Curl to make queries against Couchbase succesfully.

Thank you for the pointers. This is just a quick demo to try and demonstrate the problem I am experiencing on a much bigger project so I don’t think it’s a big deal either.

As for using the Async methods, those are the ones I’m experiencing the issue with on our larger application. I just ran a test using the synchronous methods and am receiving the same error on Query.

Hi @shickey

I’ve been trying to get my environment up and running with a simple .net core console app under docker but having some trouble. I’m not sure what the cause is, with the error I’m getting when trying to debug being:

I’ll keep digging into it. Just wanted to say I’m looking into it.

@shickey -

Given the error “Couldn’t connect to server” I would think the issue is between the client and the server, or perhaps the server is down.

-Jeff

We were able to resolve this issue by not changing the hostname of the couchbase server. The configuration for the Couchbase server we had was changing the hostname to a named address instead of letting it default to an IP address. Once we removed the hostname change, everything is working as expected.

I still find it odd that Get worked under these circumstances but Query did not. We suspect it is related to the way the IP address is resolved from the hostname, but we’re not sure. It’s possibly a bug with Couchbase where it fails lookup the IP of a named host for the index service for queries, but this is pure speculation.

Thank you for taking a look at this, @MikeGoldsmith .

@shickey -

I created a Jira ticket, since this appears to be a bug in the client or server: NCBC-1467

Thanks,

Jeff

I’m actually not certain it is. The couchbase bootstrap currently relies on the cluster serving a configuration that is reachable by the client, and that includes hostnames.

If you have the environment handy still @shickey, can you perhaps try a tool?

It’s a new experimental project that @brett19 has been working on to make it easier to diagnose environmental problems. Its working name at the moment is SDK doctor. Maybe try running it in that env?

You’ll find pre-built binaries on the release page.

Usually the summary at the end is pretty easy to interpret. In particular it reports about situations where name lookup deviates from hostnames. If you need help with interpretation, please feel free to post it or a link to it here. It’d be helpful to us to see if it catches the issue.

I ran SDK Doctor in a docker container and got the following results.

|====================================================================|
|          ___ ___  _  __   ___   ___   ___ _____ ___  ___           |
|         / __|   \| |/ /__|   \ / _ \ / __|_   _/ _ \| _ \          |
|         \__ \ |) | ' <___| |) | (_) | (__  | || (_) |   /          |
|         |___/___/|_|\_\  |___/ \___/ \___| |_| \___/|_|_\          |
|                                                                    |
|====================================================================|

Note: Diagnostics can only provide accurate results when your cluster
 is in a stable state.  Active rebalancing and other cluster configuration
 changes can cause the output of the doctor to be inconsistent or in the
 worst cases, completely incorrect.

13:51:42.152 INFO ▶ Parsing connection string `couchbase://localtest.me/default`
13:51:42.152 INFO ▶ Connection string was parsed as a potential DNS SRV record
13:51:42.220 INFO ▶ Connection string identifies the following CCCP endpoints:
13:51:42.221 INFO ▶   1. localtest.me:11210
13:51:42.221 INFO ▶ Connection string identifies the following HTTP endpoints:
13:51:42.221 INFO ▶   1. localtest.me:8091
13:51:42.221 INFO ▶ Connection string specifies bucket `default`
13:51:42.227 WARN ▶ Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your app                                  lications fault-tolerance
13:51:42.227 INFO ▶ Performing DNS lookup for host `localtest.me`
13:51:42.228 INFO ▶ Bootstrap host `localtest.me` refers to a server with the address `127.0.0.1`
13:51:42.228 INFO ▶ Attempting to connect to cluster via CCCP
13:51:42.228 INFO ▶ Attempting to fetch config via cccp from `localtest.me:11210`
13:51:42.230 ERRO ▶ Failed to fetch configuration via cccp from `localtest.me:11210` (error: dial tcp 127.0.0.1:11210: getsockopt: connection refused)
13:51:42.230 INFO ▶ Attempting to connect to cluster via HTTP (Terse)
13:51:42.230 INFO ▶ Attempting to fetch terse config via http from `localtest.me:8091`
13:51:42.232 ERRO ▶ Failed to fetch terse configuration via http from `localtest.me:8091` (error: Get http://localtest.me:8091/pools/default/b/default: dial tcp 127.0.0.1:8091                                  : getsockopt: connection refused)
13:51:42.232 INFO ▶ Attempting to connect to cluster via HTTP (Full)
13:51:42.232 INFO ▶ Failed to connect via HTTP (Full), as it is not yet supported by the doctor
13:51:42.232 ERRO ▶ All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible
13:51:42.232 INFO ▶ Diagnostics completed

Summary:
[WARN] Your connection string specifies only a single host.  You should consider adding additional static nodes from your cluster to this list to improve your applications fau                                  lt-tolerance
[ERRO] Failed to fetch configuration via cccp from `localtest.me:11210` (error: dial tcp 127.0.0.1:11210: getsockopt: connection refused)
[ERRO] Failed to fetch terse configuration via http from `localtest.me:8091` (error: Get http://localtest.me:8091/pools/default/b/default: dial tcp 127.0.0.1:8091: getsockopt:                                   connection refused)
[ERRO] All endpoints specified by your connection string were unreachable, further cluster diagnostics are not possible

Found multiple issues, see listing above.

We have a host entry for localtest.me that resolves to 127.0.0.1 somewhere on our network. Our docker instance with the couchbase server exposes port 8091 (as well as ports 8091-8094, 11207, 11210-11211 and 18091-18093).

Please let me know if you need more information to determine if this is a couchbase or environment issue.