Sync Function does not update

PlutoHD · January 26, 2023, 2:14pm

For some reason I can’t explain, I can no longer change the sync function for my server. I am using the Couchbase Postman Collection (Postman Collections for Sync Gateway Administration - The Couchbase Blog), even just re-imported it completely and entered the credentials, but when I try to change the sync function, I get a status 200, but the sync function does not change (as on this screenshot). I have checked that the admin credentials are correct, the account has Full Admin permissions. In the console it prints the following:

2023-01-26T13:55:03.209Z [INF] HTTP+: #1243:     --> 200   (218.0 ms)
2023-01-26T13:55:02.993Z [INF] HTTP:  #1243: GET /*bucketname*/_config/sync (as <ud>admin</ud> as ADMIN)
2023-01-26T13:55:02.993Z [INF] Auth: #1243: User <ud>admin</ud> was successfully authorized as an admin

So no error here either. Now I don’t understand why the Sync Function doesn’t change (no change seen in the _sync:syncdata document either).

mreiche · January 26, 2023, 6:57pm

You are doing a GET, the operation requires a PUT.

Mike

PlutoHD · January 26, 2023, 7:22pm

As you can see in the screenshot: Postman Screenshot, it is a PUT operation. I have not changed anything about the Postman Collection and it worked some time ago without any problems. But something must have happened since then so it does not work anymore.

mreiche · January 26, 2023, 7:31pm

I see a GET. Or that’s not related to the issue?

PlutoHD · January 26, 2023, 7:42pm

That’s what is printed in the Sync Gateway console at the time I run the PUT operation. I can’t tell why it says GET there, but the operation is definitely PUT.

mreiche · January 26, 2023, 7:56pm

The server is logging it was a GET.
double-check the GET/POST/PUT… drop-down before you hit send.

PlutoHD · January 26, 2023, 8:41pm

As you can see in the screenshot, PUT is definitely selected. After some research I found out that for some status codes, it may be that the request is changed to a GET operation. That would explain what we are seeing here. When I run the request using the curl command that I can copy from Postman, I get this error:

{"error":"Unauthorized","reason":"Login required"}

I have triple checked that the credentials are correct. Postman includes these like this in the curl request:

--header 'Authorization: Basic *** BASE64 ENCODED 'username:password' ***'

I have also checked that the user has the required permissions. As I use the CE, the only role available is Full Admin which I granted to the user.

In conclusion: It is possible that this is an authentication issue that is hidden by a redirect to a GET operation. Is the way Postman authenticates wrong or do you have any other idea why it is not working?

mreiche · January 26, 2023, 9:10pm

Maybe try taking the database offline and then back online? Sync Gateway _config REST endpoint not working as expected

Otherwise :

Yes. But I can’t tell if it PUT was selected when the Send was done.

After some research I found out that for some status codes, it may be that the request is changed to a GET operation.

A redirect is HTTP 301, 302 or 307. Are you seeing any of those in the logging?

When I run the request using the curl command that I can copy from Postman, I get this error:

{"error":"Unauthorized","reason":"Login required"}

You’ll need to add the -u username:password option to curl.

PlutoHD · January 26, 2023, 9:57pm

I have made sure that it is set to PUT, I have clicked Send many times.

That’s what I’ve found (GitHub), but I think this does not apply to this problem as I see no reason for a 301, 302 or 307 and I am not seeing any of those in the logging.

I now tried to run:

curl --location --request PUT '*DOMAIN ADMIN REST API*/_config' \
--header 'Content-Type: application/json' \
-u admin:*PASSWORD* \
--data-raw '{
    "logging": {
        "console": {
            "log_level": "debug",
            "log_keys": [
                "*"
            ]
        }
    }
}'

and it still returns: {"error":"Unauthorized","reason":"Login required"}.
The log shows 401 Login required.

Yes, this is a different API endpoint now, but I’ve seen that this issue applies to other endpoints too when I tried to change the log_level.

Currently it seems to be an authentication issue. I have made sure that the admin user exists (Cluster > Security), it has the Full Admin role and the password is correct. Now I have no idea why this authentication issue occurs.

priya.rajagopal · January 26, 2023, 11:06pm

Can you try one of the GET admin calls with admin API to check if its write access issue. Example

curl --location --request GET 'https://sync-endpoint:4985/db/_user' -u username:password

Can you share screenshot of your RBAC user config on server. Double check that you provided access to all buckets

Also, you mentioned CE - wondering if you can retry w/ EE to see if anything amiss in the CE permissions. Note that EE is free to use for dev and testing.

mreiche · January 26, 2023, 11:54pm

Does the password contain a special character that the shell might interpret into something else? (for instance, the shell would interpret mypass$1 into mypass (as $1 would evaluate to nothing). If so, put the username:password in single-quotes.
Also - curl needs the --location-trusted (instead of just --location) to forward credentials in a redirect. Although I don’t know if sync gateway needs that.

PlutoHD · January 27, 2023, 1:14pm

I just made a fresh Couchbase Server and Sync Gateway installation. The first command I run (users are created during setup):

curl --location-trusted --request GET '*ADMIN REST API*/_config?include_runtime=false' \
-u 'admin:password'

works and returns me the following:

{
    "bootstrap": {
        "server": "couchbase://couchbase-server",
        "username": "sync_gateway",
        "password": "xxxxx",
        "use_tls_server": false
    },
    "api": {
        "public_interface": ":4984",
        "admin_interface": ":4985",
        "metrics_interface": ":4986",
        "https": {},
        "cors": {}
    },
    "logging": {
        "console": {
            "rotation": {},
            "log_level": "info",
            "log_keys": [
                "*"
            ]
        },
        "error": {
            "rotation": {}
        },
        "warn": {
            "rotation": {}
        },
        "info": {
            "rotation": {}
        },
        "debug": {
            "rotation": {}
        },
        "trace": {
            "rotation": {}
        },
        "stats": {
            "rotation": {}
        }
    },
    "auth": {},
    "replicator": {},
    "unsupported": {
        "http2": {}
    }
}

However, if I then run this:

curl --location-trusted --request PUT '*ADMIN REST API*/_config' \                      
--header 'Content-Type: application/json' \                                                                                                                                                                         
-u 'admin:password' \
--data-raw '{
    "logging": {
        "console": {
            "log_level": "debug",
            "log_keys": [
                "*"
            ]
        }
    }
}'

to increase the log level, I get this error: {"error": "Internal Server Error", "reason": "Internal error: EOF"}.

In the log of Couchbase Server nothing appears regarding the operations, in the log of Sync Gateway it prints this:

2023-01-27T13:04:27.729Z [INF] Auth: #005: User <ud>admin</ud> was successfully authorized as an admin
2023-01-27T13:04:27.729Z [INF] HTTP:  #005: PUT /_config (as <ud>admin</ud> as ADMIN)
2023-01-27T13:04:27.729Z [ERR] #005: EOF -- rest.(*handler).writeError() at handler.go:1041
2023-01-27T13:04:27.729Z [INF] HTTP: #005:     --> 500 Internal error: EOF  (2.5 ms)

Here is a screenshot of the users (image upload not working here).

PlutoHD · January 27, 2023, 1:17pm

No, just numbers and letters, no special characters.

I tried this anyways, does not change anything.

I changed this in my requests, but this does not fix anything unfortunately.

gabriel · January 27, 2023, 5:54pm

In the last curl invocation, you posted, there is a { too much in the argument for --data-raw before "debug".

If that is the case, Sync Gateway might choke on that because it can’t parse it as JSON correctly. Again, if that is the case, it would be better if Sync Gateway returned an error message describing the problem and HTTP status code 400.

PlutoHD · January 27, 2023, 6:07pm

Sorry, I just checked, that wasn’t there when I ran the command. That somehow got in there when I was writing the post. So that didn’t cause the error.

gabriel · January 27, 2023, 7:35pm

I copied your curl command straight in to the command line. When executing it, I also got {"error":"Internal Server Error","reason":"Internal error: EOF"}.

The issue was that there was some white space after the \ new line delimiters. When executing curl with -v I noticed that the content length was only 64 bytes. In VSCode I use the Trailing Space extension, which is generally helpful to avoid random whitespace, and I usually write command lines which I have trouble with into an executable .sh file.

After fixing the whitespace issue and running the command again, the reported content length is 151, and I get back status code 200.

PlutoHD · January 27, 2023, 7:53pm

Based on how I understand you, I changed the command to a single line command to prevent such issues:

curl --location-trusted --request PUT '*ADMIN REST API*/_config' --header 'Content-Type: application/json' -u 'admin:password' --data-raw '{"logging": {"console": {"log_level": "debug", "log_keys": ["*"]}}}'

but I still get the same error. I don’t think that this issue is about syntax in the API requests, because I used Postman before and I have not changed anything in Postman. For whatever reason, both requests using Postman and curl don’t work anymore. I am sure that I am using the correct credentials. The user has the permissions required. GET requests work without any problems (so the credentials work). I have no idea what could cause this issue as I have no further logs or any other information.

PlutoHD · January 29, 2023, 7:42pm

After a phone call with @gabriel, the problem is mostly solved. By executing the request with curl with -v we got more information. You could not have come up with the solution here in the forum, since the problem was that I had only used the domain, without https:// in front of it. This apparently worked at some point, but now the redirect made the request not work for some reason. After I added https:// the requests in curl and Postman worked.
However, we noticed two things that are documented incorrectly or do not work as expected:

The documentation of the PUT /_config endpoint shows in the example value that JSON is expected which is the logging part of the configuration. However, as it turned out, the JSON with the logging configuration must be nested inside logging. So instead of:

{
      "console": {
          "log_level": "debug",
          "log_keys": [
              "*"
          ]
      }
}

as it is stated in the documentation, you have to send the following to the endpoint:

{
    "logging": {
        "console": {
            "log_level": "debug",
            "log_keys": [
                "*"
            ]
        }
    }
}

While the logging configuration changes, for example if I set the log level higher, I get more logs, but the response from the GET /_config endpoint doesn’t change. After I changed the log_level to debug, the response still says info. I don’t think that this is intended.

However, this solves my problem. Thanks for the help!