Panic: runtime error: invalid memory address or nil pointer dereference

Couchbase Server
,
1

Hello,
I have Couchbase Server 4.5 CE, i have a bucket with 20M records.
I have a 3 machine cluster with Data/Index/Query.
I have a simple index on the bucket, the index is on machine 1.
CREATE INDEX index ON bucket(action,field, field2) USING GSI WITH {“nodes”:“xxx.xxx.xxx.xx1:8091”};

When i do the query:
select * from bucket where action = 3 limit 1

On Machine 1 (where index is)
I get the panic runtime error.

if i do the query on machine 2 or 3 everything works fine.

This error start when i migrate from Couchbase 4.1 CE to 4.5 CE.
Any ideas why this is happening?
Can this be a Bug?

Thanks

2

Reinstalling Couchbase on the machine solve the panic error.
But now when i do the query some times i have results, some times i don’t.
Don’t know what can cause this behavior?

Any help?

Thanks

3

Hi @rafael.felix,

Could you provide more detail about the N1QL queries that you are having problems with?

4

Hello,
The index is:
CREATE INDEX index ON bucket(action,field,field2) USING GSI WITH {“nodes”:“xxx.xxx.xxx.xx1:8091”};
and the query is:
select * from bucket where action = 3 limit 1

Nothing complex, sometimes i get results, some times i don’t, this on one specific machine.
Like i said before, i reinstalled couchbase, but still have the problem.

5

@rafael.felix,

Could you try an EXPLAIN on that SELECT, and see what index is being used? Also, when you don’t get results, are you just getting an empty array, or are you getting an error?

6

Explain:
[
{
“plan”: {
"#operator": “Sequence”,
"~children": [
{
"#operator": “Sequence”,
"~children": [
{
"#operator": “IndexScan”,
“index”: “omnibeesevents_action_propertyid_createddate_3”,
“index_id”: “dc1854b6e47c81bb”,
“keyspace”: “OmnibeesEvents”,
“limit”: “1”,
“namespace”: “default”,
“spans”: [
{
“Range”: {
“High”: [
“successor(3)”
],
“Inclusion”: 1,
“Low”: [
“3”
]
}
}
],
“using”: “gsi”
},
{
"#operator": “Parallel”,
"~child": {
"#operator": “Sequence”,
"~children": [
{
"#operator": “Fetch”,
“keyspace”: “OmnibeesEvents”,
“namespace”: “default”
},
{
"#operator": “Filter”,
“condition”: “((OmnibeesEvents.action) = 3)”
},
{
"#operator": “InitialProject”,
“result_terms”: [
{
“expr”: “self”,
“star”: true
}
]
},
{
"#operator": “FinalProject”
}
]
}
}
]
},
{
"#operator": “Limit”,
“expr”: “1”
}
]
},
“text”: “select * from OmnibeesEvents where action = 3 limit 1”
}
]

Before reinstall sometimes i got the panic error, other times i get an empty array.
After reinstall i get empty array, no error.

7

@rafael.felix,

Could you try select b.* from bucket b where b.action = 3 limit 1 and see if that returns the records you expect? If not, I’m going to have to refer you over to the N1QL forums to talk to N1QL expert @geraldss.

8

hi,
To resolve the problem until i can replicate and find a solution in development environment, i disable query and index service on that machine.
When i add that services again i will test the query and get back to you.
Thanks for your help.

9

hello @matthew.groves we are also getting same problem in different situation. cbq engine is not starting automatically . trying to start cbq services using below cmd one of the node is throwing this error.

can i get any clue why it is behaving like this in your words.

[root@crdl-db-ha1-crdl-02 /]# /opt/couchbase/bin/cbq-engine --datastore=http://[::1]:8091 --http=:8093 --configstore=http://[::1]:8091 --enterprise=true --ipv6=true --https=:18093 --certfile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyfile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem
2023/02/26 02:56:29 cbauth: key: fts/remoteClients registered for tls config updates
2023/02/26 02:56:29 cfg_metakv: RunObserveChildren, err: Get /_metakv/fts/cbgt/cfg/?feed=continuous: Unable to initialize cbauth’s revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
_time=2023-02-26T02:56:29.789+09:00 _level=ERROR _msg=Unable to initialize cbauth. Error Unable to initialize cbauth’s revrpc: cbauth environment variable CBAUTH_REVRPC_URL is not set
_time=2023-02-26T02:56:29.789+09:00 _level=WARN _msg=Unable to parse url http://[::1]:8091: Unusable url http://[::1]:8091. No user information.
_time=2023-02-26T02:56:29.789+09:00 _level=WARN _msg=Unable to initialize cbAuth, access to couchbase buckets may be restricted
_time=2023-02-26T02:56:29.797+09:00 _level=ERROR _msg=Cannot connect url http://[::1]:8091 - cause: HTTP error 401 Unauthorized getting “http://[::1]:8091/pools”:
_time=2023-02-26T02:56:29.797+09:00 _level=ERROR _msg=Shutting down.
[root@crdl-db-ha1-crdl-02 /]# ps -ef|grep cbq
root 8198 7886 0 02:56 pts/0 00:00:00 grep --color=auto cbq
[root@crdl-db-ha1-crdl-02 /]#

for that i have export the variable

[root@crdl-db-ha1-crdl-02 /]# export CBAUTH_REVRPC_URL=“http://[20b:c0e:104:5450:9727:2:7:b02]:8091 -u Administrator -p abc123xyz”

after export the variable getting below error

[root@crdl-db-ha1-crdl-02 /]# export CBAUTH_REVRPC_URL="http://[240b:c0e0:104:5450:9727:2:7:b12]:8091 -u Administrator -p mAv@cb123 "
[root@crdl-db-ha1-crdl-02 /]# /opt/couchbase/bin/cbq-engine --datastore=http://[::1]:8091 --http=:8093 --configstore=http://[::1]:8091 --enterprise=true --ipv6=true --https=:18093 --certfile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyfile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem
2023/02/26 02:58:49 cbauth: key: fts/remoteClients registered for tls config updates
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xc99400]

goroutine 80 [running]:
github.com/couchbase/cbauth/metakv.doCallInner(0xc0001196d0, 0x1f7f37c, 0x3, 0x1f8c3ec, 0xe, 0xc000113bc8, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:94 +0xa7
github.com/couchbase/cbauth/metakv.doRunObserveChildren(0xc0001196d0, 0x1f8c3ec, 0xe, 0xc0001181d0, 0xc0005f3b60, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:273 +0x142
github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(0xc0001196d0, 0x1f8c3ec, 0xe, 0xc0001181d0, 0xc0005f3b60, 0x1, 0xc000118000)
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 +0x58
github.com/couchbase/cbauth/metakv.RunObserveChildren(0x1f8c3ec, 0xe, 0xc000118000, 0xc0005f3b60, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 +0x85
github.com/couchbase/cbgt.NewCfgMetaKv.func1(0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/cfg_metakv.go:149 +0x82
github.com/couchbase/cbgt.ExponentialBackoffLoop(0x1fa1e9d, 0x1d, 0xc000223960, 0xc8, 0x3fc00000, 0x1388)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/misc.go:130 +0x31
created by github.com/couchbase/cbgt.NewCfgMetaKv
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbgt/cfg_metakv.go:147 +0x2c1
[root@crdl-db-ha1-crdl-02 /]#

10

You can’t start cbq-engine manually

service couchbase-server stop
service couchbase-server start

or find process id kill -9 and it will automatically start

1 Like
11

If your cbq-engine process is not starting automatically after configuring your node to run the Query service and re-starting Couchbase using the OS service management, then please have a look in the query.log (in /opt/couchbase/var/lib/couchbase/logs) for clues as to why it isn’t starting, and post them here for us to assist with.

If you’re uncertain you’ve configured the Query service on a node, you could examine the output from:

curl -su Administrator:password http://localhost:8091/pools/default/nodeServices | jq

(the piping to jq is of course optional) (Ref: https://docs.couchbase.com/server/current/rest-api/rest-list-node-services.html)

1 Like