[12008] Error performing bulk get operation - cause: unable to complete action after 6 attemps

Hi,

We are encountering on our production cluster from time to time the below error when executing N1QL query:
“[12008] Error performing bulk get operation - cause: unable to complete action after 6 attemps”

  • We have no idea why it happens. We found no errors on the Couchbase logs or any indications for issues.
  • When executing it in the Couchbase UI Query editor we validated that it’s using an index . Also, the query take 3.02ms.

Here are a some details on our configuration:

  • Couchbase version: 6.0.2
  • We have 3 nodes
  • Minimum replica is 2
  • 6 cpu
  • 60GB RAM. 20 % reserved for operation system

Here is an example to one of the codes which executing N1QL query that gets the error, but again, it randomly happens in other N1ql queries too and only from time to time.
(I changed the method variables/names so it’s a pseudo)

func GetActivities(userId string) (*entity.Activity, error) {
q := gocb.NewN1qlQuery(SELECT * FROM users WHERE type = $1 AND userId = $2 ORDER BY createdAt DESC)

rows, err := bucket.ExecuteN1qlQuery(q, []interface{}{entity.DocTypeActivity, userId})
if err != nil {
	return nil, err
}

var activities []entity.Activity
    var activity = entity.Activity

for rows.Next(&activity) {
	activity = data["users"]
	activities = activities.append(activities, activity)
}
err = rows.Close()
if err != nil {
	return nil, err
}

return &res, nil

}

I found in the forum other posts that related to this error but no solution other than “try to upgrade the Couchbase”.

What is the index definition. Can you change type value as static

SELECT * FROM users WHERE type = "xyz" AND userId = $1 ORDER BY createdAt DESC
CREATE INDEX ix1 ON ( userId, createdAt DESC) type = "xyz";
If you can't
SELECT * FROM users WHERE type = $1 AND userId = $2 ORDER BY createdAt DESC
CREATE INDEX ix1 ON ( type, userId, createdAt DESC) ;

The index is defined with static type.
Anyway I don’t think it’s related to the index, the query is using the index and works really fast.

The error says trying to get the specific keys and not able to complete after 6 attempts. Check query.log any other errors around that time.
Node down or network issue or overloaded node.

Found the request log when when query the system:completed_requests.
I see that the query completed, but with error. I still unable to understand the error and the cause.
(I changed some values such remoteAddr, etc… in terms of security)

{
“completed_requests”: {
“elapsedTime”: “1.516068724s”,
“errorCount”: 1,
“node”: “xxxx:xxxx”,
“phaseCounts”: {
“fetch”: 1,
“indexScan”: 1
},
“phaseOperators”: {
“authorize”: 1,
“fetch”: 1,
“indexScan”: 1,
“sort”: 1
},
“remoteAddr”: “xxxx:xxxx”,
“requestId”: “5776b0d1-268b-44d3-996e-ef4699a669cf”,
“requestTime”: “2019-09-09 08:44:41.250762849 +0000 UTC”,
“resultCount”: 0,
“resultSize”: 0,
“scanConsistency”: “unbounded”,
“serviceTime”: “1.515905275s”,
“state”: “completed”,
“statement”: “\n\t\tSELECT * FROM users WHERE type = $1 AND userId = $2 ORDER BY createdAt DESC\n\t”,
“userAgent”: “Go-http-client/1.1”,
“users”: “xxxxxxxxxxx”
}

Could not get the reason for error. Restart couchbase on that node where it has issue and see if the problem persist.

I am facing similar issue in my Enterprise tool. Mostly, the query works in <10ms with expected results. Intermittently, the query throws the error. Attaching two screenshots for reference.


Corresponding log from Couchbase:

Service 'query' exited with status 2. Restarting. Messages:
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_index3.go:131 +0x4d3
github.com/couchbase/query/util.(*Once).Do(0xc00222b788, 0xc001903f88)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/util/sync.go:54 +0x4a
github.com/couchbase/query/execution.(*IndexScan3).RunOnce(0xc00222b680, 0xc0058d8f00, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/scan_index3.go:52 +0x82
created by github.com/couchbase/query/execution.(*base).runConsumer.func1
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/query/execution/base.go:708 +0x2d6

goroutine 1098577 [select]:
net/http.(*persistConn).writeLoop(0xc002c50fc0)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.4/go/src/net/http/transport.go:1339 +0x966