Perform query: panic: runtime error: slice bounds out of range


#1

Hello everyone.
we are using Couchbase Community Edition 6.0.0 build 1693 in 2 node configuration. After load 1.8 mio documents we are observing strange behaviour, when running query (doesn’t matter if from our application connected to CB or from CB web console):
Query from time-to-time ends with:
{
“status”: “Query stopped on server.”
}"
“msg”:Panic: runtime error: slice bounds out of range",
“code”:5001

We found on CB logs following stacktrace:

stack:
goroutine 204247260 [running]:
github.com/couchbase/query/execution.(*Context).Recover(0xc433c8e420)
        goproj/src/github.com/couchbase/query/execution/context.go:505 +0xbc
panic(0xe22d80, 0x1850150)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/panic.go:489 +0x2cf
github.com/couchbase/query/datastore/couchbase.doFetch(0xc4282bb9c0, 0x14, 0xc48ba35560, 0xed388bf3e, 0xc4087ae0a8)
        goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1063 +0x554
github.com/couchbase/query/datastore/couchbase.(*keyspace).Fetch(0xc4211fbea0, 0xc45322a800, 0x1, 0x40, 0xc44935f920, 0x187d1c0, 0xc433c8e420, 0x0, 0x0, 0x0, ...)
        goproj/src/github.com/couchbase/query/datastore/couchbase/couchbase.go:1037 +0x779
github.com/couchbase/query/execution.(*Fetch).flushBatch(0xc424cee5a0, 0xc433c8e420, 0x100000000aefc00)
        goproj/src/github.com/couchbase/query/execution/fetch.go:115 +0x533
github.com/couchbase/query/execution.(*Fetch).afterItems(0xc424cee5a0, 0xc433c8e420)
        goproj/src/github.com/couchbase/query/execution/fetch.go:72 +0x35
github.com/couchbase/query/execution.(*base).runConsumer.func1()
        goproj/src/github.com/couchbase/query/execution/base.go:551 +0x296
github.com/couchbase/query/util.(*Once).Do(0xc424cee698, 0xc467061738)
        goproj/src/github.com/couchbase/query/util/sync.go:51 +0x68
github.com/couchbase/query/execution.(*base).runConsumer(0xc424cee5a0, 0x187aa20, 0xc424cee5a0, 0xc433c8e420, 0x1899780, 0xc421ebfda0)
        goproj/src/github.com/couchbase/query/execution/base.go:552 +0xaf
github.com/couchbase/query/execution.(*Fetch).RunOnce(0xc424cee5a0, 0xc433c8e420, 0x1899780, 0xc421ebfda0)
        goproj/src/github.com/couchbase/query/execution/fetch.go:56 +0x5c
created by github.com/couchbase/query/execution.(*Sequence).RunOnce.func1
        goproj/src/github.com/couchbase/query/execution/sequence.go:95 +0x404

Could anyone help what is wrong? We already tried to remove data and drop indexes, but after load same data and create same indexes, issue happened again.

Thanks in advance
Stefan


#2

Could you please post the query that caused the panic.
Also check query.log little above panic there might be query text that caused panic.


#3

Hello vsr1,
taks for quick response. This is query:
SELECT
{a_b_c.name, a_b_c.entityUniqueId, a_b_c.entityType, a_b_c.entityDefinitionVersion} as dataa_b_c,
{a_b.entityUniqueId, a_b.entityType, a_b.entityDefinitionVersion} as dataa_b,
{a.entityUniqueId, a.entityType, a.entityDefinitionVersion} as dataa,
{‘entityUniqueId’ : a.entityUniqueId} as gdsCursorPagingSupport
FROM documents a
LEFT JOIN documents a_b ON tostring(a.b.cbsID) = a_b.entityUniqueId AND a_b.entityType = ‘Account’
LEFT JOIN documents a_b_c ON tostring(a_b.c.externalClientID) = a_b_c.entityUniqueId AND a_b_c.entityType = ‘Client’
WHERE a.entityType=“a”
ORDER BY a.entityUniqueId ASC
LIMIT (10)
OFFSET (0)

I changed it little bit (obfuscating) - when we try to help ourself, it looks like problem with joins. We had them in the past but we used index joins - after migrating to CB we rewrited them to ANSI joins.

And here is requested part of the log before stack:
I can’t past part of log here - it exceeded max lenth of comment and I can’t upload it as file because “new users can’t upload attachments” :slight_smile:

Hope it helps.

Thanks
Stefan


#4

try to past at least part:
2018-11-22T15:17:42.033+01:00 [ERROR] Invalid opaque Value. Debug info : Res.opaque : 5542(4294964026), Keys 1, Response received MCResponse status=SUCCESS, opcode=GET, opaque=5542, msg: {“senderAccount”:“PK08zANW6358819430538993”,“clientAccount”:{“entityUniqueId”:“77287”,“entityType”:“Account”,“cbsID”:“77287”},“clientBankCode”:“Yarely”,“chargeDetail”:{“entityUniqueId”:“SwiftCharge_8”,“entityType”:“SwiftCharge”,“code”:“SwiftCharge_8”},“orderNumber”:897510158,“partnerAccountBankCode”:“Unsought”,“documentNumber”:829218163,“operationCode”:“INERRANT”,“channel”:{“channel”:“Channel_1”,“entityUniqueId”:“Channel_1”,“entityType”:“Channel”},“creditExtraInformation”:“UNPOLICED”,“postingDate”:“1992-09-14T22:00:00.000Z”,“merchantName”:“Richie Maybin”,“cbsMovementID”:“Movement_10038”,“senderName”:“August Kalla”,“sendarBankName”:“Nubecula satanic neologised”,“pin”:“Blushingly”,“movementType”:{“entityUniqueId”:“AccountMovementGroup_7”,“entityType”:“AccountMovementGroup”,“groupID”:“AccountMovementGroup_7”},“entityDefinitionVersion”:“1.3”,“directionDRCR”:0,“additionalInformation”:“Nukes moistly afresh”,“merchantCode”:{“merchant”:“MerchantCode_18”,“entityUniqueId”:“MerchantCode_18”,“entityType”:“MerchantCode”},“IDDocumentDateofIssue”:“2002-01-15”,“entityType”:“Movement”,“IDDocumentValidTo”:“1999-09-30”,“transactionCode”:{“entityUniqueId”:“PFMTransactionCode_28”,“entityType”:“PFMTransactionCode”,“pfmTransactionCode”:“PFMTransactionCode_28”},“counterAmount”:0.6737551732031611,“IDDocumentType”:{“documentTypeId”:“DocumentType_12”,“entityUniqueId”:“DocumentType_12”,“entityType”:“DocumentType”},“mechantLocation”:“Quito”,“debitExtraInformation”:“This recomposes taler”,“subcategory”:{“pfmSubCategory”:“PFMSubCategory_43”,“entityUniqueId”:“PFMSubCategory_43”,“entityType”:“PFMSubCategory”},“orderDate”:“2018-01-27”,“status”:21,“documentDate”:“2017-01-04T23:00:00.000Z”,“partnerAccountName”:“Hearth hinder bye”,“rateAmount”:0.9681888383213978,“payerId”:“UNCOUTH”,“receiverBankName”:“Brokers revetted undrawn”,“description”:“Weaponless abrades thaws”,“entityUniqueId”:“Movement_10038”,“batchId”:“BATCH_10038”,“receiverAccount”:“PS64fpbc605666125202601735306”,“receiverTaxPayerId”:“MANDATORY”,“IDDocumentNumber”:“DN106538871374274870”,“counterCurrency”:{“currencyID”:“Currency_1”,“entityUniqueId”:“Currency_1”,“entityType”:“Currency”},“client”:{“entityUniqueId”:“Client_7888”,“entityType”:“Client”,“externalClientID”:“Client_7888”},“payerName”:“Janeth Naugle”,“chargeAmount”:0.3997522538652063,“currency”:{“currencyID”:“Currency_15”,“entityUniqueId”:“Currency_15”,“entityType”:“Currency”},“senderTaxPayerId”:“BASCULE”,“receiverBankCode”:“RUNAGATES”,“amount”:0.18985221890236492,“partnerAccountIBAN”:“MK50154mlpYXJ5mTt85”,“partnerName”:“Kellie Mudget”,“receiverName”:“August Lenior”,“IDDocumentIssuedBy”:“Carunculate”,“clientBankName”:“Powerfully comfortably abandon”,“senderBankCode”:“BENEATH”,“valueDate”:“1972-02-04T23:00:00.000Z”,“rateIsReverse”:true,“balanceAfterMovement”:0.26408398057737126,“transactionType”:{“transTypeID”:“AccountMovementType_37”,“entityUniqueId”:“AccountMovementType_37”,“entityType”:“AccountMovementType”},“geminiPaymentID”:“ub8XwAHzrSsCITsVBiglxh5l3r3YmdBUK5nbV1nPXxqN1cuGEVd0oC7kcOa1Prtz”,“IDDocumentCountry”:{“entityUniqueId”:“Country_26”,“entityType”:“Country”,“countryID”:“Country_26”},“merchantCategoryCode”:“ADVERTIZED”,“partnerAccountBankName”:“Greyly unbraces foul”}
key list [Account::100493] this key {“senderAccount”:“PK08zANW6358819430538993”,“clientAccount”:{“entityUniqueId”:“77287”,“entityType”:“Account”,“cbsID”:“77287”},“clientBankCode”:“Yarely”,“chargeDetail”:{“entityUniqueId”:“SwiftCharge_8”,“entityType”:“SwiftCharge”,“code”:“SwiftCharge_8”},“orderNumber”:897510158,“partnerAccountBankCode”:“Unsought”,“documentNumber”:829218163,“operationCode”:“INERRANT”,“channel”:{“channel”:“Channel_1”,“entityUniqueId”:“Channel_1”,“entityType”:“Channel”},“creditExtraInformation”:“UNPOLICED”,“postingDate”:“1992-09-14T22:00:00.000Z”,“merchantName”:“Richie Maybin”,“cbsMovementID”:“Movement_10038”,“senderName”:“August Kalla”,“sendarBankName”:“Nubecula satanic neologised”,“pin”:“Blushingly”,“movementType”:{“entityUniqueId”:“AccountMovementGroup_7”,“entityType”:“AccountMovementGroup”,“groupID”:“AccountMovementGroup_7”},“entityDefinitionVersion”:“1.3”,“directionDRCR”:0,“additionalInformation”:“Nukes moistly afresh”,“merchantCode”:{“merchant”:“MerchantCode_18”,“entityUniqueId”:“MerchantCode_18”,“entityType”:“MerchantCode”},“IDDocumentDateofIssue”:“2002-01-15”,“entityType”:“Movement”,“IDDocumentValidTo”:“1999-09-30”,“transactionCode”:{“entityUniqueId”:“PFMTransactionCode_28”,“entityType”:“PFMTransactionCode”,“pfmTransactionCode”:“PFMTransactionCode_28”},“counterAmount”:0.6737551732031611,“IDDocumentType”:{“documentTypeId”:“DocumentType_12”,“entityUniqueId”:“DocumentType_12”,“entityType”:“DocumentType”},“mechantLocation”:“Quito”,“debitExtraInformation”:“This recomposes taler”,“subcategory”:{“pfmSubCategory”:“PFMSubCategory_43”,“entityUniqueId”:“PFMSubCategory_43”,“entityType”:“PFMSubCategory”},“orderDate”:“2018-01-27”,“status”:21,“documentDate”:“2017-01-04T23:00:00.000Z”,“partnerAccountName”:“Hearth hinder bye”,“rateAmount”:0.9681888383213978,“payerId”:“UNCOUTH”,“receiverBankName”:“Brokers revetted undrawn”,“description”:“Weaponless abrades thaws”,“entityUniqueId”:“Movement_10038”,“batchId”:“BATCH_10038”,“receiverAccount”:“PS64fpbc605666125202601735306”,“receiverTaxPayerId”:“MANDATORY”,“IDDocumentNumber”:“DN106538871374274870”,“counterCurrency”:{“currencyID”:“Currency_1”,“entityUniqueId”:“Currency_1”,“entityType”:“Currency”},“client”:{“entityUniqueId”:“Client_7888”,“entityType”:“Client”,“externalClientID”:“Client_7888”},“payerName”:“Janeth Naugle”,“chargeAmount”:0.3997522538652063,“currency”:{“currencyID”:“Currency_15”,“entityUniqueId”:“Currency_15”,“entityType”:“Currency”},“senderTaxPayerId”:“BASCULE”,“receiverBankCode”:“RUNAGATES”,“amount”:0.18985221890236492,“partnerAccountIBAN”:“MK50154mlpYXJ5mTt85”,“partnerName”:“Kellie Mudget”,“receiverName”:“August Lenior”,“IDDocumentIssuedBy”:“Carunculate”,“clientBankName”:“Powerfully comfortably abandon”,“senderBankCode”:“BENEATH”,“valueDate”:“1972-02-04T23:00:00.000Z”,“rateIsReverse”:true,“balanceAfterMovement”:0.26408398057737126,“transactionType”:{“transTypeID”:“AccountMovementType_37”,“entityUniqueId”:“AccountMovementType_37”,“entityType”:“AccountMovementType”},“geminiPaymentID”:“ub8XwAHzrSsCITsVBiglxh5l3r3YmdBUK5nbV1nPXxqN1cuGEVd0oC7kcOa1Prtz”,“IDDocumentCountry”:{“entityUniqueId”:“Country_26”,“entityType”:“Country”,“countryID”:“Country_26”},“merchantCategoryCode”:“ADVERTIZED”,“partnerAccountBankName”:“Greyly unbraces foul”}
_time=2018-11-22T15:17:42.068+01:00 _level=SEVERE _msg=panic: runtime error: slice bounds out of range

    request text:
    <ud>SELECT 
     {`Account_client_adviser`.`firstName`, `Account_client_adviser`.`entityUniqueId`, `Account_client_adviser`.`entityType`, `Account_client_adviser`.`entityDefinitionVersion`} as `dataAccount_client_adviser`, 
    {`Account_client`.`entityUniqueId`, `Account_client`.`entityType`, `Account_client`.`entityDefinitionVersion`} as `dataAccount_client`, 
    {`Account`.`cbsID`, `Account`.`entityUniqueId`, `Account`.`entityType`, `Account`.`entityDefinitionVersion`} as `dataAccount`, 
    {'cbsID' : `Account`.`cbsID`, 'entityUniqueId' : `Account`.`entityUniqueId`} as `gdsCursorPagingSupport` 
    FROM  `documents` `Account`  
    LEFT JOIN `documents` `Account_client` ON tostring(`Account`.`client`.`externalClientID`) = `Account_client`.`entityUniqueId` AND `Account_client`.`entityType` = 'Client'
     AND /* begin authorization filter */ (true=false OR ((`Account_client`.`entityType` = 'Client'))) /* end authorization filter */ 
    LEFT JOIN `documents` `Account_client_adviser` ON tostring(`Account_client`.`adviser`.`advisorID`) = `Account_client_adviser`.`entityUniqueId` AND `Account_client_adviser`.`entityType` = 'ClientRelationManager'
     AND /* begin authorization filter */ (true=false) /* end authorization filter */ 
    WHERE `Account`.`entityType`=$entityType
     AND /* begin authorization filter */ (true=false OR ((`Account`.`entityType` = 'Account'))) /* end authorization filter */
     AND /* begin regular filter */ `Account`.`cbsID` IN [1,2] OR (`Account`.`cbsID` = 5 OR `Account`.`cbsID` <= 3) AND `Account`.`cbsID` <= 4 /* end regular filter */ 
    ORDER BY `Account`.`cbsID` ASC, `Account`.`entityUniqueId` ASC 
    LIMIT (1) 
    OFFSET (1)</ud>

stack:

#5

Small update,
I rewrite select to use index joins and:

  1. it is faster - ANSI 40ms - INDEX - 30ms
  2. it works like charm - without problem.

So it looks like problem with ANSI joins. But we need them. Can I anyhow help You ?

Stefan


#6

I don’t think this related to ANSI JOIN. So happened to hit in that path. It looks like there is outbound data. It should have retried instead it return error nil and causes panic.

Opened MB-32118 . If this happens rarely you can kill cbq-engine process and it will automatically restart

If you need ANSI JOIN did you tried 5.50 ?


#7

As far as I know, there is no Community Edition v5.5 - only 5.1.1 (without ANSI joins) and then 6.0.0 (with ANSI joins).
Or is it somewhere hidden - not available for public ?


#8

May be you are right Only Major Community Edition released same time as Enterprise, others will released later. cc @raju, @keshav_m

One option u can try is query parameter pipeline_batch to 1 for that query. response time might be longer.
https://docs.couchbase.com/server/6.0/settings/query-settings.html


#9

Should I set it for every query or only for this particular?
At first sight it looks like performance issue:

pipeline-batch int 16	
[Optional] Controls the number of items execution operators can batch for Fetch from the KV.

isn’t it?


#10

try the query that caused. I have mentioned performance earlier post


#11

When I set pipeline_batch to 1 - query stops working at all (all requests ends with panic). It is unusable. Any other suggestions?


#12

I don’t have any suggestions. Unless we are able to repro it is difficult pinpoint root cause.