Index problem with flush + select ... [request_plus | not_bounded]

Hello!

Keep in mind Loading...
UPDATE: “Cannot Reproduce” for MB-16957 should be revised now, i think
@jliang , @prataprc for your attention :wink:

  1. Cluster 4.1.0-EE (4CPU, 4Gb) x 3
  2. Initialize a cluster,
  3. Create default bucket (1024M, no password, value ejection, 1 replica, view index replicas, high priority, flush enable)
  4. Edit attached script (inside .zip) with host, password and bucket name and run it (sorry, this is not python, it’s bash, but at least it’s not a Brainfuck - Wikipedia, so, i think, any python-user can handle with it ;).

Or write your own:

  1. Create 3 indexes (identical: “create index i[1…3] on default(a,b) where a is valued”)
  2. Insert 1 doc
  3. Flush
  4. Insert 3 docs
  5. Flush again
  6. Make a NB-select
  7. Make a RP-select and enjoy 2-min hang

I also attached logs for 3 of my runs (inside .zip); one listed below (end of this post)

Problem shown : “Select with request_plus” hangs up to 2 minutes (time to remember about MB-16957)

UPDATE 2: “select” with “not_bounded” is not affected

And, if there is a bug (or MB-16957 going to be revised deeper), please, consider about including 4.1.X branch to “Fix version(s)” :slight_smile:

UPDATE 3: use “open in new tab” to download: https://www.couchbase.com/forums/uploads/short-url/oCJpycpKypsjBAq04bb1IE95YTB.zip

One on my runs:

  1. Dropping indexes
    {
    “requestID”: “a5626155-282b-45d2-80c5-45e159d62f60”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “271.619938ms”,
    “executionTime”: “271.542842ms”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
    {
    “requestID”: “9ecd08d5-456a-4483-8aca-a45bd4b6e6b9”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “268.067829ms”,
    “executionTime”: “268.000893ms”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
    {
    “requestID”: “dac2aa7f-980e-4d58-8015-ce4f05288400”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “302.418355ms”,
    “executionTime”: “302.352748ms”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
  2. Creating indexes
    {
    “requestID”: “fb58ea5a-6ac2-4d5c-8bde-0c625dd46cd2”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.646142048s”,
    “executionTime”: “2.646032522s”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
    {
    “requestID”: “51ff9cb4-c87e-47a2-8983-86bd2b95e91f”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.637722272s”,
    “executionTime”: “2.637654906s”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
    {
    “requestID”: “21442ac9-2d06-4f81-a4e0-d033c68986b4”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.709900182s”,
    “executionTime”: “2.709830446s”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
  3. Flushing-1 (this may take a while)
  1. Creating 1 doc
    {
    “requestID”: “f4ef731d-169c-43e8-88d9-47a39e56d56f”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.915995ms”,
    “executionTime”: “2.827083ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }
  2. Flushing-2 (this may take a while)
  1. Now there should be no docs in default
    {
    “requestID”: “31f8fda1-ce47-478e-95b4-7d48ff524aa9”,
    “signature”: {
    “a”: “json”,
    “b”: “json”,
    “id”: “json”
    },
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “6.101749ms”,
    “executionTime”: “5.952055ms”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
  2. Creating 3 docs
    {
    “requestID”: “d6253c87-3066-4c65-a00b-8bd689b229ed”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.531974ms”,
    “executionTime”: “2.46499ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }
    {
    “requestID”: “8794db63-fc83-4e39-aa12-d61dcf3fb102”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “1.653396ms”,
    “executionTime”: “1.590182ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }
    {
    “requestID”: “10878a28-b233-4f29-be97-22c30a468ea6”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.246653ms”,
    “executionTime”: “2.20457ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }
  3. NB-query result:
    {
    “requestID”: “eaf8c782-f424-4417-aefd-6a353d826f05”,
    “signature”: {
    “$1”: “number”
    },
    “results”: [
    {
    “$1”: 0
    }
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “10.267603ms”,
    “executionTime”: “10.206712ms”,
    “resultCount”: 1,
    “resultSize”: 31
    }
    }
  4. RP-query result:
    {
    “requestID”: “8d99ba60-61e1-4d09-9519-2d64858b771c”,
    “signature”: {
    “$1”: “number”
    },
    “results”: [
    {
    “$1”: 3
    }
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2m0.290964014s”,
    “executionTime”: “2m0.290880718s”,
    “resultCount”: 1,
    “resultSize”: 31
    }
    }
  5. Now there should be 3 docs in default
    {
    “requestID”: “482809b2-404b-415c-b3a0-04b5ff6bd1d6”,
    “signature”: {
    “a”: “json”,
    “b”: “json”,
    “id”: “json”
    },
    “results”: [
    {
    “a”: “Hello”,
    “b”: “World!”,
    “id”: “docA”
    },
    {
    “a”: “Hello”,
    “b”: “World!”,
    “id”: “docB”
    },
    {
    “a”: “Hello”,
    “b”: “World!”,
    “id”: “docC”
    }
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “7.382695ms”,
    “executionTime”: “7.277643ms”,
    “resultCount”: 3,
    “resultSize”: 267
    }
    }

reuploaded: n1ql-query-index-hangs.zip (4.4 KB), but still unavailable even for me (it’s strange; if needed, i can send it via e-mail)

I guess you will have to send it via email. Thanks,

@prataprc,
could you please write me you e-mail via personal message ?

n1ql-query-index-hangs.zip (4.4 KB)

:frowning: it’s useless to try to publish, i think

Index problem with flush + select ... [request_plus | not_bounded]n1ql-query-index-hangs.zip

n1ql-query-index-hangs.zip

it’s useless to try to publish .zip :frowning:

@prataprc,
oh, i made it at last [but use “open in new tab”, then download begins]

https://www.couchbase.com/forums/uploads/short-url/oCJpycpKypsjBAq04bb1IE95YTB.zip

@prataprc,
Are there any news ?

Looks like it is passing for me. Please note that I am using
Platform: ubuntu-14.04
Couchbase: couchbase-server-enterprise_4.1.0-5008-ubuntu14.04_amd64.deb
single node: data,query,index enabled

  1. Dropping indexes
    {
    “requestID”: “7e6e7d3a-a0b8-4b4c-881e-a34b4c54b0b7”,
    “errors”: [
    {
    “code”: 5000,
    “msg”: “GSI index iQA not found.”
    }
    ],
    “status”: “fatal”,
    “metrics”: {
    “elapsedTime”: “28.661593ms”,
    “executionTime”: “28.181516ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “errorCount”: 1
    }
    }
    {
    “requestID”: “a58c4998-953a-4afc-977b-89c419672f63”,
    “errors”: [
    {
    “code”: 5000,
    “msg”: “GSI index iQB not found.”
    }
    ],
    “status”: “fatal”,
    “metrics”: {
    “elapsedTime”: “501.322µs”,
    “executionTime”: “462.246µs”,
    “resultCount”: 0,
    “resultSize”: 0,
    “errorCount”: 1
    }
    }
    {
    “requestID”: “20f78ba1-9127-4737-bb5f-09eea97fe361”,
    “errors”: [
    {
    “code”: 5000,
    “msg”: “GSI index iQC not found.”
    }
    ],
    “status”: “fatal”,
    “metrics”: {
    “elapsedTime”: “500.613µs”,
    “executionTime”: “461.513µs”,
    “resultCount”: 0,
    “resultSize”: 0,
    “errorCount”: 1
    }
    }

  2. Creating indexes
    {
    “requestID”: “294de24a-9a5b-4e3d-83ea-295b9ad4fb72”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “1.644426338s”,
    “executionTime”: “1.64437922s”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
    {
    “requestID”: “3a8ec700-b532-43e7-bd02-07bb330034ff”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.673306789s”,
    “executionTime”: “2.673262784s”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }
    {
    “requestID”: “974abb4a-1915-4c4f-9e47-2c9be8525d89”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.637891129s”,
    “executionTime”: “2.63784578s”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }

  3. Flushing-1 (this may take a while)
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:–:-- 0:00:04 --:–:-- 0

  4. Creating 1 doc
    {
    “requestID”: “c271b1a0-996e-4774-8e2c-177bcdef818d”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.886553ms”,
    “executionTime”: “2.826766ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }

  5. Flushing-2 (this may take a while)
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:–:-- 0:00:05 --:–:-- 0

  6. Now there should be no docs in default
    {
    “requestID”: “5da51a8f-c98d-4ab6-87e6-de50603b102c”,
    “signature”: {
    “a”: “json”,
    “b”: “json”,
    “id”: “json”
    },
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “6.927649ms”,
    “executionTime”: “6.845871ms”,
    “resultCount”: 0,
    “resultSize”: 0
    }
    }

  7. Creating 3 docs
    {
    “requestID”: “bbe796a0-8285-4a7d-9876-7533fd52bc93”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “2.295937ms”,
    “executionTime”: “2.231976ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }
    {
    “requestID”: “65aa88b7-2360-47f3-8855-0d173540d788”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “1.031256ms”,
    “executionTime”: “984.32µs”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }
    {
    “requestID”: “cdad89e2-44d9-4a19-910b-f513a1dfa8bd”,
    “signature”: null,
    “results”: [
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “1.042539ms”,
    “executionTime”: “1.009712ms”,
    “resultCount”: 0,
    “resultSize”: 0,
    “mutationCount”: 1
    }
    }

  8. NB-query result:
    {
    “requestID”: “ca19c3c7-41e6-4fb8-b9ad-149ca5cdfdc4”,
    “signature”: {
    "$1": “number”
    },
    “results”: [
    {
    "$1": 0
    }
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “4.045408ms”,
    “executionTime”: “4.010225ms”,
    “resultCount”: 1,
    “resultSize”: 31
    }
    }

  9. RP-query result:
    {
    “requestID”: “e35e30a5-d66c-419a-af15-a76d434ef81b”,
    “signature”: {
    "$1": “number”
    },
    “results”: [
    {
    "$1": 3
    }
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “3.819558972s”,
    “executionTime”: “3.819513327s”,
    “resultCount”: 1,
    “resultSize”: 31
    }
    }

  10. Now there should be 3 docs in default
    {
    “requestID”: “1288060f-23fa-46b6-ab0b-2f488926c7a7”,
    “signature”: {
    “a”: “json”,
    “b”: “json”,
    “id”: “json”
    },
    “results”: [
    {
    “a”: “Hello”,
    “b”: “World!”,
    “id”: “docA”
    },
    {
    “a”: “Hello”,
    “b”: “World!”,
    “id”: “docB”
    },
    {
    “a”: “Hello”,
    “b”: “World!”,
    “id”: “docC”
    }
    ],
    “status”: “success”,
    “metrics”: {
    “elapsedTime”: “5.038419ms”,
    “executionTime”: “4.987359ms”,
    “resultCount”: 3,
    “resultSize”: 267
    }
    }

@prataprc,
i use .5005. I’ll try 5008

@prataprc
.5008, Ubuntu 14.04 amd64 [.deb md5 = 65154b436214bb45ae856468299258af]
I was needed 2 tries; first was successful, second failed:

9 . RP-query result:
{
“requestID”: “69da9ca2-c660-433c-a865-5db79f69ceed”,
“signature”: {
“$1”: “number”
},
“results”: [
{
“$1”: 4
}
],
“status”: “success”,
“metrics”: {
“elapsedTime”: “2m0.280813824s”,
“executionTime”: “2m0.280742215s”,
“resultCount”: 1,
“resultSize”: 31
}
}

Recommendations:

  1. Try more then once
  2. Use 3-nodes-cluster, not just one.

Tried 5 times with single node. All of them succeeded. In multi node what configuration should I use ? All services enabled in all nodes ? In which node do you want me to run the script ?

Thanks,

Okay with 3 nodes all services enabled on all nodes, it can reproduce. Thanks.

@prataprc,
For https://issues.couchbase.com/browse/MB-16957, could you please add 4.1.2 as “Fix version(s)”.
Unavailabilty of “flush without deep stuck” makes development really hard.

@prataprc,
spock, i.e. 2017 ? really great :frowning: seems like i will become grey and old before bugfix …