PrefixQuery results null fields

imre.g.kovacs · November 15, 2018, 3:56pm

Dear All,

I’m creating a PrefixQuery. Everything seems to be fine with it, but the result (ISearchQueryResult) doesn’t includes the fields.

        PrefixQuery query = new PrefixQuery("test");
        SearchQuery searchQuery = new SearchQuery().Fields( "name", "status");

        if (filterDefinition.Projection != null)
        {
            if (filterDefinition.Projection.Paging != null)
            {
                ISearchParams parameter = new SearchParams()
                    //.Fields("name", status)
                    .Limit(filterDefinition.Projection.Paging.Limit)
                    .Skip(filterDefinition.Projection.Paging.Page);
                     
                searchQuery.SearchParams = parameter;
            }
        }

        searchQuery.Query = query;
        searchQuery.Index = "default_fti";

        ISearchQueryResult results = Bucket.Query(searchQuery);

Did I miss something?

Thanks in advance for any kind of help!

Imre

jmorris · November 15, 2018, 5:33pm

@imre.g.kovacs -

Are you sure the request returns the data? You can check in the CB management console or use a HTTP debugger like Fiddler. If they are being returned, I would suspect a bug in the client.

imre.g.kovacs · November 16, 2018, 10:38am

@jmorris
Yes, I’m sure. There are ids retrieved, but the Fields is null. Please see the attached screenshot.
Untitled

jmorris · November 16, 2018, 4:05pm

Hi @imre.g.kovacs -

That screenshot is the client view after the query has executed; what I was asking for was a snapshot of what is on the network between the client and the server - the HTTP response using something like Fiddler (or the console response). In the response, we could see what the server returns and if the parameters are being returned which allows to isolate between a client deserialization error or something on the index or server side.

If you can post the index definition and example data where the fields “name” and “status” have a prefix of “test”, I can test myself.

imre.g.kovacs · November 16, 2018, 5:33pm

Hi Jeffry,

I debug the application on localhost, so the server and the client is on the same machine right now. I tried to setup the fiddler, but it seems doesnt capture localhost requests (only if I customize the ruleset). I’m going to try a workaround and get back to you with the results.

To answer your question, I generated example documents. A document has 11 fields where 10 fields are “general” fields (like int, string, datettime). The 11th field can be huge: a json with ~5000-8000 key/value pairs, where a value has 3 key-value pairs. The number of the documents should be around 200K.
Because they are generated documents most of my field values are very simple: “fieldname_counter” (so in case of the name field name_1, name_2, …). The status field can have only 5 different values (“planned”, “started”, …).

In my initial post I specified a “test” string to simplify my question.

My index definition is:
{
“type”: “fulltext-index”,
“name”: “default_fti”,
“uuid”: “34d94de493dbcc6d”,
“sourceType”: “couchbase”,
“sourceName”: “mobile”,
“sourceUUID”: “27bd2e05b7941694f409a54203f8e11e”,
“planParams”: {
“maxPartitionsPerPIndex”: 171
},
“params”: {
“doc_config”: {
“docid_prefix_delim”: “”,
“docid_regexp”: “”,
“mode”: “type_field”,
“type_field”: “type”
},
“mapping”: {
“analysis”: {},
“default_analyzer”: “en”,
“default_datetime_parser”: “dateTimeOptional”,
“default_field”: “_all”,
“default_mapping”: {
“dynamic”: true,
“enabled”: true
},
“default_type”: “_default”,
“docvalues_dynamic”: true,
“index_dynamic”: true,
“store_dynamic”: false,
“type_field”: “_type”
},
“store”: {
“indexType”: “scorch”,
“kvStoreName”: “”
}
},
“sourceParams”: {}
}

Hope you can help me based on my description above.

Thanks in advance,
Imre

jmorris · November 16, 2018, 5:47pm

@imre.g.kovacs -

Thanks! Last question(s), what SDK version and Couchbase Server version? Once I have that, I’ll try to reproduce.

jmorris · November 16, 2018, 8:48pm

@imre.g.kovacs -

You need to add child field to your index to retrieve specific fields, otherwise only the document Id is returned:

Hover over the default Type Mappings and there is a “+” which opens a menu and select “Insert child field” and select the “store” checkbox. This SO has some info as well: Full Text Search - Returning Document Content

-Jeff

imre.g.kovacs · November 19, 2018, 11:58am

Thank you Jeffry, it was worked. I really missed them to include into the index.
However one field is still null, the “survey” one. It’s the huge json string. I’ve included it also in the index. Please find my index definition below:
{
“name”: “default_fti”,
“type”: “fulltext-index”,
“params”: {
“doc_config”: {
“docid_prefix_delim”: “”,
“docid_regexp”: “”,
“mode”: “type_field”,
“type_field”: “type”
},
“mapping”: {
“default_analyzer”: “en”,
“default_datetime_parser”: “dateTimeOptional”,
“default_field”: “_all”,
“default_mapping”: {
“dynamic”: true,
“enabled”: true,
“properties”: {
“asp”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “asp”,
“store”: true,
“type”: “text”
}
]
},
“category”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “category”,
“store”: true,
“type”: “text”
}
]
},
“customerSiteId”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “customerSiteId”,
“store”: true,
“type”: “text”
}
]
},
“date”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “date”,
“store”: true,
“type”: “datetime”
}
]
},
“market”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “market”,
“store”: true,
“type”: “text”
}
]
},
“name”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “name”,
“store”: true,
“type”: “text”
}
]
},
“projectName”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “projectName”,
“store”: true,
“type”: “text”
}
]
},
“siteName”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “siteName”,
“store”: true,
“type”: “text”
}
]
},
“state”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
“include_in_all”: true,
“include_term_vectors”: true,
“index”: true,
“name”: “state”,
“store”: true,
“type”: “text”
}
]
},
“survey”: {
“enabled”: true,
“dynamic”: false,
“fields”: [
{
"include_in_all": true,
** “include_term_vectors”: true,**
** “index”: true,**
** “name”: “survey”,**
** “store”: true,**
** “type”: “text”**
}
]
}
}
},
“default_type”: “_default”,
“docvalues_dynamic”: true,
“index_dynamic”: true,
“store_dynamic”: false,
“type_field”: “_type”
},
“store”: {
“indexType”: “scorch”,
“kvStoreName”: “”
}
},
“sourceType”: “couchbase”,
“sourceName”: “mobile”,
“sourceUUID”: “27bd2e05b7941694f409a54203f8e11e”,
“sourceParams”: {},
“planParams”: {
“maxPartitionsPerPIndex”: 171,
“numReplicas”: 0
},
“uuid”: “4220046bcc85d4de”
}

What is the recommendation for indexing? If I want to retrieve only some fields I need to include them into the SearchParams and use this index above, or shall I create a new index for this, that includes only these fields?

Thanks again!

Best regards,
Imre

jmorris · November 19, 2018, 5:53pm

It should be there if “store” is true…I am not 100% sure, but there may be size issue if the field is huge. One approach is to simple return the Id and then use the K/V API to directly fetch the document.

imre.g.kovacs · November 21, 2018, 3:19pm

Yes, it seems this is the only way to retrieve it at this moment. Thanks for your help Jeffry!