Not able to get filtered data on sync gateway from Couchbase server

Mobile Sync Gateway
,
#1

Hi there,

I am trying to filter data on sync gateway from couchbase server to get only specific records but am not able to do that. there are two entries on my server “apple” and “avocado” wherein I am trying to filter those with text as “apple” and show them on sync gateway. Please guide me with the same.

Below is my syncgateway configuration file.

“databases”: {
“grocery-sync”: {
“server”: “http://10.55.18.1:8091”,
“bucket”: “grocery-sync”,
“username”: “user”,
“password”: “password”,
“enable_shared_bucket_access”: true,
“import_docs”: true,
“num_index_replicas”: 0,
“users”: {
“GUEST”: {“disabled”: false, “admin_channels”: ["*"] }
},
“import_filter”: function(doc) { if (doc.text == "apple") { return } },
“allow_conflicts”: false,
“revs_limit”: 20
}
}
}

Also, if I add data to my device it is getting reflected on sync gateway and couchbase server but when I add or delete document on the couchbase server only the sync gateway pulls it, the device doesn’t pull it. Please help me .

Not able to connect Sync Gateway with Couchbase server
#2

You have two separate questions that are not related . Please create two separate posts. Also use the right tags and category so right teams can review (I’ve updated the categories for this question).

I am going to answer your first question here

What exactly do you mean by “filter data” and “show them on sync gateway”. Perhaps an example of your use case will help.

When you post your second question, pls. include details of your couchbase lite replicator code and logs.

#3

suppose there are 10 records on my couchbase server and from there I want to filter and fetch(by means of sync gateway channel or something. I am not sure how to attain that) only some specific records on my sync gateway then how should I proceed on doing that?
Kindly guide me.

#4

By “fetch” do you mean sync to Couchbase lite clients? And are you filtering per user ? So if you have 10 users, do you want user1 to get subset of docs and User 2 to get subset of docs and so on ).
Or did you want to filter the documents that are available for Mobile sync? ( as in you have a bucket of N documents and you want just a subset of it available for sync gateway clients )

#5

I have a bucket of N documents and want just a subset of it available for sync gateway as of now.

#6

so you only want subset of documents to be available to sync gateway clients - Then follow example outlined in setting up import filters . Would recommend reading up on the docs that describes import process and shared bucket access so you are familiar with the concepts

#7

The documentation says that “The filter function takes the document body as parameter and is expected to return a boolean to indicate whether the document should be imported.” Where will the boolean be returned? will it not return the filtered documents?

my syncgateway config file is as below

{
“adminInterface”: “127.0.0.1:4985”,
“interface”: “0.0.0.0:4984”,
“databases”: {
“grocery-sync”: {
“server”: “http://10.46.200.168:8091”,
“bucket”: “grocery-sync”,
“username”: “user”,
“password”: “password”,
“enable_shared_bucket_access”: true,
“import_docs”: true,
“num_index_replicas”: 0,
“import_filter”: function(doc) { if (doc.type != "apple") { return false } return true } ,
“users”: {
“GUEST”: {“disabled”: false, “admin_channels”: ["*"] }
},
“allow_conflicts”: false,
“revs_limit”: 20
}
}
}

Below are the 2 entries on my couchbase server. I want to filter and retrieve only “apple” doc on my device instead of both. How would I achieve that?

Screenshot (1)
Screenshot (1)1213×337 44 KB

#8

Your import filter (although convoluted) should work. Instead it’s probably easier to define - “If doc.type is apple then return true else return false”)

How are you verifying that the documents are not getting filtered ?

I suspect that your import filter was specified later -i.e. after the sync gateway was started, you probably changed the filter later and restarted Sync Gateway, But by then the documents in the bucket were already imported by the sync gateway so the filter had no impact

Would suggest the following

  • stopping sync gateway,
  • deleting bucket on server
  • recreating documents in server bucket and
  • then restarting sync gateway with the import filter
    Then post results of what you get when you query for all documents via REST API. Only imported documents will show up in list.

That said, I am still unclear of your use case…you indicate that

You can do the above via channels . Put apple docs in apple channels and so on and have client side replicator filter on apple channels.

The import filter filters documents from being available to any of the mobile clients - I am unsure on whether thats what you wanted when you say "available for sync gateway "

.

#9

I tried your solution. I am able to get filtered documents (documents of type “apple”) on my sync gateway but still they are not getting pulled on my mobile client (on couchbase lite). I am not sure why is that happening.

#10

How are you checking that ?

Check your couchbase lite logs and ensure it is able to actually get to the sync gateway and pull any data at all. What does the replicator config look like on lite side. .
Also, look at sync gateway logs and see if any replication errors reported.

#11

I am getting this error in the logs

7:45:44.535+05:30 [ERR] gocb: memdClient read failure: read tcp 10.46.202.45:64142->10.46.202.45:11210: wsarecv: An established connection was aborted by the software in your host machine. – gocbcore%!e(MISSING)v7.logExf() at logging.go:144
2020-12-07T17:45:44.535+05:30 [ERR] gocb: memdClient read failure: read tcp 10.46.202.45:64141->10.46.202.45:11210: wsarecv: An established connection was aborted by the software in your host machine. – gocbcore%!e(MISSING)v7.logExf() at logging.go:144
2020-12-08T11:14:06.215+05:30 [ERR] c:#104 #104: Post http://10.46.202.45:8093/query/service: dial tcp 10.46.202.45:8093: connectex: A socket operation was attempted to an unreachable host. – rest.(*handler).writeError() at handler.go:732
2020-12-08T11:14:10.099+05:30 [ERR] c:#112 #112: Post http://10.46.202.45:8093/query/service: dial tcp 10.46.202.45:8093: connectex: A socket operation was attempted to an unreachable host. – rest.(*handler).writeError() at handler.go:732
2020-12-08T11:14:16.206+05:30 [ERR] c:#095 #095: GetWithXattr 4: operation has timed out – rest.(*handler).writeError() at handler.go:732

Also, the data on the sync gateway expires after a particular duration of time but can be seen on the Couchbase server ( even though TTL is NOT enabled).

#12

Those errors (read failure, operation has timed out) indicate network connectivity problems between your Sync Gateway and Couchbase Server.

#13

I am really clueless as to why it is taking “favicon.ico” as a database name though my database name is “grocery-sync” which i have mentioned everywhere.

2020-12-09T15:08:41.784+05:30 [INF] HTTP+: #011: --> 200 (0.0 ms)
2020-12-09T15:08:41.828+05:30 [INF] HTTP: #012: GET /favicon.ico/ (as ADMIN)
2020-12-09T15:08:41.828+05:30 [INF] HTTP: #012: --> 400 invalid database name “favicon.ico” (0.0 ms)

#14

Are you by any chance using the sync gateway admin UI which as I’ve mentioned to you is unsupported and must not be used. And we’d need more context on what you are doing that is triggering this error?

I think you are chasing after many different issues and it is getting harder to assist. The log you shared in previous the post was an issue related to network connectivity to server (which I hope you have had the chance to address) and that log snippet is not related to the original issue in the post which is about documents not syncing to clients.

I would encourage you to walk through one of our Getting Started tutorials .