Spark Connector: No results returned when invoking sqlContext.read.couchbase()

Iā€™m using spark connector 1.1.0 for scala 2.10. If I invoke sqlContext.read.couchbase() without the schemaFilter option inside couchbase(),like this:

val airline = sqlContext.read
.option(ā€œbucketā€,ā€œtravel-sampleā€)
.option(ā€œschemaFilterā€, "type = ā€˜airlineā€™")
.couchbase()

I think it should be something like select * from travel-sample where type = ā€˜airlineā€™, but in fact there will be no result returned because the N1QL generated is:
INFO N1QLRelation: Executing generated query: ā€˜SELECT FROM travel-sample WHERE nullā€™.

@cjycjy so you are saying if you do it like

val airline = sql.read.couchbase(schemaFilter = EqualTo("type", "airline"))  

it works, but when you do it like

val airline = sqlContext.read
.option("bucket","travel-sample")
.option("schemaFilter", "type = 'airline'")
.couchbase()

then the underlying N1QL query is generated wrongly and as a result you canā€™t get the records?

Exactly. couchase() will invoke buildFrame(null, null, None) and it will cover the original schemaFilter with null. So the underlying N1QL query will be wrong.

@cjycjy Are you using the Spark Connector with the Couchbase 4.5 developer preview? There have been some changes in the query API that we havenā€™t yet reflected in the Spark Connector. We typically wait until the product is fully baked before we put out compatible releases of the connectors. Iā€™m not sure thatā€™s what the problem is, but itā€™s possible.

No, My Couchbase version is 4.1.0.
I think it is a small bug in Spark Connector. The problem is that in the class N1QLRealation, when the queryFilter is constructed like

val queryFilter = if (parameters.get("schemaFilter").isDefined) { "WHERE " + parameters.get("schemaFilter").get } else { "" }

when the couchbase() is invoked with no schemaFilter, the (ā€œschemaFilterā€->null) will be set. So the if condition above will be true because itā€™s value is null.

I went ahead and filed a bug in Jira. Thanks for the report.

You are welcome.:wink: