Iām using spark connector 1.1.0 for scala 2.10. If I invoke sqlContext.read.couchbase() without the schemaFilter option inside couchbase(),like this:
val airline = sqlContext.read
.option(ābucketā,ātravel-sampleā)
.option(āschemaFilterā, "type = āairlineā")
.couchbase()
I think it should be something like select * from travel-sample where type = āairlineā, but in fact there will be no result returned because the N1QL generated is:
INFO N1QLRelation: Executing generated query: āSELECT FROM travel-sample WHERE nullā.
Exactly. couchase() will invoke buildFrame(null, null, None) and it will cover the original schemaFilter with null. So the underlying N1QL query will be wrong.
@cjycjy Are you using the Spark Connector with the Couchbase 4.5 developer preview? There have been some changes in the query API that we havenāt yet reflected in the Spark Connector. We typically wait until the product is fully baked before we put out compatible releases of the connectors. Iām not sure thatās what the problem is, but itās possible.
No, My Couchbase version is 4.1.0.
I think it is a small bug in Spark Connector. The problem is that in the class N1QLRealation, when the queryFilter is constructed like
val queryFilter = if (parameters.get("schemaFilter").isDefined) { "WHERE " + parameters.get("schemaFilter").get } else { "" }
when the couchbase() is invoked with no schemaFilter, the (āschemaFilterā->null) will be set. So the if condition above will be true because itās value is null.