Spark Streaming Receiver opens wrong bucket on cluster


#1

On local mode with local Couchbase, the Spark Streaming works just fine with the following configurations:

-Dcom.couchbase.bucket.beer-sample=""
-Dcom.couchbase.nodes=couchbase.my.domain:8091
-Dcom.couchbase.connectTimeout=100000

On the YARN cluster, the receivers try to connect to the default bucket:

16/07/07 13:28:37 INFO streaming.CouchbaseReceiver: Starting Couchbase (DCP) Stream against Bucket default

What is the problem here? The configuration should be read from SparkConf by the Spark Streaming client API as I’ve read from the internals.

Thanks!


#2

The problem is that configurations from JVM environment are not transferred to cluster.
You’ll have to set programatically with .set.


#3

@zoltan.zvara yeah I think you need to use the spark-provided ways to set options since otherwise they are only on the executor JVM. Is your issue solved by now?


#4

Yes, thanks, problem solved! :slight_smile: