Not able to read Couchbase(3.0.1) data through Kafka Connect(3.2.0)


#1

Hi,
We are using below configurations of database and connect

Couchbase 3.0.1 (42 machine cluster)
Couchbase Kafka connect (3.2.0 and 3.0.0)

We were able to read records through couchbase kafka connect for some time now.

When we restarted our connect cluster to read all the data again, we are not getting any db records pushed through kafka connect. We are seeing just below logs at connect side and no errors.

[2018-07-10 16:23:15,115] INFO WorkerSourceTask{id=cb-connector1-4} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:328)
[2018-07-10 16:23:15,115] INFO WorkerSourceTask{id=cb-connector1-4} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:345)

FYI
We have XDCR enabled at Couchbase cluster.
We tried both in Standalone and distributed mode.

When we are starting kafka connector cluster, we could see in Couchbase UI that new DCP connections are being formed but no data flowing. Let me know if any other info needed.


#2

Hi @mk12,

The Kafka Connect framework periodically saves a “bookmark” of which Couchbase changes have been published to Kafka. Is it possible you’re not seeing any new Kafka messages because there haven’t been any changes in Couchbase since the last time the connector saved a bookmark?

You could verify this in standalone mode by deleting the offset storage file. Its location is specified by the worker (not connector) configuration property offset.storage.file.filename and defaults to /tmp/connect.offsets . (In distributed mode, the offsets are stored in a Kafka topic. We could dig up the details if necessary.)

Version 3.2.2 of the connector introduced a configuration property couchbase.stream_from which you can set to BEGINNING if you want to ignore any saved state. Remembering our earlier conversation, I believe connector 3.2.2 is the most recent version that’s still compatible with Couchbase 3.0.1.

Thanks,
David


#3

Hey @david.nault,
Version 3.2.2 with config couchbase.stream_from worked for us.
Thanks for quick reply.