I am trying to run the Kafak Source Connector. When I start it using the connect-standalone.sh command from the quickstart documentation, I can see from logs that the bootstrapping of the Couchbase cluster nodes is successful, but then it trying to open a connection to CB on port 11210 and fails with below message and just repeats this failure continuously for all four nodes. I have verified that there is no firewall rule preventing the connection and the port is accessible from the server where I am running the Kafka Connector and they are on the same domain.
What else can cause this error? Do you have any troubleshooting suggestions?
INFO Connect attempt to <host>/10.249.176.27:11210 failed
Could not open DCP Connection: Failed in the OPEN step, response status is 0x0024 (No access / insufficient permissions)
Thank you David! It was the DCP Reader role that was not enabled for the user. After fixing that, the Connector starts successfully and is polling every 30 secconds per config.
However, I donât see any messages being sent after I add new documents to the bucket. I donât see any errors in the Connector logs, but inserting into the bucket does not trigger a message.
Whatâs the value of your couchbase.persistence_polling_interval setting? Some versions of the connector shipped with a quickstart config that has an inappropriate value. You probably want something like 100ms for this, or 0 to disable persistence polling.
How are you watching for Kafka messages? Are you sure youâre watching the same topic the connector is publishing to?
Thanks for the response. Iâm sure I am missing something simple again. The polling interval is set at 30 seconds as noted in my last post and I can see a log ouput from the connector every 30 seconds that looks like this:
[2020-07-29 17:49:16,029] INFO WorkerSourceTask{id=test-couchbase-source-1} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:424)
[2020-07-29 17:49:16,029] INFO WorkerSourceTask{id=test-couchbase-source-1} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:441)
The default config uses a topic named âtest-defaultâ. This topic never gets created in kafka (I would expect it to be created once there is a message generated). However, I thought maybe the problem was that I need to pre-create the topic, so I did that. I created a new topic called âtest-topicâ and re-configured the Connector to publish to that topic. I also sent a few test messages to the topic to ensure it was working. I ran a consumer from the command line as follows in order to monitor the messages:
My test messages were output, but the Connector never published anything to this topic. I inserted at least a half dozen new documents into the bucket that is the same one the Connector is configured on, but nothing.
The log messages about flushing 0 outstanding messages for offset commit are for saving the connectorâs progress, and are unrelated to persistence polling. (See the description of offset.flush.interval.ms in the Kafka Connect config documentation.)
I would try disabling persistence polling by setting couchbase.persistence_polling_interval to 0. The results will give us more information.
Incidentally, is the source bucket an âephemeralâ bucket?
It is working now. I saw the comment in the config file about ephermeral buckets which require setting polling to 0. Once I did that, the messages began streamingâŚlots of them
I needed to change user permissions to enable the âData DCP Readerâ role for the âconnection.usernameâ specified in the source config properties file.
I needed to set the value of âcouchbase.persistence_polling_intervalâ in the source config file to 0 because the bucket specified in âconnection.bucketâ was ephemeral as noted in the comment in the config file: If the source is an ephemeral bucket (which never persists documents) this value must be set to 0 to disable the persistence check.
