I am using the Couchbase Kafka Connector to stream bucket changes to a Kafka topic and was testing the case of Kafka being down, therefore not able to write the change to it.
First I was surprised to not see a lot of error logs when this happen, so was not sure that the write were not going through! Confirmed when restarting Kafka and looking at the messages in the topic - the latest DCP messages were missing.
Then, it looks like the vBucket state is updated in Zookeeper even if writing to Kafka fails, therefore, even restarting the connector from its last known state will not pick up the missed changes?
Am I missing something?
What is the intended usage in this case? Are we expected to replay the DCP stream from beginning of time? And in this case, I guess to avoid duplicate, will have either to purge the current topic and stream the changes to another topic.
Any help and guidance will be appreciated.