Make the kafka connector consuming only the latest events



Hello everyone !

I started using kafka-connect-couchbase and I’m wondering if there is any option allowing to poll only the newest events. For example, My couchbase keeps keys of less than 30 days (as the TTL= 30 days ). how I should configure my kafka connect to make it polling only the newest keys (1 hour in maximum before starting the kafka connector).

I’m looking for a solution without using the filters. because the main problem is that my CB instance keeps a lot of history keys. and polling all data takes a huge time even with only the filter/discart
Thanks in advance for your help.


Hi devhmed,

It’s not possible at the moment. You may want to watch and vote for this Jira issue:

KAFKAC-64 Specify start/end condition for streaming

If you could configure the connector to “start from now” (not an hour ago), would that be an acceptable solution?



Hello David,

That’s will be great. is it feasible with current version ?



Not with the current version, but the “start from now” feature is scheduled for inclusion in 3.2.2 to be released on December 19th.