Make the kafka connector consuming only the latest events

Kafka Connector
1

Hello everyone !

I started using kafka-connect-couchbase and I’m wondering if there is any option allowing to poll only the newest events. For example, My couchbase keeps keys of less than 30 days (as the TTL= 30 days ). how I should configure my kafka connect to make it polling only the newest keys (1 hour in maximum before starting the kafka connector).

I’m looking for a solution without using the filters. because the main problem is that my CB instance keeps a lot of history keys. and polling all data takes a huge time even with only the filter/discart
Thanks in advance for your help.

2

Hi devhmed,

It’s not possible at the moment. You may want to watch and vote for this Jira issue:

KAFKAC-64 Specify start/end condition for streaming

If you could configure the connector to “start from now” (not an hour ago), would that be an acceptable solution?

Thanks,
David

1 Like
3

Hello David,

That’s will be great. is it feasible with current version ?

Thanks
devHmed

4

Not with the current version, but the “start from now” feature is scheduled for inclusion in 3.2.2 to be released on December 19th.