Using DCP with external subscribers, at scale


#1

We’d like to implement subscription to data changes, using DCP.
I have a few questions regarding it’s features/performance:

  1. Is it available for external subscribers?
  2. How many concurrent live subscriptions can it handle? We’d ideally like to create 1 subscription per entity.
  3. What’s the latency? Does it degrade when number of subscription rise?

Thanks,
Leonid


#2

I’m afraid DCP is not currently a published, supported API. I believe the indent is to make it so in future, but it isn’t currently.


#3

Thank you for a quick reply!

In your opinion, if DCP becomes supported in the future, is it likely allow subscribing for changes of a certain entity, or will it be a global subscription for all changes?


#4

I can’t comment on what will or won’t be available when it’s a published API, but currently the finest level of granularity DCP handles is per-vBucket.


#5

Agreed with @drigby’s comments there, but can you give us a better idea of what you mean by “certain entity”? For instance, are you looking for all changes to the user “leonidb” or all of the changes to documents with a particular shape or a particular attribute set.

It’d be useful to get your use case as we make plans for DCP and related interfaces in this area.


#6

Yes

We use akka-persistence with a home made Couchbase plugin.
Every session is persisted as a sequence of events. Those events are used for session recovery when needed. They can also be streamed to long term storage/dashboards. There are some additional advantages of event sourcing, like retroactive report generation, auditing, troubleshooting, etc.

Additional possible usage is that several services will subscribe for those events and react in real time/near real time to those events. For this use case it would be useful to be able to subscribe for session’s events, at session granularity.

I doubt there is a single data store today capable of meeting the defined requirements (1M live subscriptions, latency of ~20msec, average).
I do think it would be very useful for applications using event sourcing.