Hi Team! I have a monolithic application currently using a single instance of a couchbase cluster for data persistence. Time’s come for us to break down this big monolith into smaller pieces as microservices. As a rule of thumb, in a microservice architecture, each service should have its own datastore. What would be a good design pattern for this? Should we have multiple instances of couchbase-server running along different services, or still maintain a single cluster (maybe replicated to multiple servers).
It’s very hard to say with just the definition of ‘microservices’ and without knowing access pattern, data set, update rate. Let me just toss out the things I’d consider in design.
One consideration is how you’re going to need to handle resource allocation for the dataset behind a service. If you need to keep a particular working set against an overall dataset and the resources you add/remove over time change, you’ll probably want that in a bucket unto itself.
However, if you don’t really know how you’ll need to allocate resources and/or your dataset isn’t all that large, you can probably come up with a key convention or the ‘type’ convention (described in the documentation on data modeling) with something for each service to combine data into fewer buckets. As you probably know, there is some fixed cost overhead to each bucket you add-- mainly in management and replication.
Finally, with the current system architecture, you may want to consider isolating some data for another reason: the cost of index recalculations. As an example, if you have a bucket with one microservice whose persistent data isn’t changing very fast but is frequently queried via some secondary index on N1QL and you have another microservice that has fast changing data that is never queried, you may have lower overhead by separating those out between buckets. There is some small, but very real, fixed overhead to evaluating changes in a bucket for the various indexes which are defined.
It’s not all about buckets and resources though, we could look at it from an atomicity and consistency perspective too! Are services possibly going to need to have a coherent view of a piece of data? There are ways to achieve this across multiple buckets or even multiple clusters, but it may take more work to do so.
Hope that helps. Glad to give you a sense of what I think of a planned design, if you’ve worked up to that.
p.s.: sorry for the delayed reply- I was unwell for a bit and traveling; recovering now!
Thanks for this awesome response. Hope you’re better now?
Glad it helped. Doing a bit better now-- just tired, thanks!