Which is a better for building a cluster- fewer nodes with more RAM, or more nodes with less RAM?

OVERVIEW
In this scenario, I’m building a small general purpose cluster to support a variety of projects. Ranging from hosting mostly static websites to a little search & ranking engine that will make heavy use of views. But overall load on the cluster will likely be relatively light.

I’m meaning to ask a general question, given that the Couchbase Manual has relatively high recommendations for capacity for the machines running couch base. For smaller operations, we are limited to renting machines at hosting companies, and this presents us with a trade off given the way these machines are rented. (The capital cost of 16GB of RAM for an enterprise building their own server is trivial given other costs, but for us, it’s a significant price difference every month.)

I am asking a general question, but often without context general questions are hard to answer, so I’m going to give the specifics I’m confronted with right now. I’m not really asking for a choice between these two specific providers (say one has a bad rep and the other doesn’t) as the offerings on the market change all the time.

I’m really trying to understand the tradeoffs between fewer beefier nodes and more, less beefy nodes, at the same cost, given an external performance. (EG: we have N users hitting our website each month in both cases and one of these clusters behind it supporting it.) I also recognize that this is the low end of the recommended range for couchbase, may not be officially supported, etc. etc. But it’s the reality for a small business building innovative apps on top of couchbase.

SPECIFIC EXAMPLE

Looking at dedicated hosting I see two inexpensive options:

Kimsufi KS16G at 20 euros a month http://www.ovh.ie/dedicated_servers/kimsufi.xml
Hetzner EX40 at 50 euros a month http://www.hetzner.de/en/hosting/produktmatrix/rootserver-produktmatrix-ex (or the EX50 at 60 euros has SSDs which should boost view performance)

So, for the same monthly budget, we could build:

  1. A Cluster of 3 Hetzner EX40s, with 32GB RAM each, and quad core i7 CPUs at 150 euros
  2. A Cluster of 7 Kimsufi K16Gs, with 16GB RAM each and quad core i5 CPUs at 140 euros

RESOURCES COMPARISON

I see several areas of performance: RAM, CPU, Redundancy, etc.

For RAM, we have 96GB serving those N queries a month, vs 112GB. IF load is distributed in both cases, the Kimsufi cluster with more servers will likely have more of the dataset cached.

Looking at the CPUs bot hare ~3Ghz, but one is dual threaded and the other is single thread. Not sure how significant CPU level threads are, but assuming it represents a %50 boost, we have 28 threads for kimsufi and effectively 18 threads for hetzner.

While kimsufi clearly has some better pricing here, the resources allocated for the whole cluster in both scenarios are not that far apart.

So, in which situation does couchbase perform better?

CONJECTURE / CONCLUSION

My guess is that the better solution is the Hetzner case with a few big machines. That performance issue comes when a node is starved of resources, and ramping up capacity across a whole cluster, evenly distributed, will hit the “starved node” issue much quicker when the individual nodes are smaller. While there are more of them and thus overall capacity is not that different, the transient issues that crop up will be more problematic for the cluster with smaller nodes.

Alas, I don’t have access to resources to test this scenario.

So, I must appeal to the perspective of the experts.

Hello,

Usually with Couchbase it is better to have more nodes than few nodes for the following reasons. I am saying that considering that the sum of the RAM allocated to Couchbase on all the nodes will be the same.

So it is better to have more nodes because:

  • you distribute the read and writes on more nodes: so you will be handle a bigger number of request
  • you will cache the same number of documents since you have the same amount of RAM
  • in case of failover and rebalance it will be faster on with a bigger number of nodes, since you have less data to move.

Some interesting resources:

Regards
Tug
@tgrall

2 Likes