Feasability of couchbase for applications with large number of complex views/pages with lots of filters


#1

I was evaluating couchbase for this particular use case, and I hit the view explosion barrier.
That is, creation of a separate view for each filter/sort combination. This means if I have x views in my app and I introduce a new filter then, I need to add x more filters. Am I missing something, or couchbase (as of now) is just not designed to handle these scenarios?


#2

Id say you might want to evaluate what you are doing in the system. If you have that many views, you might be conflating server side issues with client side issues. For example lets say you were providing a list of all the stocks in the market then defining views so they can be filtered by symbol, minimum price and so on. All of that filtering should be done client side ideally and you should be able to create one view where the keys are something parametric that can be searched with a query. Define all the criteria for your filters and then code that into the keys for the view. It just feels like your question should be about changing the design and not capabilities of the server.

The pure answer to your question is that it depends on how much memory and cpu you can deploy on the back end. That will be your true limiting factor.


#3

Are you saying I should have a single view that returns all the data, then I send that huge amount of data to the client, across the network and then client handles most of the filtering? I don’t think that is even practical.

And I don’t agree its a design issue, this is a requirement of n number of systems and something that a database, that projects having secondary indexes as its strength, should be capable of handling. And if it wasn’t, couchbase team wouldn’t put so much effort in implementing N1QL.

Again I do not understand how memory and cpu are the limiting factor, when the issue is about inflexibility of views and restricted querying capabilities. Are you suggesting that I should get thousands of documents from database into memory and do the filtering in my application code?

You tell me how could you implement any products page with 4 filters and 4 sort fields using views on a 100 million dataset in couchbase?


#4

Sounds like you are trying to use sql-query-style in Couchbase - “NOsql” Databases are not intended to behave like RDBMS nor have good performance on sql-like-querying
You should use a RDBMS or use a NoSQL-DB as it is intended to be used: Heavily by direct document lookups.

As to N1QL: This is not related to your question direclty (Views <> N1QL) and N1QL is still in development.


#5

I don’t want to get into a fist fight. May be your experience with NOSQL databases is limited, I get that. Ever heard of mongoDB or aerospike? these two have features that I am talking about and guess what they are “NOSQL databases”. N1QL is not related to my question, haha, nice one :slight_smile: Isn’t N1QL designed to solve exactly my problem?


#6

If you want to talk about N1QL then you need to specify that in your question. As you did not and did not give any other specific information i have to answer with that limited input you provided.
And the answer to the limited information of your initial posts still remains: Your using the NoSQL-Idea inaccurate.

“Isn’t N1QL designed to solve exactly my problem?” Partially, if your goal is to have lots of filtering/selecting aso. a RDBMS is still far superior in the current development status.

Please dont try do judge my experience without any knowledge about it.


#7

I too had same qestion in my mind. We need to filter out from gbs of data from couchbase. we are getting all data in couchbase and filtering out which is using lot of ram. I too was thinking there should be way to mutiple filter and sorting in couchbase. If we want to filter and sort data from 10TB we will need to get 10TB in our application and than filter out?.

Please guide me. Thanks


#8

Based on my experience with couchbase I would say Couchbase definitely doesn’t match your use case, it isn’t designed for complex queries, and it definitely isn’t designed for changing queries. With the amount of data we now have we can only add or update a view in off-peak hours, since indexing a new view makes the server unusable for the hour or so it takes to index (that’s with about 2 million documents). You can’t do joins, but you can sort of do queries on multiple keys in the same document but it’s hacky and very limited since the key gets converted to one big string.