In short, the ability to tell our web application to only send N1QL queries to a select few nodes in our cluster.
Here’s the back-story for perspective: As our investment in N1QL grows, we have two sides of the house:
Our Production Web Apps - .NET code sending queries to the cluster to serve our Prod web application
Reporting - Some more formalized in .NET or PowerBI, and some in raw adhoc N1QL queries
Our end-goal is a “light” level of workload isolation. We don’t want to build out an entirely separate cluster, that would be overkill for this scenario. We have a four-node cluster, and occasional we hit a “Yahtzee” scenario where a Business Analyst is refreshing a large report in PowerBI really puts the screws to a query service on an individual node in the cluster.
Then for the unlucky population of users in our web app that happen to have their query from the Prod web app land on the same node, the Prod web app response times take a hit.
So it’s easy enough to point PowerBI the adhoc tools at a single query node, but is that possible on the .NET web application side?
I am not .NET expert, But you can directly send n1ql statements to query service by specifying :8093 instead of 8091 port. Your application may loose workload balancing and availability. cc @ingenthr
Ya, that’s a great point… I’m sure there’s a way to send a query to a single node, but we’re hoping to maintain the ability to specify multiple nodes primarily for load balancing.
Here’s another question: Are there any metrics passed between the SDK and the query nodes such that the SDK is aware of how “busy” a particular query node is? Or is the query node just chosen at random? We couldn’t find much in the online documentation around how query load balancing takes place.
At the moment, scheduling is not based on how busy particular query nodes of the cluster are. @jmorris or @MikeGoldsmith could probably offer more thoughts here on how it’s handled.
One possibility is that we could introduce a feature to allow an interceptor on the configuration. We have an internal feature of sorts for this in some places. In fact, that was used to add N1QL itself during the developer previews when it shipped unbundled.
That said, the challenge here is that the complexity this kind of scheduling might be quite high. Whether at the app level or at the SDK level, something will need to consider index selection and index use. For your ad-hoc users, if they happen to hit indexes hard that overlap with operational users, you might have an impact. The query node itself will be pretty bored, but those other resources are shared.
We have a few options in the platform for you here. We might be able to introduce a feature like this to segment query nodes into a logical pool or transform the config for an instance in the connection. Would you have isolated app servers for this as well, or would those resources be shared between the ad-hoc and the operation
The other option might be to use Couchbase Analytics, which supports the same querying but is more focused on ad-hoc use cases. With the Multi-Dimensional Scaling, you could easily have these on isolated nodes, so the operational workloads continue to use the Query/Indexing service.
I might encourage you to try Analytics first if that’s a possibility.