Hello,
I’m using Couchbase Node.js Client and Ottoman.js packages in my Express application to work with Couchbase Server. For debugging purposes I need to log each query being run by Ottoman.js. So far, when I used other databases, it was just a simple flag, like"debug": true, which enabled logs in terminal. For Couchbase Server, I couldn’t find such an option. What I found is Query Monitoring in Admin Console. However when I run, for example, an insert query using Ottoman.js, it doesn’t show the query neither in Active Queries, nor in Completed Queries. But the query is being run for sure as it creates the document. Please note that I’ve used { "completed-threshold": 0 }, which according to the docs, should monitor ALL queries.
Can someone please help me to understand why it doesn’t monitor? Also, if you know a way to enable monitoring in terminal that would be great.
completed-threshold=0 should log queries that takes execution more than 0 ms. This setting will impact memory usage
and completed-limit controls how many statements it logged. You should check
SELECT * FROM system:completed_requests;
This setting is in memory i.e if query service restarted you must set again. You must set for all query services separately.
completed-threshold=0 should log queries that takes execution more than 0 ms
It’s not correct. From the docs:
Specify 0 to track all requests independent of duration.
SELECT * FROM system:completed_requests;
This is the same as clicking on the “Completed” tab in Query Monitoring (AFAIK), which doesn’t show the query.
You must set for all query services separately.
What do you mean for all? I’ve set it according the docs, using Admin Rest API, and when I request to this URL: http://localhost:8093/admin/settings, it returns the JSON config containing this line: "completed-threshold": 0,
Just a hunch here, but if you query using ottoman, the request gets logged, correct?
Also, could I please have version, CE or EE, and the number of query nodes?
Hi @Karlen91 - follwoing up from my previous question, I have just had the confirmation from the SDK team that ottoman.js does mutations through the KV, which clearly wouldn’t be logged through the N1QL completed requests.
You say you do an insert query - implication being that somehow you are using N1QL? - I’m not entirely sure how that would work, as, as far as I can tell, the only way to use N1QL is indirectly through the find method and filters.
Could you show me exactly what you are doing?
Hey @Marco_Greco, and thanks for your response.
So, I did a bit of investigation and also tried some stuff. Here’s my update:
No, it’s not. I’ve created an index just like in ottoman example, called findByName and tried to call User.findByName, which returned the right object. However there was no log. I also tried to use User.find, as well as to set n1ql as an index type. The only log I could see was the index creation i.e. CREATE INDEX ..., but it also wasn’t always logged.
I found that on ottoman.store has a debug flag, which you can just set like ottoman.store.debug = true; and it will show some useful output in the console for EVERY query.
By digging into the Couchbase.js code, I found that when I run ottoman.insert, under the hood the Bucket.insert is being called, which I’m not sure how works. I thought Query Monitoring in Admin Console should capture all queries regardless it’s n1ql or not. And I wonder, if it’s not n1ql, then what is it? How it runs queries in general?
And my last question is about indices. When I run my app, it creates and ensures the index and in Admin Panel I can see it, but on that UI it is marked as red and doesn’t show “build progress: 100%” compared to the default indices created by ottoman.
Would appreciate any help on the above bullets. Thanks!
A Bucket.insert uses Couchbase’s KV API. It is in the admin console in the overall statistics for the bucket. Since it’s not managed by or running in the query engine, it’s not in Query Monitoring.
You may want to read the Architecture Overview and some other parts of the Learn section there in the docs to understand how Ottoman is working with different parts of the underlying system.
This is because Ottoman.js is smart enough to know to use the KV API, which is more efficient, when you don’t need to go through the query service.
This is to unify logging with the underlying libcouchbase it’s based on. With this, following the docs, you can log what is happening at your node.js server.
Hm, it might just be a question of polling for updates in the UI. Or, it could be a problem with Ottoman.js’s ensure indexes. Do you see anything that indicates the index isn’t ready?
Hi @Karlen91, sorry, I’m just back from holiday.
As I said, it seems like ottoman uses the KV interface, hence you wouldn’t see any n1ql logging.
As for the other questions, they are beyond my area expertise, hence why I engaged the SDK team.
Hi @Marco_Greco and @ingenthr, and thank you for the help and information. @ingenthr I will dig deeper into the Learning section for sure and the information you provided is really helpful.
However, it seems Ottoman.js is not smart enough as you said or there’s some issue with Query Monitoring. I checked the code of Mode.find() and Model.count(). Both methods are sending N1QL queries through CbStoreAdapter. So this is the high-level call order:
Model.find(), which calls
CbStoreAdapter.find(), which eventually does
// ...
var query = this.couchbase.N1qlQuery.fromString(fullQs);
// ...
this.bucket.query(query, function (err, res) {
callback(err, res);
});
And similar call chain works for Model.count(). In both cases I get the right result, however, nothing is being shown in Query Monitoring.
Also, this is the log of Ottoman.js (which I talked about above):
CbStoreAdapter::find n1ql User { filter: { name: 'test2' } }
As you see, the type of query is logged as n1ql.
So my only guess here is that Query Monitoring is not able to catch it somehow. Can you please suggest any solution for this? Or perhaps tell me another way to send N1QL queries to Couchbase without Ottoman.js. This way I will be able to identify the issue, i.e. is it something with Ottoman.js or Query Monitoring.
As for this:
No. I don’t see any message when I do find using that index and when I open in workbench and try to create with the same props, it says that the index already exists.
It definitely looks like Ottoman is using N1QL indexes to do the find that you are looking for, however if my memory serves me right, the query monitor effectively ‘polls’ the server for active queries. Since the Ottoman.js queries execute relatively quickly, I don’t believe it’s likely that you would see the query show up unless there was some issue which caused the query to take an extraordinary amount of time to complete.
Hey @brett19,
Thanks for your reply. That makes sense. I did some more tests and it shows pretty strange results in Query Monitoring. I believe that there’s some issue, either in Monitoring engine or UI. Or if it really doesn’t show because of the ‘polling’ as you said, still it’s not documente like that.
So in any case there are some open questions, which should be properly investigated and answered. And I think simple logging functionality is being used by lots of devs and it’s really useful no matter it takes more or less time. This functionality should pretty easy to use as with other DBs.
@brett19@Marco_Greco Pretty late to the party. Was going through this thread and want to know if there is any update on this. We have a similar requirement in our project where we are using couchbase 6.6 and queries are getting fired from a springboot webservice where we are using Spring Data Couchbase Api. I had set the threshold in query monitoring engine as 0 and was able to get the logs of N1ql queries getting fired. But most of the times we are using Bucket.upsert() function to insert a record through java sdk, and as you can guess logs for those operations are not being shown in the output of completed_requests rest Api.
So my question if anyone would be kind to answer is that is there a workaround or a different approach to get those logs ,i.e, for bucket.upsert() operations?
Thanks in anticipation,
Abhishek