Couchbase indexing with Apache Storm

urbangroove · July 10, 2014, 6:47am

Hello,

I am planning to use Couchbase with Apache Storm. Since Storm inserts data into Couchbase heavily(i.e. 2~3k updates or inserts per second), view’s indexing also keeps updated and restarted endlessly with the default auto-indexing settings. So querying with stale=false is always stopped by the runtime exception saying “failed to access the view”, and it seems like querying with stale=false is impossible in this case. Of course, with stale=ok, I can get a result but cannot make sure if the result includes the data I want to map/reduce or not since the indexing is not done yet.

So is there a way to get a correct result that is the same as the result when indexing is 100% in this case? Or a way to know what data was processed with stale=ok up to? For your information, Couchbase version is 2.2.0 community edition, and Java SDK version is 1.4.1

Thanks in advance!

ingenthr · July 11, 2014, 6:06am

While it may be too slow with stale=false to suit your needs, it shouldn’t give you “failed to access view”. That seems unusual. Do you have any errors in the log at the cluster?

The best way perhaps is stale=update_after or to have the auto-index do your update and just cycle the query times.

Also, note that 2.2.0 is pretty dated at this stage and 2.5.1 would be better, if you can upgrade. 3.0 (currently under development) is expected to have some improvements here as well. Stay tuned for that in the near future.

urbangroove · July 11, 2014, 9:00am

Thanks for your answer! I checked the log of my cluster again but there were no relevant errors. Here is part of the stack trace I had from Java SDK when the exception occured so would you please have a look at this and let me know why this happened?

java.lang.RuntimeException: Failed to access the view at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:785) ... Caused by: java.util.concurrent.ExecutionException: OperationException: GENERAL at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:98) at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:82) at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:72) at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:778) ... 28 more Caused by: OperationException: GENERAL at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:74) at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:103) at com.couchbase.client.http.HttpResponseCallback.completed(HttpResponseCallback.java:51) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115) at org.apache.http.nio.protocol.HttpAsyncRequester$RequestExecutionCallback.completed(HttpAsyncRequester.java:376) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115) at org.apache.http.nio.protocol.BasicAsyncClientExchangeHandler.responseCompleted(BasicAsyncClientExchangeHandler.java:179) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:349) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:236) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267) at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:165) at org.apache.http.impl.nio.DefaultHttpClientIODispatch.onInputReady(DefaultHttpClientIODispatch.java:51) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:662)

daschl · July 11, 2014, 3:45pm

Can you share the relevant codeparts? Also, it looks like the server is returning an error here. Were you doing rebalance/failover or the kind during the view query?

urbangroove · July 14, 2014, 2:37am

Thanks for your answer! I wasn’t doing nothing other than the querying job. Here is part of my code slightly edited. To make it clear, I am using a hashmap for each bucket connection, and since the indexing job is time-consuming so I extended the timeouts in the connection initialization. Hope this is helpful.

initial connection parts


List hosts = new ArrayList();
for (String server : servers) {
	try {
		URI base = new URI(String.format("http://%s/pools", StringUtils.trim(server)));
		hosts.add(base);
	} catch (URISyntaxException e) {
		LOGGER.warn(e.getMessage());
		continue;
	}
}
if (!hosts.isEmpty()) {
	if (!clientMap.containsKey(bucket)) {
		CouchbaseConnectionFactoryBuilder cfb = new CouchbaseConnectionFactoryBuilder();
		cfb.setOpTimeout(1000 * 60 * 30);
		cfb.setOpQueueMaxBlockTime(5000);
		cfb.setViewTimeout(1000 * 60 * 60);
		CouchbaseClient client = new CouchbaseClient(cfb.buildCouchbaseConnection(hosts, bucket, password));
		clientMap.put(bucket, client);
	}
	LOGGER.debug("[" + bucket + "] initialization was complete");
}


query parts


String designDoc = "design";
String viewName = "view";
Query query = new Query();

query.setRange(

ComplexKey.of(“d”, “a”, “2014”, “07”, “10”, “12”),

ComplexKey.of(“d”, “a”, “2014”, “07”, “10”, “12”, “\u0fff”)

);

query.setLimit(20);

query.setReduce(true);

query.setGroupLevel(8);

query.setStale(Stale.FALSE);
for (int i = 0; i < SEARCH_RETRY_COUNT; i++) {

View view = clientMap.get(bucket).getView(designDoc, viewName);

try {

response = clientMap.get(bucket).query(view, query);

if (response != null) {

break;

}

} catch (RuntimeException re) {

re.printStackTrace();

if (re.getMessage().contains(“Failed to access the view”)) {

try {

Thread.sleep(SEARCH_RETRY_SLEEP_MILLI);

} catch (InterruptedException ie) {

LOGGER.info(ie.getMessage());

break;

}

}

LOGGER.info("retry count = " + (i + 1));

}

}

daschl · July 14, 2014, 5:33am

Thanks, can you also set logging to FINEST and share that? It looks like we need to inspect the server response.

urbangroove · July 14, 2014, 9:08am

Since I’m using log4j, I tried to get more verbose log by setting TRACE level but there was nothing changed compared to the below log I’ve already posted. If I missed anything, please let me know.

daschl · July 14, 2014, 9:22am

Hey,

no worries. Btw you can check here for more info on configuring logging properly with the couchbase client: http://nitschinger.at/Logging-with-the-Couchbase-Java-Client

One thing I’d like to get if possible: can you set a breakpoint where it throws the GENERAL exception and print the details? If I’m not mistaken its a json parsing error because the server returns a body error msg. I plan to make this logged in the next SDK version but for now we have to take what is there. Would be awesome, thanks!

urbangroove · July 14, 2014, 12:24pm

Thanks for the useful information! I applied the FINEST level and had the following error that kept repeating. Unfortunatley, I also set a breakpoint on the GENERAL exception line as you said but it didn’t work. In addition, I tested the same query with stale=ok, and it worked well so I guess it’s not just a server error. Any ideas or suggestions?


2014/07/14 20:55:30 com.couchbase.client.http.HttpResponseCallback retryOperation
Retrying HTTP operation from node (10.32.124.***:8092), Request: GET /analytics-service/_design/query_referer/_view/rank?reduce=true&limit=20&stale=false&group_level=8&startkey=%5B%22d%22%2C%22a%22%5D&endkey=%5B%22d%22%2C%22a%22%2C%22%E0%BF%BF%22%5D HTTP/1.1
2014/07/14 20:55:30 com.couchbase.client.http.HttpResponseCallback completed
Operation returned, but needs to be retried because of: HTTP/1.1 500 Internal Server Error

daschl · July 14, 2014, 12:39pm

Yes, what I would suggest is:

when you run these queries, afterwards run cbcollectinfo (to collect cluster logs), create a MB ticket http://www.couchbase.com/issues/browse/MB with that info. Please post the link here so I can follow up with the server team if they need more info on the client side. It could be something that we are already tracking but I don’t know yet.

Also helpful for me would be to set a breakpoint at com.couchbase.client.protocol.views.ViewOperationImpl#76 and in there print the “json” variable. So we can see whats in the actual body. Note that I already have the patch to print that automatically for the next release: http://review.couchbase.org/#/c/39347/

urbangroove · July 15, 2014, 2:57am

Thanks for your suggestions. Okay, I will do it later.

Also set a breakpoint at the line and got the result at this time. The “json” variable was just empty string("") , and I also got “400 bad request” but the request seemed to be correct. Then, I had the “Error parsing JSON” exception. Hope this is helpful.