SASL Authentication Failure while using kafka-connect-couchbase-3.3.0 with Couchbase Server 3.0.1 Community Edition

mk12 · 2018-06-09 05:47:25 UTC

Hi,

I’m using kafka-connect-couchbase-3.3.0 to read data into kafka from Couchbase server 3.0.1 Community Edition. I’m using standalone connect server. Its giving below error. I have checked bucket name, username and password and everything seems to be right, I’m using admin user for couchbase connection. What can be possible issues to look for in this case? are there any compatibility issues with the versions?

[2018-06-09 10:54:42,577] WARN Got error during connect (maybe retried) for node /xx.xx.xxx.xx:11210 (com.couchbase.client.dcp.conductor.Conductor:160)
com.couchbase.client.core.endpoint.kv.AuthenticationException: SASL Authentication Failure
at com.couchbase.client.dcp.transport.netty.AuthHandler.checkIsAuthed(AuthHandler.java:182)
at com.couchbase.client.dcp.transport.netty.AuthHandler.channelRead0(AuthHandler.java:114)
at com.couchbase.client.dcp.transport.netty.AuthHandler.channelRead0(AuthHandler.java:54)
at com.couchbase.client.deps.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

david.nault · 2018-06-11 23:05:51 UTC

Hi Mk12,

Does this note from the Quickstart guide hold the answer?

Note: For Couchbase Server versions prior to 5.0, leave the username blank. Set the password property to the bucket password, or leave it blank if the bucket does not have a password. The sample buckets do not have passwords.

Thanks,
David

mk12 · 2018-06-12 04:49:03 UTC

Hi david.nault,
Thanks for response.
This note definitely helped me to get rid of the error I posted, but now I’m getting new error like below.

[2018-06-12 10:09:57,433] WARN Got error during connect (maybe retried) for node /xx.xxx.xx.xx:11210 (com.couchbase.client.dcp.conductor.Conductor:160)
java.lang.IllegalStateException: Could not configure DCP Controls: 0x0004 (Invalid arguments)
at com.couchbase.client.dcp.transport.netty.DcpControlHandler.channelRead0(DcpControlHandler.java:109)
at com.couchbase.client.dcp.transport.netty.DcpControlHandler.channelRead0(DcpControlHandler.java:40)
at com.couchbase.client.deps.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

My config files look like:

connect.config
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=1000
plugin.path=/custom-source-handler-1.0-SNAPSHOT-jar-with-dependencies.jar

couchbase-source-config:

name=test-couchbase-source
tasks.max=2
topic.name=couchbaseDatatxt-small
connection.cluster_address=xx.xx.x.x
connection.timeout.ms=2000
connection.bucket=beer-sample
connection.username=
connection.password=password
use_snapshots=false
dcp.message.converter.class = com.couchbase.connect.kafka.example.CustomSourceHandler
connector.class=com.couchbase.connect.kafka.CouchbaseSourceConnector
event.filter.class=com.couchbase.connect.kafka.filter.AllPassFilter
couchbase.stream_from=BEGINNING
couchbase.compression=DISABLED
value.converter = org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true

david.nault · 2018-06-12 23:13:41 UTC

This happens when the client sends a DCP control the server does not understand. I took another look at the connector code, and I think any version of Couchbase Server >= 3.0.1 should support all of the required controls, but maybe something’s wrong.

I’ll try to reproduce the issue locally. In the mean time, please can you provide the debug log output from the connector ? (add this line to connect-log4j.properties):

log4j.logger.com.couchbase=DEBUG

I’m interested in just the log lines containing “Negotiating DCP Control” so we can see which control the server is choking on.

Thanks,
David

vs07 · 2018-06-29 00:53:13 UTC

Hi David,

Thanks for the help. I work with mk12 and here are the logs:

[2018-06-28 01:51:42,538] DEBUG Negotiating DCP Control connection_buffer_size: 20480 (com.couchbase.client.dcp.transport.netty.DcpControlHandler:95)
[2018-06-28 01:51:42,541] DEBUG Negotiating DCP Control set_noop_interval: 120 (com.couchbase.client.dcp.transport.netty.DcpControlHandler:95)

david.nault · 2018-06-29 19:35:31 UTC

Hi @vs07,

Thank you for those log lines.

Looks like Couchbase Server 3.0.1 CE doesn’t understand the set_noop_interval control. This is surprising, since it’s documented as being introduced in that version.

I’ve filed KAFKAC-114 to track this compatibility issue.

In the short term, you might try downgrading to Kafka Connector 3.2.2 which does not use connection keepalive and does not send this control parameter.

Ultimately it would be good to upgrade to a modern version of Couchbase Server, since 3.0.x reached end of life after April 2016, over a year ago.

Thanks,
David