InsertAsync stuck in a loop checking... something

When attempting to insert a new document into the cluster, it appears that the SDK is stuck in a loop:

2017-09-18 17:09:34,719 [37636] DEBUG ZOLL.Core.Couchbase.Sync.DurableSequenceRepository - Attempting to insert change feed marker document 'PcrCompleteChangeFeed_398189' (attempt 1 of 5): ZOLL.Core.Couchbase.Sync.ChangeFeedMarkerDoc
2017-09-18 17:09:35,313 [9084] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:35,313 [9084] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 2 - server Couchbase.Core.Server
2017-09-18 17:09:35,313 [9084] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.113:11210
2017-09-18 17:09:35,313 [9084] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31513
2017-09-18 17:09:35,313 [9084] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31513 for [zd01].
2017-09-18 17:09:35,313 [9084] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:35,313 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31513
2017-09-18 17:09:35,313 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31513 on thread 9
2017-09-18 17:09:35,313 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31513
2017-09-18 17:09:35,391 [35064] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:35,391 [35064] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 1 - server Couchbase.Core.Server
2017-09-18 17:09:35,391 [35064] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.112:11210
2017-09-18 17:09:35,391 [35064] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31513
2017-09-18 17:09:35,391 [35064] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31513 for [zd01].
2017-09-18 17:09:35,391 [35064] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:35,391 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31513
2017-09-18 17:09:35,391 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31513 on thread 12
2017-09-18 17:09:35,391 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31513
2017-09-18 17:09:37,813 [2832] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:37,813 [2832] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 0 - server Couchbase.Core.Server
2017-09-18 17:09:37,813 [2832] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.111:11210
2017-09-18 17:09:37,813 [2832] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31512
2017-09-18 17:09:37,813 [2832] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31512 for [zd01].
2017-09-18 17:09:37,813 [2832] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:37,813 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31512
2017-09-18 17:09:37,813 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31512 on thread 9
2017-09-18 17:09:37,813 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31512
2017-09-18 17:09:37,907 [43096] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:37,907 [43096] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 2 - server Couchbase.Core.Server
2017-09-18 17:09:37,907 [43096] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.113:11210
2017-09-18 17:09:37,907 [43096] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31513
2017-09-18 17:09:37,907 [43096] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31513 for [zd01].
2017-09-18 17:09:37,907 [43096] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:37,907 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31513
2017-09-18 17:09:37,907 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31513 on thread 12
2017-09-18 17:09:37,907 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31513
2017-09-18 17:09:40,329 [28268] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:40,329 [28268] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 1 - server Couchbase.Core.Server
2017-09-18 17:09:40,329 [28268] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.112:11210
2017-09-18 17:09:40,329 [28268] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31513
2017-09-18 17:09:40,329 [28268] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31513 for [zd01].
2017-09-18 17:09:40,329 [28268] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:40,329 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31513
2017-09-18 17:09:40,329 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31513 on thread 9
2017-09-18 17:09:40,344 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31513
2017-09-18 17:09:40,407 [32540] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:40,407 [32540] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 0 - server Couchbase.Core.Server
2017-09-18 17:09:40,407 [32540] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.111:11210
2017-09-18 17:09:40,407 [32540] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31512
2017-09-18 17:09:40,407 [32540] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31512 for [zd01].
2017-09-18 17:09:40,407 [32540] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:40,407 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31512
2017-09-18 17:09:40,407 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31512 on thread 12
2017-09-18 17:09:40,407 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31512
2017-09-18 17:09:42,844 [29592] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:42,844 [29592] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 2 - server Couchbase.Core.Server
2017-09-18 17:09:42,844 [29592] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.113:11210
2017-09-18 17:09:42,844 [29592] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31513
2017-09-18 17:09:42,844 [29592] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31513 for [zd01].
2017-09-18 17:09:42,844 [29592] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:42,844 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31513
2017-09-18 17:09:42,844 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31513 on thread 9
2017-09-18 17:09:42,844 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31513
2017-09-18 17:09:42,907 [29964] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking configs...
2017-09-18 17:09:42,907 [29964] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Using index 1 - server Couchbase.Core.Server
2017-09-18 17:09:42,907 [29964] DEBUG Couchbase.Core.Server - Sending Config with key  using server 10.100.62.112:11210
2017-09-18 17:09:42,907 [29964] INFO  Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Checking config with revision #31513
2017-09-18 17:09:42,907 [29964] DEBUG Couchbase.Core.ClusterController - Queueing config rev#31513 for [zd01].
2017-09-18 17:09:42,907 [29964] DEBUG Couchbase.Configuration.Server.Monitoring.ConfigMonitor - Waiting to check configs...
2017-09-18 17:09:42,907 [CT] DEBUG Couchbase.Core.ClusterController - Processing config rev#31513
2017-09-18 17:09:42,907 [CT] DEBUG Couchbase.Configuration.Server.Providers.ConfigProviderBase - 1. Checking config with rev#31513 on thread 12
2017-09-18 17:09:42,907 [CT] INFO  Couchbase.Configuration.Server.Providers.ConfigProviderBase - Ignoring config with rev#31513

Any idea as to what might be happening here? The document is never inserted.

Also getting this when attempting to run a query:

2017-09-18 17:05:37,156 [41900] INFO  Couchbase.Core.Buckets.CouchbaseRequestExecuter - System.BadImageFormatException: Bad class token.
   at Couchbase.Core.Buckets.CouchbaseRequestExecuter.<>c__DisplayClass21_0`1.<<SendWithRetryAsync>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Couchbase.Core.Buckets.CouchbaseRequestExecuter.<>c__DisplayClass21_0`1.<SendWithRetryAsync>b__0(IQueryRequest e, IConfigInfo c)
   at Couchbase.Core.Buckets.CouchbaseRequestExecuter.<RetryQueryEveryAsync>d__7`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Couchbase.Core.Buckets.CouchbaseRequestExecuter.<SendWithRetryAsync>d__21`1.MoveNext()
2017-09-18 17:05:37,156 [41900] DEBUG ZOLL.Core.Couchbase.Sync.DurableSequenceRepository - Query result has no rows for sequence number.

Hi @pkramer

Please can you provide SDK & cluster versions, any custom couchbase configuration and a code example?

I’ve not seen this behaviour before from bucket.InsertAsync.

@MikeGoldsmith Thanks, Mike.

Late yesterday we found out that AppDynamics on the server was causing this behavior. Apparently it forces the application to load some additional dependencies which were producing the “BadImageFormatException” above.

1 Like