LiveQuery returning no rows *sometimes*

Hi All,

The app I’m developing (OSX, iOS) uses CBLLiveQuery objects to monitor for changes. I have a set of views and pull different types of data based on a “type” field.

Things work well - the UI updates as changes occur in the data. But quite regularly, the CBLLiveQuery object returns no rows when I know there to be rows. When I get no rows, I’ve added a hack to create a new CBLQuery check the results - and that always returns the right data. Is there anything I am doing that could cause CBLLiveQuery to be unreliable?

The code looks like this:
First establishing the live query:

  let query = gDB.viewNamed("objectsByType").createQuery()
  query.keys = ["book"]
  liveQuery = query.asLiveQuery()
  liveQuery!.addObserver(self, forKeyPath: "rows", options: [.New, .Old, .Initial], context: nil)
  liveQuery!.start()

In my observer code - wait for the object to be equal to my liveQuery object, and if it does, call reloadLive

override func observeValueForKeyPath(keyPath: String?, ofObject object: AnyObject?, change: [String : AnyObject]?, context: UnsafeMutablePointer<Void>) {
    if object! as? NSObject == liveQuery {
        reloadLive()
    }
}

The reloadLive function repopulates a cache of objects built from the query results:

func reloadLive() {
    if var rows = liveQuery!.rows {
        cache.removeAll()
        // Walk the rows creating new Book objects from the property maps, adding them to the cache.
        while let row = rows.nextRow() {
            if let props = row.document!.properties {
                let data = Book(fromMap:props)
                cache.append(data)
            }
        }
        sortCache()
}

My hack to work around the problem is: at the end of the function check for 0 rows, and if there are none I load the data from a regular query to confirm:

if cache.count == 0 {
    let query = gDB.viewNamed("objectsByType").createQuery()
    query.keys = ["book"]
    if let rows = try? query.run() {
        // Walk the rows creating new Book objects from the property maps, adding them to the cache.
        for let row = rows.nextRow() {
         if let props = row.document!.properties {
            let data = Book(fromMap:props)
            cache.append(data)
        }
    }
}

This correctly returns the right data, every time.

For the moment, I’ve turned syncing off to make sure that is not causing any issues, but that has not helped.
I think my views are created ok because regular queries work just fine.
I’ve checked the DB contents using Couchbase Lite Viewer, and I can see my data there.
I’m not touching the liveQuery object (CBLLiveQuery) anywhere else in the app.

Does anyone have some insight as to why this might be happening?
Thanks for your help.
Paul.

Sounds like a bug, although it’s surprising that LiveQuery would be doing something that wrong. Please file an issue on Github.

I’m a little surprised too - in my hunting there doesn’t seem to be other examples of this affecting anyone else.

We have a reasonable body of code, to which we have added a number of views, live views and tons of UI interacting with it all. I’m going to regress the code somewhat - by disabling all but one of the LiveViews (the one that seems to fail most visibly/obviously). I’m also going to restrict its use to one UI area (disconnecting all the rest) and assess its reliability. I’m either going to get to a smaller reproducible use case that can be used to file a bug, or it will work, in which case I can slowly add the rest back in.

I’ve got a suspicion that there is something in the code where we have done something stupid that causes the LiveView to misbehave. Plus I’m going to beef up the automated UI tests to specifically look for this “zero” returns issue.

I’m crossing my fingers.

Ok. I think I have the answer - its a timing issue.

We keep a cache of objects constructed from the property maps provided by CBLQuery.
The cache is an observer of the LiveView - so as soon as new data is inserted into the DB,
the LiveView notifies the cache, which reads the LiveView and builds any new objects as required.

Problem is that sometimes the LiveView can be a little slow to send out notifications - e.g. we
insert a new object and sometimes we need the cache to reflect that immediately.
So as an optimisation, when an insert is undertaken we immediately update the cache. When the
liveView notifies us later… nothing will happen because that new object is already in the cache.

Seems that sometimes the following happens:

  1. insert object into DB. Cache is immediately updated
  2. Notification of a change arrives from LiveView. LiveView has no objects in it (yet), so we update our cache to reflect the liveView - i.e. empty. (Weirdly… if we create a new CBLQuery, it does reflect the new object)
  3. Later, LiveView sends another update probably from the earlier insert. We update the cache to reflect the LiveView which now contains the object inserted earlier.

So LiveView is working… eventually everything catches up and the Cache correctly reflects the objects in the DB. But there appears to be a change notification that is arriving early - before the insertion is reflected in the LiveView.

Cheers.
Paul.

I think what may be happening here is:

  1. You make a change in the database
  2. LiveQuery notes the change and starts a timer
  3. Timer fires, LiveQuery starts a background database query
  4. You make a second change in the database
  5. Background query finishes, and LiveQuery stores result into its rows property, triggering a KVO notification.
  6. You store the query results, which don’t include the change made in step 4.

The upshot is that a LiveQuery may not reflect the current state of the database, even at the moment it sends its notification. Instead it’s eventually consistent.

This is a trade-off between performance, accuracy and latency. At one extreme, the LiveQuery could run its query synchronously (as it used to long ago), which means it’s 100% up to date but at the expense of blocking the thread for potentially a long time.

At another extreme, the LiveQuery could check the database before posting its notification, and if any changes had occurred since it ran the query, it could instead start a new query. The problem is that if database changes are happening rapidly (e.g. during a first-time pull replication), and that rate is faster than the query takes to run, the LiveQuery would never post a notification; it’d just keep burning CPU time running queries and then throwing them away.

We had a case where we needed it to be synchronous and found the waitForRows method useful.

@jens

I think you are bang on. We have re-engineered our code to better live with the ‘eventual consistency’, and I think the overall result is much stronger. We are relying heavily on data coming in via the sync gateway, and need to have first class support for data arriving at any time.

@combinatorial
As I read it, the waitForRows was more about pausing for the first read to complete. Until that happens rows can return nil. It doesn’t help with keeping liveQueries up to date later, which is where my problem lay. Having said that - thanks for the reminder of that. Rather than working around nil values in rows, I’ve now added waitForRows. Much better!

Thank you for the assistance.
Paul.


As an unrelated aside: we’ve been using Swift for this. Its a wonderful language and a huge step up from objective-c… but we found a doozy. As part of tracking down this issue, I had removed some code from a function by using an early return from a function that returns VOID. Basically:

    func myFunc() {
        // some code
        return
        cache.removeAll()
        // more code that is now skipped.
    }

I couldn’t figure out why my cache kept getting zapped.
Turns out… the expression after the return statement is included in the return, even if the function is effectively VOID return and even if that statement is on the next line. So the cache.removeAll() is executed and its result is returned. Note that newline is supposed to be a statement separator in swift.

Ick.
Live and learn.

Without going too far off topic — newline doesn’t force the end of a statement, otherwise you’d never be able to break a long line. It’s more the other way around, that the end of a statement requires a newline (or semicolon) to follow. Looks like you hit an ambiguous situation where the return could be parsed either alone or with the next line, and the parser decided on the latter. Which I agree is unexpected! You should be able to work around it by adding a semicolon after the return.

(Actually I have learned not to put in early returns like this, because in Obj-C the compiler will complain that the code below is unreachable. Instead I always use Cmd-/ to comment out the lines below.)

FYI, I found out that the return statement issue is already known, and the Swift 1.3 compiler emits a warning when this happens.

Interesting that swift 1.3 raised a warning… swift 2.2 indicates that code is unreachable - but doesn’t tell you that it will silently execute the next line!.

I think it is ambiguous (or at least misleading) behaviour. In other respects, swift has a lot going for it.

Thanks for the help @jens.

Cheers.
Paul.

Oops, I meant Swift 3. The next version of Swift.

Ok. That makes sense.

I think I am experiencing the same issue as topic starter.
What is the proper solution to handle it?
Is the hack author invented the best solution for now?

I am using couchbase-lite 1.4.0

Actually I think I am experiencing different issue with similar results.

My documents disappears from live query for a second.
At the same moment I see such log in Xcode console:

WARNING: CBLRestPusher[https://sync.dev.to.firstfoundry.net/db] removePending: sequence 393 not in set, for rev {venue-1-tab-99E9E100-C2FA-4795-85E6-F17121F707C4 #34-702aa072063e0d7a2730076f60bb093b DEL} {at -[CBLRestPusher removePending:]:

Could anybody explain what does it mean?