Delete vs setting up TTL

Couchbase Server
1

Hi
I need to delete 30 million documents from a bucket in Couchbase 6.6.2. Which will be a better option to delete?

  1. Fire a delete N1QL query
  2. Set the TTL to 1 minute for 30 million documents

I need to delete any random 30 million documents(not any specific document)

Thanks

2

If you know the document keys Set TTL and let expiration take care without strain on system
Also explore eventing Delete v Expiry | Couchbase Docs

use similar technique What is the fastest way to update several million Docs cc @jon.strabala

3

Hi @rajib761

I think this should fit the bill (using Eventing) to try it out load the `travel-sample` data set, it only takes about 6 lines of JavaScript code (no comments) to do what you want.

Below I do an approximate deletion of 25% from 24024 docs but it will work for 30M or greater. The Eventing function only relies on the data service or KV it does not use N1QL.

Look at the # routes via N1QL you should see exactly 24,024 docs in travel-sample

select 24024 as init, count(*) as cur, count(*)/24024 as ratio 
    from `travel-sample` where type = "route"

The set up the below Eventing Function and deploy it.

// Example of deleting 25% of the documents from a bucket 
// in a random fashion via Eventing.

// Version 6.6.X
//   "Source Bucket".            `travel-sample`
//   "MetaData Bucket".          `metadata`
//   Feed Boundary:               Everything
//   Workers (expand the settings caret). 
//.        12 (for performance set to the # cores)
//   Binding(s)
//       "binding type",  "alias name.",   "bucket",      "Access"
//.       ==========================================================
//    1.  "bucket alias",  "src_bkt",    "travel-sample"  "read and write"

function OnUpdate(doc, meta) {
    // comment out if a filter on "type" is not needed
    if (doc.type !== "route") return;
    
    // There are 24024 route docs in `travel-sample` assume we have 
    // say we want to delete 25% of them or about 6006 doc.
    if (Math.random() <= 0.25) {
        delete src_bkt[meta.id];
    }
}

Wait until done and undeploy the above Eventing Function.

Look at the # routes via N1QL again you should see approx 75% of 24,024

select 24024 as init, count(*) as cur, count(*)/24024 as ratio 
    from `travel-sample` where type = "route"

Hope this helps

Jon Strabala

1 Like