Save Document as String

Couchbase Server
1

Hi,
I am using a couchbase elastic connector to move data from CB to elastic, It’s working fine.
I have one query if I want to modify the document before pushing it to elastic how can I do that?

for ex: from the below document, I want to convert the items array into a string before pushing to elastic.
is there any way I can achieve this? I read about the couchbase templates in older versions of plugins, can we use them in the latest plugin too?

{
   "id":"12131",
   "type":"page",
   "items":[
      {
         "id":"12131",
         "props":"sdads"
      },
      {
         "id":"2323",
         "props":"sdads"
      }
   ]
}

@oguzhan @ingenthr @vsr1 @WillGardella @david.nault @jon.strabala

2

Hi @Ashish_Mishra,

I am sure you could perform what you need in Java via the couchbase elastic connector, however I am focused on Eventing so my answer will be specifically for Eventing type solutions, I will leave a Java coding solution to others like @david.nault - but without trying anything out, I think you could follow the example here Jackson 2 - Convert Java Object to / from JSON - Mkyong.com

From the Eventing perspective you could write a simple function like

function OnUpdate(doc, meta) {
    // skip non-interesting docs
    if (!doc.type && doc.type !== "page") return;
    doc.items = JSON.stringify(doc.items);
    // This is a write to an binding in settings could be
    // the source bucket/keyspace or another bucket/keyspace
    tgt_bkt[meta.id] = doc; 
}

If your bucket binding alias (in the Eventing function’s settings) is the same as the Eventing source bucket it will convert you documents to something like:

{
  "id": "12131",
  "type": "page",
  "items": "[{\"id\":\"12131\",\"props\":\"sdads\"},{\"id\":\"2323\",\"props\":\"sdads\"}]"
}

If your bucket binding alias (in the Eventing function’s settings) is to another the bucket you would get a duplicate item but in the way you want and your Kafka connector could listen to that bucket (or keyspace in 7.X).

Alternatively is space is not a concern you could do the following and just enrich the source documents in place

function OnUpdate(doc, meta) {
    // skip non-interesting docs
    if (!doc.type && doc.type !== "page") return;
    doc.items_as_string = JSON.stringify(doc.items);
    // This is a write to an binding in settings could be
    // the source bucket/keyspace or another bucket/keyspace
    tgt_bkt[meta.id] = doc; 
}

This second function will store your array in both forms a true JSON array and a stringified array.

{
  "id": "12131",
  "type": "page",
  "items": [
    {
      "id": "12131",
      "props": "sdads"
    },
    {
      "id": "2323",
      "props": "sdads"
    }
  ],
  "items_as_string": "[{\"id\":\"12131\",\"props\":\"sdads\"},{\"id\":\"2323\",\"props\":\"sdads\"}]"
}

Best

Jon Strabala
Principal Product Manager - Server‌

2 Likes
3

encode_json(items)
decode_json(encode_json(items))

https://docs.couchbase.com/server/current/analytics/8_builtin.html#JSONFunctions

SELECT d.*, encode_json(d.items) AS items_str
FROM default AS d
WHERE .....
1 Like
4

Hi Ashish,

With the Elasticsearch connector, the recommended way to transform documents is to use Elasticsearch ingest pipelines .

Ingest pipelines let you perform common transformations on your data before indexing. For example, you can use pipelines to remove fields, extract values from text, and enrich your data.

Arun Vijayraghavan wrote a blog article showing how it’s done: Using Elasticsearch Connector with Ingest Node Pipeline | The Couchbase Blog

Thanks,
David

1 Like