Java Sdk 3.1.5:Multiple Keys for encrypting documents in the same application

We store 5 different documents (doc1 , doc2, doc3, doc4, doc5) in couchbase.
As part of new requirement, we have to encrypt few fields in above docs with different keys (doc1 fields with key1, doc2 fields with key2, doc3 fields with key3 , doc4 fields with key4, doc5 fields with key5).

I have written a sample programm for encrypting all the docs using one encryption key (by setting crypto manager at clusterenvironment) but not able to find a way to encrypt/decrypt different keys for different docs. Please advice.

Hi Raghav,

You can handle this by registering a separate encrypter for each key:

// Keyring containing key1, key2, key3, key4, and key5
Keyring keyring = ...

AeadAes256CbcHmacSha512Provider provider = AeadAes256CbcHmacSha512Provider.builder()
    .keyring(keyring)
    .build();

CryptoManager cryptoManager = DefaultCryptoManager.builder()
    .decrypter(provider.decrypter())
    .encrypter("enc1", provider.encrypterForKey("key1"))
    .encrypter("enc2", provider.encrypterForKey("key2"))
    .encrypter("enc3", provider.encrypterForKey("key3"))
    .encrypter("enc4", provider.encrypterForKey("key4"))
    .encrypter("enc5", provider.encrypterForKey("key5"))
    // you can also specify a default encrypter, but it might be safer
    // to omit this next line so you don't accidentally use the default.
    .defaultEncrypter(provider.encrypterForKey("otherKey"))
    .build();

When writing encrypted fields, specify the encrypter to use:

// EXAMPLE: Writing encrypted field using a specific encrypter

JsonObject doc1 = collection.get("doc1").contentAsObject();
JsonObjectCrypto crypto = doc1.crypto(collection)
    .withEncrypter("enc1");
crypto.put("secretField", "secretValue");
collection.upsert("doc1", doc1);

When reading an encrypted field, the SDK automatically selects the correct key; you only need to specify the encrypter when writing the field. Here’s what reading the encrypted field might look like:

// EXAMPLE: Reading an encyrpted field

JsonObject doc1 = collection.get("doc1").contentAsObject();
// no need to specify encrypter, since we're only reading the field
JsonObjectCrypto crypto = doc1.crypto(collection);
System.out.println(crypto.get("secretField"))

If you’re blending databinding and encryption by using the @Encyrypted annotation, you can specify the encrypter as an attribute of the annotation. However, that might not work for you if documents doc1 through doc5 are bound to the same Java class, since the encrypter to use is hard-coded into the annotation. In that case, you’ll probably need to reowork your code to do encryption using JsonObjectCrypto like in the above examples.

Thanks,
David

Thanks a lot for your quick response and it really helps.
I am using annotation based logic, but as you explained I can’t use annotation for this use case.

Sorry, I have follow-up questions…

  1. In the writing logic, why there is a get operation on the collection and then upsert. I want to encrypt more than 4 fields while doing upsert operation first time.
  2. You are writing the secret fields into the json doc, is ther a way to just pass the json doc(with some indication for encrypted fields or else pass the to_be_encrypted field list) and encrypter name to the SDK api or crypto while doing upsert.
  3. In Case of Reading doc also… once you fetch the doc from collection, we should be passing it to crypto and crypto should return the json doc with decrypted values.
  4. Can you please share some link which explains the usage/examples of crypto api for better understanding?

Appreciate your support in clarifying the above points.

Sorry, that’s my fault for writing a confusing example. You don’t need to get the document first. If you’re creating a new document, you can do:

JsonObject doc = JsonObject.create();
JsonObjectCrypto crypto = doc.crypto(collection)
    .withEncrypter("enc1");
crypto.put("firstSecretField", "secretValue");
crypto.put("secondSecretField", "secretValue");
// etc.
// Maybe you also want to add some non-secret fields?
doc.put("notASecretField", "notSecretValue")
collection.upsert("doc1", doc);

The SDK does not have a feature like this, but you could do it yourself with a function like:

public void encryptSensitiveFields(Collection collection,
                                   JsonObject doc,
                                   Set<String> fieldNames,
                                   String encrypterAlias) {
  JsonObjectCrypto crypto = doc.crypto(collection)
      .withEncrypter(encrypterAlias);

  for (String fieldName : fieldNames) {
    Object value = doc.get(fieldName);
    if (value != null) {
      doc.removeKey(fieldName); // remove unencrypted
      crypto.put(fieldName, value); // add encrypted
    }
  }
}

Sure, just modify the above encryptSensitiveFields function to decrypt instead.

Here’s the documentation:
https://docs.couchbase.com/java-sdk/current/howtos/encrypting-using-sdk.html

If you’d like to dive deeper into the design of the encryption framework, you could peruse the Couchbase Field-Level Encryption RFC.

Thanks,
David

Thanks a ton David.

Sorry I am mixing up the topics here…
Right now we have 32 bytes key for encrypting the field values but the algorithm which the SDK is using mandates 64 byte key.

How do I overcome this?

  1. How do I generate 64 bytes key? The AES java program which we are using can generate only 32 byte key.
  2. Is it correct to append the same 32 byte key twice, so that it will become 64 byte key and pass this to cryptomanager settings.

Hi Raghav,

Right now we have 32 bytes key for encrypting the field values

Where did these 32-byte keys come from? Are you upgrading from Couchbase SDK 2.x’s implementation of field-level encryption? If so, then you should follow the steps in the migration guide.

If the 32-byte keys came from somewhere else, then there’s no expectation that Couchbase SDK’s default encryption algorithm is compatible with that other system. For compatibility with an external crypto system, you’d need to implement your own Decrypter and Encrypters. There’s not much documentation on that topic, but the source code for AeadAes256CbcHmacSha512Provider is a good place to start.

How do I generate 64 bytes key?

The keys are just random bytes. The docs have an example of how to create a 64-byte key.

Is it correct to append the same 32 byte key twice, so that it will become 64 byte key and pass this to cryptomanager settings.

I wouldn’t recommend it. It might “work”, but the whole point of using a 64-byte key is to avoid sharing key material between the AES and HMAC steps.

I should probably take a moment to mention that deploying Field-Level Encryption requires an Enterprise Subscription License Agreement, which typically entitles you to support. If it turns out you need to heavily customize the encryption framework, you might want to get in touch with our support team through the usual channels so we can appropriately escalate your case.

Thanks,
David

If you want to use that program (because it’s been blessed by your security team, or for some other reason) you could generate two 32-byte keys and concatenate them.

Thanks David, will work on the solutions you provided. Thanks.

Hi David,
the above snippets you shared allow me to add/remove fields from JsonObject at root level.

  1. What if the “to_be_encrypted” fields are at nested level.
  2. Duplicate field names can present at different level in a nested json objects (eg: modifiedTime). How do we handle this. Please suggest.

I have managed to pass the fieldNames while doing a get operation with below code.
decryptSensitiveFields(crypto,readItBack, crypto.getEncryptedFieldNames());

For upsert operation, planning to introduce a custom annotation("@toBeEncrypted") to find the fieldNames so that i can pass it to below behaviour.
encryptSensitiveFields(collection, doc, fieldNames, encrypterAlias)

but still trying to find solution for couple of things…

  1. to_be_encrypted field at nested level.

class Employee{
@toBeEncrypted
private string name;
private string id;
private Salary sal;
}

class Salary{
private String currency;
@toBeEncrypted
private Double value;
}

  1. duplicate field names in a nested json doc.

class Employee{
@toBeEncrypted
private string name;
private string id;
private Salary sal;
private Department dep;
}

class Department{
@toBeEncrypted
private String name;
private long id;
}

Final option would be going with our own encrypt and decrypt logic instead of using couchbase SDK.

Thanks,
Raghav

Hi Raghav,

Let’s brainstorm a little bit. You need to specify a specific field to be encrypted, and that field could be at any depth. Couchbase has the notion of a “subdocument path” that identifies a specific field. The examples given in that documentation are name , addresses.billing.country, and purchases.complete[0]. These paths are very much like JSON pointers, but with slightly different syntax.

Instead of passing in a set of simple field names, you could pass in a set of subdocument paths (or JSON pointers, if you prefer). The code would parse a path/pointer into its components, and use the components to navigate the JsonObject structure. When you’ve navigated all the way to the last path component, you’ve found the field to encrypt/decrypt.

This all assumes that at the point in the code where you want to do the encryption/decryption, you know what kind of document you’re working with (Employee vs. Department, for example) and you can pass in a different set of field names/paths/pointers depending on the document type.

Duplicate field names can present at different level in a nested json objects (eg: modifiedTime). How do we handle this. Please suggest.

A JsonPointer or subdocument path uniquely identifies a field, so there would be no ambiguity if you went with that approach.

I can’t really comment on the annotation-based strategy; seems like it would be quite a bit of work.

Final option would be going with our own encrypt and decrypt logic instead of using couchbase SDK.

Feel free to use as much or as little of the Couchbase encryption framework as you want. The Couchbase CryptoManager interface and DefaultCryptoManager are part of the committed public API. You could use them to encrypt/decrypt stuff even if you don’t want to use the JsonObjectCrypto or the @Encrypted annotation. At the lowest level, I suppose you could even call directly into AeadAes256CbcHmacSha512Cipher (although it’s not technically public API, so it might change without notice).

In my opinion, at this time the primary benefit of using the Couchbase Field-Level Encryption framework is that the provided encryption algorithm is tried and true (if not exactly cutting-edge) and portable between all of the Couchbase SDKs for different programming languages.

Whatever you do, don’t roll your own crypto system :slight_smile:

Thanks,
David

1 Like

Thanks for the inputs.

I have spent some time and tried couple of options (nested json usecase) but not able to find a proper solution. Let me share here and see if you can throw some light on this…
1. upsert - employee.department.name (to_be_encrypted_field) **
** - not able to find SDK 3 api /class which will take path as input and return couchbase JsonObject/ or String

2
** Tried with JsonPath library to parse and get to the last component and encrypt the field but facing other issues while updating back to the original couchbase Json object.**

f> or ( Map.Entry<String, String> fieldToPathEntry :fieldToPathMap.entrySet()){

        DocumentContext jsonContext = JsonPath.parse(rootObject);
        logger.info("fieldToPathEntry==>"+fieldToPathEntry);
        leafNode = jsonContext.read(fieldToPathEntry.getValue());
        logger.info("leafNode==>"+leafNode);
        leafNodeCrypto = leafNode.crypto(collection).withEncrypter(encrypterAlias);
        logger.info("leafNodeCrypto==>"+leafNodeCrypto);
        Object value = leafNode.get(fieldToPathEntry.getKey());
        logger.info("value==>"+value);
        leafNode.removeKey(fieldToPathEntry.getKey()); // remove encrypted
        logger.info("leafNode removeKey==>"+leafNode);
        leafNodeCrypto.put(fieldToPathEntry.getKey(), value); // add encrypted
        logger.info("leafNode put==>"+leafNode);
        if(!"$".equalsIgnoreCase(fieldToPathEntry.getValue())) {
            jsonContext.set(fieldToPathEntry.getValue(), leafNode);
        }else{
            rootObject = leafNode;
        }
        logger.info("Mutated rootObject==>"+rootObject.toString());
    }

Hi @Raghav,

Please open a support ticket (mentinoing this forum thread) so we can get the support team and your support representative involved.

Thanks,
David

Thanks David, sure will do that.

Hi David,
We have successfully implemented with some custom logic to handle nested json.

one quick clarification required, another requirement is to configure the secret keys dynamically with out application restart.
some thing like this…

  1. Initially, application will initialize the SDK with default encrypter and standby encrypter with two different keys.
  2. Application will listen to secret key update events.
  3. on Receiving the new key, application needs to make current default encypter key as standby key and new key as default encrypter and refresh the SDK.

-----the question is, does it require application restart or is there a mechanism to refresh this configuration in SDK without any impact. Current ongoing operations shouldn’t be impacted with this configuration refresh.

Hi Raghav,

I’m glad to hear you’re getting closer to the finish line :slight_smile:

another requirement is to configure the secret keys dynamically with out application restart.

I would recommend using a custom Keyring that supports key rotation.

The Couchbase Field-Level Encryption RFC has a section on key rotation that describes why you’d want to do this and how you might implement it.

There’s also a Keyring.rotating(Keyring) helper method that takes an existing keyring and decorates it with key rotation behavior. It’s worth reading the Javadoc for that method, even if you don’t end up using it. The basic idea is the Encrypter asks the Keyring for a key using the “base name”, and the Keyring returns the latest version of the key with that base name.

Incidentally, the Keyring interface is designed to be narrow and composable; check out some of the other decoration methods like Keyring.caching and Keyring.reloading.

Okay. So let’s focus on using these building blocks to meet your requirement.

The first task is to decide on a naming convention for your keys. Let’s assume you want to keep things simple and follow the examples, which use ISO 8601 dates for the version strings, and -- as the delimiter that separates a key’s base name from the version.

Next, implement the backing Keyring. Because we want to use the rotating decorator method, the keyring also needs to implement the ListableKeyring interface (which just provides a method to get the names of all the keys in the keyring).

To keep things really simple, you could continue reading the encryption keys from the filesystem using KeyStoreKeyring, and wrap it with a reloading decorator so the keys are periodically reloaded from disk. (NOTE: the keyring returned by Keyring.reloading does not implement the ListableKeyring interface required by Keyring.rotating, so the following code sample includes a reloadingListable decorator that does.)

Here’s what all of this might look like in code.

KeyStoreKeyring loadBackingKeyringFromDisk() { ... }

Keyring myRotatingKeyring() {
  Duration reloadInterval = Duration.ofMinutes(5);
  ListableKeyring reloadingKeyring = reloadingListable(
      reloadInterval,
      () -> loadBackingKeyringFromDisk()
  );

  String versionDelimiter = "--";
  Comparator<String> versionComparator = Comparator.naturalOrder();
  Keyring rotatingKeyring = Keyring.rotating(
      versionDelimiter,
      versionComparator,
      reloadingKeyring 
  );

  // Keyring.rotating uses linear search to find latest key version.
  // This can be expensive if there are many keys, so...
  Duration cacheExpiry = reloadInterval;
  int maxCacheEntries = 10_000;
  return Keyring.caching(
      cacheExpiry,
      maxCacheEntries,
      rotatingKeyring
  );
}

/**
 * Returns a listable keyring wrapper whose backing keyring is
 * periodically refreshed by calling the given supplier.
 */
public static ListableKeyring reloadingListable(
      Duration reloadInterval,
      Supplier<? extends ListableKeyring> loader
  ) {
  requireNonNull(loader);
  Cache<String, ListableKeyring> cache = Caffeine.newBuilder()
      .expireAfterWrite(reloadInterval)
      .maximumSize(1)
      .build();
  return new ListableKeyring() {
    private ListableKeyring getBackingKeyring() {
      return cache.get("", ignore -> loader.get());
    }

    @Override
    public Collection<String> keyIds() {
      return getBackingKeyring().keyIds();
    }

    @Override
    public Optional<Key> get(String keyId) {
      return getBackingKeyring().get(keyId);
    }
  };
}

One caveat is that you might see minor performance hiccups when the keyring is reloaded or there are cache misses. This can be addressed down the road by writing a custom Keyring optimized for your use case instead of relying on the decorators.

tl;dr: Have a look at the Keyring source code on GitHub, and experiment with the above sample code. Remember that keys in the keyring should have names like myKey--2021-06-25 and the Encrypter should be configured with just myKey.

Let me know how it goes.

Thanks,
David

Thanks for the inputs David, will share the updates with you.

Hi David, Yesterday i have implemented the key rotation/reloading and It works as expected.

  • However, this approach mandates us the keynaming pattern which is not in our control.
  • I can rename the key while adding to the key ring but that will create another problem where i have to save the actual keyname vs modified keyname mapping.

Requirement → initially during server startup, application should initiate the SDK with one key and while processing documents, encrypt all the fields before persisting them into couchbase. But when the key is compromised, application will receive new key. Post that, application should use this new key for encrypting the fields for all the documents.

Tried another approach- change encrypter alias name dynamically

Add new active key to keyring, change encrypter alias name in @Encrypted annotation in runtime using reflection but i couldn’t find a way to configure the new encrypteralias into crypto manager.

Another approach - using JsonObjectCrypto object

I should be able to solve our usecase using the below sudo code but the backdrop of this approach is, I may end up writing complex code to encrypt the fields at nested level as we discussed already above.

JsonObjectCrypto crypto= jsonObject.crypto(collection).withEncrypter(encrypterAlias);
Object value = node.get(fieldName);
node.removeKey(fieldName);
crypto.put(fieldName, value);

Please suggest if there is any better approach you can think of.

Hi Raghav,

You can do this with a custom Keyring. Remember:

  • The key returned by the keyring can have a different name than the name used to look up the key.

Use this to your advantage. Configure the encrypter to use a key with the special name “latest”. In your keyring, if the requested key name is “latest” then return the latest version of the key, using the actual key name. Otherwise do a real lookup by key name and return the key with the same name the caller asked for.

When the key is compromised, tell your Keyring there’s a new “latest” key with a different name.

Thanks,
David

Hi David, Great suggestion. It worked. Thanks