Vertex AI TensorBoard instances created without a customer-managed key

What training logs reveal

TensorBoard logs look harmless, but they carry more than loss curves. Google’s CMEK resource table describes a TensorBoard key as covering all data from uploaded logs: scalars, histograms, graph definitions, images and text. Images logged during training are often samples of the training data itself, and graph definitions describe the model architecture.

The TensorBoard setup guide is explicit about timing: if you want the data encrypted with CMEK, you must enable the key when creating the instance.

Listing TensorBoard instances

gcloud ai tensorboards list --region=REGION \
  --format="table(name, displayName, encryptionSpec.kmsKeyName)"

An empty key column means Google default encryption.

How ZopNight flags an instance

ZopNight inventories each TensorBoard instance and records whether its encryption settings include a Cloud KMS key name. A confirmed absence raises the finding. Experiment count, log volume and last-upload time do not affect it.

What it leaves alone

Instances created with a key are silent. If the encryption setting was not collected, no finding is produced. Other Vertex AI resources are checked by their own rules, for example GCP Vertex AI vertex-model Without CMEK for the models these runs produce.

Compliance, not cost

There is no saving. The gap is training artefacts, sometimes including sample data, held outside the key policy that covers the rest of the pipeline.

Moving experiments to a keyed instance

1. Create a key in the instance’s region and grant the Vertex AI service agent, service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com, the roles/cloudkms.cryptoKeyEncrypterDecrypter role.

2. Create the new instance with the key:

   gcloud ai tensorboards create --region=REGION --display-name=NAME \
     --kms-key=KEY --kms-keyring=KEYRING --kms-location=REGION --kms-project=KMS_PROJECT

3. Point training jobs and the SDK at the new instance for all future runs.

4. Re-upload any historical logs you need to keep, then delete the old instance.