Dremio admin backup job error with GCS storage via S3 API

1

Hi,

I am using the below command from dremio master POD to store dremio backups into GCS bucket via S3 API:

/opt/dremio/bin/dremio-admin backup -a -u <dremio_user> -p <dremio_password> -d s3a://dremio-backup/admin-backup.

My S3 end point is set to storage.googleapis.com.

After writing most of the backup files into the GCS bucket dremio-backup/admin-backup, the job eventually fails with below error:

===============================================================
2024-05-24 18:25:20,756 [qtp296800048-10237]** **INFO c.dremio.dac.util.BackupRestoreUtil - Tables and uploads Backup** **finished. Backup of 58 tables and 4 uploads
2024-05-24 18:25:20,760 [qtp296800048-10237] ERROR** **c.d.d.server.GenericExceptionMapper - Unexpected exception when processing POST** **http://dremio-master-0.dremio-cluster-pod.dremio.svc.cluster.local:9047/apiv2/backup** **: java.io.IOException: Multi-part upload with id** **'ABPnzm4u1pPva7siacGr9LZZzALnG_6rXvWSz6LQwYm5kIQwJNDumWUpGWOuCnIbciKcMTg' to** **admin-backup/dremio_backup_2024-05-24_18.24/jobs_backup.pb failed:** **java.lang.IllegalArgumentException: Invalid base 16 character: 'L'
java.io.IOException: Multi-part upload with id** **'ABPnzm4u1pPva7siacGr9LZZzALnG_6rXvWSz6LQwYm5kIQwJNDumWUpGWOuCnIbciKcMTg' to** **admin-backup/dremio_backup_2024-05-24_18.24/jobs_backup.pb failed:** **java.lang.IllegalArgumentException: Invalid base 16 character: 'L'
===================================================================

Please suggest how to fix this issue.

Thanks,
Bijoy

2

Where is Dremio hosted and how are you authenticating dremio-admin to talk to GCS-via-S3? Feels rather unsupported (as for sources/dist we require S3-compatibility to be specified for non-AWS S3 locations)

I’d advise creating backup locally and moving it to GCS separately.

3

My Dremio is hosted on a Kubernetes cluster on-prem and from there I am talking to GCS via S3 api using HMAC keys.
Please note that as I said in my original post,most of the files are getting backed up in GCS,the error is mainly coming while backing up jobs_backup.pb file.

4

Does the log file contain the whole exception stack? If not, let’s enable debug logging and see if we can get it.

5

Also getting vibes of PutObjectRequest Unable to unmarshall response (Java 2 SDK) · Issue #7756 · minio/minio · GitHub which comes down to “compatibility mode is not set” so we are back to “how are you configuring the connection to GCS?” question.