S3 key prefix - dir0, dir1...dirN but no object/file name?

1

Hi there,

Is it possible to get the S3 object (/file) name when connecting Dremio to S3 as a source?

I create a physical dataset on s3://my-bucket/bar. My database dumps ndjson files in with a partitioning scheme like: s3://my-bucket/bar/yyyy-mm-dd/hh/001000.ndjson

I see yyyy-mm-dd and hh as dir0 and dir1 respectively, but I don’t get to see “001000.ndjson” anywhere when I query. I need this to be able to discern what the latest updates are.

Thanks!

2

Hey @J_C, Welcome to the Dremio community.

You can enable the support key dremio.store.file.file-field-enabled in Settings -> Support -> Support Keys to see the filenames in $file column

image
image1618×972 118 KB

3

Perfect - thank you so much!