I want to experiment with using AWS s3tables, and I’d really like to be able to query them from dremio. s3tables writes catalog information to a glue catalog, but not the default catalog for the AWS account; instead it goes into a catalog named “s3tablescatalog”. Is there any way to specify which catalog to use when adding a Glue Catalog source in dremio?
@tborchert If file-based tables, we have to know whether the tables are stored, e.g.,S3. In that case, we need to use S3 catalogs. If you want to use Glue, need to create tables in Glue, and then might use CTAS to create/copy table into Glue table.
S3, Azure, etc, need credentials to verify access rights. Just, different engines can access, but still use the same catalogs and credentials.
This is the new s3tables feature: Tabular Data Storage At Scale - Amazon S3 Tables - AWS . The data is iceberg tables in S3 and published to glue catalog.
@tborchert Currenty Dremio is not supporting S3tables, this is something that will be planned for the future
@tborchert thanks for bringing this topic, interested and exploring but stopped short because Dremio does not support. @balaji.ramaswamy hoping this can queued to product team
@balaji.ramaswamy is there any timeline for aws s3tables support by dremio, considering it is based on apache iceberg which dremio also is built, it would be great to see this prioritized