Do Dremio support Redshift spectrum with External table in S3?
i tried to connect my RedShift JDBC, but those External table cannot be found.
Please help. Thanks
For this scenario we would normally recommend querying the data in S3 directly using Dremio’s S3 connector. Is this something you can try?
Yes. i tried to load S3 directly, and can load the data from parquet files. However, this will handover the data processing from Redshift clusters to Dremio itself. Also, seems there is no way to create all Dremio datasets from S3 bucket using AWS Glue meta schema, but just can create one-by-one using the UI portal from the folders?
That’s correct. This would have Dremio doing the query execution, which should be faster and more cost effective.
Currently we do not yet support the Glue catalog, but this is something we are watching closely. You can try using Dremio’s REST API to create the virtual datasets. You could read from the Glue catalog and write to Dremio’s catalog.