And have managed to create a physical dataset, with everything, including the glue job working fine. In the relevant S3 bucket, I can also see the data and metadata files. However on trying to query that dataset, I am getting this error:
I have done it now, by pointing the data source in Dremio to the S3 bucket location where the AWS Glue job created the iceberg table with the data. However, I still get the same error:
dataset at [/dataset/###.db.###] not found.
My Dremio version is 21.3. Also, the AWS instance on which I am running Dremio is in Ireland, not in N. Virginia. Could these have anything to do with it?
Ah. So my Dremio instance is deployed in Ireland, but my S3 bucket and Glue job are in Virginia (I did it that way since the Iceberg connector for the Glue job works only in Virginia).
However, the issue just seems to be with iceberg tables. I created other glue databases in Virginia pointing to normal flat files in S3 and they seem to work fine on Dremio.
Could the version have something to do with it? Or is there a configuration which I may have missed?