I get Dremio and Nessie up and running, and I can see Dremio talking with Nessie.
When I run a simple “create table” statement, Dremio seems to submit the query, but it never does anything. Eventually, the following message shows up in Dremio:
org.apache.iceberg.exceptions.RuntimeIOException: Failed to write manifest list file
I see this in the dremio container logs:
Caused By (org.apache.hadoop.fs.s3a.AWSClientIOException) Writing Object on test1/metadata/snap-1332441890796237069-1-bfde68d1-d991-4112-b503-52881212ba38.avro: com.amazonaws.SdkClientException: Unable to execute HTTP request: https: Temporary failure in name resolution: Unable to execute HTTP request: https: Temporary failure in name resolution
I’ll update here if I figure something out.
From within the Dremio container, I can connect to the nessie container with this:
curl http://nessie:19120
I can also connect to the desired S3-compatible bucket via the url I have configured in conf/core-site.xml
From within the nessie container, I can connect to the s3-compatible bucket
From the Dremio web GUI, I can set up a data source that points at the nessie container using this url:
http://nessie:19120/api/v2
And then when I run this sql query it just kind of sits there for a long time with the query window grayed out, :
create table if not exists analytics.test1 (name varchar);
I can see this in the Dremio logs:
PUT /apiv2/scripts/50ac2bbf-abe8-4e23-aee9-96fed69561da
POST /apiv2/datasets/new_tmp_untitled_sql_and_run?newVersion=0000806498605355
-- and then later --
[18d54681-9c9b-a50c-9ef5-ad8d1ee2c500/0:foreman-planning] WARN c.d.e.s.i.model.IcebergBaseCommand - Not adding jobId as Iceberg snapshot update property for test1
And if I look at the jobs list I see this, and when I refresh the page I can see the “Duration” value increasing:
So, the problem in this case is that I had “https://” in the value for the Other connection property key “fs.s3a.endpoint” for the Nessie data source in the Dremio UI.
There is a doc that explicitly says to not include the “https://” part, and I completely missed it.
