Error while listing S3 objects - Caused by: java.net.UnknownHostException

I’m trying to install 22.1.1 on kubernetes using the v2 helmchart with minio as s3 storage. My dist storage setting look like

distStorage:
  # The supported distributed storage types are: local (<21.0.0 only), aws, azure, gcp, or azureStorage.
  #
  # local: (<21.0.0 only) Not recommended for production use. When using local, dist-caching is disabled.
  # aws: AWS S3, additional parameters required, see "aws" section.
  # azure: ADLS Gen 1, additional parameters required, see "azure" section.
  # azureStorage: Azure Storage Gen2, additional paramters required, see "azureStorage" section.
  # gcp: Google Cloud Storage, additional parameters required, see "gcp" section.
  type: "aws"

  aws:
    bucketName: "dremio"
    path: "/"
    extraProperties: |
      <property>
      <name>fs.dremioS3.impl</name>
      <description>The FileSystem implementation. Must be set to com.dremio.plugins.s3.store.S3FileSystem</description>
      <value>com.dremio.plugins.s3.store.S3FileSystem</value>
      </property>
      <property>
          <name>fs.s3a.access.key</name>
          <description>Minio server access key ID.</description>
          <value>redacted</value>
      </property>
      <property>
          <name>fs.s3a.secret.key</name>
          <description>Minio server secret key.</description>
          <value>redacted</value>
      </property>
      <property>
          <name>fs.s3a.aws.credentials.provider</name>
          <description>The credential provider type.</description>
          <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
      </property>
      <property>
          <name>fs.s3a.endpoint</name>
          <description>Endpoint can either be an IP or a hostname, where Minio server is running . However the endpoint value cannot contain the http(s) prefix. E.g. 175.1.2.3:9000 is a valid endpoint. </description>
          <value>minio.chariot:9000</value>
      </property>
      <property>
          <name>fs.s3a.path.style.access</name>
          <description>Value has to be set to true.</description>
          <value>true</value>
      </property>
      <property>
          <name>dremio.s3.compat</name>
          <description>Value has to be set to true.</description>
          <value>true</value>
      </property>
      <property>
          <name>fs.s3a.connection.ssl.enabled</name>
          <description>Value can either be true or false, set to true to use SSL with a secure Minio server.</description>
          <value>false</value>
      </property>

I can see subpaths and objects being created in minio

asalow@DESKTOP-86CCVB2:~$ mc ls -r staging/dremio/
[2022-10-08 15:53:46 CDT]     0B STANDARD accelerator/
[2022-10-07 15:58:26 CDT]     0B STANDARD downloads/
[2022-10-08 15:43:31 CDT]     0B STANDARD metadata/
[2022-10-07 15:56:26 CDT]     0B STANDARD scratch/
[2022-10-08 16:18:09 CDT]     0B STANDARD uploads/_staging.dremio-master-0.dremio-cluster-pod.dremio.svc.cluster.local/
[2022-10-08 16:18:09 CDT]     0B STANDARD uploads/_uploads/

but then I get the following errors and the pod never becomes ready

2022-10-08 21:31:41,614 [start-__home] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /dremio/uploads
com.amazonaws.SdkClientException: Unable to execute HTTP request: dremio.minio.chariot
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1162)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5386)
        at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:971)
        at com.dremio.plugins.s3.store.S3FSHealthChecker.healthCheck(S3FSHealthChecker.java:56)
        at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:555)
        at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:572)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.UnknownHostException: dremio.minio.chariot
        at java.net.InetAddress.getAllByName0(InetAddress.java:1287)
        at java.net.InetAddress.getAllByName(InetAddress.java:1199)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
        at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
        at com.amazonaws.http.conn.$Proxy97.connect(Unknown Source)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
        ... 20 common frames omitted
2022-10-08 21:31:41,615 [start-__home] WARN  c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __home
java.lang.Exception: Unavailable: com.amazonaws.SdkClientException: Unable to execute HTTP request: dremio.minio.chariot
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-08 21:31:41,615 [start-__home] INFO  c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-__home
2022-10-08 21:32:32,373 [catalog-source-synchronization] WARN  c.dremio.exec.catalog.PluginsManager - Skipping synchronizing source __accelerator since it's being modified
2022-10-08 21:34:19,657 [start-__accelerator] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /dremio/accelerator
com.amazonaws.SdkClientException: Unable to execute HTTP request: dremio.minio.chariot: Name or service not known
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1162)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5386)
        at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:971)
        at com.dremio.plugins.s3.store.S3FSHealthChecker.healthCheck(S3FSHealthChecker.java:56)
        at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:555)
        at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:572)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.UnknownHostException: dremio.minio.chariot: Name or service not known
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1330)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1283)
        at java.net.InetAddress.getAllByName(InetAddress.java:1199)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
        at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
        at com.amazonaws.http.conn.$Proxy97.connect(Unknown Source)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1343)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
        ... 20 common frames omitted
2022-10-08 21:34:19,657 [start-__accelerator] WARN  c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __accelerator
java.lang.Exception: Unavailable: com.amazonaws.SdkClientException: Unable to execute HTTP request: dremio.minio.chariot: Name or service not known
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-08 21:34:19,657 [start-__accelerator] INFO  c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-__accelerator
2022-10-08 21:35:32,373 [catalog-source-synchronization] WARN  c.dremio.exec.catalog.PluginsManager - Skipping synchronizing source $scratch since it's being modified
2022-10-08 21:35:39,166 [start-$scratch] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /dremio/scratch
com.amazonaws.SdkClientException: Unable to execute HTTP request: dremio.minio.chariot
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1216)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1162)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5386)
        at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:971)
        at com.dremio.plugins.s3.store.S3FSHealthChecker.healthCheck(S3FSHealthChecker.java:56)
        at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:555)
        at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:572)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.net.UnknownHostException: dremio.minio.chariot

Is dremio. supposed to be prepended to my endpoint value of mino.chariot?

look the same topic:

Ok I somewhat fixed but not by the fix above, minio gave errors when specifying a host for --address. I instread updated my core dns with rewrite name dremio.minio.chariot.svc.cluster.local minio.chariot.svc.cluster.local

It now can resolve the host but I get the following errors

2022-10-12 18:48:49,001 [scheduler-1] WARN  c.d.e.catalog.MetadataSynchronizer - Source 'sys' sync failed unexpectedly. Will try again later
java.lang.NullPointerException: Master coordinator is down
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
        at com.dremio.service.conduit.client.ConduitProviderImpl.getOrCreateChannelToMaster(ConduitProviderImpl.java:175)
        at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightClient(SysFlightStoragePlugin.java:128)
        at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightTableList(SysFlightStoragePlugin.java:258)
        at com.dremio.plugins.sysflight.SysFlightStoragePlugin.listDatasetHandles(SysFlightStoragePlugin.java:210)
        at com.dremio.exec.catalog.MetadataSynchronizer.getDatasetHandleListing(MetadataSynchronizer.java:173)
        at com.dremio.exec.catalog.MetadataSynchronizer.synchronizeDatasets(MetadataSynchronizer.java:186)
        at com.dremio.exec.catalog.MetadataSynchronizer.go(MetadataSynchronizer.java:134)
        at com.dremio.exec.catalog.SourceMetadataManager$RefreshRunner.refreshFull(SourceMetadataManager.java:441)
        at com.dremio.exec.catalog.SourceMetadataManager$AdhocRefresh.run(SourceMetadataManager.java:507)
        at com.dremio.exec.catalog.SourceMetadataManager.refresh(SourceMetadataManager.java:188)
        at com.dremio.exec.catalog.ManagedStoragePlugin.refresh(ManagedStoragePlugin.java:1119)
        at com.dremio.exec.catalog.CatalogServiceImpl.refreshSource(CatalogServiceImpl.java:377)
        at com.dremio.exec.catalog.CatalogServiceImpl.lambda$start$0(CatalogServiceImpl.java:267)
        at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:226)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,173 [start-__home] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /dremio/uploads
java.io.FileNotFoundException: Path not found: /dremio/uploads
        at com.dremio.plugins.s3.store.S3FSHealthChecker.healthCheck(S3FSHealthChecker.java:58)
        at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:555)
        at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:572)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,174 [start-__home] WARN  c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __home
java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/uploads
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,174 [start-__home] INFO  c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-__home
2022-10-12 18:48:50,225 [main] ERROR c.dremio.exec.catalog.PluginsManager - Exception while creating source.
com.dremio.common.exceptions.UserException: Source is not currently available.
        at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:890)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:436)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createSource(ManagedStoragePlugin.java:328)
        at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:174)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:424)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSourceIfMissingWithThrow(CatalogServiceImpl.java:412)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.createOrUpdateSystemSource(SystemStoragePluginInitializer.java:236)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.createSafe(SystemStoragePluginInitializer.java:216)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.pluginsCreation(SystemStoragePluginInitializer.java:136)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.initialize(SystemStoragePluginInitializer.java:105)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.initialize(SystemStoragePluginInitializer.java:59)
        at com.dremio.service.InitializerRegistry.start(InitializerRegistry.java:69)
        at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
        at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:88)
        at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
        at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:196)
        at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:202)
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:104)
Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/uploads
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
        at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1046)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:369)
        ... 16 common frames omitted
        Suppressed: java.lang.NullPointerException: null
                at com.dremio.exec.store.dfs.FileSystemPlugin.getSourceCapabilities(FileSystemPlugin.java:389)
                at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:951)
                at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1061)
                ... 17 common frames omitted
Caused by: java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/uploads
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,351 [start-__accelerator] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /dremio/accelerator
java.io.FileNotFoundException: Path not found: /dremio/accelerator
        at com.dremio.plugins.s3.store.S3FSHealthChecker.healthCheck(S3FSHealthChecker.java:58)
        at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:555)
        at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:572)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,351 [start-__accelerator] WARN  c.d.e.catalog.ManagedStoragePlugin - Error starting new source: __accelerator
java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/accelerator
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,351 [start-__accelerator] INFO  c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-__accelerator
2022-10-12 18:48:50,353 [main] ERROR c.dremio.exec.catalog.PluginsManager - Exception while creating source.
com.dremio.common.exceptions.UserException: Source is not currently available.
        at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:890)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:436)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createSource(ManagedStoragePlugin.java:328)
        at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:174)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:424)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSourceIfMissingWithThrow(CatalogServiceImpl.java:412)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.createOrUpdateSystemSource(SystemStoragePluginInitializer.java:236)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.createSafe(SystemStoragePluginInitializer.java:216)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.pluginsCreation(SystemStoragePluginInitializer.java:150)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.initialize(SystemStoragePluginInitializer.java:105)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.initialize(SystemStoragePluginInitializer.java:59)
        at com.dremio.service.InitializerRegistry.start(InitializerRegistry.java:69)
        at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
        at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:88)
        at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
        at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:196)
        at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:202)
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:104)
Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/accelerator
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
        at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1046)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:369)
        ... 16 common frames omitted
        Suppressed: java.lang.NullPointerException: null
                at com.dremio.exec.store.dfs.FileSystemPlugin.getSourceCapabilities(FileSystemPlugin.java:389)
                at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:951)
                at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1061)
                ... 17 common frames omitted
Caused by: java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/accelerator
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,446 [start-$scratch] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /dremio/scratch
java.io.FileNotFoundException: Path not found: /dremio/scratch
        at com.dremio.plugins.s3.store.S3FSHealthChecker.healthCheck(S3FSHealthChecker.java:58)
        at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:555)
        at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:572)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,447 [start-$scratch] WARN  c.d.e.catalog.ManagedStoragePlugin - Error starting new source: $scratch
java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/scratch
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,447 [start-$scratch] INFO  c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-$scratch
2022-10-12 18:48:50,449 [main] ERROR c.dremio.exec.catalog.PluginsManager - Exception while creating source.
com.dremio.common.exceptions.UserException: Source is not currently available.
        at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:890)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:436)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createSource(ManagedStoragePlugin.java:328)
        at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:174)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:424)
        at com.dremio.exec.catalog.CatalogServiceImpl.createSourceIfMissingWithThrow(CatalogServiceImpl.java:412)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.createOrUpdateSystemSource(SystemStoragePluginInitializer.java:236)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.createSafe(SystemStoragePluginInitializer.java:216)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.pluginsCreation(SystemStoragePluginInitializer.java:161)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.initialize(SystemStoragePluginInitializer.java:105)
        at com.dremio.dac.daemon.SystemStoragePluginInitializer.initialize(SystemStoragePluginInitializer.java:59)
        at com.dremio.service.InitializerRegistry.start(InitializerRegistry.java:69)
        at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
        at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:88)
        at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
        at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:196)
        at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:202)
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:104)
Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/scratch
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
        at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1046)
        at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:369)
        ... 16 common frames omitted
        Suppressed: java.lang.NullPointerException: null
                at com.dremio.exec.store.dfs.FileSystemPlugin.getSourceCapabilities(FileSystemPlugin.java:389)
                at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:951)
                at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1061)
                ... 17 common frames omitted
Caused by: java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/scratch
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

However the paths are clearly there

[centos@launcer ~]$ mc ls myminio/dremio/
Handling connection for 9000
[2022-10-12 18:52:56 UTC]     0B accelerator/
[2022-10-12 18:52:56 UTC]     0B downloads/
[2022-10-12 18:52:56 UTC]     0B dremio/
[2022-10-12 18:52:56 UTC]     0B metadata/
[2022-10-12 18:52:56 UTC]     0B scratch/
[2022-10-12 18:52:56 UTC]     0B uploads/

@ajsalow Does yur dist path in dremio.conf have the full path?

dist: "dremioS3:///<bucket_name>/<folder1>/<folder2>"

You have

    bucketName: "dremio"
    path: "/"

So as per chart it is starting at /dremio

But on Minio is it not myminio/dremio/uploads

myminio is just an alias used by minio client cli to the hostname of that minio instance. So the command mc ls myminio/dremio/ (note no subpath “upload” here) is showing contents of the root level bucket dremio. As mentioned all of those subpaths (besides the dremio subpath) were created by dremio on initialization. I’m not sure what specifying subpaths would do. As noted in the log it is not just the /uploads path that has an issue.

Caused by: java.lang.Exception: Unavailable: java.io.FileNotFoundException: Path not found: /dremio/accelerator
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:577)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-10-12 18:48:50,446 [start-$scratch] ERROR c.d.p.s3.store.S3FSHealthChecker - Error while listing S3 objects in /dremio/scratch
java.io.FileNotFoundException: Path not found: /dremio/scratch
        at com.dremio.plugins.s3.store.S3FSHealthChecker.healthCheck(S3FSHealthChecker.java:58)
        at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:555)
        at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:572)
        at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

but /dremio/uploads, /dreimio/scratch, etc. are there.

@ajsalow You’re probably running into a bug in v22.1.1
See this post for ideas: Dremio fails to start after upgrading to 22.1 - #2 by dch