Reflections failing with Exception

Hey,

I am using dremio 21.1 in a docker environment with a docker-compose file:

...
  dremio-app:
    container_name: dremio_app
    env_file: .env-dremio
    image: dremio/dremio-oss:21.1
    ports:
      - 9047:9047
      - 31010:31010
      - 45678:45678
    expose:
      - "9047"
    volumes:
      - dremio_app:/opt/dremio/data
      - ./dremio/backup:/tmp/backup:z
    restart: unless-stopped
    depends_on:
      - dremio-app-init

I have 1 external source set up and a handfull of virtual datasets. I have had configured one or more reflections for every datasource wich worked fine in version 20.0. Upgrading was no issue at all, I used dremio-admin upgrade and dremio-admin restore to move my data to the new version. Now I am having trouble because every reflection fails with an error similar to this:

UnknownFormatConversionException: Conversion = 'Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/528a8fe8-2e96-4b33-8169-66732e4c7346/3fa762bb-a1f5-4643-8612-44deefd76de8_0/metadata/3289920b-ddac-4889-ad80-f71f64dc5402.avro Error Message : No File System scheme matches'

My dremio.conf file looks like this:


paths: {
  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true,
  flight.use_session_service: true
}

I have downloaded the profile for one of the datasets where the reflection fails: f2293d2d-bb67-477f-8ad0-e840b804e2de.zip (18,6 KB)

I already deleted the volume, restored my backup, but the same problem persists.

Hope someone can help me out!

@tha Starting Dremio 21.0 PDFS is no longer supported to store reflections, see release note below

https://docs.dremio.com/software/release-notes/210-release/#2-pdfs-is-not-supported-for-distributed-storage-in-versions-2100-and-above

If you see your dremio.conf, you are writing reflections to local and commented out dist