Reflections failing with Exception

Dremio University
1

Hey,

I am using dremio 21.1 in a docker environment with a docker-compose file:

...
  dremio-app:
    container_name: dremio_app
    env_file: .env-dremio
    image: dremio/dremio-oss:21.1
    ports:
      - 9047:9047
      - 31010:31010
      - 45678:45678
    expose:
      - "9047"
    volumes:
      - dremio_app:/opt/dremio/data
      - ./dremio/backup:/tmp/backup:z
    restart: unless-stopped
    depends_on:
      - dremio-app-init

I have 1 external source set up and a handfull of virtual datasets. I have had configured one or more reflections for every datasource wich worked fine in version 20.0. Upgrading was no issue at all, I used dremio-admin upgrade and dremio-admin restore to move my data to the new version. Now I am having trouble because every reflection fails with an error similar to this:

UnknownFormatConversionException: Conversion = 'Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/528a8fe8-2e96-4b33-8169-66732e4c7346/3fa762bb-a1f5-4643-8612-44deefd76de8_0/metadata/3289920b-ddac-4889-ad80-f71f64dc5402.avro Error Message : No File System scheme matches'

My dremio.conf file looks like this:


paths: {
  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true,
  flight.use_session_service: true
}

I have downloaded the profile for one of the datasets where the reflection fails: f2293d2d-bb67-477f-8ad0-e840b804e2de.zip (18,6 KB)

I already deleted the volume, restored my backup, but the same problem persists.

Hope someone can help me out!

2

@tha Starting Dremio 21.0 PDFS is no longer supported to store reflections, see release note below

https://docs.dremio.com/software/release-notes/210-release/#2-pdfs-is-not-supported-for-distributed-storage-in-versions-2100-and-above

If you see your dremio.conf, you are writing reflections to local and commented out dist

1 Like
3

Hi, Iā€™m new to Dremio and my client is setting Dremio 22.1.1.
I keep experiencing this error message too:

image
image842Ɨ45 5.79 KB 

UnknownFormatConversionException: Conversion = 'Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/d564689c-69a9-46c2-a7da-2c16e2dab476/d5f1967e-8bd9-49d6-988a-d6b396d17a27_0/metadata/8ff37c6d-88d2-4c86-aa92-48aae6cffc0e.avro Error Message : No File System scheme matches'

This is what I see in the dremio.conf file:

paths: {\r
  # Local path for dremio to store data.\r
  local: ${DREMIO_HOME}\"/data\"\r
  # Distributed path Dremio data including job results, downloads,\r
  # uploads, etc\r
  dist: \"pdfs://\"${paths.local}\"/pdfs\"\r
}\r

I would like to advise our architect on what to do to solve this.
Could you help me here @balaji.ramaswamy ?

4

If you want to use network attached storage for your distributed store, then set paths.dist to something like:

file:///Users/bennychow/data/