Dremio 18.1 create reflection Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/

with docker running dremio 18.1. after add s3 datasource then create reflection. but view job info

UnknownFormatConversionException: Conversion = 'Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/f8e3d329-bb34-4b3d-9ba6-15fb55131a07/870f6a52-7442-4e29-b7d3-73bd22077d2b_0/metadata/17e9f356-4b84-4156-9a72-34f05eda7a86.avro Error Message : No File System scheme matches'

maybe same as Reflection Run Failure - SYSTEM ERROR but running with docker first time

@rongfengliang Can you please send us your dremio.conf?

i just run with docker

docker run -p 9047:9047   dremio/dremio-oss:18.1

inside your container, under conf folder, are you able to cat dremio.conf and send us the output?

i do some test if i enable dremio.execution.support_unlimited_splits and dremio.iceberg.enabled this exception will happen

@rongfengliang The error is coming from your scheme defined for your distributed storage like reflections. So want to check if the scheme is defined right in dremio.conf

@balaji.ramaswamy thinks. the data source is minio. if I enable dremio.execution.support_unlimited_splits and dremio.iceberg.enabled this will happen and dremio.conf just use docker default conf. I have test 18.1 and 18.2 all have the same problem

@rongfengliang So if you turn off the above 2 parameters, the issue goes away?

yes. if I disable above 2 parameters the issue goes away

@rongfengliang Yes found the issue, in order to enable Iceberg metadata Dremio needs a distributed store like S3/ADLS/HDFS

1 Like