with docker running dremio 18.1. after add s3 datasource then create reflection. but view job info
UnknownFormatConversionException: Conversion = 'Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/f8e3d329-bb34-4b3d-9ba6-15fb55131a07/870f6a52-7442-4e29-b7d3-73bd22077d2b_0/metadata/17e9f356-4b84-4156-9a72-34f05eda7a86.avro Error Message : No File System scheme matches'
maybe same as Reflection Run Failure - SYSTEM ERROR but running with docker first time
@rongfengliang Can you please send us your dremio.conf?
i just run with docker
docker run -p 9047:9047 dremio/dremio-oss:18.1
inside your container, under conf folder, are you able to cat dremio.conf and send us the output?
i do some test if i enable dremio.execution.support_unlimited_splits and dremio.iceberg.enabled this exception will happen
@rongfengliang The error is coming from your scheme defined for your distributed storage like reflections. So want to check if the scheme is defined right in dremio.conf
@balaji.ramaswamy thinks. the data source is minio. if I enable
dremio.iceberg.enabled this will happen and dremio.conf just use docker default conf. I have test 18.1 and 18.2 all have the same problem
@rongfengliang So if you turn off the above 2 parameters, the issue goes away?
yes. if I disable above 2 parameters the issue goes away
@rongfengliang Yes found the issue, in order to enable Iceberg metadata Dremio needs a distributed store like S3/ADLS/HDFS
Hi @balaji.ramaswamy , does this requirement still hold? Or can we use NAS ?
@shay I changed my reply. As Balaji mentions above, PDFS is not supported for distributed storage in versions 21.0.0 and above. Use a real distributed store like S3 or HDFS and enjoy the benefits of unlimited splits and Iceberg metadata.