with docker running dremio 18.1. after add s3 datasource then create reflection. but view job info
UnknownFormatConversionException: Conversion = 'Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/f8e3d329-bb34-4b3d-9ba6-15fb55131a07/870f6a52-7442-4e29-b7d3-73bd22077d2b_0/metadata/17e9f356-4b84-4156-9a72-34f05eda7a86.avro Error Message : No File System scheme matches'
maybe same as Reflection Run Failure - SYSTEM ERROR but running with docker first time
@rongfengliang Can you please send us your dremio.conf?
i just run with docker
docker run -p 9047:9047 dremio/dremio-oss:18.1
inside your container, under conf folder, are you able to cat dremio.conf and send us the output?
i do some test if i enable dremio.execution.support_unlimited_splits and dremio.iceberg.enabled this exception will happen
@rongfengliang The error is coming from your scheme defined for your distributed storage like reflections. So want to check if the scheme is defined right in dremio.conf
@balaji.ramaswamy thinks. the data source is minio. if I enable dremio.execution.support_unlimited_splits and dremio.iceberg.enabled this will happen and dremio.conf just use docker default conf. I have test 18.1 and 18.2 all have the same problem
@rongfengliang So if you turn off the above 2 parameters, the issue goes away?
yes. if I disable above 2 parameters the issue goes away
@rongfengliang Yes found the issue, in order to enable Iceberg metadata Dremio needs a distributed store like S3/ADLS/HDFS
Hi @balaji.ramaswamy , does this requirement still hold? Or can we use NAS ?
@shay I changed my reply. As Balaji mentions above, PDFS is not supported for distributed storage in versions 21.0.0 and above. Use a real distributed store like S3 or HDFS and enjoy the benefits of unlimited splits and Iceberg metadata.
yes, from 21.0 dremio, pdfs with local not support must use distributed store like s3、nas、hdfs…
