Unable to create reflections with Dremio Community Edition

1

We are unable to create reflections using:
Build
21.1.1-202204292111390812-57b1832f
Edition
Community Edition
the /opt/dremio/data/pdfs/accelerator folder is empty and we’re seeing the following error in the jobs listing:
UnknownFormatConversionException: Conversion = ‘Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/c55d9ef6-a693-4681-9c54-92f965a925ab/4493ca78-07d0-43e9-b460-b0999e8883c4_0/metadata/ded7085b-3236-4b07-b73d-53c204662d1b.avro Error Message : No File System scheme matches’

2

I have the same problem

3

disable dremio.iceberg.enabled
or use s3 or hdfs… as paths.dist

1 Like
4

I don’t see an option for disabling dremio.iceberg

5

I tried setting paths.dist and that didn’t work. I don’t see an option to set dremio.iceberg.enabled: false. I’m still unable to create even simple raw reflections.

UnknownFormatConversionException: Conversion = ‘Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/0ce84a34-10cd-4625-8809-12d7e213e101/c31427de-6e48-4216-9536-3997634e8665_0/metadata/1f9e55dd-2281-4942-a3de-84b684819eab.avro Error Message : No File System scheme matches’

6

https://docs.dremio.com/software/advanced-administration/support-settings/

7

Yes, see more here : https://docs.dremio.com/software/release-notes/210-release/#2-pdfs-is-not-supported-for-distributed-storage-in-versions-2100-and-above

Did they give any reason ?

8

I don’t see these options in Community Edition

9

I’m trying to save reflections to the local drive and not a distributed file system. Is this a problem?

10

yes, this is not possible since version 21.

you can also fix the configuration like this :

alter system set dremio.iceberg.enabled = true/false;

alter system set dremio.execution.support_unlimited_splits = true/false;

alter system set dremio.iceberg.ctas.enabled = true/false;
1 Like
11

I figured it out and it works! Thanks!

12

@summersmd

From 21.x PDFS is no longer supported so dist:// has to be pointed to a S3 or ADLS or HDFS or GCS

13

Disabling iceberg worked for me.

14

the best approach is use NAS, if you disable iceberg, basically are truncate list of current and future benefits of dremio

15 
paths.dist=file:///opt/dremio/data/pdfs

test ok

2 Likes
16

I am facing the same problem. How did you solve the problem?

17

@vincent_mayer Dremio 21.x and above you have to write your reflections to a DIST store or at least NAS. Can you please send your dremio.conf and we can validate?

18

Hi,

I am also facing the same problem

paths: {

the local path for dremio to store data.

local: ${DREMIO_HOME}“/data”

local: “/opt/dremio/data/”

the distributed path Dremio data including job results, downloads, uploads, etc

#dist: “pdfs://”${paths.local}“/pdfs”
}

services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true,
flight.use_session_service: true
}
registration.publish-host: “0.0.0.0”

19

I could able to solve by configuring dist as my dremio version is 24

20

@SAIDULU That is correct, going forward, dist store needs to be configured to a distributed store like S3, Azure etc