Unable to create reflections with Dremio Community Edition

Yes, see more here : https://docs.dremio.com/software/release-notes/210-release/#2-pdfs-is-not-supported-for-distributed-storage-in-versions-2100-and-above

Did they give any reason ?

I don’t see these options in Community Edition

I’m trying to save reflections to the local drive and not a distributed file system. Is this a problem?

yes, this is not possible since version 21.

you can also fix the configuration like this :

alter system set dremio.iceberg.enabled = true/false;

alter system set dremio.execution.support_unlimited_splits = true/false;

alter system set dremio.iceberg.ctas.enabled = true/false;
1 Like

I figured it out and it works! Thanks!

@summersmd

From 21.x PDFS is no longer supported so dist:// has to be pointed to a S3 or ADLS or HDFS or GCS

Disabling iceberg worked for me.

the best approach is use NAS, if you disable iceberg, basically are truncate list of current and future benefits of dremio

paths.dist=file:///opt/dremio/data/pdfs

test ok

2 Likes

I am facing the same problem. How did you solve the problem?

@vincent_mayer Dremio 21.x and above you have to write your reflections to a DIST store or at least NAS. Can you please send your dremio.conf and we can validate?

Hi,

I am also facing the same problem

paths: {

the local path for dremio to store data.

local: ${DREMIO_HOME}“/data”

local: “/opt/dremio/data/”

the distributed path Dremio data including job results, downloads, uploads, etc

#dist: “pdfs://”${paths.local}“/pdfs”
}

services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true,
flight.use_session_service: true
}
registration.publish-host: “0.0.0.0”

I could able to solve by configuring dist as my dremio version is 24

@SAIDULU That is correct, going forward, dist store needs to be configured to a distributed store like S3, Azure etc

Hi Balaji,

We have installed the Dremio on the Oracle Cloud instance and I have configured the dist with S3, but when I am running the reflections I am getting following exception.

(java.util.UnknownFormatConversionException) Conversion = ‘Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/8b5e4a94-6fd2-4048-a55f-ce857e238780/110efb81-8844-413e-9def-df165898286a_0 Error Message : No File System scheme matches’

below is the config file.
paths: {
#the local path for dremio to store data.
#local: ${DREMIO_HOME}“/data”
#local: “/storage/opt/dremio/data/”

the distributed path Dremio data including job results, downloads, uploads, etc

#dist: “pdfs://”${paths.local}“/pdfs”
dist: “dremioS3:///#bucketname#/#folder#/”

}

services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true,
flight.use_session_service: true
}
registration.publish-host: “0.0.0.0”

@SAIDULU Did you restart Dremio after the change?

HI @balaji.ramaswamy

Yes, I have done it but no luck.

I did a search in the community and disabled the support keys in the settings then it worked.

Thanks
Saidulu

@SAIDULU YEs that will work as now you are putting your reflections on local. This is not recommended/supported architecture anymore. Are you able to read a file from the S3 location you have given from AWS CLI?

@balaji.ramaswamy

I have not configured to put the reflections on locally. I have commented line in the config file but still, it didn’t work so had to disable the properties. Will there be any issue if we disable those three options.?

And yes I am able to read/access the S3 from CLI

paths: {

the local path for dremio to store data.

local: ${DREMIO_HOME}“/data”

local: “/storage/opt/dremio/data/”

the distributed path Dremio data including job results, downloads, uploads, etc

#dist: “pdfs://”${paths.local}“/pdfs”
dist: “dremioS3:///MyBucket/MyFolder/”

}

Thanks
Saidulu

@SAIDULU

Run any job in Dremio and splease send us the job profile. That should give us some indication, if possible send us the full dremio.conf