CTAS issue with 14.0 : FileNotFound exception

1

Hi,

since using Dremio Community 14, I encounter CTAS issues.

Error that is raised is
java.io.FileNotFoundException: No such file or directory
indicating an S3 parquet file path.

When looking into the contents of this directory, all other files of the partition exists excepted the one indicated in the message.
image

Here is the query profile :
e6051b02-a519-43d8-b469-6da9675ccf1c.zip (44.5 KB)

Could this come from an S3 setting ?

Each time I drop the directory (using DROP TABLE ) and retries, same message occur, but not necessary with the same file error.

THanks

1 Like
2

@dfleckinger

Have tried to refresh metadata and retry the command?

3

The parquet mentionned in the error is not part of a physical sources that the query is supposed to read.

The error message here indicates the parquet file 1_5_0 , that the CTAS is supposed to create.

4

@dfleckinger

Thanks for that info, let me check on this and get back to you

5

@dfleckinger

Is there a chance the source the CTAS is querying from has Parquet files that have different schema?

6

It is even sure ! In previous version 4.9 CE this did not prevent the CTAS from working successfully.
Some columns of the source parquet files have different formats, but the mixed type columns are corrected by a VDS and it is that VDS that is queried.
Is there a workaround, except fixing the parquet files ?

7

@dfleckinger Sorry for the back and forth but from the profile it seems you are selecting from the PDS, is that true?

8

Thanks for the follow up.
You’re right, I probably used the original query in the AS SELECT part as part of the debugging, I didn’t know this could have an impact.

Interesting finding : I’ve just found a workaround, by adding

where ts > ‘2013-01-01’

at the end of the same query, and the CTAS could work quickly and without issue. :face_with_monocle:

9

Thanks for the update @dfleckinger