Hi Guys, I have many tables on my lakehouse and just one table have a problem. This project have Delta Tables and all tables are running fine with Dremio, but sometimes one table return the following error:
IOException: /lakehouse/gold/dim_consultores/_delta_log/00000000000000000130.checkpoint.parquet is not a Parquet file. expected magic number [80, 65, 82, 49] at tail, but found [48, 49, 48, 48]
Can someone help me?
thank’s
You can Google that “expected magic number” error. In general, it usually means that the Parquet file has bad data.
@Ferrarini Is it possible the file was overwritten with the same file name but different file size? Can you try doing the below?
ALTER PDS <PDS_NAME> FORGET METADATA
ALTER PDS <PDS_NAME> REFRESH METADATA
Thanks
Bali
Yes, I 'm doing merge in my table.
@Ferrarini Did a FORGET/REFRESH help?
Refresh/forget not run with same error
@Ferrarini Sorry, I am unable to follow. Refresh/Forget not run?
Sorry, FORGET/REFRESH don’t help. I’ve some error.
@Ferrarini Are you saying same error? IF same error then can another tool read the same file? If some error, can you please send us the new error?
Hi @balaji.ramaswamy I have same error. I can read the table using Spark, but I can’t read the same table with Dremio. I’m using Dremio OSS 25.0 on Docker. Can you help me?
@Ferrarini Are you able to provide us with the checkpoint parquet file so we can investigate the issue?
/lakehouse/gold/dim_consultores/_delta_log/00000000000000000130.checkpoint.parquet
@Ferrarini We also found a bug in this path on 25.0 when C3 is enabled, Would you be able to turn off C3 and rey the query to see if it is successful?