IOException: /test-datalake/ecommerce_db/product_facts_f35397dd-2c6c-4bc3-9a17-cd759207bf1f/data/retailer=amazon.com/date=2024-02-15/00145-8-7c8c3206-f708-476e-9042-54038b84f72f-00001.parquet is not a Parquet file. expected magic number [80, 65, 82, 49] at tail, but found [0, 0, 0, 0]
I’m using PySpark, Minio, and Nessie to create an iceberg table. When I’m testing the tables and the pipeline, I find this error.
I tried the suggestion in another thread about this, but that’s not working on my issue.