How does COPY INTO transfer parquet files? Let’s say I have parquet files in S3 source and run COPY INTO Nessie on Iceberg. Will this command just transfer files to Nessie source and add metadata, or it will load data in Dremio memory first and then insert, like Spark Dataframes? Basically the question is about a computational cost. Is COPY INTO expensive? Especially with extra large parquet tables, will it require more DCU depending on a dataset size? Thanks!
@enkeipy It is difficult to exactly tell how many DCUs but COPY INTO will involve computational cost… as it requires Dremio to read the input file and write the output file. Let me give this thread to the PM who is responsible for this and get back to you with more information
1 Like
Thank you. That answers my question.