I’m new to Dremio. I found a cache file in the dremioarrow1 format. How can I read its contents, for example, in C# outside of Dremio? The apache.arrow driver doesn’t work with this file.
@Tau Those files are Dremio’s internal representation of cached blocks of Iceberg, Parquet or Delta Lake data on your distributed store so next time the query comes and data is not changed, Dremio avoids reading from object store and instead reads from local SSD or Nvme
Thank you for hour answers.
I know that. But cache file has more than 80GB. For me is faster and chipper to get file from cache than get data by API.
@Tau You do not have to worry about that, just issue the query and if the file is present in cache Dremio will read it from cache and avoid a read to your object store
@balaji.ramaswamy
sorry I didn’t explain it clearly. I know what you’re talking about. but I have many clients who would have to download the 80GB from the cloud. I don’t want to pay for multiple downloads of the same data every day by many clients. I want to download it on-prem once (this cache file) and share this data. if it were possible to read this data immediately, then it wouldn’t even be necessary to convert it to another format or load it into the database.
@Tau Understood, this is nor possible due to 2 reasons
- These files are not readable by other query engines
- They are split across executors
Can you not do a CTAS once and share the iceberg files?