Are there something like delta lake or Hive ACID transactions?

1

Hello Dremio Team, with the open sourcing of the databricks delta as delta lake I’d like to know if is there a way for dremio to recognize the delta parquet files and the updated files as the last version.
Also, I’d like to know if is there some planning to release a feature where we can update, merge parquet/orc files using dremio.

Thanks in advance.

2

We are watching Delta, especially for S3 and ADLS, which have not yet been open sourced.

In Hadoop transactional ORC is the primary approach we see, and this is something Dremio supports today.

It is unlikely Dremio will be able to update files directly in the near term (you will do this via Hive/Sark/ETL). Of course you can use CTAS today, but I don’t think that’s what you’re asking.

3

@kelly
Hi, is there a plan for delta lake supporting ?

4

@koolay

Delta lake support is planned for the first half of next year, there is not definite ETA right now

Thanks
Bali