How does one use the Hudi real-time view in Dremio to get an actual real-time view?

1

We are evaluating Dremio to see if it meets our requirements, one of which involves querying near real-time updated data. We currently intend to stream data into our data lake using Hudi and we need the real-time views produced/maintained by Hudi to be real-time - not updated on a timer as per the current Dremio metadata refresh mechanism. How does one use the Hudi real-time view in Dremio to get an actual real-time view?

There is a post "https://community.dremio.com/t/s3-streaming-json-files-in-a-bucket-as-new-documents-records/3511/8” discussing a very similar scenario, which finishes with @balaji.ramaswamy suggesting a new feature is coming that we can “Expect V1 by EOY 2020”. Is there an update on this? and will this new feature suit our scenario?

2

@jarred

Apache Iceberg implementation that does not require metadata will be later this year. For now what you can do is refresh metadata via the SQL after the lake has been refreshed and Dremio should immediately know about the data

https://docs.dremio.com/sql-reference/sql-commands/datasets.html