Programatically Load Data

1

Hi,

I’m trying to look for resources on how I can say load Dataset to Dremio via some sort of a feed that update regularly. How is this achieved? I can’t seem to find a way to write this. I have check the REST api – I saw I can create a DataSet using a catalog. But what will I use to add data to this?

Thanks for any pointers.

2

@w13bit

Dremio is not an ETL tool or data generation product. You can use Dremio to query data on your lake like Hadoop/S3/ADLS/RDBMS/MongoDB etc. We can join across sources and create reflections to accelerate query execution speeds. We can also write back results of a query back to the lake as PARQUET files

Kindly let me know if you have any other questions

Thanks
@balaji.ramaswamy

3

We have some files that is stored in some proprietary format and stored in a key/value store. Ideally I wanted this to be able to be queried by some sql ( these files contains report generated at x interval per day). I was hoping not to parse the files and load it to say a postgresqldb and have dremio access that. Is it possible if I have it transformed to csv and then have dremio read the csv files?