How to: multiple views on the same physical file?

1

Trying to work out the best way to create multiple “views” on the same source file.

Let me illustrate with an example: take an image file (jpeg) which is on S3 source.

  1. Is there a way to promote it multiple times, each time in a different format? For example: promote the image as metadata text format once, and also promote the same image file as image format next time etc.
  2. If promoting multiple times is not possible, is there a way we could achieve this through “virtual views” etc.?

The goal is to be able to offer “meta data textual” view as well as the “image” view of the same file, so that the file can be searched both by metadata, and image content also. (i.e. meta data will be indexed in a table, as well as “image” binary content also will be indexed (somehow through custom indexers etc.), separately ofcourse in two different data sets)

If currently not possible through UI, any means to achieve this through plugin/adapter etc? Any info on that would be very helpful. Thank you.

2

@KrishnaPG Currently Dremio supports only the below formats during promotion

image

Iceberg is not yet there, I am not sure, what you mean by image format?

3

Thank you @balaji.ramaswamy in that case, how do we add custom formatter options?

I guess it has to be done at the Dremio code level. How easy / difficult it is? Any info / examples would of great help.

4

Hi @balaji.ramaswamy ,

Iceberg is already supported isn’t it ? It’s Delta Lake which isn’t yet ?
I noticed that you seems to have it (Delta Lake) already, do you know if it may be available in the next release ? I expected to see end of May but it doesn’t seems to has been release yet, have you any date in mind ?

Thanks

5

@YEN Its the other way, Dremio

6

Thank you very much,
I missed this part:

image
image1307×78 14.1 KB

I will give a try, thanks!