How does index of data source work with reflection

1

After a Raw Reflection was created for a particular data source,
when i run a query, reflection is chosen and works.
But how does index of data source work, and what role it play as?

In addition, will reflection update automatically?
if so, will change of data source trigger reflection update?

Thanks!

2

Hi @jerryldh

Let me first address your second question. Change of data source and reflection refresh are 2 completely separate events

  1. When something changes in your source it is driven by the metadata refresh settings on the source, see different options below

Source Metadata Caching

  1. Reflection refresh is a separate tab under the source settings drives how frequently you want to refresh. This can also be set at the dataset level

Now to answer your first question, that purely depends on the source. For a source like RDBMS we would push down the query the reflection is built on. If for example you have an index on a date column and it is used in the where clause of the reflection definition and we end up pushing it down then we would do an INDEX_RANGE_SCAN

If the reflection is not based on a filter we would end up doing a FTS (Full Table Scan)

The alternate would be to do an incremental refresh although we have some restrictions

Incremental Refresh

What is your data source?

Thanks
@balaji.ramaswamy

1 Like
3

Thanks @balaji.ramaswamy
your reply helps a lot.
my data source is a large number of files(csv， parquet，etc.)

Thanks!