Scalability in creating reflections

1

Creating Reflections Not Performed Distributed on Nodes. Is this behavior normal?

2

@alex_lopes You mean the materialization files are heavily concentrated on one or couple of nodes and other nodes not hosting at all or less ?

Is this is a PDFS ? or HDFS ?

@Venugopal_Menda

3

I refer at the time of the creation of the reflection. When I see the execution plan I never see two or more executor clusters working on creating a reflection always one node. I’m working with one coordinator node and four executors with Deploy at Azure. Is this processing to be distributed among clusters?

4

@alex_lopes, Dremio will fully parallelize execution for many operations. Can you share a profile for one of the REFLECTION REFRESH jobs that you think isn’t using the full width of your cluster?

5

31
Captura de Tela 2019-10-29 às 14.01.31.png2588×1432 219 KB
54
Captura de Tela 2019-10-29 às 14.01.54.png2594×1438 193 KB

6

@ben
As you can see in the image, the reflection creation process is using only one node… I Have four nodes…

7

@alex_lopes, can you attach the profile, I’d like to take a look:

8

profile_reflection.zip (15,7,KB)

Thanks @ben

9

@alex_lopes, because the source is against a single table (no joins) in a RDBMS (Oracle, in this case), the read will be single threaded and subsequent operations will be single threaded. If you were reading from files on a filesystem source, your would see parallelization.

1 Like
10

@ben Makes sense! Thank you for the informations!