We have observed that the JDBC scan done by the ARP framework when reading a table (reflection creation) from the snowflake source, utilizes only one core to read the data from Snowflake. Whereas, when creating a reflection on an S3 source, because of the parallel reads it is much faster.
Is there a way to control the parallelism on an ARP-based plugin? This would really help us make our ARP plugins more performant.
If not, do you think there’s a better, more performant way to read ~300 GB of data from snowflake into Dremio to create a reflection?