Deploying Dremio on prem but S3 for distributed storage back end

1

Hello -

Does anyone have experience deploying a Dremio cluster on prem, but then using S3 as a distributed back-end? Would reflections have terrible performance if the data is not co-located with the executor nodes?

Thanks

2

Hi @jgenser

Having your reflection data co-located with executor nodes will help. You can pretty much see the time difference will be spent on wait times when the reflection files are on cloud (S3) by looking at the profiles. You can do a quick test and see the the difference in stats shown in both the profiles. Happy to go over the 2 profiles if it can be uploaded

Thanks
@balaji.ramaswamy