CSV download throws network error

1

Hi ,
I’m using latest dremio EE AWS edition and configured it behind aws loadbalancer (ALB), so 443 port redirect to dremio instance 9047. So when I’m trying to download csv it throws network error. But download works if I do it directly from instance ip and port 9047.
So what should I do to make this work with lb

Here are some logs from access.log

172.31.27.236 - - [05/Mar/2021:10:06:58 +0000] "GET /apiv2/socket HTTP/1.1" 403 0 "-" "Mozilla/5.0 
(Macintosh; Intel Mac OS X 11_2_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 
Safari/537.36"
172.31.27.236 - - [05/Mar/2021:10:07:15 +0000] "GET /apiv2/job/1fbe011e-e6bf-dd04-f5eb- 
0d24fbf1bc00/download/?downloadFormat=CSV&.token=gdrjcb3n376vpagso0scqs48al HTTP/1.1" 
200 38484 "https://dremio.recallmasters.com/space/cbs-market-smart.public/report? 
tipVersion=0000137499254264&version=0000137499254264" "Mozilla/5.0 (Macintosh; Intel Mac OS X 
10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36"
172.31.27.236 - - [05/Mar/2021:10:07:18 +0000] "GET / HTTP/1.1" 200 2785 "-" "ELB  - 
HealthChecker/2.0"
172.31.51.20 - - [05/Mar/2021:10:07:18 +0000] "GET / HTTP/1.1" 200 2785 "-" "ELB  - 
HealthChecker/2.0"
172.31.27.236 - - [05/Mar/2021:10:07:48 +0000] "GET / HTTP/1.1" 200 2785 "-" "ELB- 
HealthChecker/2.0"
172.31.51.20 - - [05/Mar/2021:10:07:48 +0000] "GET / HTTP/1.1" 200 2785 "-" "ELB - 
HealthChecker/2.0"
172.31.51.20 - - [05/Mar/2021:10:07:53 +0000] "GET /apiv2/socket HTTP/1.1" 403 0 "-" "Mozilla/5.0 
(Macintosh; Intel Mac OS X 11_2_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 
Safari/537.36"
172.31.51.20 - - [05/Mar/2021:10:07:53 +0000] "GET /apiv2/socket HTTP/1.1" 403 0 "-" "Mozilla/5.0 
(Macintosh; Intel Mac OS X 11_2_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 
Safari/537.36"
172.31.51.20 - - [05/Mar/2021:10:07:53 +0000] "GET /apiv2/socket HTTP/1.1" 403 0 "-" "Mozilla/5.0 
(Macintosh; Intel Mac OS X 11_2_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 
Safari/537.36"
172.31.51.20 - - [05/Mar/2021:10:07:54 +0000] "GET /apiv2/socket HTTP/1.1" 403 0 "-" "Mozilla/5.0 
(Macintosh; Intel Mac OS X 11_2_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.192 
Safari/537.36"
2

@VahagnBleyan

When you tried to download as CSV and it failed, can you please check the jobs page and see if you see a failed job, send us the profile

3

Thanks for reply
That job didn’t marked as failed. This is why I think problem is somewhere in loadbalancer. As I write it downloads normally when I try it directly from dremio instance IP.

Screen Shot 2021-03-08 at 2.09.29 PM
Screen Shot 2021-03-08 at 2.09.29 PM951×593 41.9 KB

Sent profile zip to you by private message.

4

@VahagnBleyan

That is a great test to rule out Dremio is not the issue and it is in the load balancer. Are there any errors in the load balancer logs?

5

Sorry for late replay, was a little busy.

so the I fix this issue by switching Desync mitigation mode from Defensive to Monitor in lb Atributes.

lb-dremio
lb-dremio958×316 17.7 KB

Maybe it will be usefull information for you in the future