# How to connect Dremio with GCS storage using proxy

**URL:** <https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200>\
**Category:** Dremio University\
**Created:** [January 19, 2023, 11:57am UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200 "2023-01-19T11:57:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pawel](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@Pawel](https://community.dremio.com/u/Pawel)\
**Post date:** [January 19, 2023, 11:57am UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/1 "2023-01-19T11:57:08Z")

</div>

Is there option to configurate Google Cloud Storage Source using proxy?

I couldn’t find similar configuration for GCS which is available for S3 ([Dremio](https://docs.dremio.com/software/data-sources/s3/#connecting-through-a-proxy-server))  
We would like to have dremio on premise with connection to S3 and GCS storage for querying files from both locations.

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [January 22, 2023, 5:22pm UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/2 "2023-01-22T17:22:52Z")

</div>

@Pawel These are added in the source’s advanced tab (add property), which should be available in all sources

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/8/8c71ea4963945249670beabe792d06f23fe6ff93.png)

---

<div class="post-metadata">

**Author:** ![Pawel](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@Pawel](https://community.dremio.com/u/Pawel)\
**Post date:** [January 23, 2023, 2:37pm UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/3 "2023-01-23T14:37:23Z")

</div>

Hi @balaji.ramaswamy thanks for respond - I saw this tab and tried to use properties: _fs.gs.proxy.address_ which should be correct for [GCS connector](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.2/gcs/CONFIGURATION.md) - unfortunately this flag doesn’t work in Dremio.  
Could you advice what configuration should be added to this properties to make proxy for GCP working?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [January 26, 2023, 4:53am UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/4 "2023-01-26T04:53:28Z")

</div>

@Pawel You would need a host and port an optional username/password

```auto
fs.gs.proxy.host and fs.gs.proxy.port 

```

---

<div class="post-metadata">

**Author:** ![Pawel](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@Pawel](https://community.dremio.com/u/Pawel)\
**Post date:** [January 26, 2023, 9:46am UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/5 "2023-01-26T09:46:09Z")

</div>

@balaji.ramaswamy We tried this (fs.gs.proxy.host and fs.gs.proxy.port) but this configuration also **doesn’t work** - there is the same UnknownHostException: [oauth2.googleapis.com](http://oauth2.googleapis.com) error.

We tested different approach with -Dhttps.proxyHost properties in dremio-env and this is working but required ugly hack which we would like to avoid.

We tested versions 19.1.0-202111160130570172-0ee00450 and 23.1.0-202211250136090978-a79618c7

So could you check why standard properties doesn’t work and why there is no documentation for GCP proxy configuration?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [January 27, 2023, 7:22am UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/6 "2023-01-27T07:22:54Z")

</div>

@Pawel I have seen that config before, are you able to send the exact `dremio-env` line or file? Let me check if that is the right one

---

<div class="post-metadata">

**Author:** ![Pawel](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@Pawel](https://community.dremio.com/u/Pawel)\
**Post date:** [January 27, 2023, 7:52am UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/7 "2023-01-27T07:52:02Z")

</div>

Hi @balaji.ramaswamy For GCS proxy any properties set in Advanced option tab doesn’t work. The only working solution for us is to add to dremio-env line  
DREMIO\_JAVA\_EXTRA\_OPTS=“-Dhttp.nonProxyHosts=localhost|127.0.0.1|\*.S3.domain|\*.localHost.domain -Dhttp.proxyHost=… -Dhttp.proxyPort=… -Dhttps.proxyHost=… -Dhttps.proxyPort=…”

Because this -Dhttp.proxyHost properties is for whole Dremio connection we had to add -Dhttp.nonProxyHosts to disable proxy for S3. Additionally Dremio use some internal calls so we have to add our local host domain to this -Dhttp.nonProxyHosts as well.

Unfortunately this proxy configuration was not pick during asynchronous accesss to parquet files so to make it work we have to disable this feature (Enable asynchronous access when possible) what have big performance influence during read parquet files.

After dremio code analyses we notice that for asynch connection AsyncHttpClient library is used and in default configuration of this lib reading standard proxy configuration is disable. To enable this feature you can edit **ahc-default.properties** file located in / **dremio/jars/3rdparty/async-http-client-2.7.0.jar** in folder org/asynchttpclient/config and set **org.asynchttpclient.useProxyProperties** to **true**

All those changes allow Dremo to work correctly with GCS and S3 but required ugly hack - could you check if there is simples working solution of this proxy issue with GCS.

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [January 28, 2023, 3:33pm UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/8 "2023-01-28T15:33:37Z")

</div>

Thanks for the detailed explanation, let me check this and get back to you

---

<div class="post-metadata">

**Author:** ![peegle](https://avatars.discourse-cdn.com/v4/letter/p/e47c2d/32.png) [@peegle](https://community.dremio.com/u/peegle)\
**Post date:** [November 2, 2023, 7:41am UTC](https://community.dremio.com/t/how-to-connect-dremio-with-gcs-storage-using-proxy/10200/9 "2023-11-02T07:41:55Z")

</div>

for me it also works to specifiy the parameter of ahc-default.properties as an addtional parameter for DREMIO\_JAVA\_EXTRA\_OPTS -Dorg.asynchttpclient.useProxyProperties=true Dhttp.nonProxyHosts=localhost|127.0.0.1|_.S3.domain|_.localHost.domain -Dhttp.proxyHost=… -Dhttp.proxyPort=… -Dhttps.proxyHost=… -Dhttps.proxyPort=…”
