# Dremio S3 Compatibilty Mode Unable to read parquet

**URL:** <https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308>\
**Category:** Uncategorized\
**Created:** [April 17, 2021, 6:45pm UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308 "2021-04-17T18:45:55Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![asakmedops](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@asakmedops](https://community.dremio.com/u/asakmedops)\
**Post date:** [April 17, 2021, 6:45pm UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/1 "2021-04-17T18:45:55Z")

</div>

We are not able to read parquet data stored in S3 Compatible Storage.  
We are able to read any other file format (csv, excel, …) stored on OVH openstack Swift storage.  
We are using OVH Openstack swift with S3 Compatitlity mode enabled.

Here are the steps to reproduce the issue :

1. In Dremio UI, add a new Data Lake source, then choose Amazon S3

2. Provide AWS Access & Secret , then switch to Advanced tab enable Compatibility mode + add extra property under Connection Properties (name:“fs.s3a.endpoint”, value: “[s3.bhs.cloud.ovh.net](http://s3.bhs.cloud.ovh.net)”) and save:  

3. Try setting format & reading a simple CSV file located under the S3 compatible Data lake source - Result Should Work

4. Try Setting the format for your parquet file - Should work  

5. After Saving the parquet format, try reading the parquet file formatted: This is where the following error will be printed:  

I downloaded the Dremio Source code from Git, and tried to get more logs. the following log file provide more details.  
[dremio-error-read-parquet-s3.zip](https://community.dremio.com/uploads/short-url/SG5GkNm6FAqMJQuXLIRkjYpYt8.zip) (3.6 KB)

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [April 20, 2021, 4:41am UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/2 "2021-04-20T04:41:03Z")

</div>

@asakmedops

Can you try to disable async and try?

---

<div class="post-metadata">

**Author:** ![asakmedops](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@asakmedops](https://community.dremio.com/u/asakmedops)\
**Post date:** [April 20, 2021, 12:54pm UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/3 "2021-04-20T12:54:06Z")

</div>

Thanks @balaji.ramaswamy it works.

I think this is the solution to all S3 Compatible related issues on the forum.

is there a config (xml) to disable async while adding S3 Compatible Obj as distributed storage(fs.s3a.endpoint kind of config) ?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [April 21, 2021, 12:22am UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/4 "2021-04-21T00:22:26Z")

</div>

@asakmedops

This is a source level property so afraid not

---

<div class="post-metadata">

**Author:** ![asakmedops](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@asakmedops](https://community.dremio.com/u/asakmedops)\
**Post date:** [April 23, 2021, 6:50am UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/5 "2021-04-23T06:50:05Z")

</div>

So the same issue on Distributed storage can’t be fixed at the moment . and that means ONLY AWS S3 & Azure storage can be used as Distributed Storage?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [April 25, 2021, 5:15am UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/6 "2021-04-25T05:15:33Z")

</div>

@asakmedops To turn off async on reflections add below in dremio.conf on executors  
`debug.dist.async.enabled: false`

---

<div class="post-metadata">

**Author:** ![asakmedops](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@asakmedops](https://community.dremio.com/u/asakmedops)\
**Post date:** [April 30, 2021, 6:34pm UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/7 "2021-04-30T18:34:31Z")

</div>

> [@balaji.ramaswamy](#):
>
> debug.dist.async.enabled: false

Finally works. I tested with latest version 15.0.0

---

<div class="post-metadata">

**Author:** ![asakmedops](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@asakmedops](https://community.dremio.com/u/asakmedops)\
**Post date:** [May 3, 2021, 7:04pm UTC](https://community.dremio.com/t/dremio-s3-compatibilty-mode-unable-to-read-parquet/7308/8 "2021-05-03T19:04:16Z")

</div>

Its not usable in production as distributed store with async off. We had to rollback to Amazon S3. We were getting a lot of timeouts.
