# Support parquet files compressed via ZSTD

**URL:** <https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849>\
**Category:** Uncategorized\
**Created:** [October 24, 2022, 8:20am UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849 "2022-10-24T08:20:02Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ikosyanenko](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ikosyanenko/32/4314_2.png) [@ikosyanenko](https://community.dremio.com/u/ikosyanenko)\
**Post date:** [October 24, 2022, 8:20am UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/1 "2022-10-24T08:20:02Z")

</div>

Is it possible to support parquet files compressed via ZSTD?

---

<div class="post-metadata">

**Author:** ![rongfengliang](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rongfengliang/32/3120_2.png) [@rongfengliang](https://community.dremio.com/u/rongfengliang)\
**Post date:** [October 24, 2022, 9:04am UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/2 "2022-10-24T09:04:04Z")

</div>

currently not supported. only snappy、gzip、none support

```nohighlight
EnumeratedStringValidator PARQUET_WRITER_COMPRESSION_TYPE_VALIDATOR = new EnumeratedStringValidator(
      PARQUET_WRITER_COMPRESSION_TYPE, "snappy", "snappy", "gzip", "none");

```

---

<div class="post-metadata">

**Author:** ![ikosyanenko](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ikosyanenko/32/4314_2.png) [@ikosyanenko](https://community.dremio.com/u/ikosyanenko)\
**Post date:** [October 24, 2022, 9:25am UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/3 "2022-10-24T09:25:26Z")

</div>

Is there any way to add it?

---

<div class="post-metadata">

**Author:** ![rongfengliang](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rongfengliang/32/3120_2.png) [@rongfengliang](https://community.dremio.com/u/rongfengliang)\
**Post date:** [October 24, 2022, 10:55am UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/4 "2022-10-24T10:55:44Z")

</div>

if you want to support ZSTD, you should modify sabot/kernel/src/main/java/com/dremio/exec/store/parquet/ParquetRecordWriter.java

and also sabot/kernel/src/main/java/com/dremio/exec/ExecConstants.java

---

<div class="post-metadata">

**Author:** ![rongfengliang](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rongfengliang/32/3120_2.png) [@rongfengliang](https://community.dremio.com/u/rongfengliang)\
**Post date:** [October 24, 2022, 1:47pm UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/5 "2022-10-24T13:47:05Z")

</div>

I modify kernel and compile one you can have a try [GitHub - rongfengliang/dremio-parquet-zstd](https://github.com/rongfengliang/dremio-parquet-zstd)

---

<div class="post-metadata">

**Author:** ![ikosyanenko](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ikosyanenko/32/4314_2.png) [@ikosyanenko](https://community.dremio.com/u/ikosyanenko)\
**Post date:** [October 24, 2022, 2:49pm UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/6 "2022-10-24T14:49:52Z")

</div>

Is it possible to merge it?

---

<div class="post-metadata">

**Author:** ![rongfengliang](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rongfengliang/32/3120_2.png) [@rongfengliang](https://community.dremio.com/u/rongfengliang)\
**Post date:** [October 24, 2022, 3:05pm UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/7 "2022-10-24T15:05:06Z")

</div>

as far as I know, it seems impossible

---

<div class="post-metadata">

**Author:** ![wundi](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/wundi/32/3920_2.png) [@wundi](https://community.dremio.com/u/wundi)\
**Post date:** [October 27, 2022, 5:07pm UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/8 "2022-10-27T17:07:05Z")

</div>

We are currently doing it for being able to _read_ ZStandard compressed parquet files

---

<div class="post-metadata">

**Author:** ![wundi](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/wundi/32/3920_2.png) [@wundi](https://community.dremio.com/u/wundi)\
**Post date:** [October 27, 2022, 5:24pm UTC](https://community.dremio.com/t/support-parquet-files-compressed-via-zstd/9849/9 "2022-10-27T17:24:39Z")

</div>

Basically, all that’s needed is adding two native libraries - a version of libhadoop.so that is compiled with ZSTD support and the ZSTD native library.

We are running Dremio using docker, so we do it like this:

```auto
---
version: '3'
services:
  executor-0:
    ...
    volumes:
      ...
      - /opt/dremio/lib/native/libhadoop.so.3.3.2:/opt/dremio/lib/libhadoop.so
      - /opt/dremio/lib/native/libzstd.so.1.5.2:/opt/dremio/lib/libzstd.so
      - /opt/dremio/lib/hadoop-common-3.3.2-dremio-202207041927090255-61c2bd1.jar:/opt/dremio/jars/3rdparty/hadoop-common-3.3.2-dremio-202207041927090255-61c2bd1.jar

```

One word of caution though. There’s a bug in the Dremio CE parquet reader library (which is free but the source isn’t open), which doesn’t release the decompressor after use. It doesn’t make a difference for Snappy, but the ZStandard decompresser uses native memory, which is then leaked. This has caused our executors to get killed by the OOM killer several times per day (~350GB memory leaked before doing so).  
That’s why `hadoop-common-3.3.2` is mapped in the above, in which I made a somewhat hacky fix that releases the native memory in a finalizer such that the leaking is somewhat contained. I have confirmed that fix contains the issue, so I’ll make a thread on it specifically (or maybe someone from Dremio will notice it here). If someone at Dremio doesn’t pick it up I’ll be doing a nicer and more permanent fix for it