# HDFS connector interpret VARCHAR as VARBINARY

**URL:** <https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206>\
**Category:** Uncategorized\
**Created:** [March 11, 2020, 6:26pm UTC](https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206 "2020-03-11T18:26:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![LucGth](https://avatars.discourse-cdn.com/v4/letter/l/858c86/32.png) [@LucGth](https://community.dremio.com/u/LucGth)\
**Post date:** [March 11, 2020, 6:26pm UTC](https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206/1 "2020-03-11T18:26:21Z")

</div>

Hi,  
connecting to an hdfs source i notice that the varchar columns are interpreted as varbinary.  
This not happend for columns with data format, like you can see from the picture.  
Another strange thing, the column dir0 has the true format. it is the partition column of the Impala table.  
Using the Hive connector this not happend.  
All this cause problems to cast these varbinary columns into varchar.  
With api requests i have extracted the infomation about the pds:  
{  
“entityType” : “dataset”,  
“id” : “8c096312-4df7-4f16-abf7-54c0b870038b”,  
“type” : “PHYSICAL\_DATASET”,  
“path” : [“HDFS”, “test”, “sample\_20200110”],  
“createdAt” : “2020-02-04T10:41:07.193Z”,  
“tag” : “31”,  
“format” : {  
“type” : “Parquet”,  
“ctime” : 0,  
“isFolder” : true,  
“location” : “/test/sample\_20200110”,  
“autoCorrectCorruptDates” : true  
},  
“approximateStatisticsAllowed” : false,  
“fields” : [ {  
“name” : “xxx”,  
“type” : {  
“name” : “VARBINARY”  
}  
}, {  
“name” : “scheda”,  
“type” : {  
“name” : “VARBINARY”  
}  
}, {  
“name” : “ts”,  
“type” : {  
“name” : “TIMESTAMP”  
}  
}, {  
“name” : “transaction\_id”,  
“type” : {  
“name” : “TIMESTAMP”  
}  
}, {  
“name” : “dir0”,  
“type” : {  
“name” : “VARCHAR”  
}  
} ]  
}

thanks!

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/a/a896c7be363f77744032382e00d6bb4ada400363.png)

---

<div class="post-metadata">

**Author:** ![ben](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ben/32/1367_2.png) [@ben](https://community.dremio.com/u/ben)\
**Post date:** [March 11, 2020, 7:43pm UTC](https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206/2 "2020-03-11T19:43:02Z")

</div>

@LucGth, what is the file type you are accessing? Parquet or ORC or something else?

---

<div class="post-metadata">

**Author:** ![LucGth](https://avatars.discourse-cdn.com/v4/letter/l/858c86/32.png) [@LucGth](https://community.dremio.com/u/LucGth)\
**Post date:** [March 11, 2020, 10:04pm UTC](https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206/3 "2020-03-11T22:04:34Z")

</div>

HI Ben, are parquet files.

---

<div class="post-metadata">

**Author:** ![ben](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ben/32/1367_2.png) [@ben](https://community.dremio.com/u/ben)\
**Post date:** [March 11, 2020, 11:10pm UTC](https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206/4 "2020-03-11T23:10:08Z")

</div>

You should have `parquet-tools` installed with your Hadoop distribution.

Try running: `$ parquet-tools meta <path/to/sample_20200110>` and attach the output here.

---

<div class="post-metadata">

**Author:** ![LucGth](https://avatars.discourse-cdn.com/v4/letter/l/858c86/32.png) [@LucGth](https://community.dremio.com/u/LucGth)\
**Post date:** [April 1, 2020, 3:59pm UTC](https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206/5 "2020-04-01T15:59:34Z")

</div>

Hi Ben, thanks for your reply.

creator: impala version 2.12.0-cdh5.16.1 (build 4a3775ef6781301af81b23bca45a9faeca5e761d)

## file schema: schema

xx: OPTIONAL BINARY R:0 D:1  
yy: OPTIONAL BINARY R:0 D:1  
zz: OPTIONAL INT96 R:0 D:1  
kk: OPTIONAL INT96 R:0 D:1

## row group 1: RC:1 TS:3039

xx: BINARY SNAPPY DO:4 FPO:33 SZ:57/53/0.93 VC:1 ENC:PLAIN\_DICTIONARY,RLE  
yy: BINARY SNAPPY DO:134 FPO:2978 SZ:2872/5957/2.07 VC:1 ENC:PLAIN\_DICTIONARY,RLE  
zz: INT96 SNAPPY DO:3059 FPO:3086 SZ:55/51/0.93 VC:1 ENC:PLAIN\_DICTIONARY,RLE  
kk: INT96 SNAPPY DO:3193 FPO:3220 SZ:55/51/0.93 VC:1 ENC:PLAIN\_DICTIONARY,RLE

---

<div class="post-metadata">

**Author:** ![LucGth](https://avatars.discourse-cdn.com/v4/letter/l/858c86/32.png) [@LucGth](https://community.dremio.com/u/LucGth)\
**Post date:** [April 17, 2020, 10:28am UTC](https://community.dremio.com/t/hdfs-connector-interpret-varchar-as-varbinary/5206/6 "2020-04-17T10:28:07Z")

</div>

is it normal in your opinion?

thanks