# Pointing Dremio to tab delimited text - ignore field \> 65535?

**URL:** https://community.dremio.com/t/pointing-dremio-to-tab-delimited-text-ignore-field-65535/7035
**Category:** Uncategorized
**Created:** [February 25, 2021, 8:52pm UTC](https://community.dremio.com/t/pointing-dremio-to-tab-delimited-text-ignore-field-65535/7035 "2021-02-25T20:52:40Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![surreynorthern](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@surreynorthern](https://community.dremio.com/u/surreynorthern)
#### Post date: [February 25, 2021, 8:52pm UTC](https://community.dremio.com/t/pointing-dremio-to-tab-delimited-text-ignore-field-65535/7035/1 "2021-02-25T20:52:40Z")

</div>

I’m looking to use Dremio to point to an ASDL hosted file which is tab-delimited text. One of the columns is a Remarks column, and sometimes they run to \> 65535 characters

Is there anyway I can tell Dremio to ignore this column when creating the format?

---

<div class="post-metadata">

### Author: ![surreynorthern](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@surreynorthern](https://community.dremio.com/u/surreynorthern)
#### Post date: [February 26, 2021, 1:52pm UTC](https://community.dremio.com/t/pointing-dremio-to-tab-delimited-text-ignore-field-65535/7035/2 "2021-02-26T13:52:06Z")

</div>

I’ve taken a look, and found the issue

For my file format, I had the following settings

Format: Text (delimited)  
Field Delimiter: Tab  
Quote: DoubleQuote (this was the root of my problem)  
Comment: NumberSign  
Escape: DoubleQuote  
Extract Fieldnames: Ticked  
Trim fieldnames: Ticked

On painfully looking through my large text file, I finally tracked it down to a field value having a doublequoted value in it… e.g.

Johnny “Pirate” Depp

This meant that the remainder of the file was considered the rest of the line.

I then changed the Quote:Doublequote to ‘custom’ value ‘~’

Is this the right way to go about it? My workaround feels hacky.

Thanks

---

<div class="post-metadata">

### Author: ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [February 27, 2021, 5:58am UTC](https://community.dremio.com/t/pointing-dremio-to-tab-delimited-text-ignore-field-65535/7035/3 "2021-02-27T05:58:10Z")

</div>

@surreynorthern

Looks like the field delimiter was the issue, and what you did was the right method, unfortunately today Dremio does not automatically suggest delimiter

Thanks  
Bali

---

<div class="post-metadata">

### Author: ![surreynorthern](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@surreynorthern](https://community.dremio.com/u/surreynorthern)
#### Post date: [March 1, 2021, 12:32pm UTC](https://community.dremio.com/t/pointing-dremio-to-tab-delimited-text-ignore-field-65535/7035/4 "2021-03-01T12:32:45Z")

</div>

As I’m working with my data, setting the formats, I’m finding most of the character I could use as delimited are actually used in the remarks - even the tilde, in some of them.

Other ETL tools can actually let you set the Quote equivalent to None since the content of a Tab delimited field, or indeed a CSV, may not have encapsulated string.

I’m concerned that picking a surrogate character e.g using a caret or a curly bracket may end up being entered by users in the data.