Handling multi-chars field separators

1

Hi,

I must work with flat files with field separators made of multiple characters ("|;|" for instance).
Is there a way to make it work with Dremio ?

Best regards,
Damien

2

Hi, any feedback ?

Dremio UI allows us to put multiple characters as column separator but it is only using the first character entered.
Is there a way around that ?

3

Hi @damiencoueron

Where are you entering multiple separation characters? Can you please send us a screenshot?

Thanks
@balaji.ramaswamy

4

Hi @balaji.ramaswamy,

Here is how my raw file looks like. You can see the field separator ( “|;|” ) :

datasetsettings-emptyseparator
datasetsettings-emptyseparator.png1313×742 137 KB

Here is what happens when I try to set a custom field delimiter. Apparently only the “|” pipe is interpreted by Dremio :

datasetsettings-multicharseparator
datasetsettings-multicharseparator.png1316×742 73.9 KB

Thanks for your help,
Damien

5

Hello @damiencoueron,

This is a known issue we’re working to address. In the mean time, what tool are you using to generate these files? Is it possible to adjust the field delimiter to a single character?

6

Hi @ben,

Actually, we generated the files this way on purpose because they need to be processed by another tool (Azure SQL Datawarehouse) that does not handle escape characters correctly. Hence the complex field delimiter !

Do you have an estimated release date for a fix ?

7

Hi @ben,
I noticed that in dremio 3.2 it is not possible anymore to set field separators with more than one character. This was not the fix I expected :confused: but at least it clarifies the UI !

Do you think this feature will be built at some point ?