I have LF (line breaks) within quoted text but dremio is not ignoring them, and showing varying behaviour depending on which fields are included in the select statement
image shows detailed walkthrough, of which the last two examples illustrate the problem parsing the linefeed
Sample file
id,field1,field2
1,"comment enclosed in quotes","comment with linefeed
enclosed in quotes"
@sm00 I just uploaded your sample data and do not see the double quotes
Please try
select id from test
and see if dremio respects the linefeed embedded in the 2nd field
The problem is that linefeeds are affecting the parsing when they are in quoted text fields. This is not evident when performing a select * statement, but shows up when selecting single fields such as id. There is a walkthrough of this in the attached image in my main post
Balaji, did you notice my reply post. The issue is not seeing the double quotes – the issue is double quotes not working as text qualifiers to keep linebreaks within text fields
@sm00 Thanks for the remainder and able to reproduce, will open a bug with our Engineering team