V18 PDS against file works, V21.x against doesn't(?)

1

In a nutshell, I have a CSV with quotes.

The file has some non-std characters in it on a column on some rows.

On V18,

select lower(addressline1) is fine.

on V20

select lower(addressline1)

throws a

GandivaException: unexpected byte \bf encountered while decoding utf8 string

The file IS the same in both, and the I’ve checked the PDS settings. These are also identical.

Is anyone aware of a change in the versions that causes this?

There are two chars in the string which in 1252 codepage get converted to ¿¿

2

@spireite Would you be able to send a part of the CSV that has the issue?

3

@balaji.ramaswamy sorry fior the delay, I was out a few days.

I can email it to you if you want? I wouldn’t want to post here.

4

@spireite Sent you an email, thanks