Python pyodbc utf-16-le error

1

I’m running Dremio 4.3.1 on a Centos 7.8 server. I’m trying to connect to Dremio using Python (3.7.1) pyodbc as describe in the Dremio docs. I’m just doing a very simple “select * from Table” statement, which resulted in the following error:

UnicodeDecodeError: ‘utf-16-le’ codec can’t decode bytes in position 2432-2433: illegal UTF-16 surrogate

I’ve search the community forum and found a couple of posts and found: this and this, and have tried setting dremio-env in the dremio config directory (tried each individually and together):

DREMIO_JAVA_EXTRA_OPTS=’-Dsaffron.default.charset=UTF-16LE -Dsaffron.default.nationalcharset=UTF-16LE -Dsaffron.default.collation.name=UTF-16LE$en_US’

DREMIO_JAVA_SERVER_EXTRA_OPTS=’-Dsaffron.default.charset=UTF-16LE -Dsaffron.default.nationalcharset=UTF-16LE -Dsaffron.default.collation.name=UTF-16LE$en_US’

But nothing worked. I still get the same error. I would love to try out Dremio, so any help would be much appreciated.

2

The problem seems to be that MySQL recommends using character set utf8mb4. Although the Dremio UI has no trouble rendering such physical/virtual datasets, it seems calcite (I presume) does. I’ve not found options that can be used in dremio-env overcome this. Instead, I’ve cloned data into tables defined as:

create table xyz ( ... ) character set utf16le;

And this seems to work. If anyone has a better solution, please do share.