Advanced Relation Pushdown not worked when dremio start with external parameter

1

Hello.
We are using dremo 14.0.0.
We have implemented our function in dremio (for example, test_f(), returned string “test_inner_f”). It is declared in the com.dremio.exec.expt.fn.impl package and other required places and works fine.
For example, on query

SELECT test_f() FROM db LIMIT 1
it returns one string: test_inner_f.

We also have an ARP-connector, in which the behavior of this function has been changed: instead of test_inner_f, it returns the string “test_arp_f”.

If we run dremio without any parameters, then the result returns the strings test_arp_f - i.e. Advanced Relation Pushdown works fine.
If we run the dremo with the parameter:

-Dsaffron.default.charset=UTF-16LE
the string test_inner_f is returned: i.e. the build-in function runs and no pushdown occurs.

Thus, when using the -Dsaffron.default.charset=UTF-16LE parameter on start Dremio, the function does not pushdown.
Is this normal behavior? Maybe it can be fixed with some settings? Or do we need to change the code?

2

@Arol

Kindly send us the profile so we can see how the ARP framework regenerated the pushdown SQL

Thanks
Bali

3

Add to profiles:

  1. With_pushodwn - start without parameter, ARP working good.
  2. Without_pushdown - start with parameter, ARP not worked.
    Name of function: kitty()

with_pushdown.zip (8.7 КБ) without_pushdown.zip (8.6 КБ)

4

I spent some time debugging the source code and this is what I found:
my test function returns a specific record (= ＾ ● ⋏ ● ＾ =).
In dremio, a String Function named kitty () has been added to the com.dremio.exec.expr.fn.impl.StringFunctions class.

When we run a project without parameters in the com.dremio.exec.planner.logical.ConstExecutor class while processing a request, when processing string expressions, when a calcite NlsString object is formed based on the result, an error occurs:

calcite failed to encode (= ＾ ● ⋏ ● ＾ = ) in charset set ISO-8859-1

, as a result, the original expression (kitty()) goes down and run in JDBC as part of request.

When dremio is started with an encoding parameter, the cast occurs normally and the result returned by our function (= ＾ ● ⋏ ● ＾ = ) going down.

It looks like we implemented our function incorrectly (in the wrong place), but everything worked correctly due to randomness (using specific symbols) when starting dremio without parameters.

So then I need to look for how to correctly implement the function and at what stage the pushdown of the request occurs.