Some question about the Function Mappings in the ARP connector

1

Hi, I have five questions about the ARP mappings. For example, consider the following mapping:

- names: - "COUNT" - signatures: - args: - "INTEGER" return: "INTEGER"

  1. The rules that I specified here about the input and output will be appended to the default Dremio rules for the COUNT function or only my rules will be considered? For example, what happens if the user use the COUNT function for a column of type other than INTEGER?

  2. The datatypes that I’m using here is supposed to be the Dremio datatypes or the Source datatypes (which has been mapped before)?

  3. How can I say the output datatype should be always the same as the input datatype

  4. If a function in Source has the same logic as a function in the Dremio but the names are different, is there any way to map these two functions? for example, the source function name is Variance and the Dremio function name is VAR_POP but both has the same logic.

  5. Finally, How can I say for example if the input was DOUBLE the output should be DOUBLE too, else the output should be INTEGER?

Thanks

2

Hi soheil08,

Before I answer your specific questions, it may be helpful to give a bit of information regarding how an ARP file is interpreted. The ARP function mappings which you put into the file act as a whitelist for what functions Dremio should attempt to push down to the underlying source. If the type or function mapping is not in the ARP file, then Dremio will not attempt to push it down. With that in mind:

  1. You are only specifying what functions are supported, if the function/signature combination is not in the ARP File and Dremio attempts to execute that function, then it will not be pushed down and Dremio will execute the function itself.

  2. The datatypes are the Dremio datatypes.

  3. You cannot, and you should not. Dremio has the type signatures (inputs and outputs) already defined for what it supports, with the ARP mapping you’re simply indicating what can be pushed down and how it should be pushed down such that it will match what Dremio supports. If you wanted to support INTEGER in and INTEGER out, you would define a signature for that function which specified that, same for any other types.

  4. Yes, you can add a rewrite element to the signature, with {0}, {1}, etc referring to the specific signature inputs. Example: rewrite: TRUNC({0})

  5. You cannot. As specified in #3 above this would not make sense in the context of ARP either.

Cheers

2 Likes
3

Thanks for your answer
Can you please give me an example for the question number #4?

4 
- names:
  - "sign"
  signatures:
  - args:
    - "integer"
    return: "integer"
    # Example rewrite (although this is not necessary here as the default is the same)
    rewrite: "SIGN({0})"
1 Like
5

Thanks for the information.
If there is a function/signature mentioned in source arp file, but the function is not supported/available in dremio. Will dremio pushed down the query to source or it will fail?

7

Hi Pranay,

Apologies for the (long) delay in the response. If something is not present in the ARP file, then Dremio will not push it down to the source and will instead execute within Dremio. The query will still succeed.