Hello!
I’m having a bit of an issue regarding trying to promote a tablea that contains JSONL (JSON Lines) files through the command:
ALTER TABLE <table_path> REFRESH METADATA
AUTO PROMOTION
Inside the folder on our datalake, the content of said folder is JSONL files and its not being detected and i’m having the error bellow:
If I manually promote through the UI or API specifying that the type is JSON, it works no problem.
The issue is that we will have the need to sometimes removing / promoting this table (more to come in the development) and were seing how we could turn this into something more automatic.
Is there any way to pass this TYPE to the command listed above? Or the only possible way is using the UI or API?
What is the dataset file actually called , is it “pds.JSON” or just “pds” ?
If it is just “pds” then change the actual file name to include the file type extension i.e. “pds.json” . You cannot pass the file type as an argument in the ALTER TABLE command.
Then wrap the whole dataset name with double quotes use lowercase for the file type i.e.
ALTER TABLE my_datalake.“pds.json” REFRESH METADATA AUTO PROMOTION
JSON in this case is a folder, that contains multiple JSONL files.
What I was wondering is if it was possible to pass the FORMAT option using the alter table command to format the folder correctly, just like we can do through the UI or API.
But from what I understand from your response, its not possible and we would need to go via API…
change your jsonl file extention to json and try again
Unfortunatly the files will be generate via AIRBYTE / JIRA integration and the output will be JSONL by default. It cant be changed normally from what we saw here.
Currently dremio not supporte auto promotion for jsonl files , maybe write one formatplugin can solve this problem or your can change dremio oss code FormatCreator.java add support for jsonl
also you can try change AIRBYTE destination data format with parquet ?
@Mauricio_Macri The issue is for AUTO PROMOTION, the extension has to be .json but since you have .pds.json, Dremio thinks the file type is .pds, YOu should be able to manually promote via API or UI
