the scenario is as following:
source: S3 bucket
PDS based on CSV file
VDS created by joining three PDSs.
one PDS can not be queried successfully,for there is one row hasing one more columns in the file based on which the PDS created.
but i still can query VDS sucessfully with output missing fields in the previous PDS i mentioned.
why this happened?The VDS should not be produced when we are in this state.
Does the VDS succeed in both preview and run mode?
I am also curious to what the error says when you try to query the PDS, can you share a query profile?
529f780b-37b8-44fd-913c-2a9798f058bf.zip (5.6 KB)
i tried to reproduce but failed this time.
1.first used a fixed csv file to test and got result as expected.
2.delete this file from S3 and upload a malformed csv,which has an extra comma in the lind end of the 4th line
3.refresh metadata(ALTER PDS “Calix Cloud Dev”.“calix-cloud-dev”.“multi-datasources”.“1234”.EquipmentAgreement REFRESH METADATA)
4.query PDS,then i got the error message.
5.this time,strangely,i can not reproduce VDS error when i query VDS.just got the same error message.
are you able to read this file from aws s3 CLI from your Dremio Coordinator?
s3a://calix-cloud-dev/multi-datasources/1234/EquipmentAgreement/2018-08-09/equipmentagreemnt_0711_fixed.csv
Thanks
@balaji.ramaswamy