Supporting MongoDB Views (Updating MongoDB Java Driver)

1

I’ve found that when working with MongoDB collections, some JSON data structures get converted to strings. When connecting to Tableau for example, these records are interpreted as strings.

MongoDB supports views, which are defined using the aggregation framework. I thought a good work-around to the above would be to create views where I can unnest my schema.

I can see my views, but when trying to access one, I got this error:

com.mongodb.MongoQueryException: Query failed with error code 166 and error message 'Namespace {database}.{view} is a view. Legacy find operations are not supported on views. Only clients which support the find command can be used to query views.' on server.

The master on GH is using mongo-java-driver 3.0.2, which is a few minor versions behind the latest 3.4.x.
Are there internal plans to support views? I haven’t spent much time on dremio, but if it’d not be complicated to update the driver, I could try it then submit a PR.

2

I’ll let others respond to the question of java driver for mongo.

Can you show an example of the BSON that is being converted to a string? Dremio has a number of ways to convert structures like arrays and sub documents into columns while preserving types, as well as working with fields of mixed type. Some of these are covered here:

http://docs.dremio.com/working-with-datasets/data-curation.html#common-tasks

The general idea of using a view to curate data makes a lot of sense. That’s essentially what a virtual dataset is in Dremio.

3

Thanks @kelly, I’ll go through the tutorial in some detail. I think it’ll make things much easier for me.

4

Hi,

I’m going to try update the MongoDB driver from 3.0.x to 3.4 or 3.5 (I think there’s major breaking changes in 3.6 so I’ll avoid that).

Can anyone give me guidance in terms of unit testing? I think it’s the only way to get Dremio to support MongoDB views.

5

@laurent I get this error when trying to run tests with an updated MongoDB version (or just trying to run mvn test on cloned repo)

"C:\Program Files\Java\jdk1.8.0_112\bin\java" -Dmaven.multiModuleProjectDirectory=D:\projects\dremio-oss "-Dmaven.home=C:\Program Files\JetBrains\IntelliJ IDEA 2017.2.6\plugins\maven\lib\maven3" "-Dclassworlds.conf=C:\Program Files\JetBrains\IntelliJ IDEA 2017.2.6\plugins\maven\lib\maven3\bin\m2.conf" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2017.2.6\lib\idea_rt.jar=58423:C:\Program Files\JetBrains\IntelliJ IDEA 2017.2.6\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\JetBrains\IntelliJ IDEA 2017.2.6\plugins\maven\lib\maven3\boot\plexus-classworlds-2.5.2.jar" org.codehaus.classworlds.Launcher -Didea.version=2017.2.6 test
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: windows
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.0
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: windows-x86_64
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Sabot - Kernel 1.3.1-201712020438070881-a7af5c8
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for com.dremio.tools:dremio-fmpp-maven-plugin:jar:1.3.1-201712020438070881-a7af5c8 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.415 s
[INFO] Finished at: 2017-12-16T03:17:43+02:00
[INFO] Final Memory: 12M/184M
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin com.dremio.tools:dremio-fmpp-maven-plugin:1.3.1-201712020438070881-a7af5c8 or one of its dependencies could not be resolved: Failure to find com.dremio.tools:dremio-fmpp-maven-plugin:jar:1.3.1-201712020438070881-a7af5c8 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

Process finished with exit code 1

Any idea on what I could do? com.dremio.tools is not on public mvn repos, so don’t know where it is, and why maven can’t find it.

6

The plugin is on Github too at https://github.com/dremio/dremio-oss/tree/master/tools/fmpp-maven-plugin. But because of the way Maven works, you need to first do mvn install in order for the plugin to be installed in your local repository.