Getting "Number of even though the dataset has 14 fields

Rohithzr · October 18, 2019, 9:19pm

I have a mongo connection where the dataset on a particular collection has only 14 fields but I still get the error “Number of fields in dataset exceeded the maximum number of fields of 800.”.

Below is the server.log and attached is the profile. Can you help me out here?
7882a5cf-90d7-4820-8bad-262445f315b1.zip (5.6 KB)

2019-10-18 21:03:37,780 [qtp1833904548-162] WARN  c.d.p.mongo.metadata.MongoVersion - Could not get the mongo compatibility version from the server. Defaulting to detected version: 3.4
com.mongodb.MongoCommandException: Command failed with error 13: 'not authorized on admin to execute command { getParameter: true, featureCompatibilityVersion: true }' on server staging-db.mongodb.net:27017. The full response is { "ok" : 0.0, "errmsg" : "not authorized on admin to execute command { getParameter: true, featureCompatibilityVersion: true }", "code" : 13, "codeName" : "Unauthorized" }
	at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:295) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:226) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:197) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:189) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:99) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:92) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.Mongo$3.execute(Mongo.java:826) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.executeCommand(MongoDatabaseImpl.java:166) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:138) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:133) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:123) ~[mongo-java-driver-3.6.4.jar:na]
	at com.dremio.plugins.mongo.metadata.MongoVersion.getVersionForConnection(MongoVersion.java:93) ~[dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.metadata.MongoTopology.getMinVersion(MongoTopology.java:219) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.metadata.MongoTopology.getClusterTopology(MongoTopology.java:167) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.connection.MongoConnectionManager.getTopology(MongoConnectionManager.java:174) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.MongoStoragePlugin.getDatasetHandle(MongoStoragePlugin.java:100) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.ManagedStoragePlugin.getDatasetHandle(ManagedStoragePlugin.java:440) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:308) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.DatasetManager.getTable(DatasetManager.java:208) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.CatalogImpl.getTable(CatalogImpl.java:130) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.SourceAccessChecker.lambda$getTable$3(SourceAccessChecker.java:103) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.SourceAccessChecker.checkAndGetTable(SourceAccessChecker.java:82) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.SourceAccessChecker.getTable(SourceAccessChecker.java:103) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.DelegatingCatalog.getTable(DelegatingCatalog.java:66) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.CachingCatalog.getTable(CachingCatalog.java:93) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287) ~[hk2-utils-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorImpl.internalInvoke(MethodInterceptorImpl.java:109) ~[hk2-locator-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorImpl.invoke(MethodInterceptorImpl.java:125) ~[hk2-locator-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorInvocationHandler.invoke(MethodInterceptorInvocationHandler.java:62) ~[hk2-locator-2.5.0-b32.jar:na]
	at com.sun.proxy.$Proxy116.getTable(Unknown Source) ~[na:na]
	at com.dremio.dac.explore.DatasetsResource.getDatasetSummary(DatasetsResource.java:258) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:134) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.dac.explore.DatasetsResource.newUntitledFromParent(DatasetsResource.java:196) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:52) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51) ~[jetty-servlets-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.Server.handle(Server.java:502) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_222]

balaji.ramaswamy · October 21, 2019, 4:07am

@Rohithzr,

Is this a nested json?

Rohithzr · October 21, 2019, 6:09am

yes here is a sample. https://gist.github.com/Rohithzr/67bce44511e7b94efe010d94e9eff35a

Sorabh_Sachdeva · October 22, 2019, 7:49am

@Rohithzr

I am able to load provided json and query from dremio without any issue.
Please share more details about dremio,mongodb version along with steps you are following to load data into mongodb.

moa101 · October 22, 2019, 6:18pm

I work with @Rohithzr;

MongoDB:

version 3.4.23.

Dremio:

Build: 4.0.2-201910020123580864-a98a0b9
Edition: Community Edition
Build Time: 10/01/2019 18:25:55
Change Hash: a98a0b935f4467740ea655c478cdf14ed26c9f0f
Change Time: 10/01/2019 16:39:46

Also when an aggregate pipeline (see below for code) on the mongodb collection I get the following:

Notice that none of our docs have more than 17 root fields.

_id: null
min-field-count:13
max-field-count:17
docs_count:1460

Code:

    {
        '$project': {
            'numFields': {
                '$size': {
                    '$objectToArray': '$$ROOT'
                }
            }
        }
    }, {
        '$group': {
            '_id': None, 
            'min-field-count': {
                '$min': '$numFields'
            }, 
            'max-field-count': {
                '$max': '$numFields'
            }, 
            'docs_count': {
                '$sum': 1
            }
        }
    }
]```

moa101 · October 22, 2019, 6:37pm

@Rohithzr

Just came across this … do you think this is related?

source: https://docs.dremio.com/release-notes/33-release-notes.html#core-services

On upgrading to 3.2, reflections on datasets with more than 800 leaf fields will fail to materialize with a NullPointerException.

Dremio supports datasets with at most 800 leaf fields. Prior to upgrade to 3.2, if there were reflections created on datasets with more than the limit, on upgrading, materializing such reflections will fail with “extended metadata (read definition) is not available” message. Administrators are recommended to drop all such reflections.