Getting "Number of even though the dataset has 14 fields

Rohithzr · October 18, 2019, 9:19pm

I have a mongo connection where the dataset on a particular collection has only 14 fields but I still get the error “Number of fields in dataset exceeded the maximum number of fields of 800.”.

Below is the server.log and attached is the profile. Can you help me out here?
7882a5cf-90d7-4820-8bad-262445f315b1.zip (5.6 KB)

2019-10-18 21:03:37,780 [qtp1833904548-162] WARN  c.d.p.mongo.metadata.MongoVersion - Could not get the mongo compatibility version from the server. Defaulting to detected version: 3.4
com.mongodb.MongoCommandException: Command failed with error 13: 'not authorized on admin to execute command { getParameter: true, featureCompatibilityVersion: true }' on server staging-db.mongodb.net:27017. The full response is { "ok" : 0.0, "errmsg" : "not authorized on admin to execute command { getParameter: true, featureCompatibilityVersion: true }", "code" : 13, "codeName" : "Unauthorized" }
	at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:164) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:295) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:226) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:197) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:189) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:99) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:92) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.Mongo$3.execute(Mongo.java:826) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.executeCommand(MongoDatabaseImpl.java:166) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:138) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:133) ~[mongo-java-driver-3.6.4.jar:na]
	at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:123) ~[mongo-java-driver-3.6.4.jar:na]
	at com.dremio.plugins.mongo.metadata.MongoVersion.getVersionForConnection(MongoVersion.java:93) ~[dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.metadata.MongoTopology.getMinVersion(MongoTopology.java:219) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.metadata.MongoTopology.getClusterTopology(MongoTopology.java:167) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.connection.MongoConnectionManager.getTopology(MongoConnectionManager.java:174) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.plugins.mongo.MongoStoragePlugin.getDatasetHandle(MongoStoragePlugin.java:100) [dremio-ce-mongo-plugin-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.ManagedStoragePlugin.getDatasetHandle(ManagedStoragePlugin.java:440) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:308) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.DatasetManager.getTable(DatasetManager.java:208) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.CatalogImpl.getTable(CatalogImpl.java:130) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.SourceAccessChecker.lambda$getTable$3(SourceAccessChecker.java:103) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.SourceAccessChecker.checkAndGetTable(SourceAccessChecker.java:82) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.SourceAccessChecker.getTable(SourceAccessChecker.java:103) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.DelegatingCatalog.getTable(DelegatingCatalog.java:66) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.exec.catalog.CachingCatalog.getTable(CachingCatalog.java:93) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287) ~[hk2-utils-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorImpl.internalInvoke(MethodInterceptorImpl.java:109) ~[hk2-locator-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorImpl.invoke(MethodInterceptorImpl.java:125) ~[hk2-locator-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorInvocationHandler.invoke(MethodInterceptorInvocationHandler.java:62) ~[hk2-locator-2.5.0-b32.jar:na]
	at com.sun.proxy.$Proxy116.getTable(Unknown Source) ~[na:na]
	at com.dremio.dac.explore.DatasetsResource.getDatasetSummary(DatasetsResource.java:258) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:134) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at com.dremio.dac.explore.DatasetsResource.newUntitledFromParent(DatasetsResource.java:196) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) ~[jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) ~[jersey-container-servlet-core-2.25.1.jar:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:52) ~[dremio-dac-backend-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51) ~[jetty-servlets-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[jetty-servlet-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:56) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.Server.handle(Server.java:502) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[jetty-server-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) ~[jetty-util-9.4.15.v20190215.jar:9.4.15.v20190215]
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_222]

balaji.ramaswamy · October 21, 2019, 4:07am

@Rohithzr,

Is this a nested json?

Rohithzr · October 21, 2019, 6:09am

yes here is a sample. https://gist.github.com/Rohithzr/67bce44511e7b94efe010d94e9eff35a

Sorabh_Sachdeva · October 22, 2019, 7:49am

@Rohithzr

I am able to load provided json and query from dremio without any issue.
Please share more details about dremio,mongodb version along with steps you are following to load data into mongodb.

moa101 · October 22, 2019, 6:18pm

I work with @Rohithzr;

MongoDB:

version 3.4.23.

Dremio:

Build: 4.0.2-201910020123580864-a98a0b9
Edition: Community Edition
Build Time: 10/01/2019 18:25:55
Change Hash: a98a0b935f4467740ea655c478cdf14ed26c9f0f
Change Time: 10/01/2019 16:39:46

Also when an aggregate pipeline (see below for code) on the mongodb collection I get the following:

Notice that none of our docs have more than 17 root fields.

_id: null
min-field-count:13
max-field-count:17
docs_count:1460

Code:

    {
        '$project': {
            'numFields': {
                '$size': {
                    '$objectToArray': '$$ROOT'
                }
            }
        }
    }, {
        '$group': {
            '_id': None, 
            'min-field-count': {
                '$min': '$numFields'
            }, 
            'max-field-count': {
                '$max': '$numFields'
            }, 
            'docs_count': {
                '$sum': 1
            }
        }
    }
]```

moa101 · October 22, 2019, 6:37pm

@Rohithzr

Just came across this … do you think this is related?

source: https://docs.dremio.com/release-notes/33-release-notes.html#core-services

On upgrading to 3.2, reflections on datasets with more than 800 leaf fields will fail to materialize with a NullPointerException.

Dremio supports datasets with at most 800 leaf fields. Prior to upgrade to 3.2, if there were reflections created on datasets with more than the limit, on upgrading, materializing such reflections will fail with “extended metadata (read definition) is not available” message. Administrators are recommended to drop all such reflections.

Topic		Replies	Views
Max Fields in dataset using MongoDB	3	1697	September 15, 2019
MongoDB IllegalArgumentException: UnSupported Bson type: UNDEFINED, was working before	3	1882	March 13, 2020
Dremio timeout connecting mongo	3	186	April 22, 2024
How to handle Dremio Limit 32000 bytes of a field Dremio University	20	8203	May 16, 2025
Field 'xyz' exceeds the size limit of 32000 bytes (though I don't have that filed in the Query) Dremio University	6	2945	November 5, 2020

Getting "Number of even though the dataset has 14 fields

Related topics