# Bug Report: Schema Discovery Problem with nested documents (Dremio 3.0.0 and MongoDB 3.6.8)

**URL:** <https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174>\
**Category:** Uncategorized\
**Created:** [November 5, 2018, 4:02pm UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174 "2018-11-05T16:02:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tid](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/tid/32/5965_2.png) [@tid](https://community.dremio.com/u/tid)\
**Post date:** [November 5, 2018, 4:02pm UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174/1 "2018-11-05T16:02:34Z")

</div>

Hi Dremio Team,

I’m facing an issue with Schema Discovery (Preview works, Run broken) when using nested JSON structures in MongoDB. The case is pretty easy to reproduce. I add three JSON documents to a Mongo collection:

```auto
[
{
  "o" : {                                   
        "text": "a",       
        "num" : 42                             
      }
    },     
      { 
    "o":
      {                                   
        "text": "b",       
        "num": 43                             
      }
    },
    {
        "o":
      {                                   
        "text": "c",       
        "num": 44                             
      }
    }
]

```

I then execute the following query in PREVIEW mode:

```auto
SELECT CONCAT("coll"."o"."text", 'myString')
FROM mysource.mydb.coll

```

This runs fine – it returns all three “text” attribute values plus “myString”.  
When I switch to “RUN” or execute via JDBC, I get:

```auto
Error: SCHEMA_CHANGE ERROR: New schema found and recorded. Please reattempt the query. Multiple attempts may be necessary to fully learn the schema.

Original Schema schema(o::struct<text::varchar>)
New Schema schema(o::struct<text::varchar, num::int32>)
SqlOperatorImpl MONGO_SUB_SCAN
Location 0:0:2
SqlOperatorImpl MONGO_SUB_SCAN
Location 0:0:2
Fragment 0:0

[Error Id: a37fca37-5532-462c-8f5e-d65d6e00a67c on MYMACHINE:31010]

  (org.apache.arrow.vector.util.SchemaChangeRuntimeException) Schema change error
    com.dremio.common.exceptions.UserException.schemaChangeError():88
    com.dremio.sabot.op.scan.ScanOperator.checkAndLearnSchema():261
    com.dremio.sabot.op.scan.ScanOperator.setupReader():178
    com.dremio.sabot.op.scan.ScanOperator.setup():163
    com.dremio.sabot.driver.SmartOp$SmartProducer.setup():560
    com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():79
    com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():63
    com.dremio.sabot.driver.SmartOp$SmartProducer.accept():530
    com.dremio.sabot.driver.StraightPipe.setup():102
    com.dremio.sabot.driver.StraightPipe.setup():102
    com.dremio.sabot.driver.Pipeline.setup():58
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():347
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():237
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():88
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():594
    com.dremio.sabot.task.AsyncTaskWrapper.run():103
    com.dremio.sabot.task.slicing.SlicingThread.run():110

SQLState: null
ErrorCode: 0

```

**When I try exactly the same with a JSON source (i.e., same data, but no MongoDB), I get no error.**

Query Profile:  
[query\_profile\_concat\_bug.zip](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/1/185d56829839701ec301453bb1035bcce235710a.zip) (41.8 KB)

Sample JSON file to import into MongoDB:  
[concat.zip](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/9/99a6af7e45810a6cada59a270fb94cbfda0dfa4b.zip) (262 Bytes)

Thanks, Tim

---

<div class="post-metadata">

**Author:** ![tid](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/tid/32/5965_2.png) [@tid](https://community.dremio.com/u/tid)\
**Post date:** [November 6, 2018, 6:36pm UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174/2 "2018-11-06T18:36:45Z")

</div>

Has anyone experienced something similar? Are there any workarounds? My team is working on a PoC and this issue is kind of a showstopper to us.

Just for completeness: I simplified the query to make it easy to reproduce. We’re actually trying to do some more complex operations involving parsing timestamp, etc.

Thanks, Tim

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [November 9, 2018, 4:21pm UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174/3 "2018-11-09T16:21:47Z")

</div>

Hi @tid

Thanks for uploading the profiles. Let me look into this and get back to you

Thanks  
@balaji.ramaswamy

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [November 9, 2018, 11:39pm UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174/4 "2018-11-09T23:39:46Z")

</div>

Hi Tim,

Looks like we are running into a minor bug here. We have started to look at this via an internal Engineering ticket and will keep you posted on the progress

Thanks  
@balaji.ramaswamy

---

<div class="post-metadata">

**Author:** ![tid](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/tid/32/5965_2.png) [@tid](https://community.dremio.com/u/tid)\
**Post date:** [November 12, 2018, 1:32pm UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174/5 "2018-11-12T13:32:06Z")

</div>

Thanks, @balaji.ramaswamy!

Glad to hear that the issue is reproducible and under investigation!

Best, Tim

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [November 13, 2018, 1:36am UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174/6 "2018-11-13T01:36:41Z")

</div>

Hi @tid

Are you using the RPM or the TAR version?

Thanks  
@balaji.ramaswamy

---

<div class="post-metadata">

**Author:** ![tid](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/tid/32/5965_2.png) [@tid](https://community.dremio.com/u/tid)\
**Post date:** [November 13, 2018, 6:48am UTC](https://community.dremio.com/t/bug-report-schema-discovery-problem-with-nested-documents-dremio-3-0-0-and-mongodb-3-6-8/2174/7 "2018-11-13T06:48:07Z")

</div>

Hi,  
We’re running on CentOS 7.5, so RPM.

Thanks, Tim