# Null values are not supported in lists by default. Please set \`store.json.all\_text\_mode\` to true to read lists containing nulls. Be advised that this will treat JSON null values as a string containing the word 'null' **URL:** \ **Category:** Uncategorized\ **Created:** [June 15, 2018, 9:10am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310 "2018-06-15T09:10:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 15, 2018, 9:10am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/1 "2018-06-15T09:10:43Z")
I downloaded this file (about 3GB): [https://data.cityofchicago.org/api/views/ijzp-q8t2/rows.json?accessType=DOWNLOAD](https://data.cityofchicago.org/api/views/ijzp-q8t2/rows.json?accessType=DOWNLOAD) Then I tried to open it by Dremio and error message was shown Null values are not supported in lists by default. Please set `store.json.all_text_mode` to true to read lists containing nulls. Be advised that this will treat JSON null values as a string containing the word ‘null’. Can you please advise how I can resolve this issue ---
**Author:** ![anthony](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/anthony/32/670_2.png) [@anthony](https://community.dremio.com/u/anthony)\ **Post date:** [June 15, 2018, 3:17pm UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/2 "2018-06-15T15:17:44Z")
If you scroll to the bottom of [http://host:9047/admin/advanced](http://host:9047/admin/advanced) under “Dremio Support”, there will be an area to copy/paste that setting \> click Show \> then toggle it ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 17, 2018, 2:35am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/3 "2018-06-17T02:35:58Z")
Thank you so much, I was able to turn this option on However, dremio is still unable to load this file, it shows error message: “Error parsing JSON - Unable to expand the buffer” Have you ever tried to open 3Gb file, is 16GB of memory computer not enough to open? ---
**Author:** ![anthony](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/anthony/32/670_2.png) [@anthony](https://community.dremio.com/u/anthony)\ **Post date:** [June 17, 2018, 12:32pm UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/4 "2018-06-17T12:32:01Z")
How do you have Dremio deployment? Windows app? Linux server? ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 17, 2018, 1:25pm UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/5 "2018-06-17T13:25:40Z")
I used linux, specifically ubuntu 16 ---
**Author:** ![anthony](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/anthony/32/670_2.png) [@anthony](https://community.dremio.com/u/anthony)\ **Post date:** [June 17, 2018, 2:18pm UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/6 "2018-06-17T14:18:39Z")
This may need further investigation. 16gb single node is indeed a bit small, and keep in mind some of that goes to heap, not direct memory. Please note we have plenty of users working with larger files though. If I have some free time maybe I’ll try to load the dataset as well… ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 18, 2018, 3:09am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/7 "2018-06-18T03:09:09Z")
So is there any option to change heap size for Dremio? ---
**Author:** ![mishai](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@mishai](https://community.dremio.com/u/mishai)\ **Post date:** [June 18, 2018, 3:14am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/8 "2018-06-18T03:14:23Z")
Yes, you can change the dremio-env file. Here is the documentation: [http://docs.dremio.com/deployment/dremio-config.html#environment-setup](http://docs.dremio.com/deployment/dremio-config.html#environment-setup) ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 18, 2018, 3:27am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/9 "2018-06-18T03:27:39Z")
Thank you, but this option doesn’t resolve problem I changed DREMIO\_MAX\_HEAP\_MEMORY\_SIZE\_MB=12000 DREMIO\_MAX\_DIRECT\_MEMORY\_SIZE\_MB=12000 I checked the RAM and it looks to me Dremio didn’t use RAM to process, MemFree is always 12645252 kB ---
**Author:** ![doron](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/doron/32/294_2.png) [@doron](https://community.dremio.com/u/doron)\ **Post date:** [June 18, 2018, 4:29am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/10 "2018-06-18T04:29:18Z")
Did you restart the node after making the change? That is required. ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 18, 2018, 5:30am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/11 "2018-06-18T05:30:30Z")
Yes, i did of course ---
**Author:** ![christy](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@christy](https://community.dremio.com/u/christy)\ **Post date:** [June 18, 2018, 7:24am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/12 "2018-06-18T07:24:50Z")
My first comment here is that given your 16Gb of ram, those settings are potentially too high. Try setting them to 8GB each and see what happens. If that still doesn’t happen, it would be interesting to see what happens if you could say bisect the file into 2 1.5 GB instances. I’d like to get a better idea as to when this becomes an issue. ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 19, 2018, 1:54am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/13 "2018-06-19T01:54:29Z")
Would you mind spending a little time to load this file to make sure it works on your machine(single machine) [https://data.cityofchicago.org/api/views/ijzp-q8t2/rows.json?accessType=DOWNLOAD](https://data.cityofchicago.org/api/views/ijzp-q8t2/rows.json?accessType=DOWNLOAD) It doesn’t make sense that analyze tool cannot process 3GB file since we’re working with big data Thank you so much ---
**Author:** ![christy](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@christy](https://community.dremio.com/u/christy)\ **Post date:** [June 19, 2018, 8:45am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/14 "2018-06-19T08:45:34Z")
Hey Hai, I can see the issue. It’s due to a very deep and wide schema in the JSON file; whilst trying to schema learn, it’s coming up against an internal limit of an buffer, which doesn’t seem to be settable via the UI. Let me talk to support. I’ll feed back once I have some more news Christy ---
**Author:** ![christy](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@christy](https://community.dremio.com/u/christy)\ **Post date:** [June 19, 2018, 8:53am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/15 "2018-06-19T08:53:18Z")
Hey Hai, Looking at the JSON file again, I noticed that the file is actually a single Object. Dremio is fundamentally a “row” based technology. Essentially, it wants an Array of Objects, so it can treat each entry as a row. Here, Dremio is trying to fit the entire file into a single row and “schema” learn across the entire file. I would suggest that in this instance, some pre-processing of the file is required to extract the data you want and turn it into a new file that is an array of objects. For example, I notice, there is a large meta section at the start of the file. You could strip this out and inside take the actual “row” data (found later in the file as the “data” property) and use that to create a new file. ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 19, 2018, 9:06am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/16 "2018-06-19T09:06:46Z")
Thank you so much Christy, it’s very informative By the way, can you please advise what public large dataset I should use in order to show Dremio’s capability to work with big data file to our data scientist team Thank you so much ---
**Author:** ![christy](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@christy](https://community.dremio.com/u/christy)\ **Post date:** [June 19, 2018, 12:49pm UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/17 "2018-06-19T12:49:34Z")
I’m glad I could help 😃 I often use the Yelp data: [https://www.yelp.com/dataset](https://www.yelp.com/dataset) It’s also a few GB in size, and multiple JSON files, so you can show joins too. Hope this helps 🙂 ---
**Author:** ![kelly](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kelly/32/11_2.png) [@kelly](https://community.dremio.com/u/kelly)\ **Post date:** [June 19, 2018, 7:19pm UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/18 "2018-06-19T19:19:04Z")
Here is a tutorial that uses Yelp, including instructions on how to make the data available: > **[Data Curation With Dremio](https://www.dremio.com/tutorials/data-curation-with-dremio/)** > > Tutorial that helps new users learn how to curate data with Dremio. ---
**Author:** ![kelly](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kelly/32/11_2.png) [@kelly](https://community.dremio.com/u/kelly)\ **Post date:** [June 19, 2018, 7:28pm UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/19 "2018-06-19T19:28:59Z")
Also, here’s a discussion on the multi-record JSON file format Christy mentioned: [Feature requests + Bug reporting](https://community.dremio.com/t/feature-requests-bug-reporting/268/4?u=kelly) ---
**Author:** ![Hai\_Pham](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/hai_pham/32/1487_2.png) [@Hai\_Pham](https://community.dremio.com/u/Hai_Pham)\ **Post date:** [June 20, 2018, 4:16am UTC](https://community.dremio.com/t/null-values-are-not-supported-in-lists-by-default-please-set-store-json-all-text-mode-to-true-to-read-lists-containing-nulls-be-advised-that-this-will-treat-json-null-values-as-a-string-containing-the-word-null/1310/20 "2018-06-20T04:16:51Z")
I changed the dataset and saw Dremio on AWS t2.medium instance can load 4GB file perfectly Thank you so much