# IndexOutofBoundsException: Index: 6, Size: 3 **URL:** \ **Category:** Uncategorized\ **Created:** [March 25, 2022, 9:23pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955 "2022-03-25T21:23:04Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![rpurdom](https://avatars.discourse-cdn.com/v4/letter/r/3d9bf3/32.png) [@rpurdom](https://community.dremio.com/u/rpurdom)\ **Post date:** [March 25, 2022, 9:23pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/1 "2022-03-25T21:23:04Z")
I’ve been kicking the tires on Dremio and I keep running into various IndexOutofBoundsException’s. I have a nested parquet dataset, 290+MM records of family, Listof , Listof\. The dataset spans 8 shards and roughly 40ish parquet files, roughly 1.2TB’s total data. I’ve created a flattend version of individuals that contains, just the familyid, and the Individuals struct. When I query from this view, “SELECT count(distinct individual[‘Individualid’]) FROM flat\_ind” I get IndexOutofBoundsException. I get this with any version of querying a column across ‘individual’. My first thought that this must be a bad file in the dataset. So, I uploaded all of it to a bucket on Google and attached the parquet as an external table in BigQuery. Though BigQuery support for Parquet is strange (inserts an extra level of struct into the data structure), I was able to execute select count(distinct) using the very same parquet files (roughly 5 seconds to run). Also, we have a 1% sample dataset with exact same schema and queries run flawlessly. Attached is the profile for the query. I’m hoping that Dremio can be a central hub for our datasets. But this is a simple use case and it only gets much more complicated from here. Robert Purdom Co-founding Managing Director Intelafy LLC [35d7efb0-77af-47e8-b762-096375f726d1.zip](https://community.dremio.com/uploads/short-url/sLltHtTyu1q7ADGnNyZrVeAdYmE.zip) (94.2 KB) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [March 31, 2022, 7:13am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/2 "2022-03-31T07:13:34Z")
@rpurdom Would like to narrow down the PARQUET file that is hitting this issue, if you give a FILTER with a partition column, do you see the error, meanwhile will check internally here ---
**Author:** ![rpurdom](https://avatars.discourse-cdn.com/v4/letter/r/3d9bf3/32.png) [@rpurdom](https://community.dremio.com/u/rpurdom)\ **Post date:** [April 3, 2022, 1:36pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/3 "2022-04-03T13:36:40Z")
@balaji.ramaswamy I don’t believe the problem is in the data. I wrote a quick Go program that simply reads the family and individual id and count’s instances of both. It runs very quickly (~4 seconds) and returns the correct count. I’m literally counting the instances, not just looking at metadata. And, as I mentioned in my original post, it also works flawlessly on BigQuery as an external table. BigQuery is not an option as really, I’m looking for a solution to in local environment to support data science and data discovery. It’s not partitioned in the traditional sense. It’s simply using hash partitions (sharding) based on family id split into 8 sub-directories and then split out into files of (4096_1024) family rows and the associated individuals and topics are lists of structs within family. Technically, individuals could have topics, but this particular file does not. The end result is 48 parquet files spread across eight ‘buckets’, seven files of 1024_4096 families and an 8th file of the remainder. However, saying this, Dremio errors at the same point every time. I’m simply running the service using docker run, though I do override memory constraints, otherwise it crashes out with memory constraint issues. I’m giving the container 64GB of memory, but it ticks just over 13GB of usage when it errors out the query. I’ve had the suspicion that I need to deploy this in Kubernetes and have perhaps one docker job just isnt a good idea. For one, it’s way too slow as I’m running it now. Just need to get to a point where I have time to continue testing. Robert ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [April 3, 2022, 5:52pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/4 "2022-04-03T17:52:52Z")
@rpurdom Is this data sensitive? Wondering if you can pass us the actual file if not sensitive, can get to the fix quickly. Meanwhile I will check internally on any open issues on this ---
**Author:** ![rpurdom](https://avatars.discourse-cdn.com/v4/letter/r/3d9bf3/32.png) [@rpurdom](https://community.dremio.com/u/rpurdom)\ **Post date:** [April 4, 2022, 1:48pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/5 "2022-04-04T13:48:10Z")
It’s very sensitive, it belongs to a client. I can probably create a scrubbed version, let me think of a quick way to knock it out. Robert ---
**Author:** ![aki0086](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/aki0086/32/4181_2.png) [@aki0086](https://community.dremio.com/u/aki0086)\ **Post date:** [May 26, 2022, 1:43am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/6 "2022-05-26T01:43:01Z")
was this problem ever resolved? it looks like I’m similar issue. I can query the table, but any time I do a distinct or some type of aggregate function, I et an IndexOutofBoundsException ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [May 28, 2022, 6:59am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/7 "2022-05-28T06:59:03Z")
@aki0086 Want to check if this iss the same stack as @rpurdom, are you able to provide the profile? ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [May 28, 2022, 7:04am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/8 "2022-05-28T07:04:57Z")
@rpurdom This stack looks very similar to an issue fixed in 20.3, would you be able to upgrade and test? ---
**Author:** ![aki0086](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/aki0086/32/4181_2.png) [@aki0086](https://community.dremio.com/u/aki0086)\ **Post date:** [June 22, 2022, 5:59pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/9 "2022-06-22T17:59:26Z")
@balaji.ramaswamy [aefbe3ae-5333-47e1-bcde-a89ebdf5c825.zip](https://community.dremio.com/uploads/short-url/gv0KMObTAoCcv86plhK735QQ70h.zip) (14.1 KB) here is the trace for the below version version commit\_id commit\_message commit\_time build\_email build\_time 20.1.0-202202061055110045-36733c65 36733c65bab0f37953474319d90a4cfff7f2a2ca [NULL] 2022-02-04T00:48:35+0000 2022-02-06T11:05:25+0000 when i query on an older version we have… dont get the same issue. the same data is loaded into both environments. version commit\_id commit\_message commit\_time build\_email build\_time 14.0.0-202103011714040666-9a0c2e10 0073ec1dd24c23e2b6ef4652ca3fea0a93e06873 [NULL] 2021-03-01T13:44:39-0500 2021-03-29T19:28:08-0400 ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [July 2, 2022, 7:09am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/10 "2022-07-02T07:09:08Z")
@aki0086 This seems to have been fixed in 20.3 or 21.x or 22.x, can you please validate? Meanwhile can you please try to disable “store.parquet.read\_column\_indexes” and see if that helps? ---
**Author:** ![aki0086](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/aki0086/32/4181_2.png) [@aki0086](https://community.dremio.com/u/aki0086)\ **Post date:** [August 3, 2022, 4:08pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/11 "2022-08-03T16:08:39Z")
Unfortunately just disabling didn’t help. For the bug you mentioned…is it referring to one of these fixes? [https://docs.dremio.com/software/release-notes/200-release/](https://docs.dremio.com/software/release-notes/200-release/) 20.4 - When attempting to download certain query results as JSON or Parquet files, the downloaded file size was zero bytes and resulted in an `IndexOutofBounds` exception. or 20.3 - Intermittent jobs were failing with an `IndexOutOfBounds` exception while preparing operator details information for runtime filters. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [August 9, 2022, 10:47pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/12 "2022-08-09T22:47:09Z")
@aki0086 Are you able to send me the profile when it failed after you disabled `store.parquet.read_column_indexes`? ---
**Author:** ![aki0086](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/aki0086/32/4181_2.png) [@aki0086](https://community.dremio.com/u/aki0086)\ **Post date:** [August 11, 2022, 4:01pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/13 "2022-08-11T16:01:33Z")
sent to you via direct msg ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [September 30, 2022, 10:11am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/14 "2022-09-30T10:11:52Z")
@aki0086 I do not seem to find the profile, would you mind sending it again please? [balaji@dremio.com](mailto:balaji@dremio.com) ---
**Author:** ![aki0086](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/aki0086/32/4181_2.png) [@aki0086](https://community.dremio.com/u/aki0086)\ **Post date:** [October 5, 2022, 2:04pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/15 "2022-10-05T14:04:24Z")
Hi @balaji.ramaswamy i’ve sent it again over direct msg. let me know if you have any issues with it ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 6, 2022, 5:17am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/16 "2022-10-06T05:17:04Z")
@aki0086 Can you try to disable “store.parquet.read\_column\_indexes” and see if the query works? If it does then this is something that is probably fixed in 21.x ---
**Author:** ![aki0086](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/aki0086/32/4181_2.png) [@aki0086](https://community.dremio.com/u/aki0086)\ **Post date:** [October 6, 2022, 7:26pm UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/17 "2022-10-06T19:26:18Z")
I did. and it didnt work. there are two traces i sent. one with it on and one with it off. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 8, 2022, 6:45am UTC](https://community.dremio.com/t/indexoutofboundsexception-index-6-size-3/8955/18 "2022-10-08T06:45:50Z")
Hi @aki0086 Did you send it via a direct message?