Could NOT config ElasticSearch on Dremio?

1

When I add ES as data source, it could be added successfully, but cannot show any index. The log show the NPE. Actually, I tried to add the ES in another dremio env, no issues. Does anyone have any idea?

2019-10-28 06:38:19,442 [metadata-refresh-SSI] WARN  c.d.e.catalog.MetadataSynchronizer - Source 'xxx' sync failed unexpectedly. Will try again later
java.lang.NullPointerException: null
        at com.dremio.plugins.elastic.mapping.ElasticMappingSet$ClusterMetadata.getIndices(ElasticMappingSet.java:765) ~[dremio-elasticsearch-plugin-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.plugins.elastic.ElasticsearchStoragePlugin.listDatasetHandles(ElasticsearchStoragePlugin.java:314) ~[dremio-elasticsearch-plugin-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.exec.catalog.MetadataSynchronizer.getDatasetHandleListing(MetadataSynchronizer.java:159) ~[dremio-sabot-kernel-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.exec.catalog.MetadataSynchronizer.synchronizeDatasets(MetadataSynchronizer.java:175) ~[dremio-sabot-kernel-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.exec.catalog.MetadataSynchronizer.go(MetadataSynchronizer.java:124) ~[dremio-sabot-kernel-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.exec.catalog.SourceMetadataManager.refreshFull(SourceMetadataManager.java:398) [dremio-sabot-kernel-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.exec.catalog.SourceMetadataManager.doNextRefresh(SourceMetadataManager.java:301) [dremio-sabot-kernel-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.exec.catalog.SourceMetadataManager.access$400(SourceMetadataManager.java:75) [dremio-sabot-kernel-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.exec.catalog.SourceMetadataManager$RefreshTask.run(SourceMetadataManager.java:498) [dremio-sabot-kernel-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) [dremio-common-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) [dremio-common-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) [dremio-common-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:187) [dremio-services-scheduler-3.2.4-201906051751050278-1bcce62.jar:3.2.4-201906051751050278-1bcce62]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_212]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_212]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2

@popejune

What version of Elastisearch is this?

3

ES version is 5.5.2.
Dremio version is 3.2.4

I found the root cause. Because I put fastjson in the 3rdparty. Why it has the conflict? Is there any workaround?

    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>fastjson</artifactId>
        <version>1.2.58</version>
    </dependency>
4

@popejune

I have no idea what fastjson does, does this work without it?

Thanks
@balaji.ramaswamy

5

It’s a kind of json parser like jackson, but more powerful and easy to use.

If I switch to jackson, it’s need a lot of dev and testing work. Could you get chance to take a look and help to fix?

FYI.

6

@popejune

Unfortunately not, Dremio is an open source project and you should be able to buil the code, change and test it yourself

Thanks
@balaji.ramaswamy