MapR - Dremio Installation

1

Hi Laurent,

I tried installing the dremio on my MapR single node cluster.

Following are the details,
OS: Centos 6
Dremio: dremio-community-1.3.0-201711211846410277_31f8d91_mapr_1.noarch.rpm
open jdk version 1.8.0_151

ERROR:
Error
[root@mcdhpoc3 bin]# tail -f /opt/dremio/log/server.out
04:53:38.496 [main] DEBUG org.apache.hadoop.conf.Configuration - Loaded org.apache.hadoop.conf.CoreDefaultProperties
04:53:38.504 [main] DEBUG org.apache.hadoop.conf.Configuration - Loaded org.apache.hadoop.yarn.conf.YarnDefaultProperties
04:53:38.512 [main] DEBUG com.dremio.datastore.StoreLoader - Loaded the following StoreCreationFunctions: [class com.dremio.dac.support.SupportService$SupportStoreCreator, class com.dremio.dac.service.datasets.DatasetVersionMutator$VersionStoreCreator, class com.dremio.service.accelerator.store.AccelerationStore$AccelerationStoreCreator, class com.dremio.service.accelerator.store.MaterializationStore$MaterializationStoreCreator, class com.dremio.exec.store.StoragePluginRegistryImpl$StoragePluginCreator, class com.dremio.service.users.SimpleUserService$UserGroupStoreBuilder, class com.dremio.provision.service.ProvisioningServiceImpl$ProvisioningStoreCreator, class com.dremio.exec.server.options.SystemOptionManager$OptionStoreCreator, class com.dremio.service.jobs.LocalJobsService$JobsProfileCreator, class com.dremio.service.namespace.NamespaceServiceImpl$NamespaceStoreCreator, class com.dremio.exec.store.CatalogServiceImpl$CatalogSourceDataCreator, class com.dremio.service.jobs.LocalJobsService$JobsStoreCreator, class com.dremio.dac.server.tokens.TokenStoreCreator, class com.dremio.service.accelerator.store.AccelerationEntryStore$AccelerationHolderStoreCreator, class com.dremio.service.namespace.NamespaceServiceImpl$DatasetSplitCreator].
04:53:38.512 [main] INFO c.d.datastore.LocalKVStoreProvider - LocalKVStoreProvider is up
04:53:38.513 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
04:53:38.526 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProvider
Exception in thread “main” java.lang.NullPointerException: No Cluster Identity found
** at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226)**
** at com.dremio.dac.daemon.DremioDaemon.checkVersion(DremioDaemon.java:115)**
** at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:151)**

Please let me know if you require further details

Regards,
Vivek

2

Hi @gvivek,

Is it an upgrade or a fresh installation?

3

Its a fresh Installation.

4

If it’s a fresh install, can you check if you have a directory named /var/lib/dremio/db (or if you changed paths.local in dremio.conf, can you check ${paths.local}/db? If a directory exists, it might cause Dremio to think there was a previous version installed and expecting an upgrade.

Since that’s a fresh install, you can simply remove the whole directory and try to restart Dremio.

5

Hi Laurent - Did remove the directory and re-started the dremio with the following command line
./dremio --config …/conf/dremio.conf restart
Still getting the same issue. Not sure if i’m making any issue with the conf file.
Let me know if you require any details from conf file.

FYI -
paths: {

the local path for dremio to store data.

local: “/var/lib/dremio”
…
}

6

Since you downloaded the rpm, I was assuming that you used service to start/stop the dremio daemon. Also, the RPM installs a default configuration under /etc/dremio but it seems you are using a different config file.

Can you post your config file, and also confirm which user do you use to start dremio and which users owns the /var/lib/dremio directory?

7

You are rite, I had tried both the ways

  • service dremio restart
  • executing the dremio script

the /var/lib/dremio folder and files under it is owned by mapr user

[root@mcdhpoc3 conf]# ls -ltr /var/lib/dremio
total 8
drwxr-xr-x 3 mapr mapr 4096 Dec 5 13:51 zk
drwxr-xr-x 5 mapr mapr 4096 Dec 5 13:51 db
[root@mcdhpoc3 conf]# ls -ltrd /var/lib/dremio
drwxr-xr-x 4 mapr mapr 4096 Dec 5 13:51 /var/lib/dremio

Following is the dremio.conf file.
[
[root@mcdhpoc3 conf]# cat dremio.conf

Copyright © 2017 Dremio Corporation

Licensed under the Apache License, Version 2.0 (the “License”);

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an “AS IS” BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

master: {

the name of the master server. If this node matches the name, it starts the master service

name: mcdhpoc3,
port: 45678
}

services: {
executor.enabled: true,
coordinator :{
enabled: true,
auto-upgrade: false,
web: {
enabled: true,
port: 9047
},
client-endpoint: {
port: 1011
}
}
}

paths: {

the local path for dremio to store data.

local: “/var/lib/dremio”

the distributed path Dremio data including job results, downloads, uploads, etc

#dist: “pdfs://”${paths.local}"/pdfs"
dist: “maprfs:///dremio/pdfs”
}
provisioning.yarn.classpath:[${DREMIO_HOME}"/jars/bundled/dremio-shimloader-1.3.0-201711211846410277-31f8d91-mapr.jar",${DREMIO_HOME}"/jars/3rdparty/maprfs-5.2.1-mapr.jar"]

services.coordinator.embedded_master_zk.enabled: true
services.coordinator.embedded_master_zk.port: 5181

zookeeper: “mcdhpoc3:5181”

the zookeeper quorum for the cluster

#zookeeper: ${master.name}"\:"${services.coordinator.embedded_master_zk.port}"

debug: {
enabled: false,
autoPort: false,
prepopulate: false,
singleNode: false,
verboseAccessLog: false,
allowTestApis: false,
forceMaster: false,
forceRemote: false,
useMemoryStorage: false,
addDefaultUser: false,

to enable remote debugging of the DremioDaemon running in YARN container

yarnremote.enabled: false
}
]

8

Thanks for sharing,

Can you try again the following steps:
1/ confirm that Dremio is not running
2/ delete /var/lib/dremio/db (and confirm it is deleted: ls /var/lib/dremio/db should say no such file or directory)
3/ start dremio using service: service dremio start
4/ if still not working, share /var/log/dremio/server.out and /var/log/dremio/server.log

9

Hi,

Following are the requested details,

[root@mcdhpoc3 ~]# service dremio status
dremio not running.
[root@mcdhpoc3 ~]# date
Tue Dec 5 14:50:40 EST 2017
[root@mcdhpoc3 ~]# ls -ltr /var/lib/dremio/db
ls: cannot access /var/lib/dremio/db: No such file or directory
[root@mcdhpoc3 ~]# date
Tue Dec 5 14:51:09 EST 2017
[root@mcdhpoc3 ~]# service dremio start
starting dremio, logging to /var/log/dremio/server.out
[root@mcdhpoc3 ~]# tail -f /var/log/dremio/server.out

Still i have the same issue,

Attached is the server log and server out file.
Couldnt get the complete one from the server copied few 100 lines.

Regards,
Vivek

10

I don’t see any attachment unfortunately

11

Previous message was an email dont know why its not showing up.
However attached it here as zip file.
Logs.zip (2.6 KB)

12

From the server.out log, it seems you also have a port binding problem. Can you change services.coordinator.client-endpoint.port to a free port higher than 1024? On Unix systems, only processes running as root can bind to a port < 1024.

Then, retry the steps above

13

Thanks Laurent for your time.
I tried the same process twice, thrice but it dint work.
But today morning before reaching out to you and your team thought of trying it again fresh.
It worked without any issues.
Last few days

  • updated my Java from 1.7 to 1.8
  • upated my Python from 2.6 to 2.7

Currently i was able to connect to

  • MySQL
  • MapRFS
  • Hive

Wanted to check with you if Dremio supports ORC files too and how to connect to MapRDB
I will start a new thread too.