Installation Issue while installation via RPM

rajtej · November 20, 2019, 10:40am

Hi . I have installed dremio in linux via RPM in my namenode.
I have ran sudo service dremio start, it is showing dremio is running.
when i tried to open UI using web browser:
localhost:9047/
but UI is not opening??
I have changed the localhost name in dremio.conf with the name node hostname:namenode1.xyzcompany.com but still the UI is not opening. Can you please let me know how to continue further??

nuwan1 · November 20, 2019, 2:18pm

#Install script

sudo groupadd -r dremio

sudo useradd -r -g dremio -d /var/lib/dremio -s /sbin/nologin dremio

sudo mkdir /opt/dremio

sudo mkdir /opt/dremio/log && sudo chown dremio : dremio /opt/dremio/log

sudo mkdir /opt/dremio/run && sudo chown dremio : dremio /opt/dremio/run

sudo mkdir /opt/dremio/data && sudo chown dremio : dremio /opt/dremio/data

# Latest version link can be obtained on https://www.dremio.com/download/ under 'TAR'

cd /tmp

wget https : //download.dremio.com/community-server/dremio-community-LATEST.tar.gz

sudo tar -xvf dremio-community-LATEST.tar.gz -C /opt/dremio/ --strip-components=1

sudo ln -s /opt/dremio/conf /etc/dremio

sudo cp /opt/dremio/share/dremio.service /etc/systemd/system/dremio.service

# Add service

sudo systemctl daemon-reload

sudo systemctl start dremio

# Optionally, set Dremio service to start on boot:

sudo systemctl enable dremio

sudo mount /tmp -o remount,exec

echo “65536” > /proc/sys/fs/file-max

vi /etc/sysctl.conf

fs.file-max = 65536

cd /opt/dremio/data

mkdir ~/dremio-backup

mv db ~/dremio-backup/

service dremio restart

balaji.ramaswamy · November 20, 2019, 4:29pm

@rajtej

Kindly send us the server.log, we can check if there were any errors during startup

Thanks
@balaji.ramaswamy

ben · November 20, 2019, 5:01pm

@rajtej

I have changed the localhost name in dremio.conf with the name node

Can you share your dremio.conf file? It’s unclear to me what you changed. Is Dremio located on the same host as the name node?

In Dremio’s server.log do you see a line like:
2019-11-20 16:54:37,240 [main] INFO com.dremio.dac.server.WebServer - Started on http://localhost:9047

This would indicate that the web service is started and – if you are accessing from the same host machine as the Dremio installation – you should be able to go to your browser and access the UI at http://localhost:9047

rajtej · November 21, 2019, 5:36am

[root@namenode01 dremio]# cat /var/log/dremio/server.log
2019-11-21 04:53:42,908 [main] INFO com.dremio.exec.util.GuavaPatcher - Google’s Stopwatch patched for old HBase Guava version.
2019-11-21 04:53:43,129 [main] INFO com.dremio.exec.util.GuavaPatcher - Google’s Closeables patched for old HBase Guava version.
2019-11-21 04:53:43,386 [main] INFO com.dremio.common.config.SabotConfig - Configuration and plugin file(s) identified in 238ms.
Base Configuration:
- jar:file:/opt/dremio/jars/dremio-common-2.0.5-201806021755080191-767cfb5.jar!/sabot-default.conf

Intermediate Configuration and Plugin files, in order of precedence:
- jar:file:/opt/dremio/jars/dremio-common-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-yarn-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-backend-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-common-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-accelerator-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-elasticsearch-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-hive-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-coordinator-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-jdbc-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-datastore-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-mongo-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-jobs-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-sabot-scheduler-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hbase-plugin-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hive-plugin-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-jdbc-plugin-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-mongo-plugin-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-pdfs-plugin-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-users-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-provision-common-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-s3-plugin-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-logical-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-namespace-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/3rdparty/dremio-hive-exec-shaded-2.0.5-201806021755080191-767cfb5.jar!/sabot-module.conf

rajtej · November 21, 2019, 5:53am

paths: {

the local path for dremio to store data.

local: “/var/lib/dremio”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “pdfs://”${paths.local}"/pdfs"

location for catalog database (if master node)

db: ${paths.local}/db,

spilling: [${paths.local}/spill]

storage area for the accelerator cache.

accelerator: ${paths.dist}/accelerator

staging area for json and csv ui downloads

downloads: ${paths.dist}/downloads

stores uploaded data associated with user home directories

uploads: ${paths.dist}/uploads

stores data associated with the job results cache.

results: ${paths.dist}/results

shared scratch space for creation of tables.

scratch: ${paths.dist}/scratch

}

services: {
coordinator: {
enabled: true,

# Auto-upgrade Dremio at startup if needed
auto-upgrade: false,

master: {
  enabled: true,
  # configure an embedded ZooKeeper server on the same node as master
  embedded-zookeeper: {
    enabled: true,
    port: 2181,
    path: ${paths.local}/zk
  }
},

web: {
  enabled: true,
  port: 9047,
  ssl: {
    # If SSL for communication path between browsers (or REST clients) and Dremio should be enabled.
    enabled: false,

    # Allow for auto-generated certificates if keyStore option is not set
    # Auto-generated self-signed certificates are considered insecure, and this
    # option should be set to false in production environment
    auto-certificate.enabled: true

    # KeyStore and TrustStore settings default to Java keystore and truststore JVM arguments.
    # If needed to be overridden, then change the below properties

    # KeyStore type
    keyStoreType: ${javax.net.ssl.keyStoreType},

    # Path to KeyStore file
    keyStore: ${javax.net.ssl.keyStore},

    # Password to access the keystore file
    keyStorePassword: ${javax.net.ssl.keyStorePassword},

    # Password to access the key
    keyPassword: ${javax.net.ssl.keyPassword},

    # TrustStore type
    trustStoreType: ${javax.net.ssl.trustStoreType},

    # Path to TrustStore file
    trustStore: ${javax.net.ssl.trustStore},

    # Password to access the truststore file
    trustStorePassword: ${javax.net.ssl.trustStorePassword}
  },
  auth: {
    type: "internal", # Possible values are "internal", "ldap" (includes Active directory)
    # LDAP config file for configuration when auth type is "ldap"
    ldap_config: "ldap.json" # file name provided should be in the classpath.
  }
  ui: {
    # Configuration for Intercom
    intercom: {
        enabled: true
        appid:  "@dremio.ui.intercom.appid@"
    }
  }
  tokens: {
    cache: {
      # number of tokens to store locally on this coordinator node
      # (set to 0, if all requests should be made directly to the database of tokens)
      size: 100
      # time (in minutes) after which the token needs to be obtained from the token store
      expiration_minutes: 5
    }
  }
},

client-endpoint: {
  port: 31010
},

scheduler: {
  threads: 24
},

command-pool: {
  enabled: true
  size: 0 # 0 defaults to the machine's number of cores
}

},

executor: {
enabled: true
},

fabric: {
port: 45678,

memory: {
  reservation: 100M
}

},

Set up kerberos credentials in server (applicable for both coordinator and executor)

kerberos: {
principal: “”,
keytab.file.path: “”
}

web-admin: {
enabled: true,
# Port, bound to loopback interface, on which the daemon responds to liveness HTTP requests (0 == auto-allocated)
port: 0
}
}

provisioning: {
yarn: {
jvmoptions: “”,
# list of jars to be added to the main container classpath
classpath: []
app {
# list of jars to be added to the Dremio application classpath
classpath: []
# list of jars to be added to the Dremio application classpath (at the front)
classpath-prefix: []
}
watchdog: {
# how long to wait for server reply before considering it failed
poll.timeout: 1000ms
# interval between two server polls
poll.interval: 10000ms
# how many consecutive failed attempts before killing server
missed.polls.before.kill: 6
# maximum attempts at killing server
max.kill.attempts: 10
# how long before reattempting killing server
kill.reattempt.interval: 1000ms
}
}
}

the zookeeper quorum for the cluster

zookeeper: “localhost:”${services.coordinator.master.embedded-zookeeper.port}
zk.client.session.timeout: 90000

debug: {
enabled: false,
autoPort: false,
prepopulate: false,
singleNode: false,
verboseAccessLog: false,
allowTestApis: false,
forceRemote: false,
useMemoryStorage: false,
addDefaultUser: false,
allowNewerKVStore: true,

to enable remote debugging of the DremioDaemon running in YARN container

yarnremote.enabled: false

UI Red Screen Of Death

rsod.enabled: false

UI File A Bug option

bug.filing.enabled: false

enable on-idle load shedding

task.on_idle_load_shed: true

enable rescheduling task on unblock

task.reschedule_on_unblock: true

Use election service to elect between multiple master candidates

has to be set to false if multiple master candidates

master.election.disabled: false,
}

These system properties are listed here to allow substitution of system property values for DAC Web SSL properties

listed in services.web.ssl section. Currently we consider only the system properties listed in this file for

substitution.

javax.net.ssl {
keyStoreType: “”,
keyStore: “”,
keyStorePassword: “”,
keyPassword: “”,
trustStoreType: “”,
trustStore:"",
trustStorePassword: “”
}

registration.publish-host: “”
paths: {

the local path for dremio to store data.

local: “/var/lib/dremio”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “pdfs://”${paths.local}"/pdfs"

location for catalog database (if master node)

db: ${paths.local}/db,

spilling: [${paths.local}/spill]

storage area for the accelerator cache.

accelerator: ${paths.dist}/accelerator

staging area for json and csv ui downloads

downloads: ${paths.dist}/downloads

stores uploaded data associated with user home directories

uploads: ${paths.dist}/uploads

stores data associated with the job results cache.

results: ${paths.dist}/results

shared scratch space for creation of tables.

scratch: ${paths.dist}/scratch

}

services: {
coordinator: {
enabled: true,

# Auto-upgrade Dremio at startup if needed
auto-upgrade: false,

master: {
  enabled: true,
  # configure an embedded ZooKeeper server on the same node as master
  embedded-zookeeper: {
    enabled: true,
    port: 2181,
    path: ${paths.local}/zk
  }
},

web: {
  enabled: true,
  port: 9047,
  ssl: {
    # If SSL for communication path between browsers (or REST clients) and Dremio should be enabled.
    enabled: false,

    # Allow for auto-generated certificates if keyStore option is not set
    # Auto-generated self-signed certificates are considered insecure, and this
    # option should be set to false in production environment
    auto-certificate.enabled: true

    # KeyStore and TrustStore settings default to Java keystore and truststore JVM arguments.
    # If needed to be overridden, then change the below properties

    # KeyStore type
    keyStoreType: ${javax.net.ssl.keyStoreType},

    # Path to KeyStore file
    keyStore: ${javax.net.ssl.keyStore},

    # Password to access the keystore file
    keyStorePassword: ${javax.net.ssl.keyStorePassword},

    # Password to access the key
    keyPassword: ${javax.net.ssl.keyPassword},

    # TrustStore type
    trustStoreType: ${javax.net.ssl.trustStoreType},

    # Path to TrustStore file
    trustStore: ${javax.net.ssl.trustStore},

    # Password to access the truststore file
    trustStorePassword: ${javax.net.ssl.trustStorePassword}
  },
  auth: {
    type: "internal", # Possible values are "internal", "ldap" (includes Active directory)
    # LDAP config file for configuration when auth type is "ldap"
    ldap_config: "ldap.json" # file name provided should be in the classpath.
  }
  ui: {
    # Configuration for Intercom
    intercom: {
        enabled: true
        appid:  "@dremio.ui.intercom.appid@"
    }
  }
  tokens: {
    cache: {
      # number of tokens to store locally on this coordinator node
      # (set to 0, if all requests should be made directly to the database of tokens)
      size: 100
      # time (in minutes) after which the token needs to be obtained from the token store
      expiration_minutes: 5
    }
  }
},

client-endpoint: {
  port: 31010
},

scheduler: {
  threads: 24
},

command-pool: {
  enabled: true
  size: 0 # 0 defaults to the machine's number of cores
}

},

executor: {
enabled: true
},

fabric: {
port: 45678,

memory: {
  reservation: 100M
}

},

Set up kerberos credentials in server (applicable for both coordinator and executor)

kerberos: {
principal: “”,
keytab.file.path: “”
}

web-admin: {
enabled: true,
# Port, bound to loopback interface, on which the daemon responds to liveness HTTP requests (0 == auto-allocated)
port: 0
}
}

provisioning: {
yarn: {
jvmoptions: “”,
# list of jars to be added to the main container classpath
classpath: []
app {
# list of jars to be added to the Dremio application classpath
classpath: []
# list of jars to be added to the Dremio application classpath (at the front)
classpath-prefix: []
}
watchdog: {
# how long to wait for server reply before considering it failed
poll.timeout: 1000ms
# interval between two server polls
poll.interval: 10000ms
# how many consecutive failed attempts before killing server
missed.polls.before.kill: 6
# maximum attempts at killing server
max.kill.attempts: 10
# how long before reattempting killing server
kill.reattempt.interval: 1000ms
}
}
}

the zookeeper quorum for the cluster

zookeeper: “localhost:”${services.coordinator.master.embedded-zookeeper.port}
zk.client.session.timeout: 90000

debug: {
enabled: false,
autoPort: false,
prepopulate: false,
singleNode: false,
verboseAccessLog: false,
allowTestApis: false,
forceRemote: false,
useMemoryStorage: false,
addDefaultUser: false,
allowNewerKVStore: true,

to enable remote debugging of the DremioDaemon running in YARN container

yarnremote.enabled: false

UI Red Screen Of Death

rsod.enabled: false

UI File A Bug option

bug.filing.enabled: false

enable on-idle load shedding

task.on_idle_load_shed: true

enable rescheduling task on unblock

task.reschedule_on_unblock: true

Use election service to elect between multiple master candidates

has to be set to false if multiple master candidates

master.election.disabled: false,
}

These system properties are listed here to allow substitution of system property values for DAC Web SSL properties

listed in services.web.ssl section. Currently we consider only the system properties listed in this file for

substitution.

javax.net.ssl {
keyStoreType: “”,
keyStore: “”,
keyStorePassword: “”,
keyPassword: “”,
trustStoreType: “”,
trustStore:"",
trustStorePassword: “”
}

registration.publish-host: “localhost”

I have changed localhost with my hostname “namenode01.xyz.com” i tried to save and run but still it dremio is not running

balaji.ramaswamy · November 21, 2019, 5:58am

@rajtej

Kindly send us the server.log in full. The cat of server.log you have sent has only a few jars loading lines

rajtej · November 21, 2019, 6:20am

I tried the above statements. When i check the status it is showing dremio is not running

rajtej · November 21, 2019, 6:57am

server.zip (534.0 KB)

balaji.ramaswamy · November 21, 2019, 2:20pm

@rajtej,

I see this,

Do you have conflicting jars on the classpath?
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.common.scanner.ClassPathScanner.verifyClassUnicity(ClassPathScanner.java:439) [dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.common.scanner.ClassPathScanner.scan(ClassPathScanner.java:414) [dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.common.scanner.RunTimeScan.fromPrescan(RunTimeScan.java:58) [dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.common.scanner.ClassPathScanner.fromPrescan(ClassPathScanner.java:454) [dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:130) [dremio-dac-daemon-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
2019-11-21 11:07:21,978 [main] INFO c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.plugins.adl.store, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.provision.yarn.service,

It is possible the previous RPM install did not remove some jars, try this

Backup your db and conf folders
yum remove rpm-name
cd DREMIO_HOME/jars
rm -rf *
cd DREMIO_HOME/plugins
rm -rf *
yum install rpm-name
copy back db and conf folders
dremio-admin uograde (if required)
start Dremio

Topic		Replies	Views
Dremio Installation	7	4516	December 5, 2019
Dremio service start issues	1	1042	September 23, 2019
Standalone linux installation Dremio University	8	1532	May 19, 2022
Issues after dremio cluster servers restart	1	1370	November 20, 2019
Fail after start dremio service	3	1518	April 27, 2018

Installation Issue while installation via RPM

the local path for dremio to store data.

the distributed path Dremio data including job results, downloads, uploads, etc

location for catalog database (if master node)

storage area for the accelerator cache.

staging area for json and csv ui downloads

stores uploaded data associated with user home directories

stores data associated with the job results cache.

shared scratch space for creation of tables.

Set up kerberos credentials in server (applicable for both coordinator and executor)

the zookeeper quorum for the cluster

to enable remote debugging of the DremioDaemon running in YARN container

UI Red Screen Of Death

UI File A Bug option

enable on-idle load shedding

enable rescheduling task on unblock

Use election service to elect between multiple master candidates

has to be set to false if multiple master candidates

These system properties are listed here to allow substitution of system property values for DAC Web SSL properties

listed in services.web.ssl section. Currently we consider only the system properties listed in this file for

substitution.

the local path for dremio to store data.

the distributed path Dremio data including job results, downloads, uploads, etc

location for catalog database (if master node)

storage area for the accelerator cache.

staging area for json and csv ui downloads

stores uploaded data associated with user home directories

stores data associated with the job results cache.

shared scratch space for creation of tables.

Set up kerberos credentials in server (applicable for both coordinator and executor)

the zookeeper quorum for the cluster

to enable remote debugging of the DremioDaemon running in YARN container

UI Red Screen Of Death

UI File A Bug option

enable on-idle load shedding

enable rescheduling task on unblock

Use election service to elect between multiple master candidates

has to be set to false if multiple master candidates

These system properties are listed here to allow substitution of system property values for DAC Web SSL properties

listed in services.web.ssl section. Currently we consider only the system properties listed in this file for

substitution.

Related Topics