ERROR: java.lang.UnsatisfiedLinkError

1

Hi,

I am running JUnit test based BaseTestQuery on Dremio 4.9.0-202010170313000159-4a37757a. It report the error below. The error is happened when init dfs/dfs_test/dfs_root in TestUtilities. It was working with old version (4.1.1-201912191016540894-05db06db or 4.5.0-202006180205030236-1a5d28da). Could you advise how to fix?

com.dremio.common.exceptions.UserException: Source is not currently available.

	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:804)
	at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:402)
	at com.dremio.exec.catalog.ManagedStoragePlugin.createSource(ManagedStoragePlugin.java:313)
	at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:166)
	at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:377)
	at com.dremio.exec.catalog.CatalogServiceImpl.access$400(CatalogServiceImpl.java:100)
	at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:705)
	at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:863)
	at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:259)
	at com.dremio.exec.util.TestUtilities.addDefaultTestPlugins(TestUtilities.java:92)
	at com.dremio.BaseTestQuery.openClient(BaseTestQuery.java:282)
	at com.dremio.BaseTestQuery.setupDefaultTestCluster(BaseTestQuery.java:204)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1907)
	at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:924)
	at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:347)
	... 24 more
	Suppressed: java.lang.NullPointerException
		at com.dremio.exec.store.dfs.FileSystemPlugin.getSourceCapabilities(FileSystemPlugin.java:266)
		at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:829)
		at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:939)
		... 25 more
Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.getStat(NativeIO.java:460)
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNativeIO(RawLocalFileSystem.java:821)
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:735)
	at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:703)
	at org.apache.hadoop.fs.FileSystem.checkAccessPermissions(FileSystem.java:2610)
	at org.apache.hadoop.fs.FileSystem.access(FileSystem.java:2595)
	at org.apache.hadoop.fs.FilterFileSystem.access(FilterFileSystem.java:460)
	at com.dremio.exec.hadoop.HadoopFileSystem.access(HadoopFileSystem.java:432)
	at com.dremio.exec.store.dfs.FileSystemPlugin.getState(FileSystemPlugin.java:368)
	at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:827)
	at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$1(ManagedStoragePlugin.java:524)
	at com.dremio.exec.catalog.ManagedStoragePlugin$$Lambda$124/1025406634.get(Unknown Source)
	at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$3(ManagedStoragePlugin.java:591)
	at com.dremio.exec.catalog.ManagedStoragePlugin$$Lambda$125/1038325123.get(Unknown Source)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1582)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
2

@popejune

What source is this?

3

Hi @balaji.ramaswamy,

csv file.

Here is my code

public class DremioPluginTest1 extends BaseTestQuery {

    @Test
    public void test_simple_csv() throws Exception {
        GlobalConstants.UT_MOCK_FILE = "/tds3/cars.csv";
        String query = String.format("select \"Year\", Model from TABLE(dfs.\"%s\"(type => 'text', fieldDelimiter => ',', lineDelimiter => '\n', extractHeader => true)) where Make = 'Chevy'",
                getFilePath(GlobalConstants.UT_MOCK_FILE));
        List<QueryDataBatch> results = testSqlWithResults(query);
        printResult(results);

        testBuilder()
                .sqlQuery(query)
                .unOrdered()
                .baselineColumns("Year", "Model")
                .baselineValues("1999", "Venture \"Extended Edition\"")
                .baselineValues("1999", "Venture \"Extended Edition, Very Large\"")
                .go();
    }

    protected String getFilePath(String relativeFilePath){
        return PluginBaseTest.class.getResource(relativeFilePath).toString()
                .replace("file:/C:","file://")
                .replace("C:","file://");
    }
}
4

This issue is due to old hadoop version. It is fixed after I upgrade to Hadoop 3.1.1.

1 Like
5

Which hadoop version were you using when you hit this issue?

6

I am still hitting the error after using hadoop 3.1.1 winutils bin files. Did you happen to choose a specific version for hadoop-common dependency?

7

@zzgzy What version of Dremio are you on? Are you able to send the full stack?