Cannot build Dremio OSS because of private NPM modules

1

I’m trying to build Dremio following the instructions detailed at dremio-oss GH repository so I’m using command below:

./mvnw clean install -DskipTests -Ddremio.oss-only=true

Maven seems to install both Node and pnpm:

 [INFO] Installing node version v22.0.0
(...)
 [INFO] Installed node locally.
 [INFO] Installing pnpm version 9.3.0
(...)
 [INFO] Installed pnpm locally.

And I’m finding the error below when it tries to install the NPM deps for dremio-js:

 [INFO] --- frontend:1.14.0:pnpm (install dependencies) @ dremio-js ---
(...)
 [INFO] FetchError: request to https://repo.drem.io/repository/npm/temporal-polyfill/-/temporal-polyfill-0.2.4.tgz failed, reason: getaddrinfo ENOTFOUND repo.drem.io
 [INFO]     at ClientRequest.<anonymous> (/bitnami/blacksmith-sandox/dremio-25.1.1/ui/dremio-js/target/frontend/node/node_modules/pnpm/dist/pnpm.cjs:68823:18)
 [INFO]     at ClientRequest.emit (node:events:520:28)
 [INFO]     at TLSSocket.socketErrorListener (node:_http_client:500:9)
 [INFO]     at TLSSocket.emit (node:events:532:35)
 [INFO]     at emitErrorNT (node:internal/streams/destroy:169:8)
 [INFO]     at emitErrorCloseNT (node:internal/streams/destroy:128:3)
 [INFO]     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

It seems it tries to download the NPM modules from some private NPM registry so it’s not possible to build Dremio.

2

@juan131 ui/dremio-js/pnpm-lock.yaml remove tarball: https://repo.drem.io part rebuild again

3

Thanks so much @rongfengliang ! That helped me workaround the original issue. Now I’m facing the error below:

Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.14.0:pnpm (build minimized webpack bundle) on project dremio-dac-ui:
(...)
 [INFO] FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Is there any way I can configure Maven to increase the memory limits for JS?

4

I think you can config

export NODE_OPTIONS="--max-old-space-size=8192"

then run mvn build again

5

Hi @rongfengliang

Thanks so much for the workaround for the memory limits, it seems it worked! However, I’m still unable to complete the Dremio build due to errors with JavaScript (webpack), now I’m facing the error below:

 [INFO] > dremio-ui@ build /dremio-25.1.1/dac/ui
 [INFO] > cross-env NODE_ENV=production webpack "--output-path=/dremio-25.1.1/dac/ui/target/classes/rest/dremio_static"
 [INFO] 
 [INFO] <s> [webpack.Progress] 0%
 (...)
 [INFO] <s> [webpack.Progress] 92% sealing asset processing SwcMinifyWebpackPlugin
 [INFO] ???ELIFECYCLE??? Command failed with exit code 1
6

Maybe you can delete the installed npm dependencies and run the build again. When I build the OSS package, I only encounter a private npm problem. I don’t know your environment (os, account, sys resource)

7

Hi @rongfengliang

Could it be related to how references to repo.dremio.io were removed from pnpm-lock.yaml? In my case, I did a replacement like the one below so the “tarball” section was removed:

    .replace(
      /resolution: {integrity: ([^,]+), tarball: [^}]+}/g,
      'resolution: {integrity: $1}',
    )

Original lines:

(...)
  temporal-polyfill@0.2.4:
    resolution: {integrity: sha512-WA5p0CjQTkMjF9m8sP4wSYgpqI8m2d4q7wPUyaJOWhy4bI9mReLb2yGvTV4qf/DPMTe6H6M/Dig5KmTMB7ev6Q==, tarball: https://repo.drem.io/repository/npm/temporal-polyfill/-/temporal-polyfill-0.2.4.tgz}

  temporal-spec@0.2.4:
    resolution: {integrity: sha512-lDMFv4nKQrSjlkHKAlHVqKrBG4DyFfa9F74cmBZ3Iy3ed8yvWnlWSIdi4IKfSqwmazAohBNwiN64qGx4y5Q3IQ==, tarball: https://repo.drem.io/repository/npm/temporal-spec/-/temporal-spec-0.2.4.tgz}
(...)

After my change:

(...)
  temporal-polyfill@0.2.4:
    resolution: {integrity: sha512-WA5p0CjQTkMjF9m8sP4wSYgpqI8m2d4q7wPUyaJOWhy4bI9mReLb2yGvTV4qf/DPMTe6H6M/Dig5KmTMB7ev6Q==}

  temporal-spec@0.2.4:
    resolution: {integrity: sha512-lDMFv4nKQrSjlkHKAlHVqKrBG4DyFfa9F74cmBZ3Iy3ed8yvWnlWSIdi4IKfSqwmazAohBNwiN64qGx4y5Q3IQ==}
(...)
8

It look fine, I just modify pnpm-lock.yaml try to delete project node_modules and rebuild again

1 Like
9

That made it! Thanks so much for you help @rongfengliang