TimeoutException on uploading csv/json files

I have a problem uploading large files via ui to the server. After 30 seconds, I have an IdleTimeoutException. Try to solve this problem setting jetty option in java options using Dremio-env config file. Somebody help me solve this problem!

2019-04-22 18:36:38,296 [qtp1444629601-121] ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing POST http://localhost:9047/apiv2/home/%40sensa/upload_start/?extension=json : java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:213) ~[jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent(HttpInputOverHTTP.java:66) ~[jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.HttpInput$1.waitForContent(HttpInput.java:489) ~[jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:122) ~[jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:102) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream.read(ReaderInterceptorExecutor.java:296) ~[jersey-common-2.25.1.jar:na]

at org.jvnet.mimepull.MIMEParser.fillBuf(MIMEParser.java:445) ~[mimepull-1.9.6.jar:1.9.6]

at org.jvnet.mimepull.MIMEParser.skipPreamble(MIMEParser.java:312) ~[mimepull-1.9.6.jar:1.9.6]

at org.jvnet.mimepull.MIMEParser.access$300(MIMEParser.java:68) ~[mimepull-1.9.6.jar:1.9.6]

at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:154) ~[mimepull-1.9.6.jar:1.9.6]

at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:132) ~[mimepull-1.9.6.jar:1.9.6]

at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:228) ~[mimepull-1.9.6.jar:1.9.6]

at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:189) ~[mimepull-1.9.6.jar:1.9.6]

at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:115) ~[mimepull-1.9.6.jar:1.9.6]

at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.getMimeParts(MultiPartReaderClientSide.java:272) ~[jersey-media-multipart-2.25.1.jar:na]

at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.readMultiPart(MultiPartReaderClientSide.java:231) ~[jersey-media-multipart-2.25.1.jar:na]

at org.glassfish.jersey.media.multipart.internal.MultiPartReaderServerSide.readMultiPart(MultiPartReaderServerSide.java:91) ~[jersey-media-multipart-2.25.1.jar:na]

at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.readFrom(MultiPartReaderClientSide.java:183) ~[jersey-media-multipart-2.25.1.jar:na]

at org.glassfish.jersey.media.multipart.internal.MultiPartReaderClientSide.readFrom(MultiPartReaderClientSide.java:93) ~[jersey-media-multipart-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:256) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:235) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:155) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:74) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:155) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1085) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:874) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:808) ~[jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:234) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.media.multipart.internal.FormDataParamValueFactoryProvider$ValueFactory.getEntity(FormDataParamValueFactoryProvider.java:126) ~[jersey-media-multipart-2.25.1.jar:na]

at org.glassfish.jersey.media.multipart.internal.FormDataParamValueFactoryProvider$FormDataParamValueFactory.provide(FormDataParamValueFactoryProvider.java:280) ~[jersey-media-multipart-2.25.1.jar:na]

at org.glassfish.jersey.server.spi.internal.ParamValueFactoryWithSource.provide(ParamValueFactoryWithSource.java:71) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:90) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:127) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]

at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]

at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]

at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]

at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161) ~[jetty-io-9.2.26.v20180806.jar:9.2.26.v20180806]

at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) ~[jetty-io-9.2.26.v20180806.jar:9.2.26.v20180806]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_212]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_212]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]

… 1 common frames omitted

You may want to reconsider how you work with these files. When you upload to Dremio, you are making a copy of that file you will no longer be able to edit. Also, depending on what you mean by “large” the upload step may be problematic for a number of different reasons.

Have you considered using the NAS data source option? This will allow you to read any file that is local to where Dremio is running. It will also allow you to make edits to the file which Dremio will see on next query. In addition, it will allow you to treat a directory of files with the same layout as a single dataset.

https://docs.dremio.com/data-sources/files-and-directories.html#directories

I read about NAS and HDFS data sources, but in this case I was interested to test the functionality of uploading files to the server. Speaking of file size, I’m having problems with json files that are about hundreds of megabytes in size. And I also tried to download a zip archive size of about 4 gigabytes with which in terms of download there were no problems. I know that dremio does not work with archives, I was wondering how dremio will behave if the user will download the archive.

Dremio doesn’t work with archives?

As I understand on uploading, Dremio can work with xlsx, json, csv, text. But maybe I’m wrong.

Dremio can directly query compressed files and archives. This may not work for uploaded files, but it does work for NAS, S3, ADLS, HDFS, etc.

I don’t know that. Thank you for this information. But what about topic problem? Do you know how to change Dremio idle timeout time?