Hi, we are testing dremio to use it execute hive sqls. During concurrent testing, many ‘Stream closed’ exception threw out. When reading dremio code, i find a problem. As shown in doCommand() function. There might be concurrent problems when many requests execute doCommand() function in class ‘com.dremio.exec.store.hive.HiveClient’. For example, when ‘TException’ throw out from the try clause, client is closed. Before the client reconnected some requests come and use the closed client to execute their commands. Then ‘java.io.IOException: Stream closed’ exception will be threw out. So the ‘synchronized’ key word should outside doCommand() function, or wrap all the try-catch clause inside. Full trace:
(org.apache.thrift.transport.TTransportException) java.io.IOException: Stream closed
Correct me if I’m wrong.
ben
March 1, 2019, 2:46pm
2
Hi @hanbingwinter ,
What version of Hive are you using?
What version of Dremio?
dremio 3.1.0-201901172111160703-dc6f6e5
ben
March 3, 2019, 5:35pm
4
Hi @hanbingwinter , how many concurrent queries are you executing here? Can you share any more detail on the concurrency testing?
Total 7338 hive sql, 8 concurrent thread. First each thread convert it’s hive sql to another that dremio can recognize. Then execute "EXPLAIN PLAN without implementation FOR " + sql to test if any grammar dremio doesn’t support. During the sql compatibility concurrent testing error occurred. Apart from “java.io.IOException: Stream closed”, i think exceptions below also related to hive client concurrency. About 237 tests have these exceptions. And after we changed the code as i said above these exceptions disappeared.
get_partitions failed: out of sequence response
Cannot write to null outputStream
java.net.SocketException: Connection reset
Cannot read from null inputStream
java.net.SocketException: Broken pipe
