org.apache.hc.core5.http.TruncatedChunkException:Truncated chunk

[nanjing922]

After upgrading the ClickHouse client from version 21.11.3.6 to 24.8.14.39, then update to 26.6.1.1193, I suddenly got the following error when reading ClickHouse via Spark:
org.apache.hc.core5.http.TruncatedChunkException: Truncated chunk.
What should I do?

[nanjing922]

info: org. ache.spark,SparkEcept ion: Job abor ted due to stage foure: Tsk D in stage . falled 4 tines, mst reent fa
shangjian-spark-workero executor O): Java.sql. S0L Excption org.apache.hc.cores. http. TruncatedChunkException: Truncated chunk (expected size: 104s576: a
com.clickhouse.jdbc.SqlExceptionUtils.create(SqlExceptionUtils.java:45)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
at org.apache.spark.InterruptibleIterator .hasNext(InterruptibleIterator.scala:37
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
org. apache. spark.sql. catalyst.expressions. GeneratedclasssGeneratedIteratorFor codegenStagel.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator .hasNext(BufferedRowIterator.java:43)
at org.apache. spark.sql.execution.wholeStageCodegenExec$$anons1. hasNext(wholeStageCodegenExec.scala:760)
at org. apache.spark.sql.execut ion.UnsafeExternalRowSor ter.sort(UnsafeExternalRowsorter.java:225)
at org.apache. spark,sql.execut ion.exchange. ShuffleExchangeExecs.fanonfunsprepareShufflebependency 1o(ShuffleExchangeExec.scala:369)
at org.apache. spark.rdd. RDD.$anonfun$mapPartitionsInternal$2(RD0.scala:888)
org.apache. spark.rdd. RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD. scala:888)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeorReadCheckpoint(RDD.scala:364
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.shuffle.ShufflewriteProcessor.write(ShufflewriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:101)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:139)
at org.apache.spark.executor .Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
at org.apache.spark.executor.Executor$TaskRunner. run(Executooocaleitor
org.apache.spark.util.Utils$.tryWithsafeFinally(Utils.scala:1529
:624
at java.util.concurrent. ThreadPoolExecutor. runWorker (ThreadPoolExecutor .java:1149)
java.util.concurrent. ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
java.lang.Thread.run(Thread.java:750)
by: org.apache.hc.core5.http. TruncatedChunkException: Truncated chunk (expected size: 1048576; actual size: 634357)
at org.apache.hc.core5.http.impl. io.ChunkedInputStream. read(ChunkedInputStream. java:198)
at org.apache.hc.core5.http. io.EofSensor InputStream. read(EofSensor InputStream. java:135)
at com.clickhouse.data.stream.WrappedInputStream.updateBuf fer (WrappedInputStream. java:30)
at com. clickhouse. data.stream.AbstractByteArrayInputStream. readBytes (AbstractByteArrayInputStream.java:335)
at com. clickhouse.data.stream.AbstractByteArrayInputStream. readBuf fer(AbstractByteArrayInputStream.java:166)
at com.clickhouse.data.ClickHouseInputStream. readString(ClickHouseInputStream. java:1046)
at com. clickhouse.data.ClickHouseInputStream. readunicodeString(ClickHouseInputStream. java:1082)
at com.clickhouse.data. format.BinaryDataProcessor .readTextString(BinaryDataProcessor.java:678)
at com.clickhouse.data. format.ClickHouseRowBinaryProcessor. readAndFill(ClickHouseRowBinaryProcessor.java:254)
at com.clickhouse.data.ClickHouseDataProcessor .nextRecord(ClickHouseDataProcessor. java:204)
at com.clickhouse.data.ClickHouseDataProcessor .access$200(ClickHouseDataProcessor.java:25)
at com.clickhouse.data.ClickHouseDataProcessor$RecordsIterator.next(ClickHouseDataProcessor.java:124)
at com.clickhouse.data.ClickHouseDataProcessorsRecordsIterator. next(ClickHouseDataProcessor.java:110)
at com.clickhouse.jdbc.ClickHouseResultSet.next(ClickHouseResultSet. java:727)
more

[nanjing922]

class ClickHouseSource(option: Map[String, Any], spark: SparkSession) extends JDBCSourceBase(option, spark) {

override val driver: String = "com.clickhouse.jdbc.ClickHouseDriver"
override def url: String = s"jdbc:clickhouse://$host:$port/$db?compress=0"

[nanjing922]

jdbc : 0.7.2
spark: 3.4.2

[nanjing922]

This issue occurs sporadically, but it can be reproduced consistently (100%) when the data volume exceeds 50 million records.

[nanjing922]

I have attempted to upgrade the JDBC driver to version 0.9.2, yet the problem persists. What would be the recommended way to fix this?

[chernser]

Good day, @nanjing922 !

I think this is related to the fact that http compression is enabled by default. We have fixed in in the https://github.com/ClickHouse/clickhouse-java/releases/tag/v0.9.4 . If it is fine for you - please upgrade to latest client version.

[chernser]

@nanjing922

I see you are using client V1.
Try to disable http compression by sending enable_http_compression = 0 in server settings (custom_http_params or other way you are using).

[nanjing922]

Alright, I will test it with version 0.9.4 as soon as possible. Thanks.

[chernser]

@nanjing922
0.9.4 is quite old - I would recommend using latest. If there are some issues migrating to it just let me know.

Thanks!