在windows环境下部署的Kafka,在运行一段时间后挂掉。
查看日志logs/server.log 发现如下报错:
ERROR Failed to clean up log for __consumer_offsets-44 in dir D:kafkakafka_2.13-2.8.0kafkakafka_2.13-2.8.0kafka-logs due to IOException (kafka.server.LogDirFailureChannel) java.nio.file.FileSystemException: D:kafkakafka_2.13-2.8.0kafkakafka_2.13-2.8.0kafka-logs__consumer_offsets-44 0000000000000000000.timeindex.cleaned -> D:kafkakafka_2.13-2.8.0kafkakafka_2.13-2.8.0kafka-logs__consumer_offsets-44 0000000000000000000.timeindex.swap: 另一个程序正在使用此文件,进程无法访问。 at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:904) at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:210) at kafka.log.LazyIndex$IndexValue.renameTo(LazyIndex.scala:155) at kafka.log.LazyIndex.$anonfun$renameTo$1(LazyIndex.scala:79) at kafka.log.LazyIndex.renameTo(LazyIndex.scala:79) at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:496) at kafka.log.Log.$anonfun$replaceSegments$4(Log.scala:2402) at kafka.log.Log.$anonfun$replaceSegments$4$adapted(Log.scala:2402) at scala.collection.immutable.List.foreach(List.scala:333) at kafka.log.Log.replaceSegments(Log.scala:2402) at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:613) at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:538) at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:537) at scala.collection.immutable.List.foreach(List.scala:333) at kafka.log.Cleaner.doClean(LogCleaner.scala:537) at kafka.log.Cleaner.clean(LogCleaner.scala:511) at kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:380) at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:352) at kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:332) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:321) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) Suppressed: java.nio.file.FileSystemException: D:kafkakafka_2.13-2.8.0kafkakafka_2.13-2.8.0kafka-logs__consumer_offsets-44 0000000000000000000.timeindex.cleaned -> D:kafkakafka_2.13-2.8.0kafkakafka_2.13-2.8.0kafka-logs__consumer_offsets-44 0000000000000000000.timeindex.swap: 另一个程序正在使用此文件,进程无法访问。 at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301) at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287) at java.nio.file.Files.move(Files.java:1395) at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:901) ... 20 more原因
当消费者偏移日志清理,或者日志文件滚动时,由于文件重命名失败,Kafka broker 将会shut down。这是因为在windows中不能修改正在使用中的文件 造成的。
解决方案在server.properties 配置文件中:
- 修改log.retention.hours=-1
- 文件末尾增加log.cleaner.enable=false
- 重启即可
GitHub中有类似issue并且有PR,但是没有被合并,笔者用的是kafka_2.13-2.8.0 版本是有此问题的。不建议在win环境使用Kafka。



