问题:
datanode启动失败,查看datanode日志
org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /var/lib/docker/overlay2/2f1ec1e560a2d5d6d51fa8bb23bc3d0fe7dbcd07e6460273da8f70b7bc472af0/merged/hadoop/hdfs/data at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:119) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:99) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:128) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:44) at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:127) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-06-24 14:23:47,782 WARN checker.StorageLocationChecker (StorageLocationChecker.java:check(208)) - Exception checking StorageLocation [DISK]file:/var/lib/docker/overlay2/de2f6b02326e3ebe8f209474868b6c0d55bb0c5fe3c798f9aa5bd5048ed38dbc/merged/hadoop/hdfs/data/ org.apache.hadoop.util.DiskChecker$DiskErrorException: Directory is not readable: /var/lib/docker/overlay2/de2f6b02326e3ebe8f209474868b6c0d55bb0c5fe3c798f9aa5bd5048ed38dbc/merged/hadoop/hdfs/data at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:119) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:99) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:128) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:44) at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:127) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-06-24 14:23:47,782 ERROR datanode.DataNode (DataNode.java:secureMain(2691)) - Exception in secureMain org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 4, volumes configured: 6, volumes failed: 2, volume failures tolerated: 0 at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:216) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2583) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2492) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2539) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2684) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2708)
原因:
查看服务器文件系统和挂载发现挂载的磁盘不存在
df -h
查看hdfs数据目录 hdfs-site.xml的配置dfs.datanode.data.dir数据目录和dfs.datanode.failed.volumes.tolerated磁盘容错
dfs.datanode.data.dir file://目录 dfs.datanode.failed.volumes.tolerated 0
解决:
因为没磁盘换,可将容错数量提高
dfs.datanode.failed.volumes.tolerated 2
或者将磁盘目录从配置项中删除



