HDFS读写流程以及多节点、单节点磁盘负载均衡

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

[mao@JD hadoop]$ hadoop fs
Usage: hadoop fs [generic options]
	[-appendToFile  ... ]
	[-cat [-ignoreCrc]  ...]
	[-checksum  ...]
	[-chgrp [-R] GROUP PATH...]
	[-chmod [-R]  PATH...]
	[-chown [-R] [OWNER][:[GROUP]] PATH...]
	[-copyFromLocal [-f] [-p] [-l]  ... ]
	[-copyToLocal [-p] [-ignoreCrc] [-crc]  ... ]
	[-count [-q] [-h] [-v] [-x]  ...]
	[-cp [-f] [-p | -p[topax]]  ... ]
	[-createSnapshot  []]
	[-deleteSnapshot  ]
	[-df [-h] [ ...]]
	[-du [-s] [-h] [-x]  ...]
	[-expunge]
	[-find  ...  ...]
	[-get [-p] [-ignoreCrc] [-crc]  ... ]
	[-getfacl [-R] ]
	[-getfattr [-R] {-n name | -d} [-e en] ]
	[-getmerge [-nl]  ]
	[-help [cmd ...]]
	[-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [ ...]]
	[-mkdir [-p]  ...]
	[-moveFromLocal  ... ]
	[-moveToLocal  ]
	[-mv  ... ]
	[-put [-f] [-p] [-l]  ... ]
	[-renameSnapshot   ]
	[-rm [-f] [-r|-R] [-skipTrash]  ...]
	[-rmdir [--ignore-fail-on-non-empty]  ...]
	[-setfacl [-R] [{-b|-k} {-m|-x } ]|[--set  ]]
	[-setfattr {-n name [-v value] | -x name} ]
	[-setrep [-R] [-w]   ...]
	[-stat [format]  ...]
	[-tail [-f] ]
	[-test -[defsz] ]
	[-text [-ignoreCrc]  ...]
	[-touchz  ...]
	[-usage [cmd ...]]

Generic options supported are
-conf      specify an application configuration file
-D             use value for given property
-fs       specify a namenode
-jt     specify a ResourceManager
-files     specify comma separated files to be copied to the map reduce cluster
-libjars     specify comma separated jar files to include in the classpath.
-archives     specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

1
2
3
4
5
6
7
8
9

[-cat [-ignoreCrc]  ...]
[-chmod [-R]  PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l]  ... ]
[-copyToLocal [-p] [-ignoreCrc] [-crc]  ... ]
[-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [ ...]]
[-mkdir [-p]  ...]
[-put [-f] [-p] [-l]  ... ]
[-rm [-f] [-r|-R] [-skipTrash]  ...]

1
2
3
4
5

        fs.trash.interval
        100
    

1
2
3

[mao@JD hadoop]$ hdfs dfs -rm /wordcount/input/1.log
19/12/06 00:21:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/12/06 00:21:16 INFO fs.TrashPolicyDefault: Moved: 'hdfs://JD:9000/wordcount/input/1.log' to trash at: hdfs://JD:9000/user/mao/.Trash/Current/wordcount/input/1.log

1
2
3

[mao@JD root]$ hdfs dfsadmin -safemode enter
19/12/06 18:54:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Safe mode is ON

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

[mao@JD software]$ hdfs dfs -put mao.log /
19/12/06 19:41:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put: Cannot create file/mao.log._COPYING_. Name node is in safe mode.
[mao@JD software]$ hdfs dfs -ls /
19/12/06 19:42:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
drwx------   - mao supergroup          0 2019-12-02 19:40 /tmp
drwx------   - mao supergroup          0 2019-12-06 00:21 /user
drwxr-xr-x   - mao supergroup          0 2019-12-02 19:40 /wordcount
[mao@JD software]$ hdfs dfs -cat /wordcount/output1/part-r-00000
19/12/06 19:44:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1	2
2	2
3	2
a	3
aaaa	1
afei	1
b	1
bcd	1
c	1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

[mao@JD root]$ hdfs dfsadmin -safemode leave
19/12/06 19:56:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Safe mode is OFF

[mao@JD root]$ hdfs fsck /
19/12/06 19:57:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connecting to namenode via http://JD:50070/fsck?ugi=mao&path=%2F
FSCK started by mao (auth:SIMPLE) from /192.168.0.3 for path / at Fri Dec 06 19:57:07 CST 2019
......Status: HEALTHY
 Total size:	175079 B
 Total dirs:	18
 Total files:	6
 Total symlinks:		0
 Total blocks (validated):	5 (avg. block size 35015 B)
 Minimally replicated blocks:	5 (100.0 %)
 Over-replicated blocks:	0 (0.0 %)
 Under-replicated blocks:	0 (0.0 %)
 Mis-replicated blocks:		0 (0.0 %)
 Default replication factor:	1
 Average block replication:	1.0
 Corrupt blocks:		0
 Missing replicas:		0 (0.0 %)
 Number of data-nodes:		1
 Number of racks:		1
FSCK ended at Fri Dec 06 19:57:07 CST 2019 in 4 milliseconds

The filesystem under path '/' is HEALTHY

1
2
3
4
5

[mao@JD sbin]$ start-balancer.sh 
starting balancer, logging to /home/mao/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-mao-balancer-JD.out
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
实际shell脚本的代码
[mao@JD root]$ cat /home/mao/app/hadoop/sbin/start-balancer.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR ConDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/hdfs-config.sh

# Start balancer daemon.

"$HADOOP_PREFIX"/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs start balancer $@

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

[mao@JD root]$ hdfs
Usage: hdfs [--config confdir] COMMAND
       where COMMAND is one of:
  dfs                  run a filesystem command on the file systems supported in Hadoop.
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  journalnode          run the DFS journalnode
  zkfc                 run the ZK Failover Controller daemon
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  diskbalancer         Distributes data evenly among disks on a given node
  haadmin              run a DFS HA admin client
  fsck                 run a DFS filesystem checking utility
  balancer             run a cluster balancing utility
  jmxget               get JMX exported values from NameNode or DataNode.
  mover                run a utility to move block replicas across
                       storage types
  oiv                  apply the offline fsimage viewer to an fsimage
  oiv_legacy           apply the offline fsimage viewer to an legacy fsimage
  oev                  apply the offline edits viewer to an edits file
  fetchdt              fetch a delegation token from the NameNode
  getconf              get config values from configuration
  groups               get the groups which users belong to
  snapshotDiff         diff two snapshots of a directory or diff the
                       current directory contents with a snapshot
  lsSnapshottableDir   list all snapshottable dirs owned by the current user
						Use -help to see options
  portmap              run a portmap service
  nfs3                 run an NFS version 3 gateway
  cacheadmin           configure the HDFS cache
  crypto               configure HDFS encryption zones
  storagepolicies      list/get/set block storage policies
  version              print the version

Most commands print help when invoked w/o parameters.

1
2
3
4
5
6

[mao@JD root]$ start-balancer.sh
starting balancer, logging to /home/mao/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-mao-balancer-JD.out
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved 
[mao@JD root]$ start-balancer.sh -threshold 5
starting balancer, logging to /home/mao/app/hadoop-2.6.0-cdh5.16.2/logs/hadoop-mao-balancer-JD.out
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved

1
2

[mao@JD root]$ stop-balancer.sh
no balancer to stop