Ambari2.75默认是没有flume组件，这里需要自定义集成。（ambari的安装可参考我上一篇文章：最新版Ambari2.75安装及HDP3.1.5集群搭建）
一：使用如下github地址，将编译的包，以及将flume添加到ambari-server的web页中的包，克隆到ambari-server所在服务器的一个目录中。
git clone https://github.com/maikoulin/ambari-flume-service.git
二：去官网下载flume的tar包：https://flume.apache.org/download.html。下个 apache-flume-1.9.0-bin.tar.gz 的tar.gz包，或者直接使用如下命令：
wget https://dlcdn.apache.org/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
三：将apache-flume-1.9.0-bin.tar.gz 放到ambari-flume-service/buildrpm/rpmbuild/SOURCES 这个相对目录下。
四: ambari-flume-service/buildrpm 目录下执行:
sh buildrpm.sh
到此，rpm包编译完成。
五：将 flumerpm/ambari-flume-service 相对目录下的FLUME 复制到 ambari-server 主机的 /var/lib/ambari-server/resources/stacks/HDP/3.1/services/ 目录下并重启ambari-server:
ambari-server restart
到此，ambari的web界面就能识别出来flume了。
六：创建flume的本地yum源
关于本地yum源的设置可参考我上一篇文章：最新版Ambari2.75安装及HDP3.1.5集群搭建
mkdir /var/www/html/flume/
// 创建yum源
createrepo /var/www/html/flume/
// 将上面生成的rpm包拷贝到此
cp ambari-flume-service/buildrpm/rpmbuild/RPMS/noarch/flume-1.9.0-1.el7.noarch.rpm /var/www/html/flume/
七、创建repo
cd /etc/yum.repos.d/
cp centos.repo flume.repo
vim flume.repo
[flume-1.9.0]
name=flume-1.9.0
baseurl=http://hadoop01/flume/
gpgcheck=0
enabled=1
八、将flume.repo复制到需要安装的子节点上
scp flume.repo hadoop05:/etc/yum.repos.d/
九：通过ambariUI安装flume
1）点击 add service，勾选flume
点击 next。
2） 分配客户端
点击next。
3） 填写flume agent配置文件
可以在这里填写配置，也可以不填，等安装完成再填写。点击next。
4）预览页面
直接点击 deploy。
5）安装成功
点击next。
6）完成
点击complete。
7）填写agent配置文件
如果刚刚在安装过程过程中没有填写 配置文件，可在此点击 “Configure Agents”，填写配置文件：
（1）通过avro方式接受数据，最后通过日志形式输出。
logger.sources = r1 logger.sinks = k1 logger.channels = c1 # Describe/configure the source logger.sources.r1.type = Avro logger.sources.r1.bind = 0.0.0.0 logger.sources.r1.port = 6666 # Describe the sink logger.sinks.k1.channel=c1 logger.sinks.k1.type=logger #Spillable Memory Channel logger.channels.c1.type=SPILLABLEMEMORY logger.channels.c1.checkpointDir = /data/flume/checkpoint logger.channels.c1.dataDirs = /data/flume # Bind the source and sink to the channel logger.sources.r1.channels = c1
（2）通过avro方式接受数据，输出到kafka。
logger.sources = r1 logger.sinks = k1 logger.channels = c1 # Describe/configure the source logger.sources.r1.type = Avro logger.sources.r1.bind = 0.0.0.0 logger.sources.r1.port = 6666 # Describe the sink logger.sinks.k1.channel=c1 logger.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink logger.sinks.k1.brokerList=hadoop03:6667,hadoop04:6667,hadoop05:6667 logger.sinks.k1.topic=buriedLogger logger.sinks.k1.serializer.class=kafka.serializer.StringEncoder logger.sinks.k1.serializer.appendnewline=false #Spillable Memory Channel logger.channels.c1.type=SPILLABLEMEMORY logger.channels.c1.checkpointDir = /data/flume/checkpoint logger.channels.c1.dataDirs = /data/flume # Bind the source and sink to the channel logger.sources.r1.channels = c1
保存之后，点击“ACTIONS”>“Restart All”。到此安装完成。
问题：
1.Failed to execute command '/usr/bin/yum -y install flume', exited with code '1', message: 'Error: Nothing to do'
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLUME/package/scripts/flume_handler.py", line 134, inFlumeHandler().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLUME/package/scripts/flume_handler.py", line 53, in install self.install_packages(env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 853, in install_packages retry_count=agent_stack_retry_count) File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install self._pkg_manager.install_package(package_name, self.__create_context()) File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package shell.repository_manager_executor(cmd, self.properties, context) File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor raise RuntimeError(message) RuntimeError: Failed to execute command '/usr/bin/yum -y install flume', exited with code '1', message: 'Error: Nothing to do'
解决办法：没有配置好flume的yum源，按照六、七、八步骤操作即可。一定要把flume.repo复制到需要安装的子节点上，比如：要在hadoop01、hadoop02节点安装flume，则需要把flume.repo复制到以上两个节点。
2.Package flume-1.9.0-1.el7.noarch.rpm is not signed
解决办法：由于flume的rpm包时在本地编译的，没有对应的签名，将flume.repo中的gpgcheck设置为0即可。 gpgcheck表示使用gpg文件来检查软件包的签名。
3.安装了flume的节点spark-shell启动报错：
hdfs@hadoop05 spark2-client]$ ./bin/spark-shell Traceback (most recent call last): File "/bin/hdp-select", line 453, inlistPackages(getPackages("all")) File "/bin/hdp-select", line 271, in listPackages os.path.basename(os.path.dirname(os.readlink(linkname)))) OSError: [Errno 22] Invalid argument: '/usr/hdp/current/flume-server' ls: cannot access /usr/hdp//hadoop/lib: No such file or directory Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx at org.apache.spark.launcher.Main.main(Main.java:118)
解决办法：将/usr/hdp/current下的flume-server文件夹复制到/usr/hdp/3.1.5.0-152下，然后将/usr/hdp/current下的flume-server文件夹删除，创建软链接过去
cd /usr/hdp/current
cp -r flume-server ../3.1.5.0-152/
rm -rf flume-server
ln -s /usr/hdp/3.1.5.0-152/flume-server flume-server
