先更新系统和安装 pip3
sudo apt update sudo apt upgrade sudo apt install python-setuptools sudo apt install python3-pip
安装基本库
sudo apt install libmysqlclient-dev sudo apt install libssl-dev sudo apt install libkrb5-dev
更新本地 pip3 package
注: 要加 ~/.profile export PATH=/home/kazuya/.Local/bin:$PATH
pip3 list --outdated pip3 list --outdated --format=freeze | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip3 install -U
安装 python 虚拟环境
sudo apt install python3-virtualenv
建立 airflow 虚拟环境 和 数据存放 目录
sudo mkdir /opt/airflow sudo chown -R kazuya:kazuya /opt/airflow
输出 AIRFLOW_HOME
vi ~/.profile export AIRFLOW_HOME=/opt/airflow
建立 airflow 虚拟环境
sudo virtualenv /opt/airflow_env sudo chown -R kazuya:kazuya /opt/airflow_env
起动 python 虚拟环境
cd /opt/airflow_env/bin source activate
更新虚拟环境 pip3 package 和 安装 apache-airflow
pip3 list --outdated pip3 list --outdated --format=freeze | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip3 install -U pip3 install apache-airflow pip3 install typing_extensions
查看 apache-airflow 安装路径 Location: /opt/airflow_env/lib/python3.8/site-packages
$ pip3 show apache-airflow Name: apache-airflow Version: 2.1.4 Summary: Programmatically author, schedule and monitor data pipelines Home-page: https://airflow.apache.org/ Author: Apache Software Foundation Author-email: dev@airflow.apache.org License: Apache License 2.0 Location: /opt/airflow_env/lib/python3.8/site-packages Requires: flask-wtf, dill, python-dateutil, sqlalchemy, sqlalchemy-jsonfield, pendulum, python-nvd3, markdown, apache-airflow-providers-ftp, pygments, docutils, cattrs, pyjwt, importlib-metadata, termcolor, pandas, argcomplete, apache-airflow-providers-imap, numpy, blinker, openapi-spec-validator, setproctitle, httpx, markupsafe, importlib-resources, flask-caching, tabulate, psutil, gunicorn, iso8601, alembic, pyyaml, attrs, croniter, flask-appbuilder, jsonschema, python-daemon, flask, swagger-ui-bundle, lazy-object-proxy, itsdangerous, jinja2, apache-airflow-providers-http, python3-openid, flask-login, cryptography, colorlog, marshmallow-oneofschema, unicodecsv, clickclick, python-slugify, tenacity, inflection, lockfile, rich, apache-airflow-providers-sqlite, werkzeug, graphviz Required-by:
初始数据库和建立管理账号
airflow db init airflow users create --username admin --firstname kazuya --lastname kazuya --role Admin --email XXXXXXX@gmail.com Password
起动 airflow 排程器
airflow scheduler
再开一个 session 和 source activate 虚拟环境 起动 airflow 服务器
airflow webserver -p 8080
Ctrl-C 停止服务
Ctrl-C to quit webserver Ctrl-C to quit scheduler
退出 python 虚拟环境
deactivate
