34 lines
635 B
Markdown
34 lines
635 B
Markdown
# Apache Airflow Investigation
|
|
|
|
## How to run
|
|
in airflow directory:
|
|
`docker-compose up -d`
|
|
|
|
login:
|
|
airflow:airflow
|
|
|
|
Questions:
|
|
- Predefined Modules like "nifi processors"?
|
|
|
|
xcom= cross communication between DAG components
|
|
|
|
ti= task instance
|
|
|
|
test task on CLI:
|
|
airflow tasks test aoe_draft_dag
|
|
|
|
|
|
webserver runing in Gunicorn
|
|
Redis needed for xCom
|
|
|
|
OpenFaas Provider
|
|
https://airflow.apache.org/docs/apache-airflow-providers-openfaas/stable/index.html
|
|
|
|
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html
|
|
|
|
|
|
Lessons learned
|
|
Airflow stack takes A LOT of RAM: ~ 9GB
|
|
|
|
|