Query = db.select().where(study_圜ol2 =my_name ) Study_table = db.Table('my_table', metadata, autoload=True, autoload_with=engine) To learn quickly SQLAlchemy: I used this blog for the select and this blog for the insert, 1 hour later the below sample code was born.Įngine = get_name_from_airflow_db(my_name): ImportError: this is MySQLdb version (1, 2, 4, 'beta', 4), but _mysql is version (1, 2, 5, 'final', 1) I tried using SQLAlchemy because I assumed since airflow is using it, the packages will be set. Push1 > pull1 > push2 > pull2 > push3 > pull3 > push4 > pull4Įventually, it was so frustrating using XCom, started checking how fast and simple would be to query the MySQL db directly from the dag (using a pythonOperator). Go over airflow DAG – “example_xcom” trigger the DAG For each PythonOperator – and view log –> watch the Xcom section & “ task instance details“įor push1 –> key: “value from pusher 1″, value:””įor push2: –> key=”return_value”, value=,dag=dag) Fix pythonOperator import if needed (based on specific airflow and python version your are running).From the example- push1 and puller are missing provide context=True.And it makes sense because in taxonomy of Airflow, XComs are communication mechanism between tasks in realtime: talking to each other while they are running be sure to understand: context becomes available only when Operator is actually executed, not during DAG-definition.Be sure to understand the documentation of pythonOperator.Go over the official example and astrnomoer.io examples.
0 Comments
Leave a Reply. |