CheckClusterSubDagTransformer

class ditto.transformers.subdag.CheckClusterSubDagTransformer(dag, defaults)

Bases: ditto.api.SubDagTransformer

Transforms a create-if-not-exists subdag pattern commonly used in EMR airflow DAGs to the corresponding pattern for an Azure HDI airflow DAG

Parameters

• dag (DAG) – the source dag where subdags are to be found

• defaults (TransformerDefaults) – the default configuration for this subdag transformer

check_for_existing_emr_cluster

the python callable which checks if the cluster exists.

get_sub_dag_matcher()

Implement this method to construct and return a DAG of matchers for ditto to then try to use and find a subdag in the source airflow DAG

Return type

List[TaskMatcher]

Returns

list of root nodes of the TaskMatcher DAG

transform(subdag, parent_fragment)

Implement this method to do the actual transformation of the subdag found using the matchers returned by get_sub_dag_matcher()

Parameters

• subdag (DiGraph) – DAGFragment representing subdag of source DAG operators matched

• parent_fragment (DAGFragment) – This will give you a linked-list of parent DAGFragment’s, uptil the root. See parent_fragment.

Return type

DAGFragment

Returns

the DAGFragment of transformed airflow operators