EmrJobFlowSensorTransformer

class ditto.transformers.emr.EmrJobFlowSensorTransformer(target_dag, defaults)

Bases: ditto.api.OperatorTransformer

Transforms the sensor EmrJobFlowSensor

Parameters

• target_dag (DAG) – the target to which the transformed operators must be added

• defaults (TransformerDefaults) – the default configuration for this transformer

transform(src_operator, parent_fragment, upstream_fragments)

This transformer assumes and relies on the fact that an upstream transformation of a EmrCreateJobFlowOperator has already taken place, since it needs to find the output of that transformation to get the cluster_name and azure_conn_id from that operator (which should have been a AzureHDInsightCreateClusterOperator)

Creates a AzureHDInsightClusterSensor in non-provisioning mode to monitor the cluster till it reaches a terminal state (cluster shutdown by user or failed).

Warning

We do not have a way to tell the HDInsight cluster to halt if a job has failed, unlike EMR. So the cluster will continue to run even on job failure. You have to add a terminate cluster operator on step failure through ditto itself.

Return type

DAGFragment