EmrStepSensorTransformer

class ditto.transformers.emr.EmrStepSensorTransformer(target_dag, defaults)

Bases: ditto.api.OperatorTransformer

Transforms the sensor EmrStepSensor

Parameters

• target_dag (DAG) – the target to which the transformed operators must be added

• defaults (TransformerDefaults) – the default configuration for this transformer

transform(src_operator, parent_fragment, upstream_fragments)

This transformer assumes and relies on the fact that an upstream transformation of a EmrCreateJobFlowOperator has already taken place, since it needs to find the output of that transformation to get the cluster_name and azure_conn_id from that operator (which should have been a AzureHDInsightCreateClusterOperator)

This transformer also requires than there would already be transformations of EmrAddStepsOperator to LivyBatchOperator or AzureHDInsightSshOperator in the upstream_fragments which can then be monitored by the output tasks of this transformer. It needs to search for those ops upstream to find their task IDs

Adds LivyBatchSensor if it was a livy spark job. There’s no sensor required for a transformed AzureHDInsightSshOperator as it is synchronous.

Return type

DAGFragment