S3KeySensorAdlsGen1OperatorTransformer

class ditto.transformers.s3.S3KeySensorAdlsGen1OperatorTransformer(target_dag, defaults)

Bases: ditto.api.OperatorTransformer

Transforms a S3KeySensor into a AzureDataLakeStorageGen1WebHdfsSensor

Parameters

• target_dag (DAG) – the target to which the transformed operators must be added

• defaults (TransformerDefaults) – the default configuration for this transformer

transform(src_operator, parent_fragment, upstream_fragments)

You need to add the adls_conn_id to the source operator (or preferably DAG) for this to work. The glob_path for the ADLS sensor is coped from the bucket_key of the s3 sensor, so make sure that is templatized for changing between s3:// and adls:// paths using config

Return type

DAGFragment