S3KeySensorBlobOperatorTransformer

class ditto.transformers.s3.S3KeySensorBlobOperatorTransformer(target_dag, defaults)

Bases: ditto.api.OperatorTransformer

Transforms a S3KeySensor into either a WasbWildcardPrefixSensor or a WasbPrefixSensor

Parameters

• target_dag (DAG) – the target to which the transformed operators must be added

• defaults (TransformerDefaults) – the default configuration for this transformer

transform(src_operator, parent_fragment, upstream_fragments)

You need to add the wasb_conn_id to the source operator (or preferably DAG) for this to work. The container_name and prefix for the blob based sensors are coped from the bucket_name and bucket_key of the s3 sensor, so make sure they are templatized for changing between s3:// and wasb:// paths, etc. using config

Return type

DAGFragment