CheckClusterEmr2HdiDagTransformerTemplate

class ditto.templates.CheckClusterEmr2HdiDagTransformerTemplate(src_dag, *args, **kwargs)[source]

Bases: ditto.templates.emr_hdi_template.EmrHdiDagTransformerTemplate

A template which uses the CheckClusterSubDagTransformer

See also

See examples/example_emr_job_flow_dag_2.py for an example. You can find more examples in the unit tests at tests/test_dag_transformations.py

Parameters

  • target_dag – ditto allows you to provide a pre-fabricated airflow DAG object so that you can set essential parameters like it’s schedule_interval, params, give it a unique dag_id, etc. outside of ditto itself, instead of ditto copying the attributes of the DAG over from the source DAG. This gives more flexbility.

  • transformer_defaults – allows you to pass a map of transformer type to their default configuration. This is helpful to pass things like a default operator to use when the transformer cannot transform the source operator for some reason, or any other configuration required by the transformer

  • transformer_resolvers – resolvers to use to find the transformers for each kind of operator in the source DAG.

  • subdag_transformers – subdag transformers to use for converting matching subdags in the source DAG to transformed subdags

  • debug_mode – when True it will render the intermediate results of transformation using networkx and maplotlib so that you can debug your transformations easily.