Hey,
I'm using databricks.yml
in conf in order to generate the yaml to deploy on databricks workflow, with the command run kedro databricks bundle
, I wanted Let's say I have something like :
<my_job>: schedule: # Run at 12:50 quartz_cron_expression: '00 30 10 * * ?' timezone_id: Europe/ParisThis works fine and the file is well generated
<my_job>: schedule: # Run at 12:50 quartz_cron_expression: '00 30 10 * * ?' timezone_id: Europe/Paris tasks: - task_key: default run_if: AT_LEAST_ONE_SUCCESSEvery tasks from
my_job
has the run_if conditions. However, I just wanted that a specific task inherits this run_in condition :<my_job>: schedule: # Run at 12:50 quartz_cron_expression: '00 30 10 * * ?' timezone_id: Europe/Paris tasks: - task_key: <my_task> run_if: AT_LEAST_ONE_SUCCESBut this is not correctly converted to my ressource file from this job
depends_on is automatically generated by the bundle it is indeed present in my file generated by run kedro databricks bundle
However, I don't manage to add the run_if : AT_LEAST_ONE-SUCESS
for this given task
I haven't tried but check if tasks_override
works or you can separate the tasks into different Databricks jobs.