Join the Kedro community

Updated last week

Configuring Kedro Databricks Workflow Schedules and Task Run Conditions

Hey,
I'm using databricks.yml in conf in order to generate the yaml to deploy on databricks workflow, with the command run kedro databricks bundle , I wanted Let's say I have something like :

<my_job>:
  schedule:
    # Run at 12:50
    quartz_cron_expression: '00 30 10 * * ?'
    timezone_id: Europe/Paris
This works fine and the file is well generated
Same, if i'm doing something like :
<my_job>:
  schedule:
    # Run at 12:50
    quartz_cron_expression: '00 30 10 * * ?'
    timezone_id: Europe/Paris
  tasks:
    - task_key: default
      run_if: AT_LEAST_ONE_SUCCESS
Every tasks from my_job has the run_if conditions. However, I just wanted that a specific task inherits this run_in condition :
<my_job>:
  schedule:
    # Run at 12:50
    quartz_cron_expression: '00 30 10 * * ?'
    timezone_id: Europe/Paris
  tasks:
    - task_key: <my_task>
      run_if: AT_LEAST_ONE_SUCCES
But this is not correctly converted to my ressource file from this job

Do you have any idea on how I can solve this ? Cheers !

J
L
3 comments

Hey Luc, Have you tried depends_on ?

depends_on is automatically generated by the bundle it is indeed present in my file generated by run kedro databricks bundle
However, I don't manage to add the run_if : AT_LEAST_ONE-SUCESS for this given task

I haven't tried but check if tasks_override works or you can separate the tasks into different Databricks jobs.

Add a reply
Sign up and join the conversation on Slack