Join the Kedro community

Updated 3 days ago

Table a: Appending and Updating Rows with Primary Keys

tableA:
  type: pandas.SQLTableDataset
  table_name: tableA
  load_args:
    schema: my.dev
  save_args:
    schema: my.dev
    if_exists: "append"
    index: False
<strike><br />I have a table, TableA, which is the final output of my pipeline. The table already contains primary keys.<br /><br />If the primary key values are not yet in the table, I want to append the new rows. However, for rows where the primary key values already exist, I want to update those rows with the new results.<br /><br />What would be the right save_args to use in this case? I tried 'replace' for if_exists, but this keeps deleting the whole table and only the current results are stored. If I use 'append', despite the primary keys, duplicated results will still be inserted into the table<br /><br /></strike><strike>https://docs.kedro.org/en/0.18.14/kedro_datasets.pandas.SQLTableDataset.html#kedro_datasets.pandas.SQLTableDataset</strike>

A
1 comment

Solved. Basically, I have to ensure primary keys are created after the first insertion to the table.

When I experimented with replace, the whole table seems to have lost the primary key definition. And it's fixed when I redefine the primary keys

Add a reply
Sign up and join the conversation on Slack