tableA: type: pandas.SQLTableDataset table_name: tableA load_args: schema: my.dev save_args: schema: my.dev if_exists: "append" index: False<strike><br />I have a table, TableA, which is the final output of my pipeline. The table already contains primary keys.<br /><br />If the primary key values are not yet in the table, I want to append the new rows. However, for rows where the primary key values already exist, I want to update those rows with the new results.<br /><br />What would be the right save_args to use in this case? I tried 'replace' for if_exists, but this keeps deleting the whole table and only the current results are stored. If I use 'append', despite the primary keys, duplicated results will still be inserted into the table<br /><br /></strike><strike>https://docs.kedro.org/en/0.18.14/kedro_datasets.pandas.SQLTableDataset.html#kedro_datasets.pandas.SQLTableDataset</strike>
Solved. Basically, I have to ensure primary keys are created after the first insertion to the table.
When I experimented with replace, the whole table seems to have lost the primary key definition. And it's fixed when I redefine the primary keys