Hey team, what is a good way of checking whether all the input tables for the nodes that I want to run, are accessible. I am having issues with permissions in BigQuery and testing is cumbersome. Is there a way to run a validation of all external datasets in the catalog?
I was thinking of adding a hook and a metadata tag that identifies the datasets as external.
My main concerns are
For different dataset types, you would need to adjust the check accordingly maybe use load_args to limit rows or using a custom dataset that queries just one row?
For different dataset types, you would need to adjust the check accordingly maybe use load_args to limit rows or using a custom dataset that queries just one row?
Just wonderingâwhat are you using to work with BigQuery?
Ibis has good BigQuery support (the BigFrames team at Google helped build and maintain the Ibis backend, and they use it under the hood of BigFrames). Especially on the second point, Ibis has a lazy/deferred execution model, so it ensures tables exist and let you examine schema without loading any data.
Kedro's Ibis integration is on the newer side, but improving quickly.
I am using Ibis, yes. My issue is that I pull in external tables into different nodes and want to make sure I know whenever the SA running the code does not have access to any dataset.
I think an after_catalog_loaded or before_pipeline_run hook could make sense.
