I was trying to ser a variable in a dbt model.
However I am struggling to overcome a issue that I encounter. From my understanding dbt_utils.get_column_values() doesn't work with CTE, however for overcoming that, i build an auxiliary model to mimic the cte and build a table.
From this table (see print shot bellow), I want to fetch the maximum levels between a string, the leves would be split by /
so for that I build this model:
--/models/dim_product/stg/stg_aux_calculate.sqlselect max(LENGTH(VALUE) - LENGTH(REPLACE(VALUE, '/', '')))::integer as max_splitsfrom {{ source('schema', 'category_tbl') }}
And this will generate this table (using snowflake):
From here I create a new model, that wants to pick that value 5 in set it as variable
-- will need to add 1 to account for the original string (tackle after) --/models/dim_product/stg/stg_category_model.sql{%- set max_splits = dbt_utils.get_column_values( table = ref('stg_aux_calculate'), column = 'max_splits')[0] -%}with split_values as ( select value_id, value, {% for i in range(1, max_splits) %} split_part(value, '/', {{ i }}) as value_component_{{ i }}, {% endfor %} ATTRIBUTE_ID, ENTITY_ID, STORE_ID from {{ source('schema', 'category_tbl') }})select value_id, value, {% for i in range(1, max_splits) %} max(value_component_{{ i }}) as value_component_{{ i }}{% if not loop.last %},{% endif %} {% endfor %} , ATTRIBUTE_ID, ENTITY_ID, STORE_IDfrom split_valuesgroup by value_id,value, ATTRIBUTE_ID, ENTITY_ID, STORE_ID
The error that I get is TypeError: 'Undefined' object cannot be interpreted as an integer
I know that this works because if I hardcode the variable with:
{% set max_splits = 6 %}
It will return what I expect:
On snowflake the stg_aux_calculate.sql model gets created like this
create or replace view DEV_etc.DBT_my_name_xyz.stg_aux_calculate( MAX_SPLITS) as ( select max(LENGTH(VALUE) - LENGTH(REPLACE(VALUE, '/', '')))::integer as max_splitsfrom DEV_RAW_INT.conf.CATALOG_CATEGORY_conf );
Any suggestion for overcome this?