Quantcast
Channel: Recent Questions - Stack Overflow
Viewing all articles
Browse latest Browse all 12111

alternative solution to dbt_utils.get_column_values() for a dbt model

$
0
0

I was trying to ser a variable in a dbt model.

However I am struggling to overcome a issue that I encounter. From my understanding dbt_utils.get_column_values() doesn't work with CTE, however for overcoming that, i build an auxiliary model to mimic the cte and build a table.

From this table (see print shot bellow), I want to fetch the maximum levels between a string, the leves would be split by /

enter image description here

so for that I build this model:

--/models/dim_product/stg/stg_aux_calculate.sqlselect max(LENGTH(VALUE) - LENGTH(REPLACE(VALUE, '/', '')))::integer as max_splitsfrom {{ source('schema', 'category_tbl') }}

And this will generate this table (using snowflake):

enter image description here

From here I create a new model, that wants to pick that value 5 in set it as variable

 -- will need to add 1 to account for the original string (tackle after) --/models/dim_product/stg/stg_category_model.sql{%- set max_splits = dbt_utils.get_column_values(    table = ref('stg_aux_calculate'),     column = 'max_splits')[0] -%}with split_values as (    select        value_id,        value,        {% for i in range(1, max_splits) %}            split_part(value, '/', {{ i }}) as value_component_{{ i }},        {% endfor %}        ATTRIBUTE_ID,        ENTITY_ID,        STORE_ID    from {{ source('schema', 'category_tbl') }})select    value_id,    value,    {% for i in range(1, max_splits) %}        max(value_component_{{ i }}) as value_component_{{ i }}{% if not loop.last %},{% endif %}    {% endfor %}    ,    ATTRIBUTE_ID,    ENTITY_ID,    STORE_IDfrom split_valuesgroup by value_id,value, ATTRIBUTE_ID, ENTITY_ID, STORE_ID 

The error that I get is TypeError: 'Undefined' object cannot be interpreted as an integer

I know that this works because if I hardcode the variable with:

{% set max_splits = 6 %}

It will return what I expect:

enter image description here

On snowflake the stg_aux_calculate.sql model gets created like this

    create or replace view DEV_etc.DBT_my_name_xyz.stg_aux_calculate(    MAX_SPLITS) as (    select max(LENGTH(VALUE) - LENGTH(REPLACE(VALUE, '/', '')))::integer as max_splitsfrom DEV_RAW_INT.conf.CATALOG_CATEGORY_conf  );

Any suggestion for overcome this?


Viewing all articles
Browse latest Browse all 12111

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>