The Data Loop Challenge

We are on a journey together, right? How to break the data loop? I've been talking about this so much that I decided to create a diagram to illustrate the loop:

Here is the mermaid code in case you want to adjust it a bit:

flowchart LR
    %% Define node styles
    classDef leader fill:#D4F1F9,stroke:#05445E,stroke-width:2px,color:#05445E,font-weight:bold
    classDef analyst fill:#FFE6CC,stroke:#D79B00,stroke-width:2px,color:#783F04,font-weight:bold
    classDef engineer fill:#D5E8D4,stroke:#82B366,stroke-width:2px,color:#3D6143,font-weight:bold
    classDef decision fill:#F8CECC,stroke:#B85450,stroke-width:2px,color:#6E0000,font-weight:bold,border-radius:10px
    
    %% Nodes with improved descriptions
    A[/"Business Leader<br/>asks a question"/]
    B["Data Analyst<br/>searches for data"]
    C{"Is the data<br/>available?"}
    D["Data Analyst<br/>models data"]
    E["Data Engineer<br/>creates new data point"]
    F["Data Analyst<br/>updates model"]
    G["Data Analyst<br/>answers question"]
    H[/"Business Leader<br/>has more questions"/]

    %% Applying styles
    class A,H leader
    class B,D,F,G analyst
    class E engineer
    class C decision
    
    %% Connections with labels and styles
    A ==> B
    B ==> D
    D ==> C
    
    %% Data missing branch
    C -- "No" --> E
    E -- "New table/view<br/>created" --> F
    F --> D
    
    %% Data available branch
    C -- "Yes" --> G
    G ==> H
    
    %% Close the loop with styled edge
    H -. "New question" .-> A
    
    %% Add a title
    subgraph "The Data Loop Challenge"
    end

In order to reduce this loop, we are trying to empower analysts or even business leaders to ask questions without having to go through the entire cycle. But to get there, we concluded that we need to clean up our data assets (as discussed last week) and ensure data accuracy. Here's the deal: if we want to empower non-technical people to interact with data without supervision, we need to be sure that our data is reliable enough that no one needs to "double-check" it.

From 7% to 60%: Our Quality Testing Journey

You might be shocked to know that out of our entire data warehouse, only 7% of our assets had some sort of quality check to validate the data. The way we were operating was simply developing new queries with the data analyst, validating the data at that point, and if everything passed our initial validation, we would deploy the new asset to our warehouse.

But surprise, surprise, data changes. An asset can start reporting bad data, and people can make decisions with it without noticing a subtle change in the backend causing a metric to go in the wrong direction.

For that reason, we set a goal to raise our data coverage from 7% to 100%. We're actually working toward that goal right now. We are currently at 60% coverage and increasing.

How Dataform Assertions Work

What does that mean in practice?

At our church, we use Dataform to transform our data. It's a very helpful tool similar to dbt but with the advantage of being within our BigQuery environment.

The way Dataform works, you define data models in a SQLx file, where a config header is required. This config header includes many settings including what are called "assertions," which are quality tests that run right after a SQLx file is executed.

There are a few useful assertions that we can apply to our data. Some are very simple, such as defining what a unique row looks like in this model (e.g., IDs or a combination of columns) and columns that should never be null for each particular model. I know this is database 101, but when you use a data warehouse such as BigQuery that is very flexible with database structure, quality checks like these can go unnoticed, causing your data to not behave as expected.

The assertions also give you the ability to add row condition tests. While the body of your SQLx defines the business logic with SQL-like code, you can set your row condition assertion to test it. And I'll be honest, it feels redundant sometimes because it feels like you're already excluding bad data with proper joins and WHERE clauses. Many times I thought to myself, "OK, this assertion is not needed," but guess what? It was needed. It caught things in the data that I wasn't even aware existed.

Surprising Findings from Quality Testing

Another common row assertion logic that we're putting in place is to catch bad data entry. Our church started in 1996, and we're in 2025. How is it possible that someone got baptized in the year 0225? I know, I know, this type of bad entry should have been caught further left in the system that allowed the entry, but nevertheless, the data made it through our data warehouse, causing our reports to be inaccurate. This extended the data loop because business leaders were questioning the integrity of the data. Without an assertion looking out for us, it would be painful to find those eight bad records claiming someone was baptized in 225 A.D.

Another lesson we learned is that if we already tested all the underlying tables of a view, there's no need to test the view itself, right? We were surprisingly wrong about that assumption. Having assertions on the view level requires the view to actually run, and we found that we had a few views that had good tables behind them but, thanks to bad syntax, the views weren't actually running. Again, these views were published and live in the data warehouse, meaning it's possible that a dashboard was connected to such a view. If a business leader tried to open that dashboard, it would trigger the execution of the view, which would have failed, leading the business leader to come back to us with questions—triggering the whole data loop again.

This rigorous testing approach directly supports our goal by ensuring that ministry leaders can trust the data they're using for decision-making.

Alfred: Our AI Assistant for Test Coverage

Now, let me be honest with you, adding assertions to assets is not fun. We need to understand the data model, think of ways to test it, and having to fix the issues the assertions are finding is even worse. So we had to come up with a way to speed up this process.

That's why we created Alfred. Alfred is not an acronym for something smarter; it's just a reference to the Batman movies where Alfred was a major helper. Alfred is a Claude Agent that we created to be another data engineer on our team. We have provided Alfred with all the context we have documented (we'll talk about documentation next week) and our standards for quality testing. Now, every time we need to create assertions, we can start by copying and pasting the SQLx to Alfred so we can work together on developing ways to test if the business logic is being implemented correctly.

Alfred has definitely sped up our process. We were at 7% coverage two months ago and have already hit the 60% mark thanks to Alfred.

The Interconnected Warehouse: Unexpected Benefits

One side effect of adding assertions to every model in our data warehouse is that every time we try to publish a new model, all assertions are tested to make sure we're not breaking anything while deploying something new. And I don't know how we made it this far without the assertions, because we're finding out that our warehouse is so interconnected. A merge request is taking longer to be approved because all these assertions are thankfully catching issues before they become problems.

That's how we're making progress toward our vision of an AI-ready and self-service data warehouse—reducing obsolete data assets and making sure the ones that remain are well-tested. This approach directly supports our goal by ensuring accurate data that truly represents our ministry impact.

Next step? We need to talk about documentation. Stay tuned for next week's post!

The Data Loop Problem

Last week I mentioned something that has been stuck in my mind for days: the data loop. The long back and forth process from leaders wanting to be data-driven, to analysts, to data engineers, back to analysts and finally back to leaders after days, sometimes weeks after a simple question was asked.

This is simply not acceptable anymore. AI has arrived and everyone around us expects this process to be faster and better, and they are right.

We want our church to reach more people and we believe data is a tool to help us get there, so data insights must be ready when leaders ask questions.

For that reason we are committed to creating a data warehouse that is AI-ready and suitable for self-service analytics. But getting there is not easy. Here are a few elements of our strategy to help us get there.

Step 1: Clean-up

We have been doing data for over 10 years at our church, which means many, many data assets have been created over time. Many of these assets became obsolete over time, but they are still available in our data warehouse.

We are on a mission of eliminating unused data assets and combining assets that can be combined.

The Materialized Views Challenge

The clean-up task was particularly challenging. We use BigQuery to build our data warehouse and decided to implement our golden assets (the final assets consumed by our clients such as app, Tableau, etc.) as materialized views instead of tables. This approach allows us to change the underlying structure of the view without any of the destinations noticing any restructuring of the backend.

This was a great decision that gave the engineering team enough flexibility to make bold moves, but on BigQuery we have no way to use the logs to find out what view is being consumed. If view vw_my_view is referencing table_a and table_b in a simple query such as:

select * from table_a, table_b

Google Cloud Platform (GCP) will provide logs that table_a and table_b are being used, but not vw_my_view. And this is a problem, because both tables might be still important for other useful views, but vw_my_view itself became obsolete because it has not been used for months.

Our Technical Solution

Still, GCP does provide the query text, and with the help of AI that was enough:

INSERT INTO 
`project_id.dataset_id.views_usage`
WITH
 model_results AS (
SELECT
 ml_generate_text_llm_result,
 queried_at,
 user_email
FROM
 ML.GENERATE_TEXT( MODEL `dataset_id.gemini`,
(
SELECT
CONCAT( 'For the given query, generate the following metadata for each referenced table or view and return ONLY a valid JSON array with objects containing these exact fields: project_id, dataset_id, table_id. Query: ', query) AS prompt,
 queried_at,
 user_email
FROM
 `dataset_id.jobs` ),
STRUCT( 0.2 AS temperature,
TRUE AS FLATTEN_JSON_OUTPUT ) ) )
SELECT
 JSON_EXTRACT_SCALAR(table_json, "$.project_id") AS project_id,
 JSON_EXTRACT_SCALAR(table_json, "$.dataset_id") AS dataset_id,
 JSON_EXTRACT_SCALAR(table_json, "$.table_id") AS table_id,
 queried_at,
 user_email
FROM
 model_results,
UNNEST(JSON_EXTRACT_ARRAY(REGEXP_REPLACE(ml_generate_text_llm_result, r'```json\n|\n```', ''))) AS table_json

The query above is capable of taking the entire query text found in the logs, and then combines the query text with pre-made instructions I give to Gemini specifying what I want from the query text and what format I want the results in. Gemini then takes the entire prompt with the query text (an unstructured data) and returns all the table/view IDs being called in the query. Finally, I have the last step of the query to make sure that each table/view identified by Gemini is represented with its own row in the query results. Voilà. That's how we found a way to catch materialized views that have not been used by any client.

Results and Risk Management

Thanks to this view catcher, we now have a scheduled job searching daily for unused views which then triggers an alert to the data engineers to start the process of decommissioning them.

Over the past few weeks, we were able to delete over 3,000 assets.

Think about this number and how it can lead to outdated answers to leaders and contribute to AI hallucination. These are over 3,000 tables or views that haven't been used in a while. All these data assets might have old and obsolete metadata such as descriptions or even metrics that no one uses anymore. If I provide all these assets to AI, AI would not be able to discern what's good or bad, so if a pastor uses an AI solution to ask about a data point we might have in store, AI might answer with one of the 3,000 bad data points that no one uses anymore. That's why this is a reason to celebrate and that's why if you are also pursuing an AI-ready database, you also need to find ways to identify the lifecycle of the assets of your data warehouse as well.

We definitely broke some things in the process, don’t be surprised if you break some things on your end as well. Every time we're about to delete an object from our data warehouse, we move these objects to what we call a "dumpster dataset" within BigQuery. This dumpster has an expiration period of 30 days. Yup, all tables or views moved to this dataset auto-delete in 30 days, but this is enough time for the "scream test" to notify us (if we move an asset and it breaks something important, someone or something will scream, hence "scream test").

Next Steps

This is definitely a lot of work, but this is taking us much closer to an AI-ready data warehouse where only relevant data is stored so that our users, either pastors or data analysts, can ask questions about our data without all the noise from unused or outdated data.

This is just step one from our strategy to building an AI-ready and self-service-ready data warehouse. Our next step is expanding test coverage and following the "document-first" approach when building data assets. Let's unpack these other steps in the coming weeks.

Tableau is a major player when it comes to data visualization, and they just hosted a conference in San Diego, California to launch new features and explain their agentic era: how a data visualization company is positioning itself for the AI revolution, and a word of advice to data analysts on how to adapt in this transforming season the industry is going through.

Here are my main takeaways as an engineer attending a conference that used to be focused primarily on data analysis:

The "move to the left" movement

The usual diagram to describe the flow of data usually starts with all the data sources on the far left, and then multiple arrows go from there, passing through many boxes in the middle where the ETL (Extract, Transform, and Load) processes happen – or in our case ELT – and the final step is the data destinations such as Tableau where all the data is visualized for impactful ministry decisions.

One mistake we (and many others in the industry) have made over the years was to build business logic on the visualization tools. They are usually hidden in a form of calculated fields within published data sources in Tableau or, worse yet, inside isolated Tableau workbooks and views that are being used on final dashboards for leaders to use and make executive and ministry decisions.

The issue here is that these calculated fields are gold with precious business logic locked in a data visualization tool, that if done well is in a published data source so that other workbooks and views can benefit from it or worse, these precious business logic can be hidden in a workbook, so that if another data analyst needs that logic they will have to rebuild it, creating what all of us data professionals hate the most: same metric, different results.

The advice here is to move all the business logic to the left, closer to the data. Rebuild those precious and rich calculated fields packed with CASE and IF statements and move them away from the visualization layer, back to the data layer.

The main immediate benefits here are:

• Unlocking the business logic from the visualization layer and making it available to all data destinations that might benefit from knowing key metrics that our organization cares about

• It eliminates or minimizes the risk of having to recreate the same metric which significantly increases the risk of the same metric reporting different numbers

• Bringing the logic to the data layer allows agents (GPTs) to read our entire diagram codified with all the declared data sources, SQL transformations, and finally all the business logic with the proper description so that it quickly answers leader questions like "what is this metric?" and lineage questions such as "where did you get this data?" and even for other data engineers, it helps us to build other queries using existent components found in the diagram.

Now, let's address the elephant in the room. If you are a data analyst, even the idea of eliminating calculated fields probably makes you cringe a bit. I get it. I will confess that my immediate reaction when I heard this advice was like "That's it! Let's all stop the calculated fields for good", but a more non-nuclear approach proposed in the conference was to keep giving analysts the freedom to do what they do best: create with leaders all the business logic they need using the tools and language they are used to; and we, the engineers, simply need to pay attention if those new dashboards will take off and have high visibility; if so, then that's the signal for us to go ahead and move those rich business logic back to the data layer so that other analysts, data destinations and GPTs can benefit from it.

Action Step for us:

• Explain to leaders and other data analysts the benefits of this approach

• Find the most popular dashboards and start transitioning those calculated fields to the data layer

• Implement a process to continually look for new and popular dashboards and migrate any calculated fields to the data layer

Metadata, Metadata, Metadata

I am sure you have been there before; you have a problem to solve and have spent hours finding the solution. Now it's time to double-check with a know-it-all GPT to ensure you are not going crazy with this novel and perfect solution you found out. But quickly you realize that whatever GPT you are using, knows nothing about you or the problem. It has access to your solution, but it also has no idea of what other data assets you have available to use to give you the most efficient answer or even find a much better and more concise solution to your problem.

And this is not an issue just for the data professionals; if you are a leader or a pastor trying to understand what data points are available for you to make a huge impact on your organization; if one of them goes to a GPT they are in a much worse place because they might not have any tools to provide GPT with the context it needs.

In this AI revolution you often hear that data is gold, well if that's the case, then metadata must be the harvesting tool we need.

The advice here is to ensure that all data assets you have are packed with metadata because all this metadata will later become part of your prompt to a GPT; it will give GPT all the context – data source, ETL process, business logic and data destination – so that when you come with a problem to solve or a simple question about what data point is available it has all the context it needs to assist you to find the answer quickly and effectively.

And according to the conference and my own research, something like dbt or Dataform provides you the tools you need to metadata everything, but then dbt and Cube Dev go one step beyond by offering a way for you to build a semantic layer, a universal layer that allows you to move your business logic and definitions everywhere you go while providing a standard framework to write documentation as code where all metrics, entities, and relationships of your data are declared, which is exactly what a GPT needs to shine at its job.

This is an image I saw often at the conference that explains well that all the standard ETL/ELT process is still needed, and the semantic layer comes on top of it to allow what they called "data portability" where you "define once, use everywhere" on Tableau or any other data destinations including LLMs to answer simple questions and help engineers build new data models.

Another benefit of having a semantic layer is that it breaks what I call the data loop: someone asks for an analysis, the data analyst tries to find a dashboard or a table in the warehouse that answers that question, but nothing does, so they contact the engineer who will then find in the raw tables something that needs to be transformed properly, tested and deployed to the warehouse, so that the analyst can take it, analyze it and finally answer the question to the leader who now has four new questions.

The semantic layer helps us reduce or even eliminate the data loop because it empowers analysts to safely join things, it empowers GPTs to assist engineers in developing new queries, and it equips non-technical people to browse a comprehensive data dictionary and even, in a future not too far away, to safely query our data.

Action Steps for us:

• The "garbage in, garbage out" rule is still true, so we need to get ready for the semantic layer by simplifying our data models and adding all the metadata to them.

• Find and implement a semantic layer solution

It's time to govern like never before

If you thought that data governance was important, it just became a lot more critical. The main idea of Tableau is to democratize data, and the "move to the left" and metadata approach definitely invites more people to do data with us, which also raises the risk of bad analyses and bad queries.

That's why we will have to focus more and more on data governance so that GPTs and others can get their answers safely.

Tableau provided this framework about the core components of data governance that we should pursue:

As I reflected on these components and attended multiple sessions, I heard and saw enough things sprinkled here and there to help build a more solid picture of what this looks like in real life.

The main ones that caught my attention were:

Data Lifecycle Management: We all love to create new things, but archiving obsolete data that no one else uses should be part of our routine, and as exciting as launching a new and shiny thing.

Data Quality and Master Management: Test coverage was my takeaway here. The recommendation is to break long queries into small queries (they call them data models), which allows us to reuse these small pieces of logic in more places and test them individually to ensure the data pipeline is moving from sources to the dashboard as expected.

Data Discovery and Understanding: It goes back to the semantic layer and the metadata concept. It reinforces the idea of documenting everything, so that we can search for data points using natural language and combining all business logic in one place; "define once, use everywhere."

Action Steps:

• Add metadata to all data assets

• Expand test coverage to 100%

• Define and implement data lifecycle

• Launch a data dictionary