2,233 Indonesian Price Records Modeled into Snowflake Marts

In collection, I use Python, requests, BeautifulSoup, and pandas to fetch public pages, parse source tables, normalize raw fields, and prepare the data before it moves into the warehouse flow. This gives the project a structured data foundation instead of a one-off spreadsheet or hardcoded dashboard file.

For cloud landing, I use AWS CLI, AWS S3, boto3, and partitioned raw objects to store raw evidence in a traceable landing area, keep files organized by source and run date, and make reruns easier to audit.

In the warehouse layer, I use Snowflake, RAW/STAGING/MARTS/AUDIT schemas, and X-Small warehouse with auto-suspend to keep raw, staging, mart, and audit data separated so the final tables are easier to validate and consume. This gives the project a structured data foundation instead of a one-off spreadsheet or hardcoded dashboard file.

For transformation, I use dbt Core, dbt Snowflake, SQL models, and source freshness to turn loaded observations into clean staging models, dimensions, fact tables, and freshness-aware analytics outputs. This confirms the warehouse source is recent enough for the dashboard and analysis use case.

For orchestration and delivery, I use Airflow DAG, Docker Compose, Streamlit, and Plotly to make the pipeline repeatable and expose the final modeled data through a dashboard experience. This is the part a stakeholder can open to understand trends without reading the pipeline code.

For quality and developer workflow, I use pytest, dbt tests, PowerShell helper scripts, and Git-ready docs to test assumptions, document the run, and keep the project reproducible from local development to portfolio review. These tests act like guardrails that catch broken relationships, missing values, or suspicious prices before the data is trusted.

Raw source responses are preserved before transformation so final metrics can be traced back to source pages. In simple terms, this keeps the original evidence available so the final dashboard numbers are not treated as a black box.

S3 objects are partitioned by source and run date, making reruns and audits easier to inspect. This makes it easier to find which source and run produced a file when the pipeline needs to be reviewed.

Snowflake loads include LOAD_RUN_DATE to avoid uncontrolled duplicates when the same run is repeated. This helps separate one pipeline run from another, so repeated loads are easier to control.

dbt test suite passed 14 checks covering accepted sources, non-null fields, uniqueness, relationships, and price sanity. These tests act like guardrails that catch broken relationships, missing values, or suspicious prices before the data is trusted.

dbt source freshness passed for RAW.MARKET_PRICES. This confirms the warehouse source is recent enough for the dashboard and analysis use case.