DE Portfolio — a real data platform
This portfolio is not mockups — it is a live homelab running a production-grade data engineering stack. Every chart, table and metric on this site comes from actual pipelines processing real financial and market data.
How the pipeline works
Ingest
APIs polled by Airflow DAGs every 15–60 min. Binance WebSocket streams to Kafka in real time.
Stream
Kafka topics buffer raw trade data. ClickHouse consumer writes 25M+ rows directly from broker.
Transform
dbt models build staging → mart layers in Neon PostgreSQL. Idempotent, tested, documented.
Visualize
Grafana reads ClickHouse for low-latency metrics. Superset queries PostgreSQL for BI reports.
Tech Stack
Homelab
Data Services
BI & Monitoring
Data Sources
Live services
PostgreSQL Schema
staging schema
staging.stg_crypto_prices
coin_id, symbol, current_price_usd, price_change_pct_24h, market_cap_usd
staging.stg_forex_rates
currency_code, rate_kzt, units, rate_date
staging.stg_news
title, category, published_at, url
mart schema
mart.mart_daily_crypto_kzt
trade_date, symbol, avg_price_usd, avg_price_kzt
mart.mart_forex_trend
rate_date, currency_code, rate_per_unit, day_change