Skip to content
View myps6415's full-sized avatar

Block or report myps6415

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
myps6415/README.md

👋 Hi, I'm John Tung (@myps6415)

Data Lead · Principal Data Engineer · Data/AI & cloud platforms on GCP · OSS Contributor

⭐ Follow me on GitHub →

🌐 English · 中文

📄 View my CV →


🚀 About Me

  • 🌏 Based in Taoyuan, Taiwan. Data Lead / Principal Data Engineer at a media group, responsible for the data platform, the AI built on it, and the team
  • 🛠️ Built the group's Data/AI platform from zero: cross-platform ingestion, dbt models, three production tools used daily by executives and editors, and generative AI with quality gates
  • ☁️ I build and run the infrastructure under it too: a re-runnable GCP environment setup, least-privilege IAM cut down from audit logs, CI/CD with keyless deploys and post-deploy checks, and monitoring that catches jobs that succeed but write nothing
  • 🧰 Production experience on GCP, AWS and Azure, with Airflow and PySpark. I pick the stack by fit; right now that is mostly GCP
  • 👥 Led a four-person Data Team for two and a half years (Scrum, mentoring, hiring, vendor work) while still writing code. I work the same way now
  • 🔬 Background in Chinese text mining, NLP and sentiment analysis, which carries straight into the current LLM work
  • 🎯 Prefer small, observable, low-cost stacks (BigQuery, Cloud Run Jobs, dbt) over heavy orchestrators, and spend as much time on the unglamorous parts: token refresh, idempotent jobs, small bills
  • 🏠 Run a homelab on a MacBook server: Docker stacks for photo backup, Prometheus + Grafana monitoring, a family LINE assistant and a self-hosted agent runtime, each with its own watchdog jobs
  • 📚 Currently focused on making AI reliable enough for a newsroom: grounded generation, evals that fail the job when quality drops, agents that work within explicit rules, and repo rules shared by people and AI agents

🛠️ Technologies & Tools

Data Engineering BigQuery dbt Cloud Run Airflow PySpark Apache Iceberg Fluentd

Cloud GCP AWS Azure Kubernetes Cloudflare Pages

AI / GenAI / LLMOps Gemini scikit-learn LightGBM NLP

Data Analysis & Viz Streamlit Superset Plotly pandas

DevOps / CI-CD Docker Cloud Build GitHub Actions Ansible Prometheus Grafana Poetry pytest Ruff

Languages & Database Python SQL MySQL MongoDB Redis Elasticsearch

Working with — patterns and tools I reach for
  • Ingestion: Cloud Run Jobs + Cloud Scheduler (chosen over Airflow / Composer for cost)
  • Infra & delivery: re-runnable gcloud environment setup, least-privilege IAM (audit-log based, IAM conditions, resource-level bindings), Cloud Build push-to-deploy, GitHub Actions with Workload Identity Federation, container smoke tests and post-deploy version checks
  • Modeling: dbt-bigquery — append-only raw ingestion, then dbt staging (views) / marts (tables) with QUALIFY ROW_NUMBER() dedup
  • APIs: Meta Graph API (Facebook / Instagram), Threads API (OAuth 2.0), YouTube Data API v3
  • Secrets & auth: GCP Secret Manager, OAuth 2.0 long-lived token refresh automation
  • LLM-augmented pipelines: Gemini via BigQuery AI.GENERATE for classification + grounded generation (shipped), classification of what each negative comment targets, Vertex-embedding RAG retrieval with LLM-as-judge eval gating (shipped), a four-tool newsroom agent with mandatory self-check, hard cost caps and weekly replay eval (shipped)
  • Past production: fluenthub, a Fluentd collection service I built from zero on GKE (cluster, 6-replica Deployment with rolling updates, Ingress + static IP, gcsfuse-mounted schemas; load-tested by replaying 400k real articles through Spark, ~1,900 req/s peak with zero errors), Apache Iceberg + Spark data lake, multi-cloud (AWS ↔ GCP) ETL with Ansible-driven deployment, classical ML (XGBoost / RandomForest / SVM) for user-segment prediction
  • Other: Poetry, pytest, ruff (lint + format in CI), GitHub Actions, SendGrid / SMTP email-reporting framework

💼 Recent Work

Shipped at work

  • Cloud infrastructure: one re-runnable script builds the whole GCP environment from zero (service accounts, IAM, Secret Manager, Artifact Registry, schedules, alerts). Cloud Build deploys on push in 85s, down from 302s, and its build cache stopped about 46GB a month of image growth. When a shared project needed a security review, I narrowed our service account to what the audit logs showed it actually used (IAM conditions plus resource-level bindings, with plan / apply / rollback), testing each case with a throwaway account first
  • Multi-platform ingestion: Facebook, Instagram, Threads, YouTube and GA4 into BigQuery through 34 Cloud Run Jobs on staggered schedules, append-only with dedup on read. dbt source freshness, account-level checks and Cloud Monitoring catch jobs that succeed but write nothing. Postmortem: when a missing value must be NULL, not 0 →
  • Three production tools for non-engineers: a group revenue dashboard for executives, a social performance dashboard for marketing and editors, and a tool that lays out monthly revenue tables as print-ready newspaper PDFs (Streamlit + ReportLab). All run on Cloud Run behind one Google login; managers add and remove users themselves through a three-tier permission system with an audit log
  • Inherited pipeline rewrite: replaced a Composer + Apps Script revenue pipeline with BigQuery External Tables and Scheduled Queries, cutting monthly cost from about USD 300 to under USD 1. At this scale Cloud Scheduler + Cloud Run Jobs does the job of a managed orchestrator; the migration path is written down for when that changes
  • Threads OAuth: two accounts, eight scopes, and a weekly Cloud Run Job that refreshes the 60-day tokens through Secret Manager, so nobody re-authorizes by hand. Editors grant access through a static self-serve page on Cloudflare Pages
  • LLM classification in the warehouse: Gemini through BigQuery AI.GENERATE classifies comment sentiment and post topics incrementally, using the newsroom's own section labels; a shared cache keeps on-demand inference cost flat. How the bill stayed flat →
  • Comment analysis and RAG: every negative comment is classified by what it is aimed at (our reporting, the news event itself, or spam), which replaced an earlier KMEANS clustering that could only group by topic. The comment RAG layer (Vertex embeddings) is checked weekly by an LLM-as-judge eval that fails the job when quality drops; the third alert so far was real (duplicate comments filling the top-k) and was fixed in retrieval, not by lowering the threshold
  • Newsroom agent: four tools (topic suggestions, viral cases, competitor radar, source-based drafting), no free-form SQL, a faithfulness check after every draft, cost caps per run and per user, and a weekly replay eval for tool choice, faithfulness and refusals
  • Competitor and cross-BU analysis: competitor data from the official YouTube and Instagram APIs for topic share of voice, joined with the stock index and the business-cycle signal; Mann-Kendall, Sen's slope, seasonal decomposition and lead-lag correlation separate market tailwinds from real gains
  • Governance: a data-governance inventory that lists six open gaps and the decisions they need, cost labels on every job and service, decision records, staged onboarding, a team knowledge base, and a machine-readable repo contract (no-go zones, acceptance commands, guards) that people and AI agents both follow
  • Requirements pipeline PoC (Sept 2026, paused): a multi-agent requirements-engineering pipeline. A hook writes every user input verbatim into a hash-chained evidence log, requirements live as typed records linked by ID, and six checker agents on different models audit each round, while a deterministic validator handles the mechanical checks. In an A/B with criteria fixed before the run, interview coverage tied with the existing setup (25.9% each); the existing setup's AI auditor spent 53% of its findings on mechanical checks that the validator catches in seconds at no token cost. 147 regression tests

Side Projects

  • realping.tw: Taiwan property transactions cleaned into comparable prices per usable area (實坪), built and run solo. dbt-duckdb builds the dataset; one query core serves a REST API (FastAPI) and an MCP server (FastMCP, listed in the official MCP Registry), with API keys, usage quotas, self-serve signup and Cloudflare in front. CI builds the images and smoke-tests a running container; deploys use keyless Workload Identity Federation, then confirm the live version; a daily production smoke test also flags a live service that has fallen behind main for too long. A LightGBM valuation model (median error about 12%) is built and waiting for demand. Live at api / mcp.
  • bopomofokids.com: a free Zhuyin (ㄅㄆㄇ) learning site for young kids that my wife started and I built out. A static React SPA with seven game modes, no backend or accounts, and Ministry of Education recordings for the readings. Two rendering bugs, one Safari-only and one print-only →
  • LINE family assistant (July 2026): a self-hosted LINE group bot (FastAPI behind a Cloudflare Tunnel) with 14 function-calling tools, including weather, quotes, reminders that @-mention people and price alerts gated by market hours, at zero marginal cost per message. Six watchdog jobs run outside Docker (liveness, daily health digest, delivery audit, verified backups, regression probes), because a webhook bot usually fails silently. Series →

🌱 Open Source Contributions

  • apache/airflow#67592 (Merged Oct 2026) — My first Apache Airflow PR. After the Airflow 3 upgrade, task.queued_duration stopped being sent: workers now start tasks through the Execution API, and the only code that sent the metric sat on the old path. I added it back in ti_run (+38 / −1, plus 360 lines of tests; closes #63503 and #66067). Most of the 4.5 months of review went into one question: when should a sample be recorded? Both guards in my first version were removed. It ended at one sample per queue wait, sent only after every step that can return a 5xx, so retries don't count twice. Merged by a PMC member, milestoned for 3.4.0. Full write-up →
  • openclaw/openclaw#87291 (Filed May 2026) — Filed a source-level root cause for a silent 500-character truncation in OpenClaw's reply-context sanitizer: one cap covered both short metadata fields and multi-paragraph bot bodies, which clobbered tail content after idle session reset and left Telegram users with "amnesiac" bots and no error trace. Proposed splitting the body cap from the field cap; maintainers later landed a more sophisticated variant of the same split — head+tail truncation with additional ReplyChain / inline ReplyToBody coverage — in commit 3753c5e2c8. Full postmortem →
  • openclaw/openclaw#84890 (Merged May 2026) — Root-caused a SIGUSR1-listener / dynamic-import deadlock that left the gateway unable to restart after in-place package upgrades. Shipped a +28 / −1 surgical fix (eager-load lifecycle runtime + listener .catch) backed by production logs and a live kill -USR1 reproduction on the patched build. Full postmortem →

📞 Connect with Me

LinkedIn Email Medium

Popular repositories Loading

  1. NLP-marathon NLP-marathon Public

    Learn NLP knowledge

    Jupyter Notebook 2

  2. Crawler_Project Crawler_Project Public

    Jupyter Notebook 1 1

  3. LineBot-for-my-mom LineBot-for-my-mom Public archive

    Python

  4. Data-Analysis-Project Data-Analysis-Project Public archive

    Step by step go on data analysis

    Jupyter Notebook

  5. myps6415.github.io myps6415.github.io Public

    MDX

  6. myps6415 myps6415 Public

    Config files for my GitHub profile.