Part I
Overview and monitoring
SPOF, risks, how to watch each hop.
CrazyGames data engineer test · Daniil Kovekh
Pipeline as in the brief first. Then monitoring. Only later three narrow changes.
Synthetic demo. Crash a buffer: {"crash_buffer":true}
Slide · Task
Part I
SPOF, risks, how to watch each hop.
Part II
What to speed up, and how, without burning the warehouse.
Part III
Safe company-wide SQL with cost and permission control.
Deliverable: this deck plus a small running demo of the same hops. Reasoning is mine. AI helped with layout and wording.
Architecture today
Left to right: clients → Analytics → Memory → S3 → Redshift → dbt → readers. Pink rim = silent loss before S3.
Strength
game_start.I would protect that spine. A second “play” in a stream is how readers disagree.
Part I · SPOF
The endpoint already took the event. S3 has no file. Deploy, OOM, or task kill drops the batch. Redshift stays green because there was nothing to copy.
Mitigation detail is under Changes: write S3 first, then forget. Flush on shutdown. Two ECS tasks. Client retry.
Part I · Risks
event_id.received_at.Part I · Monitoring
Scheduler green is not enough. Demo snapshot: 36.612 accepted, 4.092 rejects.
Clients vs same hour yesterday. Buffer = oldest unflushed age. S3 file without loaded_at > 40 min. Lag = now − newest received_at.
Before any redesign
Not because batch “feels old.” Three pressures only.
A
Memory before S3. Correctness bug. Fix before any stream.
B
Developer plays wait for flush + 20 min COPY + hourly dbt. Training does not.
C
Shared warehouse password is a leak. Access design, not new ingest.
Part II · Streaming
Developer play counts after a release. Ads and page views are most of the volume and are not watched live. Recommendation training is daily.
Full stream of all types at this size buys duplicates and bill. Keep the fast path narrow.
Part II · Structure
After A: every accept writes S3 first. A narrow consumer reads game_start / game_end into a stats table (~1 min). Portal reads it. Hourly dbt corrects by event_id.
Pitfalls: distinct event_id, upsert not append, no double backfill, replay from S3.
Part III · AI
One query tool. Runs as the signed-in person (SSO). Read-only. 30 s timeout. 10.000 row cap. Mart schema only. No raw, payments, or PII. Developer roles filtered to their games.
Cost: sum Serverless RPU per person per day. Hard stop near a normal Preset afternoon. Show last-query cost. Short metric catalog; missing metric → stop.
A shared warehouse password is a leak. This does not change ingest.
Proposal · only now
Still almost all volume and training. One play definition in dbt.
Today: accept → RAM → maybe S3. After: accept → S3 → forget. Removes silent loss.
Same flush → stats table (~1 min) → portal. dbt hourly corrects. Not a second play formula.
SSO, allowlist, row cap, RPU stop. No raw. Does not replace Preset.
What does not happen. No delete of the 20-minute job. No live stream of ads/page views. No shared Redshift password for Claude. No second play metric in the stream.
Live demo
Postgres stands in for Redshift. JSON directory for S3. Views for dbt. Six tasks every 20 minutes. Crash flag discards the buffer on purpose.
After two days: 36.612 accepted, 4.092 rejects. Boards need a koveh login.
{"crash_buffer":true}
FAQ
Yes — unlisted SHA-256 link on koveh.com. Read-only: one DAG, runs and task states, no triggers. Open crazygames_events_pipeline.
Fine for a narrow consumer after flush. Not a day-one replacement for the all-types 20-minute COPY. Debezium is DB CDC; this pipe is client events via Analytics.
Layout and wording drafted with AI. Thinking and problem solving are mine. Prose started as speech-to-text; AI cleaned structure. It did not invent the architecture.