Keep the events

CrazyGames data engineer test · Daniil Kovekh

Keeptheevents

Pipeline as in the brief first. Then monitoring. Only later three narrow changes.

Synthetic demo. Crash a buffer: {"crash_buffer":true}

Slide · Task

What was asked

Part I

Overview and monitoring

SPOF, risks, how to watch each hop.

Part II

Closer to real time

What to speed up, and how, without burning the warehouse.

Part III

Claude access

Safe company-wide SQL with cost and permission control.

Deliverable: this deck plus a small running demo of the same hops. Reasoning is mine. AI helped with layout and wording.

Architecture today

As the brief draws it

Portal GF SDK Ads Analytics ECS · validate Memory lossy on kill S3 Redshift COPY 20 min dbt Preset Stats Model

Left to right: clients → Analytics → Memory → S3 → Redshift → dbt → readers. Pink rim = silent loss before S3.

Part I · SPOF

Single point of failure

Yes: the in-memory batch after accept

The endpoint already took the event. S3 has no file. Deploy, OOM, or task kill drops the batch. Redshift stays green because there was nothing to copy.

Accept Memory · kill = loss S3 (maybe) COPY

Mitigation detail is under Changes: write S3 first, then forget. Flush on shutdown. Two ECS tasks. Client retry.

Part I · Risks

What can go wrong

  • Filter drift — stricter bots eat real players; accepted volume falls while the site looks busy.
  • Integrity — broken SDK signature looks like a bot wave unless rejects split by reason.
  • Schema — one client change can fail a whole COPY if types share a table.
  • Duplicates — retries inflate plays unless you count distinct event_id.
  • Time — client clocks lie; liveness uses server received_at.
  • Shared RPU — Preset during COPY looks like pipe lag.

Part I · Monitoring

One series per hop

Scheduler green is not enough. Demo snapshot: 36.612 accepted, 4.092 rejects.

Clients
4/4
Endpoint
ok
Buffer
<2 m
S3 → COPY
<40 m
Raw lag
<45 m
dbt marts
plays

Clients vs same hour yesterday. Buffer = oldest unflushed age. S3 file without loaded_at > 40 min. Lag = now − newest received_at.

Before any redesign

Why touch anything?

Not because batch “feels old.” Three pressures only.

A

Accepted events can vanish

Memory before S3. Correctness bug. Fix before any stream.

B

One screen is too slow

Developer plays wait for flush + 20 min COPY + hourly dbt. Training does not.

C

Claude needs a gate

Shared warehouse password is a leak. Access design, not new ingest.

Part II · Streaming

What to speed up

Developer play counts after a release. Ads and page views are most of the volume and are not watched live. Recommendation training is daily.

4.000.000
DAU (brief)
~100.000.000
events / day (mid estimate)
~1.400.000
events / 20 min
~5,5–14 GB
raw / window at 4–10 KB

Full stream of all types at this size buys duplicates and bill. Keep the fast path narrow.

Part II · Structure

Batch spine + one branch

After A: every accept writes S3 first. A narrow consumer reads game_start / game_end into a stats table (~1 min). Portal reads it. Hourly dbt corrects by event_id.

S3 flush COPY 20 m dbt Preset / model narrow ~1 min Dev portal

Pitfalls: distinct event_id, upsert not append, no double backfill, replay from S3.

Part III · AI

Claude behind a tool

One query tool. Runs as the signed-in person (SSO). Read-only. 30 s timeout. 10.000 row cap. Mart schema only. No raw, payments, or PII. Developer roles filtered to their games.

Cost: sum Serverless RPU per person per day. Hard stop near a normal Preset afternoon. Show last-query cost. Short metric catalog; missing metric → stop.

A shared warehouse password is a leak. This does not change ingest.

Proposal · only now

What I would change

4 clients Analytics flush then forget S3 · replay COPY 20 m + dbt marts narrow plays ~1 min Claude · SSO
  • Keep

    Clients → Analytics → S3 → 20 min COPY → dbt → Preset / model

    Still almost all volume and training. One play definition in dbt.

  • Change

    Memory drops only after S3 write

    Today: accept → RAM → maybe S3. After: accept → S3 → forget. Removes silent loss.

  • Add

    Narrow path for game_start / game_end

    Same flush → stats table (~1 min) → portal. dbt hourly corrects. Not a second play formula.

  • Add

    Claude query tool on marts

    SSO, allowlist, row cap, RPU stop. No raw. Does not replace Preset.

What does not happen. No delete of the 20-minute job. No live stream of ads/page views. No shared Redshift password for Claude. No second play metric in the stream.

Live demo

Synthetic copy of the hops

Postgres stands in for Redshift. JSON directory for S3. Views for dbt. Six tasks every 20 minutes. Crash flag discards the buffer on purpose.

After two days: 36.612 accepted, 4.092 rejects. Boards need a koveh login.

{"crash_buffer":true}

FAQ

Short answers

Public read-only Airflow?

Yes — unlisted SHA-256 link on koveh.com. Read-only: one DAG, runs and task states, no triggers. Open crazygames_events_pipeline.

Kafka / Spark / Debezium?

Fine for a narrow consumer after flush. Not a day-one replacement for the all-types 20-minute COPY. Debezium is DB CDC; this pipe is client events via Analytics.

How this page was written

Layout and wording drafted with AI. Thinking and problem solving are mine. Prose started as speech-to-text; AI cleaned structure. It did not invent the architecture.