Keep the events

CrazyGames data engineer test. Daniil Kovekh

Keeptheevents

Long read of the brief’s pipeline. First the hops as drawn. Then monitoring. Only later: why anything should change, and what exactly changes.

Pink node = memory can lose accepted events. Drag the scene. Crash demo: {"crash_buffer":true}

Pipeline today

This is what the brief describes. No redesign in this section. Events leave crazygames.com and the developer portal, pass one Analytics service, land on S3, then enter Redshift on a schedule.

Portalweb
GFgame frame
SDK
Ads
↓ same traffic into one door
Analytics on ECSbot filter · integrity · validate · enrich
↓ accepted events only
In-memory batchgrouped by event type · SPOF for silent loss
↓ flush when the process writes
S3 JSONfiles split by event type
↓ every 20 minutes · manifest + COPY
Redshift Serverlessraw tables
↓ dbt hourly and daily
dbt modelsone definition of a play
↓ readers of processed tables
PresetBI / explorations
Stats / portal / graphprogrammatic
Model cronrecommendations
1 ClientsPortal GF SDK Ads
2 AnalyticsECS validate
3 Memorylossy on kill
4 S3JSON by type
5 RedshiftCOPY 20 min
6 dbthourly / daily
7 ReadersPreset stats model
  1. 1

    Four clients send events

    Portal, game frame, SDK, and Ads. Same product traffic: page views, game start and end, heartbeats, ad impressions, and similar.

  2. 2

    Analytics endpoint on ECS

    One place for bot filter, integrity check, field validation, schema cleanup, and server-side enrichment. Accepted events are ready to store. Rejected events stop here with a reason.

  3. 3

    In-memory batch by event type

    Accepted events sit briefly in process memory, grouped by type to cut cost. Then the task writes JSON files. If the process dies before that write, those accepted events never become a file. The warehouse never sees them. The COPY job stays green.

  4. 4

    S3 JSON files

    Durable objects. Split by event type so a bad ads payload does not share a file with game_start.

  5. 5

    Scheduled job every 20 minutes

    Builds a manifest of buffered S3 files and COPY-loads them into Redshift Serverless. You pay for the load, not for an idle cluster sitting between runs.

  6. 6

    dbt hourly and daily

    Transforms raw into processed tables inside the same warehouse. Portal, model, Stats, and Preset share one definition of a play.

  7. 7

    Readers

    Preset is BI on marts. Stats / userportal / graph are programmatic. Model-cron trains on daily output. They query processed tables, not Analytics memory.

What this design already gets right

Four million daily players already justify a batch warehouse, and the shape you have is the right one. One endpoint means one implementation of the bot filter, the integrity check, and the schema. Splitting JSON by event type is a cost decision and an isolation decision at the same time. Redshift Serverless fits a copy that runs every 20 minutes. dbt in the same warehouse means every reader shares one play metric. That spine is the part I would protect. A second definition of “play” in a stream is how the portal and the model start to disagree.

Part I

Overview and monitoring

Questions from the brief about this architecture. The long answers below stay on what is wrong and how to watch it. The redesign sits under What changes.

Is there a single point of failure? If yes, how can it be improved or mitigated?

Yes, for silent data loss: step 3, the memory batch. The endpoint has already accepted the event. The 20-minute job has not seen a file yet. A deploy, an out-of-memory kill, or a task replacement drops the batch, and the warehouse stays green because there was nothing to copy. S3, the manifest job, and Redshift can all retry a file that exists. They cannot retry a batch that lived only in RAM.

That is the only place an accepted event can disappear without leaving a file. Mitigate it before any stream: the process may forget a batch only after the JSON is on S3, and it flushes on shutdown. Client retry covers a short endpoint blip. Two ECS tasks so one death is not a collection outage. Detail and the diagram of the fix are under What changes.

Accept Memory (today) S3 (maybe) COPY kill here = silent loss

What are the risks related to how we handle such an architecture?

The other risks sit on either side of the memory hop.

Filter drift

A stricter bot filter eats real players. Accepted volume falls while the site still looks busy.

Integrity / SDK

A broken signature looks like a bot wave unless rejects are split by reason.

Schema

One client change can fail a COPY of every type if the load treats the prefix as one table. Split files by event type.

Duplicates

Client retries inflate game_start if plays are a sum, not distinct event_id.

Time

Client clocks lie. Liveness uses server received_at. Player timing uses event time.

Shared RPU

Preset and model-cron share Serverless with the COPY. A heavy dashboard in the same window looks like lag.

How would you monitor each part of the pipeline?

Each hop gets its own series. The scheduler’s green check is not one of them. Demo snapshot on our synthetic copy (Postgres stand-in for Redshift): 36.612 accepted, 4.092 rejects over two days.

1 Clients
4/4
2 Endpoint
ok
3 Buffer
<2m
4–5 S3 COPY
<40m
Raw lag
<45m
6–7 Marts
plays

Clients. Four series: portal, game frame, SDK, ads. Compare each to the same hour yesterday. One source at zero while the other three hold is a release. All four at zero is the endpoint.

Endpoint. Accepted, rejected by reason, error rate, running tasks. Page when accepted falls and a cheap front-door counter on the site does not. Bot rejects and integrity rejects stay on separate lines.

Buffer. Age of the oldest unflushed batch. Page past a couple of minutes, and when a flush writes an empty file after a non-zero accept.

S3 / load. A file with no loaded_at older than 40 minutes is a missed COPY plus one retry. Publish rows in the file vs rows inserted. The gap is duplicates or a bad row.

Lag I trust. now minus the newest received_at in raw. Page past 45 minutes. A model that finishes with no plays for an hour that has raw game_start is a failed model, even when dbt itself is green.

Readers. RPU and queue time by Preset / stats / model next to the COPY window, so a heavy exploration is visible beside the load.

Grafana boards (login) mirror these hops. The bars above are the public view of the same idea.

What sits after Redshift: dbt, Preset, and other services?

Raw lands in Redshift Serverless. dbt runs hourly and daily and writes processed tables. That is the single definition of a play. Preset is BI on those marts. Stats / userportal / graph query marts programmatically. Model-cron trains on daily dbt output.

We do not point every service at raw. Marts isolate load, permissions, and metric grain. A wrong Preset query should hurt RPU, not corrupt raw files.

Why touch anything at all?

I would not replace the 20-minute spine because it “feels old.” Only three pressures from the brief force a change. Each change is narrow. Until these are true, staying on today’s map is correct.

  1. A

    Accepted events can vanish

    Memory before S3. That is a correctness bug, not a latency preference. Fix this before any stream.

  2. B

    One screen is too slow for a human

    A developer watching plays after a release feels flush + 20 min COPY + hourly dbt. Daily training does not. Speed up that screen only, not every event type.

  3. C

    Company-wide Claude needs a gate

    Dashboards cannot cover every cut. An LLM on a shared warehouse password is a leak. That is an access design, not a new ingest path.

What a developer feels today for a play count (not to scale exactly — order of magnitude):

Today
mem
COPY ~20 min
dbt hour

Ads, page views, and daily training do not sit on that feeling. That is why a full stream of every type is the wrong default.

Part II

Streaming

Only after A (no silent loss). The question is which delay hurts a person, and how to structure a narrow speedup without burning the warehouse.

Which part of the pipeline would benefit most from a speedup?

The screen a person is watching: developer play counts after a release. That number currently waits for the memory flush, then for the 20-minute copy, then for the next hourly dbt run. The recommendation model trains on a daily schedule and does not feel it. Ad impressions and page views are most of the volume, and nobody is staring at them minute by minute.

How would you structure closer-to-real-time ingestion?

Keep steps 1–2 and 4–7. After every accept, S3 is written first (fixes A). Then a second, small consumer reads the same flush for game_start / game_end only and upserts a stats table (game × hour) within about a minute. The portal reads that table. Hourly dbt still owns the wide models and corrects by event_id. The stream does not write the tables dbt owns.

Optional later: the same flush can fan out to Kafka (or Kinesis) and a small Spark/Flink job for that narrow table. I would not move the whole COPY onto a full stream of every type on day one.

S3 flush COPY 20m dbt Preset / model narrow ~1 min Dev portal
Portal plays
~1 min
batch still repairs

Portal gets the fast path. Preset / model stay on the batch. One play formula in dbt; stream upserts by event_id.

What volume sits in a 20-minute window?

Working model from the brief and public posts (estimates labeled). Full numbers on purpose.

4.000.000
DAU (brief)
~100.000.000
events / day mid (20–50 per DAU → 80–200M)
~1.400.000
events / 20 min
~5,5–14 GB
raw JSON / window at 4–10 KB / event

Rough share of volume vs who watches live (sketch, not a warehouse query):

Ads + page
most
game_*
watch
other
batch

Average ingest ~1.000–2.300 events/s; peak maybe 3–5×. Parallelism matters at the endpoint and at COPY. Streaming all event types at this volume buys duplicates, out-of-order repair, and bill size. Four million daily players is a reason to keep the raw files complete and the fast path narrow.

Pitfalls if you do stream

Duplicates. Client retries and consumer retries must land on the same event_id. Play count is distinct ids at the grain of the table.

Order. Events arrive late. The stats row is an upsert, not an append.

Backfill. A backfill that sums the stream and the raw table doubles the hour of the incident. One writer for the narrow table, one writer for the wide models.

Replay. A stream that cannot be rebuilt from S3 drifts from the warehouse after the first bad deploy. The 20-minute COPY stays as the repair path.

Part III

AI, cost, and access

Thousands of ad-hoc cuts will never fit in dashboards. Claude can close the gap if it never holds a warehouse password. This does not change ingest.

How would you set up the connection between Claude and Redshift?

Claude calls one query tool. The tool opens Redshift as the signed-in person (SSO). Role is read-only. Timeout 30 seconds. Cap 10.000 rows. Search path = mart schema only. Raw events, payment columns, and personal data are not granted. A game developer’s role has a row filter on that developer’s games. Finance is a different role for people who already see finance in the product. A prompt cannot widen a role. A game title that says “select payments” still runs as the developer role, which has no payment column.

A shared service account in front of Claude is a warehouse leak: every chat sees every studio, and a leaked key is the warehouse. The tool logs the person, the SQL, the row count, the duration, and the reject reason when the allowlist or the timeout stops the query.

Person Claude Query tool Marts only

How do you control cost and wrong queries?

Sum Serverless RPU per person per day from warehouse history. Hard stop near a normal Preset afternoon; raise per person when needed. Put last-query cost in the answer. Alert at about 5× that person’s median day, and on any attempt to scan raw (the allowlist should already block it).

Ship a short metric catalog: name, grain, what not to double-count, one example SQL. A play is a count of game_start at game and hour. The sum of hourly players is not the daily player count, and the list says so. Missing metric → model names the gap and stops. That list stays short on purpose.

What I would change

Only after the map above. Same spine. Three deliberate edits. Everything else stays. Read this as a delta on the brief, not a new product.

Stays (today)

Clientsunchanged
Analyticssame door
Memoryproblem hop
S3JSON
COPY 20m+ dbt
ReadersPreset model

After the three edits

Clientsunchanged
Analyticsflush then forget
S3replay source
BatchCOPY + dbt
Narrowplays ~1 min
ClaudeSSO tool
Accept S3 first Forget RAM COPY / narrow
Same 4 clients
↓
Analyticsflush JSON before forget
↓
S3durable + replay
↙ batch     ↘ narrow
Redshift + dbtmarts · 20 min spine
Stats tablegame_start/end ~1 min
↓
Preset / model
Claude toolSSO · marts only
Dev portal

What does not happen

We do not delete the 20-minute job. We do not stream ads and page views into a live warehouse. We do not hand Claude a shared Redshift password. We do not invent a second “play” metric in the stream.

Live demo (synthetic)

Small copy of the same hops. Postgres stands in for Redshift. A JSON directory stands in for S3. Views stand in for dbt. Used to prove buffer-loss and hop metrics — not production traffic.

Six tasks run every 20 minutes. Generate stands in for the four clients. Validate drops bots and rows with an empty game id, and fills country and receive time. Buffer keeps the batch, unless the run is started with a crash flag, in which case the batch is discarded and the hop is recorded as lost. Land writes one JSON file per event type plus a manifest. Load copies the manifest into the raw table and skips an event_id it already has. Transform refreshes the views the boards read.

After two days the raw table held 36.612 accepted events and 4.092 rejects. Each hop writes rows in, rows out, status, and lag. Open the boards from the hero, or the DAG grid on Airflow.

FAQ

Where is the live demo?

Buttons in the hero and under Demo. Trigger with {"crash_buffer": true} to show buffer loss. Synthetic events.

Public read-only Airflow?

Not yet. Planned later: view-only DAG list and code, no triggers.

Why Grafana needs login

Boards are behind koveh auth. This page tells the hop story without a login wall.

Kafka / Spark / Debezium?

Fine for a narrow consumer after flush. Not a day-one replacement for the all-types 20-minute COPY. Debezium is DB CDC; this pipe is client events via Analytics.

How this page was written

Layout and wording drafted with AI. Thinking and problem solving are mine. Prose started as speech-to-text; AI cleaned structure. It did not invent the architecture.