FanChat - Building an AI chat product that remembers

Overview

FanChat is a social AI product for direct messages and live group conversations with AI characters. It has reached more than 250,000 users and delivered over one million messages.

Many users are school students in non-English-speaking countries. They use FanChat to practise everyday English through conversations they actually want to continue. It was not designed as a language course, but that use case became an important signal: the conversation itself gave students a reason to keep writing.

Generating one plausible AI reply is relatively simple. Building a product that still feels coherent after hundreds of messages is not. Characters need to remember useful details, replies need to start quickly, group rooms need to remain synchronised, and slower background work cannot hold up the conversation.

We built FanChat as a complete product rather than a thin interface around a model API. The work covered the application, AI generation, long-term memory, real-time group chat, database design, background processing, analytics, deployment, and performance.

The application is written in TypeScript with TanStack Start. It runs on Cloudflare, stores its core product data in PlanetScale Postgres, and connects through Hyperdrive. Drizzle provides typed queries, while Effect keeps service dependencies and failure cases explicit.

How a message becomes a reply

When a user sends a message, the system has to gather enough context for a useful reply without sending the model everything it has ever stored.

The interactive path starts by identifying the user, character, and conversation, then validating and saving the incoming message. FanChat loads the recent exchange in its original form, the latest compacted summary, and long-term memories related to the new message. It combines those layers with the character instructions to build the model context.

Generation goes through the Vercel AI SDK and OpenRouter. The SDK provides one interface for generation and streaming; OpenRouter keeps provider-specific response formats out of the rest of the product.

The response is streamed back as it is generated. The user sees the reply begin before the model has completed the entire message, which keeps a conversation feeling responsive when generation takes several seconds. Once generation finishes, the completed assistant message is saved alongside the conversation.

Memory extraction, summary updates, and compaction are handed to background processing so they do not delay the next reply.

Architecture at a glance

Browser
   │
   ├── Direct message
   │      ↓
   │   TanStack Start on Cloudflare Workers
   │      ↓
   │   Recent messages + summary + retrieved memories
   │      ↓
   │   Vercel AI SDK → OpenRouter → model
   │      ↓
   │   Streamed response
   │
   ├── Group room
   │      ↓
   │   Durable Object + WebSockets
   │      ↓
   │   Room coordination and message broadcast
   │
   └── Product data
          ↓
       Hyperdrive
          ↓
       PlanetScale Postgres + pgvector

Background work:
Cloudflare Workflows + scheduled jobs

Each part has a narrow job. Workers handle web requests and server functions. Durable Objects coordinate live rooms. Postgres holds durable product data. Workflows and scheduled jobs move slower maintenance away from the response path.

Memory without replaying the whole conversation

Sending more chat history to the model works for a while. As conversations grow, it becomes slower, more expensive, and less reliable. Older details compete with the current conversation, while useful information can become buried inside thousands of messages.

FanChat handles this with several memory layers. Retrieval is best-effort, not a promise that every detail will always be recalled.

Recent context

Recent messages preserve the immediate conversation in their original form. This is where the model sees the exact wording, tone, replies, and short-term references needed to answer the latest message naturally.

The window stays bounded because the newest exchange usually has the highest value for the next reply.

Summaries and compaction

As a conversation grows, older history is compressed into a smaller summary. The aim is not to reproduce every line. It is to preserve durable context—what the conversation has been about, important changes, and details that would make a later reply confusing if forgotten.

Compaction carries continuity forward without repeatedly sending the complete transcript. Summary work runs outside the latency-sensitive reply path, updating the compacted context as more history accumulates.

Compression is deliberately lossy. A summary can keep the shape of a relationship and still omit a detail that later turns out to matter. That is why summaries are combined with recent messages and retrieval rather than treated as a complete record.

Long-term memory

Useful facts and recurring details are stored separately from the normal message stream. These memories are smaller units than a conversation summary: a preference, a personal detail, or another piece of context that may become relevant again much later.

The system extracts candidates during background work and keeps useful ones available for retrieval. Retaining too little makes a character forgetful; retaining everything creates noise.

Retrieval with pgvector

FanChat represents memories as embeddings and stores those vectors with the memory records in Postgres. An embedding turns text into a numeric representation that can be compared by meaning rather than exact wording.

For a new message, pgvector searches for related memories. Retrieval-augmented generation adds the selected results to the model context alongside recent messages and the compacted summary. The wording does not need to match an older message exactly.

Similarity is not certainty. A retrieved memory may be only loosely connected, and a useful memory may not rank highly enough on a particular turn. The prompt therefore treats retrieved material as context, not as an instruction to force every memory into the reply.

The model does not receive everything FanChat knows about a conversation. It receives the smallest useful set of recent context, summaries, and retrieved memories.

Real-time group conversations

A direct conversation mostly follows a request-and-response flow. A group room has several people connected at once, and everyone needs to see new activity immediately.

Each group room maps to a Cloudflare Durable Object, giving one coordinator responsibility for its live state. The object accepts WebSocket upgrades, tracks clients, and broadcasts messages, typing updates, reactions, and presence changes.

AI messages enter the same room flow as human messages. The room decides when the character should respond, publishes a typing update, and broadcasts the result to connected clients.

The WebSocket attachment and Durable Object hibernation APIs allow connections to survive while the object is not actively executing application code. When a client reconnects, the room can restore its session context and send the state needed to catch up rather than assuming the browser saw every live event.

Durable Objects coordinate live room activity. WebSockets deliver live updates. PlanetScale Postgres stores durable conversation data. The room also uses its local durable state to coordinate pending work before it is written through to the main relational store; it is not treated as a substitute for the product database.

Postgres from Cloudflare’s edge

PlanetScale Postgres is the main relational database for users, characters, rooms, messages, reactions, and generation records. Drizzle keeps the schema and application queries typed, which makes changes to the data model easier to trace through the TypeScript code.

pgvector lives with that relational data rather than in a separate memory service, keeping conversation ownership and vector search together.

Hyperdrive sits between Cloudflare Workers and Postgres. A Worker may handle many short-lived requests, while Postgres expects a smaller number of reusable database connections. Hyperdrive provides a pooled path between the application and PlanetScale instead of creating a fresh database connection for every request.

That boundary matters most under uneven traffic. A burst of chat activity should not translate directly into an equally large burst of new database connections.

Work that should not delay a reply

FanChat uses scheduled jobs for recurring maintenance and Cloudflare Workflows for multi-step background operations that need durable progress. These mechanisms solve different problems.

Scheduled jobs start recurring maintenance at known intervals. Workflows carry memory extraction, summary updates, and compaction through several steps without keeping a web request open, with failures and retries visible at the operation level.

The interactive path should do enough work to start a useful reply. Slower maintenance can happen afterward without making the user wait. This separation also prevents a temporary problem in summary or memory processing from automatically turning into a failed chat message.

Keeping the model layer replaceable

FanChat should not need a rewrite every time the preferred model changes. The Vercel AI SDK provides the application-facing generation and streaming interface, while OpenRouter provides access to multiple models through a consistent API.

Model configuration stays behind that boundary. The application can evaluate response quality, time to first token, context-window size, reliability, and cost without changing how the interface consumes a streamed result or how completed messages are stored.

Models are not interchangeable in practice. A cheaper model may respond quickly but lose character consistency; a larger context window may cost more without improving recall; and direct and group chats can behave differently. The boundary makes those trade-offs easier to test.

Errors are handled at the service boundary so the product can distinguish a model failure from a database or configuration problem. Retries and fallback behaviour are applied deliberately around the operation that failed rather than assuming OpenRouter will automatically choose another model.

Effect and application structure

Effect keeps service dependencies and failure cases explicit. Database access, model generation, configuration, and background operations can be described and composed as typed work instead of surfacing as unrelated exceptions from different parts of the application.

This is useful in the message pipeline, where context building, generation, persistence, and background work fail in different ways. Effect gives those steps a common structure without moving product decisions into framework code.

It also makes boundaries easier to test. A message operation can be supplied with controlled database and model services, while production wiring provides the real Cloudflare bindings and OpenRouter client.

Product analytics

PostHog records product actions such as opening a chat, sending a message, loading older messages, using reactions, and moving through onboarding. The useful signal is behaviour—the kind of chat, whether people return, and where a flow stops—not a copy of someone’s private conversation.

These events help the team see how people actually use the product. The student English-practice use case came from observing behaviour and listening to users, not from deciding in advance that FanChat should become an education product.

Why this architecture

The goal was not to use serverless infrastructure everywhere for its own sake. It was to keep the product small enough to operate while supporting persistent memory, live group rooms, streamed AI responses, and uneven traffic.

TanStack Start owns the full-stack application. Workers handle web requests and server functions. Durable Objects coordinate live rooms, while WebSockets deliver room activity. PlanetScale Postgres holds durable product data, pgvector retrieves long-term memories, and Hyperdrive manages the connection boundary from Cloudflare.

Workflows carry durable background operations, and scheduled jobs start recurring maintenance. The Vercel AI SDK and OpenRouter provide generation and streaming behind one application-facing interface. PostHog supplies product analytics without becoming part of the message path.

This kept the system practical to operate at an early stage without preventing us from addressing real performance and reliability problems as usage grew.

What we did

Technology