FanChat - Building an AI chat product that remembers

Overview

FanChat is a consumer AI application that lets you send direct message or live group chats with AI characters. It has reached more than 250k users who sent over one million messages.

We originally built fanchat for testing an AI memory tool we built internally but after a few posts in Discord groups, it went viral in some schools where even teachers started recommending it to their students as a fun way to learn English. It was not initially designed as a language course.

On the surface it seems like a simple product, but it requires a lot of independent components working together for the best experience. Just piping last X messages to an AI model for a response gets boring really fast, the character needs some context both about themselves and the user so it can ask thoughtful and engaging questions. Different AI models perform differently for this use case, we had to do some internal evals on the models to analyze best models in price/performance for roleplay and creative writing, based on our evals DeepSeek: DeepSeek V4 Flash 0731 performs the best and OpenRouter evals also agrees.

The entire application is written in Typescript with Tanstack Start, Effect, Drizzle, Openrouter. It runs on Cloudflare, using Planetscale postgres for store connected through Hyperdrive.

Architecture

Browser
   │
   ├── Direct message
   │      ↓
   │   TanStack Start on Cloudflare Workers
   │      ↓
   │   Recent messages + summary + retrieved memories
   │      ↓
   │   Vercel AI SDK → OpenRouter → model
   │      ↓
   │   Streamed response
   │
   ├── Group room
   │      ↓
   │   Durable Object + WebSockets
   │      ↓
   │   Room coordination and message broadcast
   │
   └── Product data
          ↓
       Hyperdrive
          ↓
       PlanetScale Postgres + pgvector

Background work:
Cloudflare Workflows + scheduled jobs

Memory without replaying the whole conversation

Sending more chat history to the model works for a while. As conversations grow, it becomes slower, more expensive, and less reliable. Older details compete with the current conversation, while useful information can become buried inside thousands of messages.

FanChat handles this with several memory layers. Retrieval is best-effort, not a promise that every detail will always be recalled.

Recent context

Recent messages preserve the immediate conversation in their original form. This is where the model sees the exact wording, tone, replies, and short-term references needed to answer the latest message naturally.

The window stays bounded because the newest exchange usually has the highest value for the next reply.

Summaries and compaction

As a conversation grows, older history is compressed into a smaller summary. The aim is not to reproduce every line. It is to preserve durable context: what the conversation has been about, important changes, and details that would make a later reply confusing if forgotten.

Compaction carries continuity forward without repeatedly sending the complete transcript. Summary work runs outside the latency-sensitive reply path, updating the compacted context as more history accumulates.

Compression is deliberately lossy. A summary can keep the shape of a relationship and still omit a detail that later turns out to matter. That is why summaries are combined with recent messages and retrieval rather than treated as a complete record.

Long-term memory

Useful facts and recurring details are stored separately from the normal message stream. These memories are smaller units than a conversation summary: a preference, a personal detail, or another piece of context that may become relevant again much later.

The system extracts candidates during background work and keeps useful ones available for retrieval. Retaining too little makes a character forgetful; retaining everything creates noise.

Retrieval with pgvector

FanChat represents memories as embeddings and stores those vectors with the memory records in Postgres. An embedding turns text into a numeric representation that can be compared by meaning rather than exact wording.

For a new message, pgvector searches for related memories. Retrieval-augmented generation adds the selected results to the model context alongside recent messages and the compacted summary. The wording does not need to match an older message exactly.

Similarity is not certainty. A retrieved memory may be only loosely connected, and a useful memory may not rank highly enough on a particular turn. The prompt therefore treats retrieved material as context, not as an instruction to force every memory into the reply.

The model does not receive everything FanChat knows about a conversation. It receives the smallest useful set of recent context, summaries, and retrieved memories.

Real-time group conversations

A direct conversation mostly follows a request-and-response flow. A group room has several people connected at once, and everyone needs to see new activity immediately.

Each group room maps to a Cloudflare Durable Object, giving one coordinator responsibility for its live state. The object accepts WebSocket upgrades, tracks clients, and broadcasts messages, typing updates, reactions, and presence changes.

AI messages enter the same room flow as human messages. The room decides when the character should respond, publishes a typing update, and broadcasts the result to connected clients.

The WebSocket attachment and Durable Object hibernation APIs allow connections to survive while the object is not actively executing application code. When a client reconnects, the room can restore its session context and send the state needed to catch up rather than assuming the browser saw every live event.

Durable Objects coordinate live room activity. WebSockets deliver live updates. PlanetScale Postgres stores durable conversation data. The room also uses its local durable state to coordinate pending work before it is written through to the main relational store; it is not treated as a substitute for the product database.

Postgres from Cloudflare’s edge

PlanetScale Postgres is the main relational database for users, characters, rooms, messages, reactions, and generation records. Drizzle keeps the schema and application queries typed, which makes changes to the data model easier to trace through the TypeScript code.

pgvector lives with that relational data rather than in a separate memory service, keeping conversation ownership and vector search together.

Hyperdrive sits between Cloudflare Workers and Postgres. A Worker may handle many short-lived requests, while Postgres expects a smaller number of reusable database connections. Hyperdrive provides a pooled path between the application and PlanetScale instead of creating a fresh database connection for every request.

That boundary matters most under uneven traffic. A burst of chat activity should not translate directly into an equally large burst of new database connections.

Work that should not delay a reply

FanChat uses scheduled jobs for recurring maintenance and Cloudflare Workflows for multi-step background operations that need durable progress. These mechanisms solve different problems.

Scheduled jobs start recurring maintenance at known intervals. Workflows carry memory extraction, summary updates, and compaction through several steps without keeping a web request open, with failures and retries visible at the operation level.

The interactive path should do enough work to start a useful reply. Slower maintenance can happen afterward without making the user wait. This separation also prevents a temporary problem in summary or memory processing from automatically turning into a failed chat message.

Keeping the model layer replaceable

FanChat should not need a rewrite every time the preferred model changes. The Vercel AI SDK provides the application-facing generation and streaming interface, while OpenRouter provides access to multiple models through a consistent API.

Model configuration stays behind that boundary. The application can evaluate response quality, time to first token, context-window size, reliability, and cost without changing how the interface consumes a streamed result or how completed messages are stored.

Models are not interchangeable in practice. A cheaper model may respond quickly but lose character consistency; a larger context window may cost more without improving recall; and direct and group chats can behave differently. The boundary makes those trade-offs easier to test.

Errors are handled at the service boundary so the product can distinguish a model failure from a database or configuration problem. Retries and fallback behaviour are applied deliberately around the operation that failed rather than assuming OpenRouter will automatically choose another model.

Effect and application structure

Effect keeps service dependencies and failure cases explicit. Database access, model generation, configuration, and background operations can be described and composed as typed work instead of surfacing as unrelated exceptions from different parts of the application.

This is useful in the message pipeline, where context building, generation, persistence, and background work fail in different ways. Effect gives those steps a common structure without moving product decisions into framework code.

It also makes boundaries easier to test. A message operation can be supplied with controlled database and model services, while production wiring provides the real Cloudflare bindings and OpenRouter client.

Product analytics

PostHog records product actions such as opening a chat, sending a message, loading older messages, using reactions, and moving through onboarding.

These events help the team see how people actually use the product. The student English-practice use case came from observing behaviour and listening to users, not from deciding in advance that FanChat should become an education product.