You are an expert in pagination patterns for designing robust APIs.

## Key Points

- Default to cursor-based pagination for feeds and event streams where data changes frequently.
- Set a maximum page size (e.g., 100) and a sensible default (e.g., 20) so clients cannot request unbounded result sets.
- Include `has_more` or `has_next_page` so clients know when to stop paginating without relying on an empty page.
- Using offset pagination on large, frequently-updated tables where rows shift between pages causing duplicates or skipped items.
- Computing `total_count` on every request with `COUNT(*)` on large tables; make it optional or cache it.

## Quick Example

```
GET /users?offset=40&limit=20
```

```
GET /users?limit=20&after=eyJpZCI6NDJ9
```

API Pagination — API Design

You are an expert in pagination patterns for designing robust APIs.

Core Philosophy

Overview

Pagination divides large result sets into manageable pages. The choice of strategy affects performance, consistency, and client complexity. No single approach is universally best — the right choice depends on data characteristics and access patterns.

Core Concepts

Offset Pagination

The simplest model. The client specifies a page number or offset and a limit.

GET /users?offset=40&limit=20

{
  "data": [ ... ],
  "meta": {
    "total": 523,
    "offset": 40,
    "limit": 20
  }
}

Pros: easy to implement, supports jumping to arbitrary pages. Cons: performance degrades with large offsets (OFFSET 100000), inconsistent results when rows are inserted or deleted between pages.

Cursor-Based Pagination

The client receives an opaque cursor representing a position in the result set.

GET /users?limit=20&after=eyJpZCI6NDJ9

{
  "data": [ ... ],
  "cursors": {
    "after": "eyJpZCI6NjJ9",
    "before": "eyJpZCI6NDN9",
    "has_next": true,
    "has_previous": true
  }
}

Pros: stable under concurrent writes, consistently fast regardless of position. Cons: cannot jump to arbitrary pages, requires a stable sort key.

Keyset Pagination

A transparent variant of cursor pagination where the client passes the last seen sort key directly.

GET /users?limit=20&created_after=2025-01-15T10:30:00Z&id_after=42

SELECT * FROM users
WHERE (created_at, id) > ('2025-01-15T10:30:00Z', 42)
ORDER BY created_at, id
LIMIT 20;

Pros: no opaque tokens, fast on indexed columns, stable results. Cons: requires compound sort keys for uniqueness, harder to implement for multi-column sorts.

Implementation Patterns

Cursor Encoding

Encode the sort key values into a base64 cursor so clients treat it as opaque.

import base64, json

def encode_cursor(user):
    payload = {"id": user["id"], "created_at": user["created_at"].isoformat()}
    return base64.urlsafe_b64encode(json.dumps(payload).encode()).decode()

def decode_cursor(cursor):
    return json.loads(base64.urlsafe_b64decode(cursor))

Generic Paginated Response Wrapper

interface PaginatedResponse<T> {
  data: T[];
  pagination: {
    next_cursor: string | null;
    prev_cursor: string | null;
    has_more: boolean;
    total_count?: number;   // optional, can be expensive
  };
}

Database Query for Keyset Pagination

def get_users_after(cursor_created_at, cursor_id, limit=20):
    return db.execute("""
        SELECT id, name, created_at
        FROM users
        WHERE (created_at, id) > (%s, %s)
        ORDER BY created_at ASC, id ASC
        LIMIT %s
    """, (cursor_created_at, cursor_id, limit + 1))  # fetch limit+1 to detect has_more

Best Practices

• Default to cursor-based pagination for feeds and event streams where data changes frequently.
• Set a maximum page size (e.g., 100) and a sensible default (e.g., 20) so clients cannot request unbounded result sets.
• Include has_more or has_next_page so clients know when to stop paginating without relying on an empty page.

Common Pitfalls

• Using offset pagination on large, frequently-updated tables where rows shift between pages causing duplicates or skipped items.
• Computing total_count on every request with COUNT(*) on large tables; make it optional or cache it.

Anti-Patterns

Over-engineering for hypothetical scale. Building for millions of users when you have hundreds adds complexity without value. Solve today's problems first.

Ignoring the existing ecosystem. Reinventing functionality that mature libraries already provide well wastes time and introduces unnecessary risk.

Premature abstraction. Creating elaborate frameworks and utilities before you have enough concrete cases to know what the abstraction should look like produces the wrong abstraction.

Neglecting error handling at boundaries. Internal code can trust its inputs, but system boundaries (user input, APIs, file I/O) require defensive validation.

Skipping documentation for obvious code. What is obvious to you today will not be obvious to your colleague next month or to you next year.