A guide to REST API pagination

Pagination allows you to retrieve data (REST resources) in an efficient way that doesn't strain the backend or frontend. More specifically, it allows you to partition data into digestible segments. This helps ensures smooth data transactions, which prevents server strain and enhances client experiences.

You can read on to learn more about REST API pagination, including different pagination patterns and best practices to optimize your API's functionality.

What is REST API pagination?

It's a technique used in API design and development that allows you to take a REST API endpoint's response and segment it into smaller, manageable units. The segmentation process that's used can vary, as it depends on the type of API pagination you use.

Beyond just segmenting data, pagination also enhances the user experience and ensures that your API is efficient. This is especially important in data-heavy applications. Without pagination, users might face long loading times or even timeouts when retrieving large amounts of data. It also conserves server resources, preventing overloading and potential system failures.

Related: What is REST API authentication?

Different types of API pagination

Pagination works like a skilled librarian who's efficiently organizing and presenting books (in this case, data) so that you can easily find what you are looking for. And similar to how there are different ways to catalog books, there are different ways to segment, or paginate, data.

Each pagination type has advantages and focuses on different characteristics, like the size of the data set, performance requirements, data volatility, ease of implementation, user experience, and resource constraints.

We'll help you learn about each by exploring how they work, looking at examples, and reviewing their respective pros and cons.

Offset pagination

Offset pagination is the most straightforward method of pagination. It's similar to skipping chapters in a book to reach a desired section.

It utilizes two parameters: offset (the starting point in the data set) and limit (the number of records to return). The offset is usually a number indicating the distance from the first record in the data set, and the limit is the maximum number of records to be returned in the response.

For example, consider how you would make a call to the HubSpot API to fetch a list of contacts but only starting from the twenty-first contact:

<code class="blog_inline-code">GET /crm/v3/objects/contacts?limit=10&offset=20</code>
‍

In this request:

• offset=20 means "skip the first 20 contacts" and
• limit=10 means "then, give me the next 10 contacts."

Offset pagination is often used in B2B APIs because it's straightforward to implement. This is due to its use of offset and limit parameters, which can be used directly in SQL queries.

And while this makes development easier—as the SQL server handles the pagination—it leads to scalability issues. The database must scan every row from the beginning to the offset point. As a result, its performance degrades as the offset value increases, making it unsuitable for handling vast amounts of data. There's also the possibility of navigating to nonexistent pages (e.g. you use an offset value that's larger than the data set size), which can lead to errors when retrieving data.

Related: REST API pagination best practices

Page-based pagination

Page-based navigation divides data into uniformly sized pages, simplifying data handling and enhancing the developer experience. More specifically, it allows you to specify the page number and the page size (limit). From there, the server can calculate the offset and return a page with items that match the page size.

One notable advantage of page-based pagination is its predictability. It effectively eliminates the chances of navigating to nonexistent pages—a common pitfall in offset pagination. Another benefit of splitting data into equal-sized pages is that the API can also provide metadata on how many pages there are, as well as links to previous and next pages, boosting the API consumer's user experience.

Here's a simple example of a page-based pagination request:

<code class="blog_inline-code">GET /api/articles?page=3&pageSize=10</code>

In this request:

• /api/articles is the endpoint;
• page=3 indicates that the client wants the third page of results; and
• pageSize=10 sets the number of articles per page to 10.

Despite its structured approach, page-based pagination shares some of offset pagination's disadvantages, especially when dealing with large data sets. Since it relies on offset and limit calculations, performance issues can arise when calculating offset for higher page numbers. Additionally, in environments with fast-changing data, such as social feeds, there's a risk of encountering data repetition or skipped data. That's because new data can be added or existing data can be altered or deleted during consecutive page requests. These limitations make page-based pagination unsuitable for large data sets and fast-changing data.

{{this-blog-only-cta}}

Keyset pagination (seek method)

Keyset pagination, or the seek method, is tailored for scenarios that involve large or frequently-updated data sets. It involves using a specific field (or set of fields) as a key to paginate through the data set. This key is usually a unique and sequentially ordered column—for example, a timestamp or an auto-incrementing ID.

To help bring this method to life, let's review an example request:

<code class="blog_inline-code">GET /api/events?since_id=12345&limit=10</code>

In this request:

• /api/events is the endpoint;
• since_id=12345 indicates that the client wants events newer than the event with ID 12345;
• andlimit=10 limits the number of events returned.

Since it uses sequentially ordered keys, keyset pagination can provide a consistent performance and replicable results, irrespective of the data set's size or its rate of updates. This, however, makes the method unsuitable for data that's not sequential. Additionally, this method's reliance on continuous sequences makes skipping to a random page hard, thereby restricting its flexibility  to navigate data.

Time-based pagination

Time-based pagination is a type of keyset pagination used to navigate a data set based on time criteria. It uses timestamps to segment and retrieve records; in other words, the client typically specifies a time range (start and end time) or a reference time point to retrieve records.

This method is particularly useful in B2B tools like analytics platforms or log monitoring systems, enabling precise data access within specific time windows, which is crucial for trend analysis or incident tracking.

Related: Common challenges of API pagination

Cursor-based pagination

Cursor-based pagination excels in efficiently navigating through vast data sets. The way it works is that the API provides a "cursor"—similar to a bookmark—which marks a specific item in the data set. Each request not only retrieves data but also returns a cursor pointing to the start of the next data segment.

Let's review an example of cursor-based pagination:

<code class="blog_inline-code">GET /api/messages?cursor=abc123&limit=10</code>

In this request:

• /api/messages is the endpoint;
• cursor=abc123 is the cursor pointing to where the last request left off;
• limit=10 indicates the number of records to return. This parameter is usually optional as the server maintains a default limit value.

The response to such a request would include the data and a new cursor value that the client can use to request the next page in the data set, such as the following:

```
{
  "data": [...], // The requested data
  "nextCursor": "def456" // Cursor for the next set of data
}
```

The returned cursor, def456, can then be used to retrieve the next set of results.

Unlike keyset pagination, cursor-based pagination uses the cursor—a backend-determined value not necessarily linked to any data fields—to retrieve data from our API. This cursor allows the backend to be flexible with its strategy for handling data updates and also allows for efficient  data retrieval. This makes cursor-based pagination ideal for working with large data sets, even those that might not have sequentially sortable data, and it helps reduce issues with data consistency for fast-changing data sets. Unfortunately, this also makes cursor-based pagination more complex to implement than simpler methods like offset and keyset pagination.

Cursor-based pagination can involve both forward and backward cursors. A forward cursor allows you to retrieve the next set of records, while a backward cursor allows you to navigate back to previous records. While it provides bidirectional navigation, cursor-based pagination offers limited flexibility as the client can't easily jump to a specific page.

Related: Examples of cursor pagination

Combined pagination

Combined pagination is a hybrid approach that integrates various pagination methods to optimize data retrieval and navigation in APIs.

This method might combine elements of offset, cursor, keyset, page-based, and time-based pagination, depending on the data set's nature and the application's requirements. The goal is to leverage the strengths of each method to provide a more efficient and user-friendly experience.

An example of combined pagination is how most social media feeds offer cursor-based pagination for real-time updates and support time-based pagination for historical data.

REST API pagination best practices

When implementing pagination in your REST APIs, there are some best practices that'll help ensure a seamless experience for your users as well as allow your API and backend resources to operate efficiently:

• Use standard parameter names: Employ universally recognized parameter names like page, pageSize, or offset for pagination controls. This standardization facilitates easier understanding and integration for users.
• Include pagination metadata in responses: Always provide metadata (like total pages, current page, and total records) in your API responses. This context will help your users navigate through the data set effectively.
• Provide links for easy navigation: Include links for the next, previous, first, and last pages in your API responses. This makes navigation intuitive and user-friendly.
• Allow custom page sizes where possible: Giving users the option to set their own page size (within reasonable limits) can significantly enhance the user experience. This helps them specify page sizes that are reasonable for their use case. For example, someone working on a chatbot would want a smaller page size, while someone working on a data pipeline might want as much data as possible.
• Consider setting offset limits: For offset pagination, setting a maximum limit on the offset can help optimize performance, especially for large data sets.
• Implement rate limiting: To prevent overloading your server, implement rate limiting. This controls how many requests a user can make in a given period.
• Support ordering: Allow users to sort data based on different criteria (like date, name, and size). This adds flexibility and makes your API more adaptable to user needs.

Related: REST API integration best practices

Final thoughts

When building customer-facing integrations (i.e. product integrations), you'll be dealing with a diverse set of 3rd-party APIs that use different types of pagination. This can make your integration development process all the more complicated and time consuming.

To help simplify your efforts and provide all the B2B integrations your product needs, you can just build to Merge's Unified API.

Learn more about Merge's approach to pagination, rate limiting, authentication, and more by scheduling a demo with one of our integration experts.