.svg){kind=small}
Keyset pagination: how it works, examples, and pros and cons
To help you collect API responses more efficiently, save bandwidth, reduce memory consumption, and improve performance, you can use API pagination—a method of retrieving response data in manageable chunks.
And while there’s a variety of different types of API pagination to choose from, one of the options at the top of your list is likely keyset pagination (i.e., "seek pagination").
We’ll help you determine whether you should use keyset pagination by breaking down how it works, highlighting examples of it, sharing its pros and cons, and more.
What is keyset pagination?
It’s a pagination method that uses a key (e.g., ID) to help the client and the server request and provide progressive sets of data.
More specifically, the client uses the key within the request to indicate the starting point for the next set of results; while the API response includes what the next key should be.
To help clarify our definition, let’s differentiate keyset pagination from other pagination methods.
Related: What is cursor-based pagination?
Keyset pagination vs offset pagination
While both methods use a starting point to fetch records and define how much data to fetch from a given request, keyset pagination doesn’t skip records, while offset pagination can.
Keyset pagination vs cursor pagination
The two methods are very similar to one another. The main difference between them is how they handle position tracking and the continuation of data retrieval.
Related: A guide to REST API pagination
Examples of keyset pagination
To help bring our definition to life, let’s break down a few examples of keyset pagination.
IDs
Say an application tracks a certain type of data, like a product, by its IDs.
To help you fetch the next set of products in that application's server, you can include the ID in the API request as follows: <code class="blog_inline-code">GET /api/products?last_id=1023&limit=10</code?
Timestamps
Imagine the data you want to retrieve uses timestamps that are structured in a certain format.
You can then use a key that denotes the last timestamp you fetched in your API request using that format. Here’s how it can look: <code class="blog_inline-code">GET /api/events?last_timestamp=2023-07-14T12:00:00Z&limit=10</code>
IDs and timestamps combination
In some cases, you might need to rely on both IDs and timestamps to gather data in the correct order.
When that’s the case, you can use both the last ID and the last timestamp fetched, leading your request to look something like the following: <code class="blog_inline-code">GET /api/messages?last_id=5678&last_timestamp=2023-07-14T11:59:59Z&limit=10</code>
Related: Best practices for performing API pagination
Keyset pagination pros and cons
To help you decide whether to perform keyset pagination, you’ll need to consider the approach’s pros and cons.
Benefits of keyset pagination
- Isn’t affected by most changes in the server’s database since each request’s key points to the last item retrieved and generally only needs this reference point to function properly
- Saves computational costs since each API request doesn’t require counting and skipping rows
- Offers consistent performance since each API request jumps to a defined point in the dataset and retrieves the same number of records
- Relatively easy to implement since it primarily just requires using a last seen key in the requests. That said, the implementation can vary depending on the data's structure and the complexity of the API's requirements
Drawbacks of keyset pagination
- Forces you to collect each data point. In some cases, you’ll want to skip certain parts of a data set; doing so through keys can prove technically complex when compared to, say, offset pagination
- Volatile data sets can negatively affect performance. For instance, if the last seen key is deleted, your implementation of keyset pagination may not function correctly
- Requires you to use forward navigation. Collecting previous data can be difficult to set up since keyset pagination uses forward navigation by default
- Handling non-uniform data can be difficult. For example, if data points have overlapping timestamps, your implementation of keyset pagination may fail to segment and return the data properly
Integrate at scale without worrying about pagination by using Merge
While implementing keyset pagination for one API provider might not be time and resource intensive, you’re likely looking to integrate with several other API providers. And since each provider has unique pagination requirements, your engineers will likely quickly find themselves consumed with meeting them.
To help you avoid this challenge, you can use Merge, the leading unified API solution.
Merge lets you only worry about their unified API’s approach to pagination, rate limits, and authentication, while letting you access hundreds of integrations across software categories, such as HRIS, file storage, ATS, and accounting.
You can learn more about Merge by scheduling a demo with one of our integration experts.
