Pagination is one of those problems that every developer encounters when working with APIs. While it might seem straightforward at first, implementing a clean, efficient API client that handles paginated responses can quickly become messy. Let’s explore how Ruby’s Enumerator class can help us create elegant, lazy-loading API clients.

The Pagination Problem

When building API clients, pagination creates several challenges:

Spaghetti code: Traditional approaches often lead to nested loops, manual page tracking, and scattered pagination logic throughout your codebase. You end up with code that’s hard to read and maintain.

Unnecessary data fetching: You don’t always need every page of results. Maybe you’re searching for a specific item and want to stop once you find it, or perhaps you only need the first few results. Fetching everything upfront wastes bandwidth and time.

Rate limiting: If you try to eagerly fetch all pages at once, you risk hitting API rate limits. You need a way to control the pace of requests while maintaining clean code.

The Enumerator Solution

Ruby’s Enumerator provides an elegant way to handle these challenges through lazy evaluation. Instead of fetching all pages upfront, we can create an enumerator that fetches pages on-demand as we iterate through results.

Here’s a clean implementation:

def fetch_collection(url, params: {})  
  connection = Faraday.new("https://example.com")
  
  Enumerator.new do |yielder|  
    page = params[:page] ||= 1
  
    loop do  
      result = connection.get(url, params: params.merge(page: page))  
  
      collection = result.body.dig("data") || []  
      collection.each do |item|  
        yielder << item  
      end  
  
      raise StopIteration unless result.body.dig("links", "next")  
      page += 1  
    end  
  end
end

How It Works

Let’s break down what makes this solution elegant:

Lazy evaluation: The enumerator doesn’t fetch any data until you start iterating. Each page is only fetched when needed, making the operation memory-efficient and responsive.

Clean abstraction: The pagination logic is completely hidden from the caller. Your code simply iterates over items as if they were in a regular array, without worrying about pages, cursors, or API details.

Natural control flow: You can use standard Ruby iteration methods like each, take, find, or select. The enumerator automatically stops fetching when iteration ends, whether that’s because you found what you were looking for or simply chose to stop.

Usage Examples

The beauty of this approach becomes clear when you see how simple it is to use:

# Iterate through all posts
fetch_collection("posts").each do |post|
  puts post.title
end

# Find the first matching post (stops after finding it)
post = fetch_collection("posts").find do |post|
  post.title.include?("Ruby")
end

# Get only the first 10 posts (fetches only necessary pages)
first_ten = fetch_collection("posts").take(10)

# Filter and transform on the fly
popular_titles = fetch_collection("posts")
  .select { |post| post.views > 1000 }
  .map(&:title)
  .take(5)

In each case, the enumerator fetches only what’s needed. If you’re looking for a specific item and find it on page 2, the enumerator never fetches page 3. This makes your API client both efficient and respectful of rate limits.

Benefits

Memory efficient: Large result sets don’t overwhelm your application’s memory since items are processed as they’re fetched.

Composable: Enumerators work seamlessly with Ruby’s Enumerable methods, allowing you to chain operations naturally.

Testable: The pagination logic is isolated and easy to test without making actual HTTP requests.

Flexible: The same pattern works with different pagination styles (page numbers, cursor-based, offset-based) with minimal modifications.

Extending the Pattern

You can enhance this basic pattern in several ways:

  • Add error handling and retry logic for failed requests
  • Implement caching to avoid redundant API calls
  • Support different pagination styles (cursor-based, offset-based)
  • Add rate limiting with sleep intervals between requests
  • Include progress tracking for long-running operations

Conclusion

Enumerators transform pagination from a source of complexity into an elegant abstraction. By leveraging lazy evaluation, you get clean, efficient code that naturally handles rate limiting and memory concerns. The next time you’re building an API client, consider using an enumerator. Your future self will thank you.