This is adapted from a blog post I wrote on Raise.dev.

The consistently inconsistent API problem

When you're dealing with an API, you either need to use some sort of adapter for it or write your own. There's a reason there are so many interfaces to them. No two APIs are written exactly the same, so dealing with them is awfully inconsistent.

There are so many factors to take into account. Is it using REST, or GraphQL, or SOAP? Does it respond with JSON, XML, or something entirely different? Do you have to authenticate, and if so, how? How does pagination work?

Of course, you can make your life a lot easier by using an API client, but what are the odds that it's well maintained? If you're dealing with an API that has OpenAPI or Swagger documentation, you might also think about generating one of these clients yourself. A lot of the time, though, you'll probably pull out HTTParty or Faraday to write your own client class.

There's almost no escaping the nastiness of APIs, especially if the one you're dealing with is external to your app or infrastructure. From the beginner integrating with Twitter or GIPHY to the industry-scale app that's backed by all kinds of services, APIs will almost always be different beasts with every fresh provider.

There has to be another way, right?

The dream: an easier way for Ruby developers to work with APIs

One thing you can do to protect yourself is to create a layer of abstraction. Similar to how view models bring structure to the view layer and make it easier to test, it should be possible to create a class that wraps an API client. We started out with this approach by creating service objects specifically dedicated to wrapping API calls.

This provided us with the benefit that we had a central point from which to change the implementation. However, the return types of all those methods would be key-value stores (known in Ruby as hashes) containing the response body. This resulted in a few problems of its own:

What if the response body were to change due to an unchecked upstream API change? We'd have to either replace all uses of an attribute, or manually change the response as it came out of the method. With a hash, we wouldn't have the power to alias or otherwise change the getter for that attribute.

What if we wanted to cast the incoming attributes to their own types? We'd have to put the response hashes through more classes or methods to get the data in the format we want.

What if we wanted to cache the response data locally? Well-made APIs provide caching headers like ETags, making it easier to integrate with faraday-http-cache and avoid wasting time on requests. But some don't, and you might not always want or need to ask the API for the data you're after. This is a complex problem all of its own.

But it also created a few opportunities for change, too. We began to think about what our ideal solution to this might look like so that we could set a standard, at least within our own app — and provide something to the open source community in the process. So we asked ourselves:

• What if Ruby developers had a clean standard for working with APIs? Rails previously came with Active Resource, which attempted to reach for this ideal. But, as previously mentioned, the situation here is still fraught — Rails developers haven't chosen one standard here. Perhaps you wrap things in service objects. Perhaps you have domain models. Perhaps you're working with an API client directly. Done right, a single solution could have knock-on benefits like better code readability and organization.
• What if remote resources behaved like Active Record models? It'd be so nice to be able to query remote resources in the same way as we can with the Active Record query interface, or cast the attributes from plain text to working objects we can do more with in our own app.
• What if it were easier to find and develop interfaces to APIs? If a community could be built around one library, it'd be straightforward to set standards and do things the "right" way. Rails as a whole is built around this — it provides a suite of tools that work well in their own right, but together provide a way of doing things. There's enough freedom that you're likely to find some new patterns wherever you go, but there's enough laid out and standardized that it's not intimidating.

Introducing Remote Record

Remote Record helps you interact with remote records — your data on other services — as though they're local domain objects. It's a new Ruby library from the folks here at Raise.dev which helps you to:

• structure and organize the way you deal with APIs
• integrate remote records into your own models
• abstract away the inconsistencies of APIs

APIs are diverse, but Remote Record is straightforward to configure based on your needs: tell it how to make the request, and it'll handle the rest.

Remote Record was born and raised in production to solve a problem shared by all kinds of developers — the inconsistencies of working with external APIs.

How others have looked at the problem

Chris, one of Raise.dev's expert coaches, had previously built Remote Resource as a foray into solving this problem. Development on that happened around five years ago (2016). That was never completed, but it always marked the need for a consistent interface to APIs. During Remote Record's development, John and I also realized that we weren't the only folks who wanted to see this solved.

Active Resource was once a part of Rails itself and tried to do something very similar. Development on that dates back as far as 2006. Her is a similar effort that carries the torch. That dates back to 2012, and still seems to be actively maintained. So what we’re up against here is a long-running problem.

But you may be wondering — if Active Resource was left in the dust, and Her isn't mainstream, is it really worth trying again? Here's why.

These libraries didn’t meet our use case because their defaults made assumptions that the API was written the Rails way - resourceful, RESTful routing. But the reality of APIs is that no two people will write them the same way. You can't expect that everyone will respond with the correct status codes, or use the same case in their URLs, or separate things with dashes rather than underscores, or respond with the same format. That means there's room for a new approach that bears this in mind.

This problem needs something general

The key problem we're trying to solve is to have a predictable interface to all kinds of APIs. There’s no real layer between those alternatives and the resources you’re dealing with - they’re effectively API clients. As I mentioned before, there are so many clients you can use and some that you might have to write yourself. But it'd be really nice not to have to think about this as much — an abstraction layer is what's necessary.

Before we even thought about this solution, we decided we'd need a module within our app that's dedicated to service objects that wrap the various API clients we use. An abstraction layer over all these different clients that still gives us their benefits, but puts control over them somewhere better. But, as I've discussed earlier on, that doesn't solve all of our problems — the response isn't an interactable object, just a hash.

Using Remote Record

Remote Record ran in production at Raise.dev, and it helped us to integrate a variety of data from external services. The remote method cleanly extends Active Record's scopes and accessors in a readable way, Remote Record classes themselves are lightweight and readable, and it behaves in a way that's predictable.

It solved a lot of these problems for Raise.dev — here's a quick look.

Let's say you're storing tracks on Spotify using Remote Record. You'd write a remote record class, which inherits from RemoteRecord::Base. When you call remote on an Active Record model instance, you'll get an instance of this class.

# app/lib/remote_record/spotify/track.rb
module RemoteRecord
  module Spotify
    class Track < RemoteRecord::Base
      def get
        client(authorization).get("tracks/#{remote_resource_id}").body
      end

      private

      def client(token)
        @client = Faraday.new('https://api.spotify.com/v1/') do |conn|
          conn.request :json
          conn.response :json
          conn.use Faraday::Response::RaiseError
          conn.headers['Authorization'] = "Bearer #{token}"
        end
      end
    end
  end
end

Now, to use this remote record class in a model, you could do the following:

# app/models/spotify/track_reference.rb
class Spotify::TrackReference < ApplicationRecord
  belongs_to :user
  include RemoteRecord
  remote_record do |config|
    config.authorization { |record| record.user.spotify_token }
  end
end

This record belongs to a user and has a remote record configuration. We're using some of Remote Record's defaults here, which can be changed to your liking.

By default, the class name to use will be looked up based on the model class name, minus Reference if it's present. So Spotify::TrackReference becomes Spotify::Track, which is looked up within the RemoteRecord namespace. Remote Record also assumes the ID for this record on the remote service is stored at the remoteresourceid field by default.

We've also configured an authorization hook, which will be called whenever a request is made.

Now we can access attributes from the Spotify API as follows:

track = Spotify::TrackReference.create!(remote_resource_id: '0HBrtXJohbIW4IhPZ50GmH')
track.remote.name
> "The Bidding"

Remote Record has some other neat features like fetching records in bulk, configurable memoization, and transformation of attributes. There's still a lot more to investigate, though. Add it to your app and get started. And if you have any questions, concerns, or suggestions regarding Remote Record, you're welcome to open a discussion or an issue over on GitHub.