1024px-Flatirons_Winter_Sunrise_edit_2

Brian Edgerton explains the Request-Response PatternBrian Edgerton is a JavaScript Engineer at appendTo and develops products for appendTo which are targeted for large audiences at scale. He writes today about his experience developing large scale applications, specifically, regarding the architecture patterns that are used to make appendTo’s code maintainable and extendable. In this post, well look at the request-response pattern and how to use it with postal.js.

The concept of messaging, or message passing, is a widely-used architectural pattern in software systems of all sizes. It is a pattern that we use extensively at appendTo to decouple our code allowing for components that are easy to manage and extend. This has allowed us to build applications for our clients and internal projects that are huge and complex to smaller one off applications.

Messaging is certainly no stranger to most JavaScript developers even if the developer does not recognize it as such. Many API’s provide events to which client code can subscribe and execute callbacks appropriately. The object publishes a message that an event has occurred and the subscriber reacts. If you recognize this use case, then you are already familiar with one form of messaging broadly referred to as pub-sub. There are entire books written about messaging patterns and system integration, but suffice it to say that messaging is very flexible and can be leveraged to do some powerful things. There are no absolute standards for implementing messaging within a system, but there are recognizable patterns that work well for different situations. Some implementations are very simple while others are quite complex and provide many integration points for many components.

Arguably the biggest advantage of using message passing is that it allows components (or even entire systems) to interact with each other without needing knowledge of each other. All that is necessary is knowledge of how the message passing system, or bus, works. In our examples below we will be using the postal.js message bus which works both in the browser and in Node.js. You can find more information and links to blog posts on the postal.js wiki.

In this post, we are going to look at a simple implementation of the request-response pattern and how it can be used, in this case, to abstract data access in an application. Let’s get to it.

The Usual Way

A pattern like this is used for typical database interaction. The database client is queried directly and the results are processed. This is the simplest and most straightforward approach. For many applications, it’s good enough, however, it introduces some very tight coupling between the model and data layers. If the project suddenly takes a technological change in direction to a different data back-end, a lot of changes throughout the model code will be necessary. So, how do we abstract the data access in a way that is easy to use and also flexible enough to handle changes in the future?

One answer to this question is the adapter approach in which one would create a generic database object interface. That interface could reference “subclasses” that would actually handle database interaction. While that is more flexible than the previous example (and very common), it still involves (in)direct references to the database. The model layer has to be aware of the data layer’s interface. Well, of course it does, you might say. Models are data so they have to interact directly with the database, right?

Request-Response Pattern – The Message Bus Way

The following code snippets will assume you understand the basic concepts of pub-sub, messages, envelopes, etc.

Using a message bus such as postal.js allows us to completely abstract the two layers from needing to know anything about one another. Sound crazy? It might be, but it actually works quite nicely. Let’s look at some code. We’ll go slowly and break it down piece by piece.

First, let’s modify the model code from above to work with our message bus:

If you’re paying attention, you’ll notice that the only thing that really changed was the call to the requestData method that we’re about to define. We’re going to pass it an object literal that contains the id of the user we want to retrieve.

Now, let’s define requestData.

This is a straightforward method, but there are a couple important concepts here. Just like a conventional pub-sub implementation, it publishes a message to a topic on a channel. Unlike, traditional pub-sub, it does not want to forget about the message after it is sent because the results need to be processed. It expects a response, hence the request-response name. requestData encapsulates some meta-data on the message envelope that tells the receiver where to send a response. This is effectively like the return address on a physical envelope. The publisher is saying, “Whoever receives this message, please send your response to this address.”

The other important piece here is the one-time subscription on the replyTo address. Notice that the reply topic is given a unique ID. This can be generated several different ways, but the important part is that it is somehow unique so that we can ensure that this subscription is getting the response it is expecting. If all responses were routed to the same channel and topic, the callbacks could get mixed up and be run on the wrong result sets.

The astute reader might think, “Why do we need to go through the trouble of setting up a replyTo and subscription if we’re just going to execute the callback in the same way every time? Couldn’t we just send the callback along to be processed by the subscriber?” Well, I’m glad you asked. Basically, it comes down to a matter of opinion and what works best for your application. There are arguments for doing it both ways, but they depend mostly on the problem domain. In my opinion, the technique illustrated here is better for several reasons, but we will save those for another discussion and move on.

Lastly, let’s look at the code in the data layer.

To summarize this snippet quickly, when the database layer is initialized, it will set up a subscription listening for the get topic on the model.data channel. When a message is received, it will extract the id, perform the query directly on the database client, and then send back a response to the designated replyTo address. The response object will contain either an error or the query result data.

We have reduced the data layer here to a service that listens for requests and responds to them. There are a few very important things to note about what we’ve done here:

  1. The model does not need an actual reference to the database layer.
  2. The database layer can receive and respond appropriately to requests from any publisher.
  3. The only contract needed is the basic format of the messages and envelopes.

The model never communicates directly with the database client; all messages pass through the bus. This means that the database can be switched out easily and the only integration points are the subscriptions on the appropriate channels. Also, in scenarios such as testing, a database connection does not even need to be present. A stand-in for serving static fixtures could be set up to respond to requests instead of using actual data from a database.

Since references to the database are no longer necessary, data requests can be easily sent from other parts of the application that have access to the message bus. The only thing that the data layer and other components need to know about each other is a simple message contract. While it’s impossible to achieve 100% decoupling, components need only to agree on a few tenants of communication in order to work together. This is much more desirable than having to pass around references and understand component API’s.

Another benefit is that this type of setup allows us to get creative in integrating some other functionality. For instance, by manipulating some of the message flow via channels, a caching layer could be introduced in front of the database layer that responds to data requests when appropriate and forwards requests on to the database when it can’t produce a response. Also, a logger or debugger could be subscribed to the data channels to monitor the requests being sent back and forth. Instead of having to encapsulate all the functionality in large methods, we can attach them independently to the message bus and keep their concerns safely separated.

Conclusion

We have looked at a straightforward implementation of the request-response pattern in Node.js. It is certainly more complex than the way data access is usually done, but there are many benefits to be reaped, especially in systems with a number of components that need to work together. As a fair warning, messaging patterns can become a bit addictive once you’ve used a couple successfully. You may find that there are a lot of interactions in your application that you can abstract to communication via a bus. Explore them and figure out what works for you. Finding the right abstraction is not always quick or easy, but it could produce major gains in terms of maintainability and even extensibility.