Windows Azure service bus messaging with publish/subscribe pattern using topics and subscriptions

In this post we will look into the publish/subscribe pattern with the Windows Azure Service Bus, which is a messaging platform in the cloud. One of the common use to build disconnected and reliable systems is the use of queues:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

The Windows Azure Service Bus allows messaging using the publish/subscribe pattern, which looks like this:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

The sender sends a message to a topic and anyone who could be interested in one of those messages, could subscribe to the topic. They could subscribe to receive any message, but they can also apply filters on the incoming messages to only receive certain messages that comply to the defined filter.

Quoted directly from Microsoft:

The Service Bus provides secure messaging and relay capabilities that enable building distributed and loosely-coupled applications in the cloud, as well hybrid application across both private and public clouds. It supports multiple messaging protocols and patterns and handles delivery assurance, reliable messaging and scale for your applications. The Service Bus is a managed service that is operated by Microsoft and has a 99.9% monthly SLA

There are many practical uses for the Service Bus.  Some common uses include:

Hybrid applications

The Service Bus enables you to securely connect and integrate enterprise systems running in your private cloud with applications running on Windows Azure.  This makes it easier to extend solutions to the cloud without having to port or migrate all of your data or code from an existing enterprise datacenter to Windows Azure. 

Mobile applications

The Service Bus enables you to easily build applications that can distribute event notifications and data to occasionally-connected clients, such as smart phones or tablets.  You can expose notifications or events from an application running either in Windows Azure or in your private cloud environment, and ensure that they are ultimately delivered to mobile devices. 

Loosely coupled architectures

The Service Bus enables you to build loosely-coupled systems that are more resilient to network failure and can more easily scale out based on demand. The Service Bus can act as the connecting broker between the different components of a system, eliminating direct dependencies between different components.  Easily leverage the Service Bus to architect applications that support application load balancing.

Service Bus messaging

  • Service Bus Queues offer a reliable, highly scalable way to store messages as they travel between systems without losing messages in the event of connectivity failure.
  • Service Bus Topics and Subscriptions implement a publish/subscribe pattern that delivers a highly scalable, flexible, and cost-effective way to publish messages from an application and deliver them to multiple subscribers.

Service Bus connectivity

  • The Service Bus Relay enables applications hosted inside Windows Azure to securely call back to private cloud applications hosted in your own datacenter behind a firewall and vice versa.  The relay service avoids the need to instantiate and setup a new connection on each call and makes connectivity faster and more reliable.  It also supports the ability to integrate applications across existing NATs and firewalls.  The relay service supports a variety of different transport protocols and Web services standards, including REST, SOAP, and WS-*.
  • Companies can use the Service Bus relay to expose just the information they want from their private cloud environment, which creates a more secure architecture than opening up a VPN.  Enterprises can use a SOA-based architecture and expose just the services they want to deliver from their on-premise data centers.

1. Scenario for publish / subscribe pattern with Windows Azure Service Bus

Suppose we have a company that takes and processes orders from customers. Customers are able to place orders for products that we sell and the orders get shipped the day after ordering. Our company works with different departments, which all have a role in the process of the order.

New orders can be placed:

  • At the website, where customers can place an order containing multiple products, of any amount.
  • In future, orders will also be able to be placed by the support desk, which will be able to create an order manually for a customer.
This means the services that can place new orders have to be extensible for in the future, as new publishers will be added. Every service that can send a message to our Orders Topic, will be a publisher. Multiple services will be able to send order messages to the Orders topic and multiple subscribers will be present on the orders topic.There are multiple departments of our company interested in the orders:
  • The warehouse where the employees have to wrap the items the customer ordered and get them ready for shipment
  • The billing that has to register each order for accountancy purposes
  • The management that is only interested in the orders with a price > 5.000$
  • The purchase department that is only interested in orders that contain products of which there are not enough in store. Those items have to be purchased at an external vendor to complete the customer’s order.
  • In future, there might be other departments which could be interested in the order messages.
This means we will have 4 subscribers to the orders. In future, if other departments would appear that could be interested in the order messages, we just plug another subscriber on the topic.

2. Windows Azure Service Bus Topics ans Subscriptions by portal

There are many possible solutions to build this sort of solution, and often it depends on personal opinion. In our case we will use the Windows Azure Service Bus using the publish / subscribe pattern to build an extensible and loosely coupled solution. This means if a subscriber goes down, the other subscribers will not know about this and will still be working. When the subscriber comes back available, it continues reading the order messages from it’s subscription. The publisher which sends the order messages to the topic, has no idea of what is happening to the message and who is subscribing to the message. The only task of the subscriber is to create new orders and place them on the topic. Having the publishers and subscribers not knowing about each other, makes our system loosely coupled. If one of the senders or receivers go down, the other services in our architecture will continue to function properly. Imagine if you have a system where most of the services are tightly coupled, making 1 of the services go down, resulting in more then half of our services to break.

If we want to add another publisher of orders, we simply add a component in our architecture which will be able to send order messages to the specified Orders topic. The other publishers do not know about this new publisher and neither should they. Having them know about each other, only increases the chance that things break when a service or component changes or goes unavailable.

If we want to add another subscriber, which could be interested to some order messages, we simple add another subscriber to our topic, apply a filter to the subscription and we are set. We do not have to change our code, we do not have to change any publisher or any subscriber. It’s what we call plug-and-play with services.

Log into your Windows Azure Subscription and go to the Service Bus, Access Control & Caching service:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Select the Service Bus service under the available services:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

At the top ribbon click the New button to create a new Windows Azure service bus namespace:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We will create a new Windows Azure service bus namespace called “robbin-cremers” and we will locate the hosting  in the West-European data-center:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

After creation of the service bus namespace, our screen should look like this:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We have a Service Bus namespace robbin-cremers present under our Windows Azure subscription and we can add Queues and Topics to our service bus namespace. If you click on the Windows Azure service bus namespace, you can see the properties at the right:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

The name you defined for the Windows Azure service bus namespace is used in the uri to communicate with the service bus relay, service bus queues or topics.

For our scenario, we want to create 1 topic “orders” that will take all the orders that the publishers create. For now we will only have 1 subscriber, which is the website that allows customers to place orders.
To create a new topic by the Windows Azure portal, you click on the topics under the service bus namespace and hit the New Topic button in the ribbon menu on top:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We will create a new Topic called “Orders”:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We set the Default Message Time To Live to 84600 seconds, which equals 1 day. In case there are no subscribers present for the topic when the message is being recieved, the message will stay on the topic. If the message stays on the topic for more then 1 day without any subscriber reading the message, the message will expire. If a message is received on the topic and there are valid subscribers, the message gets passed on to the matching subscriptions, from which the subscribers can read the message.

We set the Duplicate Detection History Time Window to 600 seconds, which equals 10 minutes. This means that as soon as a message is received, the service bus will detect any duplicate message for this message in the next 10 minutes. This avoid duplicate messages being send multiple times, like for example clicking the send button twice. After 10 minutes the duplication detection expires, so the order message could be send again. We set the Maximum Topic Size to 1GB and we enable the Requires Duplicate Detection, since we want to check for duplicate detection to avoid an order being submitted multiple times after each other.

After we created the topic, it will be visible under our Windows Azure Service Bus namespace topics:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Hit the New Subscription button and we will create a new subscription to our Orders topic for the Warehouse department:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We enable Dead Lettering on Message Expiration, which will bring messages that expire to the subscription deadletter queue, on which someone of support can check the dead letter queue why the message did not get processed and take the necessary steps. We set the Default Message Time To Live to 1 day, meaning if the message stays on the subscription for 1 day without being completed or removed, it will expire. Depending whether Dead Lettering is enabled on Message Expiration it will be moved to the subscription dead letter queue or it will be removed permanently. In our case we can not afford to lose any order message because it doesn’t get processed correctly, so we are working with Dead Lettering.

There are two ways to read messages:

  1. Receive and delete: The client retrieves the next message from the buffer and immediately removes it from the buffer after it has retrieved it.
  2. PeekLock: The client retrieves the next message from the buffer, but does not remove it from the buffer. Instead the message gets locked, meaning it can not be processed again by anyone during the lock duration. You have to complete or abandon the message to have it removed from the queue.

We set the Lock Duration of the subscription to 1 minute, which stands for the lock duration of the message when using peeklock retrieval. We also Enable Dead Lettering on Filter Evaluation Exceptions which basically means the message will be send to the subscription dead letter queue if an exception occurs while trying to the apply the subscription filter on the message.

Our screen will look like this after creating the subscription:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

One important thing to notice is that when you create a new subscription within the windows azure portal, you can not create a filter on the subscription! For that reason we will create the other subscriptions within code.

3. Building our Windows Azure Service Bus messaging solution

I created a solution that contains 5 Console Applications:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

I guess most of the project names are quite obvious. The Windows.Azure.Messaging.Website console application is the publisher in our demo case. The other 4 console applications are departments that are interested in the order messages and act as subscribers.

To use the Windows Azure service bus messaging capabilities in code we will need to import the namespace Microsoft.ServiceBus. To add the namespace to my projects I use NuGet package manager:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Search for the Windows Azure Service Bus package and install it:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

It will add the latest Microsoft.ServiceBus namespace to the project:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Add this dll to the other projects where we need it as well. For the other projects you can use the Add Reference and browse to the packages folder in your solution folder:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We will start by creating a class library called Windows.Azure.Messaging.Shared that will contain some common code that will be used through our different projects:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

The Order class contains 3 classes which we use to mimic the orders that will be placed at our website:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Our OrderRepository has 1 static function GetOrder which returns an Order object which contains some dummy data to test with:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

And then we finally have the messaging class, our core class, which contains common logic for using windows azure topics, subscriptions and reading information:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We have an operation that returns a MessagingFactory, which is used to send messages or retrieve messages to and from topics and subscriptions. The other operation returns a NameSpaceManager, which is used to created or delete topics and subscriptions. Both of these operations have common code, which is to create a TokenProvider with the Issuer and Key defined at our service bus namespace. Finally we create an Uri which will stand for the service bus uri for our namespace, which we do by the ServiceBusEnvironment.CreateServiceUri.

We also defined 4 constant fields in our class, which refer to our windows azure service bus namespace we created.

  • Namespace: the name we called our service bus namespace, which in our case is “robbin-cremers”
  • Topic: the name of the topic we work on, which in our case is “orders”
  • Issuer: The default issuer provided by the namespace. Is needed to access the windows azure service bus
  • Key: The default key provided by the namespace. Is needed to access the windows azure service bus

If you click the service bus namespace in the windows azure portal, at the right you can see the properties:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

If you click the View Default Key:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

There you can find the Default Issuer and the Default Key, which are required to be able to access the windows azure service bus namespace we created. You can also set other Issuers to use then the default, but that’s out of the scope of this post.

We decided to write the service bus namespace, issuer and key into 1 single file instead of having to write it in every project that works with the service bus namespace. We also added the operations for creating and reading messages and so forth in the same class, that way we do not have to have access to this Issuer and Key in any other location, but this messaging class. If the Issuer or Key would change in the future, it can easily be done by changing it in our single messaging class. In a real world you might want to refactor it a bit better, but for this demo it will suffice.

Finally I created a method in my shared library to send a message to our Orders topic:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Quite important to notice is that we work with BrokeredMessage, which is the message you work with when working with the Windows Azure service bus messaging. We simple create a message at our publisher and pass it along to this operation, which sends to message to our Orders topic at our service bus namespace.

Finally we added some simple code to our Windows.Azure.Messaging.Website console application to mimic our Website on which orders can be placed:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We create a new BrokeredMessage and pass our serializable order object into the BrokeredMessage. If the object you are passing along is not serializable then an exception will be thrown. One very important thing to notice is that we set Properties on the BrokeredMessage. These are properties added on the message that is being send to the topic. We will use these properties to filter on for the Management and PurchaseDepartment.

We will also need some code to create the subscriptions for Management, Billing and PurchaseDepartment, which we did not create by the windows azure portal:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We have a shared CreateSubscription method which takes a SubscriptionDescription (details of the subscription) and a RuleDescription (defines the filter). It will get the NameSpaceManager and create the subscription if it does not exist yet. Finally we also need some code to read the messages from a subscription and print them to the output window:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We get the MessagingFactory and create a MessageReceiver through the CreateMessageReceiver method which takes the location of the subscription which is “topic/subscriptions/subscription” where you want to read from. As second parameter it takes the ReceiveMode which is either PeekLock or ReceiveAndDelete, which I already covered earlier in this post. Due to using PeekLock, we use the Complete or Abandon method to complete or remove the read messages from the buffer.

Due to using some shared code, our subscription console application code will look a lot more compact. The warehouse subscriberwhich simply reads the messages from the Warehouse subscription and outputs them to the console window:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

The Billing subscriber, which reads the messages from the Billing subscription:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Since we did not create the subscription for the Billing department through the windows azure portal, we need to make sure the CreateSubscription method is called when running the Billing receiver. It will create the subscription if it does not exist yet. We create a SubscriptionDescription, which contains the details of the subscription, just as you would set it through the windows azure portal. As filter we pass null in the shared CreateSubscription method, since billing wants to be notified of each order message so we do not have to set any filter.

One of the remarkable things is that creating a subscription through code allows to set some extra properties compared to creating the subscription through the portal:

  • EnableBatchedOperations: Whether batched operations are enabled
  • MaxDeliveryCount: Defines the maximum amount a message can be delivered. If the maximum delivery count is reached, it means the message could not be processed within the attempts and something is wrong. The message will be moved to the dead letter queue.

The Purchase department subscriber:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Only difference here is that we add a new RuleDescription which contains a SqlFilter which checks whether InStore is false. InStore is one of the properties we set on the BrokeredMessage which is being sent from the publisher. The filters we apply are on the BrokeredMessage properties. If the InStore property is set to false, it means an order was placed of some products that were not in store, so the purchase department will have to be notified of these orders to be able to get those products from an external vendor to complete the customer’s order.

The Management department subscriber which is equal, but with another filter:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

4. Testing our windows azure service bus messaging solution

I first ran the 3 departments once that did not have the subcription yet, so the subscriptions would be created before we start sending out order messages with the publisher. That way the messages will be transferred to the matching subscriptions. If the subscriptions do not exist yet when the message is being send, we will not be able to read the message from the subscription afterwards.

When you refresh the Windows Azure service bus namespace, you should see the newly created subscriptions:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

If I now run the 4 console applications that read messages from the subcriptions, no messages will be read yet, but as soon as an order is placed, the matching subscribers will pick up the order:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

If we run the Windows.Azure.Messaging.Website console application, a BrokeredMessage will be created containing an order and it will be send to the Orders Topic. For every subscription of the Orders topic, the message will be passed along to the subscription if it matches the filter of the subscription. The Billing and Warehouse subscriptions have no filters, so they should receive a notification of any order coming in. The management only wants to be notified of orders with a total price of more then 5000$, which this order is not, so they shouldn’t  be receiving any notification. The Purchase Department only wants to be notified of orders where some products might not be in store and in this case there are enough products in store for the order we created in the OrderRepository containing our dummy data order.

If we run the Publisher (website) console application, everything behaves as we expect it to:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Only the Billing and the Warehouse subscriber get notified of the order, which is correct since they want to be notified of each order.

If we now change the amount of Windows Phones we place in the order to 2 instead of 1, the Purchase Department should be notified of this order since we only have 1 windows phone in store while the customer orders 2 of them, so they will need to make an additional purchase at an external vendor:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Running the solution again:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

The purchase department also gets notified of the order because the message did validate the filter we placed on the subscription.

If we now change our dummy order at the OrderRepository to an amount of 20 windows phones, while there are 50 in store, the management should be notified of this message since the total price of the order is more then 5000$, while the Purchase Department should not be notified, since there are more then enough phones in store to ship the order:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Running the solution again:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

The Management department gets notified of this order because the Total price of the order is above 5000$. The Purchase department did not get notified because there were enough windows phones in store so they do not have to intervene in the order processing. The Billing and Warehouse receive the message as well since they subscribe to each order message to take the necessary actions.

5. Reading BrokeredMessage typed content and reading dead lettering queue

Now in our code we wrote the BrokeredMessage type and MessageId to the output window. When you use a receiver on a subscription, you obviously will want to read the message content from the subscription to process it. Parsing the message back to the type we originally serialized it from:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

We use the BrokeredMessage.GetBody<T> operation to get the message serialized back to the type we want it in. Since we serialized an Order into the message send to the topic, the subscription receiver will deserialize it back into an Order. The GetBody<T> operation on the BrokeredMessage takes a generic parameter, which can be a class or a primitive type.

When a message on a subscription expires and dead lettering is enabled, the message gets moved to the dead letter queue. If a message on a subscription is received by the client and the client fails to complete the message, then most likely the message caused the client to throw an exception. If the message reaches the amount of times it gets pulled down from the subscription to be equal to the MaxDeliveryCount, the message is considered a poison message and it also gets placed to the subscription dead letter queue.

For some reason, an order might not get processed or might not get processed within the message expiration timeframe, which can mean some messages get placed on the dead letter queue. In our case, orders are important pieces of our customer satisfaction. We can not allow unprocessed orders to go unnotified. That would make our customers very unhappy. That means you will have someone in support having the check up on the orders in the subscription dead letter queue.

Ideally you will write a background service which processes the orders from the dead letter queue and when an order gets moved to the dead letter queue, our background service will read this dead order and send out an email to someone of the support with the contents of the order. Then the support can find out why this order did not get processed and take the necessary steps, like fixing the issue or contacting the customer.

To read from the deadletter queue is not much different then reading from a subscription. Reading from the subscription looked like this:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Reading from a subscription deadletter queue will look like this:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

Instead of creating a MessageReceiver for the subscription, we create a MessageReceiver for the subscription/$DeadLetterQueue, which will refer to the deadletter queue of the subscription.

6. Exploring the Service Bus queues, topics and subscriptions easily with Service Bus Explorer

At some point, you will want to have some sort of graphical interface to create a new subscription with a filter, instead of having to create it by code always. One of these free tools is the Windows Azure Service Bus Explorer which you can find here: http://code.msdn.microsoft.com/windowsazure/Service-Bus-Explorer-f2abca5a
Download the c# code, extract it somewhere and go the project bin/debug folder and run the executable to work with it.

You can add a service bus namespace and it will load and visualize the queues and topics/subscriptions at your Windows Azure service bus namespace:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

You can also see the rules that are defined for each subscription:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

For our Warehouse subscription we did not create a rule, but a default rule gets created with a Filter “1=1″, which basically means every message validates against the filter. You can also create new subscriptions at a topic and directly set a filter for the subscription, which you sadly enough can’t by the windows azure portal at the time of this writing:

Windows Azure Service Bus messaging with Publish/ Subscribe pattern with topic and subscription

For me personal, it’s a useful tool to create new subscriptions easily with a rule or add rules to existing subscriptions.

7. Summary

There is a few very nice things working with the publish/subscribe pattern and the windows azure service bus:

  • Using the windows azure service bus with topics and subscriptions is pretty straight forward
  • You can easily plug another subscription on the topic without breaking any existing code
  • You can easily unplug an existing subscription from a topic without breaking any existing code
  • The senders and receivers are loosely coupled and they don’t know about each other
  • If one receiver goes down, the other receivers can still receive their message. They won’t know who else is receiving the same message, nor if those other receivers are available.
  • You can easily send with multiple publishers to 1 and the same topic. For example we have a website who submits orders to our topic, but you might have a support desk where to also can create orders manually and put them into the order workflow
  • It allows you to build a loosely coupled, disconnected and extensible solution
  • Many more of which I can not think of now
One of the things you need to keep in mind is that the maximum message size for a message to be send over the Windows Azure service bus is 256 kb, which is including the header information. The header information, which the custom properties, can only be of a maximum of 64 kb. Meaning if you want to send large messages over the service bus, you might want to store the message content into table or blob storage and pass the blob reference as content in the message. When the subscriber receives the message, it can get the blob reference as content from the message, after which the blob can be downloaded to the subscriber. In theory you should not hit the maximum message size that easily, unless you transfer very very large objects or are transferring parts of large data, like XML or attachments.

Any suggestions, remarks or improvements are always welcome.

If you found this information useful, make sure to support me by leaving a comment.

Cheers and have fun,

Robbin

About these ads