Typesafe Activator

Akka Circuit Breaker with Java!

Akka Circuit Breaker with Java!

TNG Technology Consulting
July 7, 2014
akka java

When talking to services you might want to make sure you play nice and lighten your load on them when they become unresponsive. The circuit breaker provided in the patterns package of Akka provides a nice way to achieve this. This template explains how to use it, and provides solutions for further problems that can arise.

How to get "Akka Circuit Breaker with Java!" on your computer

There are several ways to get this template.

Option 1: Choose akka-circuit-breaker-java in the Typesafe Activator UI.

Already have Typesafe Activator (get it here)? Launch the UI then search for akka-circuit-breaker-java in the list of templates.

Option 2: Download the akka-circuit-breaker-java project as a zip archive

If you haven't installed Activator, you can get the code by downloading the template bundle for akka-circuit-breaker-java.

  1. Download the Template Bundle for "Akka Circuit Breaker with Java!"
  2. Extract the downloaded zip file to your system
  3. The bundle includes a small bootstrap script that can start Activator. To start Typesafe Activator's UI:

    In your File Explorer, navigate into the directory that the template was extracted to, right-click on the file named "activator.bat", then select "Open", and if prompted with a warning, click to continue:

    Or from a command line:

     C:\Users\typesafe\akka-circuit-breaker-java> activator ui 
    This will start Typesafe Activator and open this template in your browser.

Option 3: Create a akka-circuit-breaker-java project from the command line

If you have Typesafe Activator, use its command line mode to create a new project from this template. Type activator new PROJECTNAME akka-circuit-breaker-java on the command line.

Option 4: View the template source

The creator of this template maintains it at https://github.com/TNG/akka-circuit-breaker-activator-template#master.

Option 5: Preview the tutorial below

We've included the text of this template's tutorial below, but it may work better if you view it inside Activator on your computer. Activator tutorials are often designed to be interactive.

Preview the tutorial


In this template we will take a look at a typical problem when talking to other systems: dealing with their failure.

Especially when talking to publicly available services you cannot always operate under the assumption that the third-party service will always be available and always perform well. One reason for a third-party service not being available can be a high load that is crippling the system. Therefore just continuing to send your messages to that system will make the problem worse. A better solution would be to detect problems with third-party systems and react accordingly by stopping sending requests. But this is not only useful in scenarios in which you communicate with third-party services, you can use this to avoid overflowing your own actors, as well.

Akka provides a circuit breaker component for this exact scenario in its pattern package, and there is a nice documentation for it. But in most real-world scenarios using only the circuit breaker will not be enough. In this template we will take a look how to deal with the problems that might arise.

The Service

First we will need a Service to call. We will simulate this service using an actor. Take a look at the code in Service.java.

The Service actor will simulate a service that becomes very slow from time to time. In the constructor we set up a regular Swap message, that will be sent to the actor every 2 seconds. When receiving the Swap message the actor will change its behavior. In its normal state it will reply to each Task message with a Response message. Upon the first Swap message it will change its behavior. In this new state it will block for 1 second and ignore the message. No response will be sent.

The next Swap message will revert the Service back to the normal behavior. Please note that blocking inside your actors is not something that should be done under normal circumstances. In this case, however, it simulates exactly what we need, a service that does not respond to our requests, and does not handle new requests in the mean time. A nice side-effect is that the queued messages prevent the swap message from being handled, an indication that us sending the service messages impedes it from recovering.

Let's see what happens when we run it.

We defined a main method in the Main.java class. It creates an actor system, and the service actor, as well as a TaskCreator actor, that will create the Tasks. To do this we set up a regular Tick message that will be sent every 200 milliseconds and triggers creation of tasks. Then we wait for 10 seconds, before shutting everything down. Thanks to the log statements in the actor we can observe what happens when we execute the program. Each Task message receives incremented ids, so we can follow the messages in the log.

First we will see a number of requests and responses, after 2 seconds the first Swap message is received and we see that there is a delay of one second between request and response. After 2 seconds the second Swap message should arrive, but as there are still Task messages in the Service actors mailbox, the Swap message will be delayed. Obviously this is a problem.

Let's try to take the load off our service.

Introducing the circuit breaker

Let us tackle the problem using a circuit breaker. The relevant code can be found in SimpleCircuitBreaker.java.

Let us go through the code.

First we need to initialize the circuit breaker:

new CircuitBreaker( getContext().dispatcher(),
            RESET_TIMEOUT )
We need to specify the dispatcher the circuit breaker will be running on, as well as a scheduler. The scheduler is necessary as the circuit breaker uses Callables to send the messages to the service. Next, we set the maximum number of failed attempts before the circuit breaker opens. This setting is very specific to your situation. If the service you are talking to is on the same machine as your Akka process, you can use a low setting, as failures are most likely caused by problems within the service. On the other hand, a service you connect to through the internet may fail on occasion because there are problems with the connection, you will want to make sure that you do not stop the message flow just because one or two messages got lost. Similarly the next two settings depend very much on your unique situation. First, the call timeout specifies how long the circuit breaker is willing to wait for a response. If the response is not received in this timeframe, it will be considered failed. The last parameter is the reset timeout, the time the circuit breaker will wait after opening until it tries to send another request.

After creating the circuit breaker, we register some callbacks to be notified when the circuit breaker changes its state. In this case we will simply log the changed state, you could have more complex logic in there though.

Sending messages

Now, take a look at lines 61-67 to see how to send messages to the service using the circuit breaker. The circuit breaker provides two modes of operation, callWithCircuitBreaker will make the call asynchronously and return a future containing the result of the call, while callWithSyncCircuitBreaker allows you to perform a synchronous call. We will use the asynchronous method here. The method expects a callable object as parameter, which we will declare anonymously at this point. The callable needs to return a Future<Object>. This makes it very natural to combine the circuit breaker with the Ask Pattern, since it returns Future<Object>. We send the Task to the service, and let the circuit breaker wait for the response for us. In order to return the result to our caller, we use the PipeTo pattern to return the response once the future is done.

Time to run the example. Go to the Main.java class and comment line 13, while uncommenting line 14,

system.actorOf( SimpleCircuitBreaker.props(), "SimpleCircuitBreaker" );
Once you've done this, click Save and switch to the Run tab. Activator should run the program after compiling automatically, unless you switched this off. In that case, just click the Start button.

Using the CircuitBreaker works quite nicely for our use case. After the service switches into slow mode, it will still receive some messages until the circuit breaker realizes something is wrong, but then the flow of messages stops, and the service can recover. The circuit breaker will probe if the service is available again, and if it is, will resume sending the requests. There is one problem though. The Task messages that are sent while the Service is in slow mode are never delivered.

More on the circuit breaker

Before we move on, note that we use the call timeout of the circuit breaker for the ask timeout. This may or may not make sense for your use-case. You may want to use a longer timeout for the ask, so as to still receive responses from the service, even if the circuit breaker will treat that call as failed and cease sending further messages. Note that you will need to add handling of the messages in this case, as responses of timed out calls are not returned. Also, it is noteworthy that for the asynchronous call method the call timeout is only used to decide if the call failed or succeeded, the call will not be interrupted after the timeout. This is useful when you want to get the result, but it means that the circuit breaker will open only after the call finishes. Try setting the ask timeout to a higher value in the code (e.g. 500), and see what happens. This is not the intended behavior, and a issue is open for this.

For some use cases this simple solution is already all you need. But usually creating our messages was hard work, we want to make sure they all reach the target. Right now they are just being discarded. So we need to make sure to remember all messages for which we did not receive a response from our service yet. There are several possibilities to do this. While it is entirely possible to do this using a queue and some become magic, this is not necessary anymore. Akka 2.3 saw the introduction of Akka Persistence, which allows to persist messages, which sounds like just the thing we need.

At least once delivery

What we are trying to do here, is to accomplish at-least-once delivery. Normally Akka guarantees at-most-once delivery (read here for a detailed discussion of message delivery reliability), and this is what we saw in the earlier examples, no Task was delivered twice to the service. However, a lot of the Tasks got lost on the way. With at-least-once delivery we will make sure that every one reaches its destination.

If you have used the eventsourced library or the Akka Persistence API prior to 2.3.4, you might have used channels for this. However, in an attempt to streamline the API, the Akka team greatly reduced the amount of concepts. Now, if you want to use persistence, you will have to use UntypedPersistentActor. This actor provides event-sourcing, which means that, unlike in command-sourcing, not every message that is received will automatically be persisted. Instead you have to provide persit blocks that will persist the events that are created from received commands. This is a very nice way to recreate your actors state, as you only persist and replay events that influence the actors state, if you receive other messages (e.g. to retrieve the actor state), these do not need to be persisted and replayed. Usually you separate updating the state and any side effects like sending messages, so that the side effects are not triggered during recovery.

To use at least once delivery, Akka persistence introduces the perfectly named UntypedPersistentActorWithAtLeastOnceDelivery. This actor lets you create a deliver block, in which you can provide a message and a destination. Each message needs to have a deliveryId. This id will be used to confirm that the message has arrived. For this the receiver needs to respond with a message that contains this deliveryId. When you receive this response you then call the confirmDelivery method with the deliveryId as the argument. Akka Persistence will attempt redelivering the message until such a response is received. Both the event from which you create the message to be delivered and the response need to be persisted. This way if the actor crashes and restarts it will know which messages have already been delivered, and which need to be resent. Let us take a look of how this looks in our example.

The persisting circuit breaker

Take a look at PersistingCircuitBreaker.java The PersistingCircuitBreaker actor extends the UntypedPersistentActorWithAtLeastOnceDelivery. UntypedPersistentActor provides two abstract receive methods, onReceiveCommand and onReceiveRecover, the former is the regular receive method, while the latter will only be used when the actors starts and performs recovery of persisted events. Furthermore the UntypedPersistentActor provides methods that can be overwritten to change some settings. We use redeliverInterval and persistenceId. The persistenceId is especially important, as it will mark all messages for this actor. Per default the actor path is used, which works well in most cases, but can be a problem if you change your path or if your actor has no name (as the path will then be dependend on the order in which the actors are started).

We also create a circuit breaker actor, that is basically equal to the SimpleCircuitBreaker actor we used in the last step. Upon receiving a Task message the PersistingCircuitBreaker will wrap it into an envelope to preserve the original sender. We need to preserve the original sender, as the PersistingCircuitBreaker needs to receive the response to confirm the delivery. We then persist the envelope to be able to resend the message during actor recovery and pass it on to the updateState method. This method encapsulates the delivery part, as it needs to be performed during recovery as well as normal operation. Here we will put the TaskEnvelope into a new message type which contains the deliveryId. We specify the circuit breaker actor as a receiver. Note that we need to use the path for sending here. This is because the semantics of at-least-once delivery are more in line with sending to ActorSelection than sending to ActorRef, which is made visible by using the path instead of the ref.

The handling of DeliveryConfirmation is similar. We also need to persist it, and pass it on to the updateState method. However, before we do that, we extract the response from the service from the confirmation and send it to the original sender. We do not put it into the updateState method, as this would cause the message to be sent each time during recovery, we do not want that. Using event-sourcing it is easy to separate side-effects (sending of the response) from state updates (confirmation of the delivery). In the updateState method we simply confirm that the response was received, and that Akka does not need to try resending the message anymore.

In CircuitBreakerPersistentActor we had to add handling of TaskEnvelope messages. We extract the task from the envelope and send it to the service, and if we receive a Response message, we wrap the response into a DeliveryConfirmation message, that contains all information the PersistingCircuitBreaker actor will need.

We use the resending feature to make sure Tasks that were rejected by the circuit breaker or received no response from the service will be resent. A nice side-effect is that our messages survive a restart of the actor or the complete actor system. If you run the example twice you will see messages that were not confirm in the last run being sent around 1 second after the start.

Running the example

Go to the Main.java class and comment line 14, while uncommenting line 15,

system.actorOf( PersistingCircuitBreaker.props(), "PersistingCircuitBreaker" );
Once you've done this, click Save and switch to the Run tab. Activator should run the program after compiling automatically, unless you switched this off. In that case, just click the Start button.

Starting the program we now see that after the service recovered it will receive the messages that could not be sent during the down time. Usually you will see Tasks 23-25 before tasks 11-20 will be replayed. That is because the redelivery time is set to 1 second.

The combination of persistent at-least-once delivery and the circuit breaker is quite nice, if you are dealing with unreliable services. However, problems can arise if the service is unavailable for a long time or you send a high number of messages. As the UntypedPersistentActorWithAtLeastOnceDelivery actor stores all undelivered messages in memory you can face problems when a lot of messages are waiting for delivery. There is a setting for the at-least-once delivery, maxUndeliveredMessages which can be used to deal with the problem. Now, when the limit is reached trying to deliver new messages will result in an Exception being thrown. In such a case you use supervision to divert all messages to a temporary storage until sending is possible again. Or you could stop creating new messages or take other measures that make sense for your business case.

Further exercises

In order to gain more experience with circuit breakers you can try several things using this template:

  • Make the service respond in slow mode and check how the different implementations deal with this
  • Add a delay to responding to a Task in normal mode and look at how the different implementations deal with this
  • Use variable delays in the service, make it return or drop responses randomly, which solution is the best under this circumstances?
  • Limit the number of undelivered messages to 5 and deal with the exceptions that will occur

We hope you had fun going through this template with us. Any issues, feedback or questions can be given using the GitHub page of this template.

Happy hAkking !

comments powered by Disqus