PHP Circuit Breaker - initial zend framework proposal

Overview

UPDATE: project was moved to github a while back as an independant PHP library php circuit breaker

Circuit breaker is a component that supports high relaiability of web sites. It helps discovering, at runtime, which of the external dependencies are failing. Having that knowledge application can avaid wasting time on trying to call them untill they are back online.

Motivation

Current PHP application may depend on several databases, soap/rest web services, external cache providers or data grids, mail, ftp etc.

It is important for the application to keep on functioning even if some of these dependencies fail. To do that application has to be able to track when services become unavailable and when they become active again.

If a database or web service is down we want our application to detect it as soon as possible and react accordingly. Maybe application has a secondary slave database that can be read from? Maybe there is a way to load cached data? If all fails maybe its best just to hide some function or display message that service will be fully functional soon. What we want to avoid is making every user wait 30s for the database connection (which has to fail any way).

Circuit breaker pattern

Circuit breaker is a component that allows user code to check if external dependency is available before actually connecting to the external system.

Circuit breaker keeps track which services fail and, based on thresholds, decides if service should be used or not.

Circuit breaker also hides complexity from the end user code. It keeps statistics hiddedn and gives simple answers: available or not.

Last final cool thing that Circuit breaker does is that it will help the application detect when external service is back online so that all PHP processes can use it again.

Dependencies

Circuit breaker is standalone component that does not depend on anything except the persistance handler.

Circuit breaker needs a way to store service status between web requests (amount of failures etc). In PHP enviroment you have to store this information in some external data store as PHP requests do not share state.

Interfaces

Circuit breaker interface is keept to mimimum for simplicity. Interface allows the end users to check status of any external service before connecting to it. It also allows user code to report if connection failed or succeeded so that further requests can benefit from updated statistics.

interface Zend_CircuitBreaker_Interface 
{
    public function isAvailable($serviceName);

    public function reportFailure($serviceName);

    public function reportSuccess($serviceName);
}

The above interface is all that end user has to work with. The proposed implementation extracts persistance handler in form of a separate interface.

interface Zend_CircuitBreaker_Storage_Interface 
{
    public function loadStatus( $serviceName, $attributeName );
    
    public function saveStatus( $serviceName, $attributeName, $value, $flush = false );
}

The persistance handler takes string service name and stores it somewhere. I provide Dummy, APC and ZendCache implementation.

Circuit breaker factory will allow you to pass storage interface instance or configuration array/object to get Circuit breaker instance.

Please see the small diagram showing main components of my initial implementation of Circuit Breaker.

End user code scenarios

Here are three possible sequences of calls to circuit breaker instance.

External system is currently marked as unavailable

Cliend does not even try to connect to external service, we know it would fail. Instead client can instantly handle service failure:


External system is marked as available or circuit breaker decided to allow a retry

User code tried to connect to the external service but connection failed. User reports error back to CircuitBreaker:


External system is marked as available, connection succeeded

End user successfully connected to the service. User reports success back to CircuitBreaker:


As it is shown its up to the end user code to decide if external service connection failed or not. Circuit breaker does not knnow any details except the name and previous failures count.

Using Circuit Breaker from PHP code

As you can see below, component is designed for maximum simplicity of use. End user code does not have to worry how to handle retries, when nor which service is unavailable etc.

Client would colaborate with CB instance in a following way:

$userProfile = null;
if( $cb->isAvailable("UserProfileService") ){
    try{
        $userProfile = $userProfileService->loadProfileOrWhatever();
        $cb->reportSuccess("UserProfileService");
    }catch( UserProfileServiceConnectionException $e ){
        $cb->reportFailure("UserProfileService");
    }catch( Exception $e ){
        // do stuff that should happen if service is ok but query was incorrect
    }
}
if( $profile === null ){
    // nadle error in a gracefull way
    // for example show 'System maintenance, you cant login now.' message
    // where the user status would be shown.
}

Explanations:

  • $cb is an instance of Zend_CircuitBreaker. Can be stored in registry or passed in any other way
  • $userProfileService is a web service clinet instance i made up (used for loading of profiles)
  • "UserProfileService" is an arbitrary name you decided to use for profile service here. It could be any string
  • UserProfileServiceConnectionException is a made up exception name. Important thing is to distinguis between connection exception
    and code error or incorrect query exceptions. Circuit breaker should only be notified of connection failures. Application errors are
    not considered as dependency failures.

Failures count is a stack

Circuit breaker stores failures count up to maxFailures in the storage handler. Each time failure occurs the counter is increased. Once failures count is equal the maxFailures value, service becomes marked as unavailable.

Service is unavailable untill the retryTimeout seconds pass. Then Circuit breaker allows one of the PHP processes to go through by returning true from isAvailable method. Client does not know its a retry test (should work as usually not knowing its a sample).

If retry is successful, client will report back a success. Then circuit breaker will update stats in the storeage. Next time request comes and circuit breaker loads stats from the store it will see that failures count is maxFailures - 1 so it will allow client as well. This way if service becomes available again it will start working after max of retryTimeout seconds.

Configuration

  • You can use config array or config object with settings for each known service name.
  • Configuration can have any arbitrary service names.
  • Each service name is the first index of the array, second index is configuration attribute, value is setting value.
  • There are two configuration attributes that can be set per service: maxFailures and retryTimeout.
    • maxFailures - how many failures will be tolerated (counting as on stack) before service is marked as unavailable
    • retryTimeout - how many seconds should circuit breaker wait before allowing a retry
  • If there is no configuration value for some service name the defaults will be used.

Example of config array:

$config = array( 
	'UserProfileService' => array('maxFailures'  => 10, 'retryTimeout' => 120),
	'someDatabaseName'   => array('maxFailures'  => 30, 'retryTimeout' => 10),
);

Creation of instance

You can pass your own instance implementing Zend_CircuitBreaker_Storage_Interface to the factory if you want your own storage handler.

Different ways of instantinating circuit breaker:

$store = new Zend_CircuitBreaker_Storage_Adapter_Apc();
$store = new Zend_CircuitBreaker_Storage_Adapter_Apc(7200);
$store = new Zend_CircuitBreaker_Storage_Decorator_Array(
             new Zend_CircuitBreaker_Storage_Adapter_Apc(600, 'customPrefix'));
$store = new Zend_CircuitBreaker_Storage_Decorator_Array(
             new Zend_CircuitBreaker_Storage_Adapter_ZendCacheBackend( 
                 new Zend_Cache_Backend_File(array(
                     "cache_dir" => "tmp",
                     "file_name_prefix"=>"tmp",
                     ))));

$cb = Zend_CircuitBreaker::factory($store);
$cb = Zend_CircuitBreaker::factory($store, $config);
$cb = Zend_CircuitBreaker::factory($zendCacheBackendInstance);
$cb = Zend_CircuitBreaker::factory($zendCacheBackendInstance, $config);
$cb = Zend_CircuitBreaker::factory($zendCacheCoreInstance);
$cb = Zend_CircuitBreaker::factory(Zend_CircuitBreaker::APC_ADAPTER, $config);
$cb = Zend_CircuitBreaker::factory(Zend_CircuitBreaker::DUMMY_ADAPTER);

I hope above examples show that you can easily replace storage handler and pass on config array/object.

Once Circuit Breaker instance is created you do not mess with its guts, just use the interface.

Performance considerations

There is an obvious question to answer: how does it affect performance?

The answer is: almost not at all.

1. CB is a simple lookup and a few if statements, code itself is very simple.
2. The only slow part of CB is the code that has to load and save service statistics in the shared data store.
3. As long as you store CB statistics in APC / Memcache or other fast storage CB will be flying!

To allow maximum performance i would suggest using APC / shared memory or local memcache. They have minimal delays in rande of a millisecodn to load the stats. In most cases that should be a very low price to pay.

There are 2 generic usecases:

1. slow storage (slow meaning storage read takes 5ms or more)
2. fast storage (read is a sub millisecond operation like APC)

Circuit breaker is optimized to work best when all external system are working. We assume that most of the time that is the case. So if your external services are ok you need only one read per request. So in case of APC we are talking about sum millisecond delay.

In case of service failing delays will increase up to three fold as you will be doing read and write per request.

In case where your application has more than two external dependencies you want to track it might be good idea to use array decorator. Array decorator wraps all reads into single operation. Its especially important if your storage is slow. If you use remote memcache, file cache or database you should use array wrapper. This will significantly reduce the delays as you need only one read per web request no matter how many services you are tracking.

I have made some performance check on my windows laptop and numbers look good. See attached graph:

  • Fake: a fake Circuit breaker class that returns true on each call
  • Dummy: CB using dummy storage adapter (array)
  • APC: CB using real APC cache storage
  • Array Dummy: CB with dummy adapter wrapped in array decorator (to see overhead)
  • Array APC: Real APC cache adapter wrapped in array decorator
  • File Cache: Zend Cache adapter wrapping a Zend File Cache Backend instance
  • Array File Cache: Zend File Cache wrapped in an array

As you can see from the graphs all APC tests fall way below millisecond. Even if you probe multiple services or use array wrapper and report failures.

Tests performend on windows, i did not spend too much time so hard to say if they are really percise. I did not test concurrency, it would be interesting to see if APC or local memcache would suffer a lot on 20-50 concurrency. To be honest i would not worry about it any way.

Times averaged on thousands of calls per handler/scenario.

Going further

Code as above is simple but still forces user to explicitly use Circuit Creaker. It would be beneficial if Zend Framework components would allow wrapper classes using circuit breaker or use Circuit Breaker based on configurations. In this case user would request a CB-enabled web service instance from factory method and Circuit breaker settings would be used automatically from config file.

Users would benefit the most if they could set defaults in the ini file and all database adapters and web service components would be Circuit Breaker Enabled. Then user says use CB and this is the default threshold and retry time and all components in ZF know they should use CB before connecting to external resource. But that would cause dependency of all these components on CB classes.

Circuit breaker detects only unavailable services. If service fails in 0-50% then it would not exceed the maxFailures. It could be worth to add functionality to detect such situations. Some time based threshold should do the trick (failures per second/minute/hour limit).

Initial Implementation

Here is initial implementation of Circuit Breaker and adapters for APC zend cache backend and array decorator. I also include some unit tests but not everything is covered yet as its a initial release.

I would like you to share your opinions with me by email. You can aslo post to zf-contributors@lists.zend.com to discuss component and its adoption.

Code format and naming
I tried to follow naming and format guidelines from Zend Framework as closely as possible but let me know if you spot any inconsistencies.

Things to do
There is still some work to do especially on factory method. I saw Zend Framework uses factory methods quite often but im not sure if it would not be nicer to remove all this factory junk out to a separate class. Its a lot of code but for now its in the main class.

I also need to extend factory to allow config objects instead of arrays. I also have to make sure config names etc all follow the Zend Framework conventions.

At last i need to double check that all methods take parameters as should and return exact types registered, also need to add some more exceptions handling etc.

After i get some initial feedback i will finish off unit tests to get 100% coverage. I am not sure what is review going to bring and dont want to write tests twice : -)

Get the Circuit Breaker

Please get the code and let me know what you think. I am more than happy to answer any questions and change code to make it better.

UPDATE: project was moved to github a while back as an independant PHP library php circuit breaker

Comments

Post new comment

Image CAPTCHA

About the author

Artur Ejsmont

Hi, my name is Artur Ejsmont,
welcome to my blog.

I am a passionate software engineer living in Sydney and working for Yahoo! Drop me a line or leave a comment.

Follow my RSS