The Stress Tester

The Stress Tester is designed to put a Hydra system under load by setting up a collection of clients that post and listen for messages. The rate of posting, size of messages, and other parameters are controllable from a console app using, of course, Hydra messages. The Stress Tester is useful for determining how Hydra will behave in your environment, but is also an interesting example of Hydra code as it involves multiple applications, communicating using several different message types.

Quick start

The Stress Tester code can be found in the DotNet\Examples\HydraStressTest folder of the Hydra source code, where the HydraStressTest Visual Studio solution lives. Opening this reveals a solution containing 5 projects. To get going immediately:

  • Edit the HydraServers list in the App.config files in the HydraStressTestClient and HydraStressTestConsole projects to contain the addresses of your replicating Hydra servers.
  • Build the solution and run instances of HydraStressTestClient on some test machines, and an instance of HydraStressTestConsole somewhere convenient. Each of the client instances should show up in the console, where you can tweak parameters and watch the number of messages sent by each client. If you see many errors reported, then increase the Buffer delay parameter in the console (apply it by clicking the Update button). A value of 1500 is usually plenty in my test environment.
  • Leave it for a couple of weeks to soak.

In more detail

The Stress Tester consists of two components: a client, to be run on multiple test machines, and a console, to monitor what's going on.

The client

The client does several things:

  1. Sends test messages of configurable size, at configurable intervals.
  2. Polls Hydra for test messages from other clients.
  3. Sends error messages if incoming test messages are out of sequence.
  4. Sends regular heartbeats containing configuration information.
  5. Polls for messages from the console with configuration changes.

Let's take these in sequence:

Send test messages. The client sends out a random number of messages of the StressTestData type at regular intervals. Each message contains: a client identifier (machine name and process id), the current user and their domain, a sequence number, and a data string of random length. The interval, maximum number of messages sent, and maximum message size are all configurable. Each data string is constructed from the strings in MessageData.resx: to construct a string of length x, the system starts with the longest MessageData string of length <= x, and then repeats the process with the remaining number of characters required, until it gets to at least x characters. To get a realistic test, you should replace the MessageData strings with ones that are representative of your messages - this will impose the right sort of load on CouchDb's compression processes.

Messages are sent using a PubSubByType Publisher, using code like this:

    ...
    _stressSender = new Publisher<StressTestData>(_hydraService);
    _sendObservable = Observable.Interval(TimeSpan.FromMilliseconds(_sendIntervalMs), Scheduler.ThreadPool);
    _sendSubscription = _sendObservable.Subscribe(OnSend);
    ...

    private void OnSend(long seq)
    {
        // Send up to MaxSendBatchSize messages
        try {
            for (int i = 0; i < _rnd.Next(_maxSendBatchSize + 1); i++) {
                string data = CreateStringOfLength(_rnd.Next(_maxDataLength + 1));
                _stressSender.Send(new StressTestData {Sender = _myName, Domain = _domain, Username = _username,
                                                       Seq = _sendCount + 1, Data = data, Timestamp = DateTime.UtcNow});
                _sendCount++;
            }
        } catch (Exception) {
            // Ignore errors
        }
    }


Poll Hydra for test messages and send error messages. Using a PubSubByType Subscriber, each client polls for StressTestData messages. It keeps track of the latest sequence number from each of the other clients and, if a message arrives out of sequence, sends a StressTestError messages, using a PubSubByType Publisher.

Send heartbeats. Every minute the client sends a StressTestControl message to the console. This contains the current values of the parameters described above when sending StressTestData messages, booleans to indicate whether the client is currently sending and currently polling, and the current Hydra buffer delay. Messages use the HydraMessage Source and Destination fields to identify the sending client and the destination console app, so cannot be sent using a PubSubByType Publisher. The code used for heartbeats is like this:

    ...
    _serializer = new HydraDataContractSerializer<StressTestControl>();
    _heartbeatObservable = Observable.Interval(TimeSpan.FromMilliseconds(_heartbeatIntervalMs), Scheduler.ThreadPool);
    _heartbeatSubscription = _heartbeatObservable.Subscribe(OnHeartbeat);
    ...

    private void OnHeartbeat()
    {
        try {
            var reply = new StressTestControl {
                Listen = _listening, Send = _sending, SendBatchSize = _maxSendBatchSize, SendIntervalMs = _sendIntervalMs,
                SendMaxDataLength = _maxDataLength, BufferDelayMs = (_subscriber == null) ? 0 : _subscriber.BufferDelayMs
            };
            _hydraService.Send(new HydraMessage { Source = _myName, Destination = "StressTestConsole", Topic = typeof(StressTestControl).FullName, Data = _serializer.Serialize(reply) });
        } catch (Exception) {
            // Ignore errors
        }
    }

Poll for configuration changes. Each client also polls for StressTestContol messages sent to it from the console app. On receipt of one of these it will update its current setting to match those of the message. Clients cannot use a PubSubByType Subscriber for this, as this is for broadcast messages and does not allow differentiation of messages intended for a single recipient. It therefore uses a lower-level mechanism which does allow specification of destination when polling, as follows:

    _poller = _hydraService.GetPoller(new HydraByTopicByDestinationMessageFetcher(typeof(StressTestControl).FullName, _myName));
    _controlSubscription = _poller.Select(hydraMessage => _serializer.Deserialize(hydraMessage.Data)).ObserveOn(SynchronizationContext.Current).Subscribe(OnControlRecv);

Here we use GetPoller and HydraByTopicByDestinationMessageFetcher, which polls for message with the given Topic and Destination values. Topic is just the name of the type and matches the Topic above when sending StressTestControl messages; Destination is the name of the client - this matches the Source value used when sending messages, and is used by the console app when it wants to send messages to a particular client.

HydraByTopicByDestinationMessageFetcher returns HydraMessage objects, which the Select statement takes and deserialises the Data field into a StressTestControl object to be passed to OnControlRecv to update the configuration.

The console

The console is simpler than the client. It does the following:

  1. Polls for test messages.
  2. Polls for error messages.
  3. Polls for heartbeats.
  4. Sends configuration change messages on request.

Again taking these in sequence:

Poll for test messages. These are the StressTestData messages sent by the test clients around the network. The console maintains a collection of Windows controls, one for each known client, used to display information about that client. It console polls for the StressTestData messages using a PubSubByType Subscriber and passes each one on receipt to the control for its sender (creating the control if it does not already exist). The message is used to update a display of how many messages the client has sent.

Poll for error messages. The console sets up a PubSubByType Subscriber to StressTestError messages and again passes each one on receipt to its control, which updates its disaply of how many errors the client has received.

Poll for heartbeats. These are the StressTestControl messages above. Just as they needed a lower mecanism to send, so they need that same mechanism to receive. The console polls for StressTestControl messages directed to itself using this code:

...
    _poller = _hydraService.GetPoller(new HydraByTopicByDestinationMessageFetcher(typeof(StressTestControl).FullName, "StressTestConsole"));
    _controlSubscription = _poller.ObserveOn(SynchronizationContext.Current).Subscribe(OnControlRecv);
...

Where OnControlRecv handles the incoming message by passing it to the relevant control, which updates its display of the current values of the configuration values for the client. This is the same process that the client uses for subscribing to control messages, but the specified destination is the console rather than the client.

Send configuration messages. The Windows control for each client allows the console user to specify new values for the configuration values, and click an update button. This button causes the control to send a StressTestControl message whose destination is the client being reconfigured. This uses the same mechanism as the client does when it sends out heartbeats.

Last edited Jun 13, 2012 at 2:29 PM by NickNorth, version 9

Comments

No comments yet.