Improve performance for remote asset events

Symptom: weak performance of the censhare server

Reduced performance of a censhare server can be caused by a network with one master server and several remote servers. This can result in delayed processing of remote asset events. If a censhare server generates asset events, the server sends them as remote asset events to other servers so they can also respond. As of censhare 5.4 or 4.10, you can configure a remote thread pool in order to limit the effects of the delay. 

Log information and monitoring the remote event queue can help you keep an eye on the remote event processing of a server. 

Processing remote event assets

When a censhare server writes data for an asset, it generates so-called asset events depending on the situation. These include for example "Save and close" or "Asset workflow changed".

These events can initiate commands in censhare, for example, "Create PDF (automatic)" in the module renderer. The events have to be sent to all other censhare servers in the network. Modules can wait there for events in order to carry out certain commands. For certain events, actions may only be configured on remote servers to provide relief for the master server.

If a thread on the master server generates an event, it will transmit this event to another thread from the thread pool. This remote event thread is now responsible for sending the event. censhare servers send events over the network using the RMI protocol (remote method invocation). The events are sent either individually or in packets. RMI works synchronously. It confirms receipt of an event to the sender server immediately. If a thread sends an event, it can only continue processing after receipt of the confirmation. It doesn't matter whether modules are configured on the recipient server that is meant to process the event in question.

Cause: delayed remote event processing

If multiple events arrive at a remote server simultaneously, the server may not be able to handle receipt due to the quantity. In most cases, the master server sends multiple events because of high activity.

If the remote server can't receive any more events due to an overload, the associated remote event threads on the master server need to wait for receipt confirmation. During that period, events continue to be created on the master server and need to be sent to remote event threads. The master server thread pool can be cleared in this case. That affects users and applications on the server. A similar sequence is possible on remote servers, too, but remote servers will not generate nearly as many events as a master server.

A network with heavy delays can exacerbate this development and become a problem because it increases the time needed for transmitting events. It then takes longer for the master server to again release a remote event thread. One example of this would be connecting a remote server to the master server using a slow WAN.

The reciprocal impairment of the master and remote servers can also spread to other remote servers. Events are generated on those servers, too, that are subsequently sent to the master server. If the master server is slowed down, the receipt confirmation to other remote servers is delayed or might not even take place. These are then affected by the delay, too.

Solution: activate a separate thread pool for remote events

To combat the problem of the master server being cleared, you can activate a remote event thread pool. This solution is available as of censhare 5.4 or 4.10. The remote event thread pool is more effective and replaces the old approach. We recommend, if possible, that you use the remote event thread pool for performance problems with remote event processing. Both approaches can't be used simultaneously. For more information see the section "Previous solution: Send fewer events".

If the remote event thread pool is turned on, the master server will take a thread from that pool to use for sending remote events. This pool also contains a correspondingly long queue in order to buffer remote events with a remote event thread until they can be processed. This takes the main thread pool out of the equation for remote event processing, which keeps it from being affected at once by any problems that may occur there. But the master server is still available to users and applications. Remote server problems that occur during remote event acceptance are also not as easily passed on from the master server to other remote servers.

Recommendations for the area of application

Configuring a separate remote event thread pool on the master server is recommended only as of a certain installation size: For a master server and three to five remote servers on which some activity is taking place. Another important indication for using a remote thread pool is when you are working with a slow network such as a WAN.

It is also possible to set up a remote event thread pool on remote servers, too. This can be helpful, for example, if the communication between censhare servers is not in a star layout. In this case, the remote servers communicate with each other as well as with the master server.

You can also enable the remote event thread pool if the recommended application area does not apply or applies only partially to your censhare installation. Activation does not always improve the performance of the censhare servers, but it does not affect it.

Configuration

Set up remote event thread pools in the censhare Admin Client at Configuration | Services. Double-click the desired server folder:

• Event folder for local server:
  In the Event setup, deactivate Use asynchronous calls for remote events. If the XML file is to be edited directly:

"remote-post-async"="false"

• Remote server folder
  In the Remote server setup, activate Use remote event posting thread pool. If the XML file is to be edited directly:

  "use-post-thread-pool"="true"

Specify the number of threads in the remote event thread pool. The number should be double the number of remote servers. If the number is smaller, you run the risk of one remote server having to wait. In the field "Event posting thread pool size" enter the desired size, for example, 10 for five remote servers. XML file:

"Event posting thread pool size"="10"

Specify the size of the event queue. This should be long enough to buffer a larger amount of remote events until a thread can send them on. If the queue is too small, threads from the main pool may have to wait or not even be able to hand over their remote events to the remote events thread pool. That in turn weakens the performance of the master server. In the field "Event posting queue size", enter the size of the queue, for example, 10000. XML file:

"post-queue-size"="10000"

Monitor remote event processing

censhare provides you with two tools to monitor remote event processing on servers:

• "Remote event queue" diagram in the censhare Admin Client
• server log entries

Remote event queue diagram

The "Remote event queue" diagram shows how many remote events are waiting in the server queue to be sent by a remote event thread. It contains three sections for the last 60 seconds, 240 minutes and 96 hours. If the diagram shows no events, the remote events are currently being processed directly by a thread and no buffering is needed.

The diagram is only available if you have also set up a remote event thread pool on the server.

1. To see the diagram, log into the censhare Admin Client and go to the Status folder.
2. Double-click the Diagram entry to open it. On the left side, you see all currently selected diagrams.
   If the desired diagram isn't displayed, select it on the right. You find all available diagrams, categorized.
3. Click the Other category to show that list. Here, activate "remote-event-queue".
4. Double-click the diagram to open a detailed view. Here you can see whether the number of events has frequently exceeded the configured size of the queue and adjust the value to your needs.

Server log entries

The corresponding entries in the server log show how long the processing took for the remote events. This is the time needed from when the event was handed off to a thread for transmission until the time the confirmation was received. The time a remote event spends in the queue is not measured. Roughly every ten minutes the server records the processing time in milliseconds for all servers to which it transmitted events. Currently, the information is recorded with log level "Info", this is always. You don't need to change anything in the logging level in order to receive the information.

Previous solution: limit the number of remote events

The idea behind the previous approach was to limit the number of remote events arriving at a remote server. The number of threads on the master server that can send events simultaneously is limited. This limits the burden on remote servers so they can more quickly process events.

The solution is well suited for installations with a master server and remote servers that are not all that active. But it does not address problems that occur due to slow remote event processing. 

As of censhare 5.4 or 4.10, the remote event thread pool represents a more efficient solution. We recommend to use it. 

Configuration

Set up event limitations in the censhare Admin Client at Configuration | Utilities. Double-click the desired server folder:

• Event folder for local server:
  In the Event setup, deactivate Use asynchronous calls for remote events. If the XML file is to be edited directly:

"remote-post-async"="true"

• Remote server folder
  In the Remote server setup, activate Use remote event posting thread pool. If the XML file is to be edited directly:

  "use-post-thread-pool"="false"