Table of Contents
Just a second...

Conflation

Conflation of messages is the facility to treat two messages as being essentially the same and avoiding sending duplicate information to clients.

Updates and messages for a client are queued on the Diffusion™ server. Each client has its own outbound queue.

Conflation removes an existing message from the outbound client queue and replaces it with a newer, equivalent message either at the position of the old message or at the end of the client queue. The replacement message can either be a new update or a merge of the new update and an existing update.

Conflation is an optional feature that can be applied to all clients, clients connecting through a specific connector, or can be applied programmatically on a client-by-client basis.

Advantages of message conflation

There are many scenarios in realtime data distribution where data being communicated need not only be current, but must always be consistent. Structural conflation maximizes for concurrency while ensuring consistent delivery.

Concurrency

Both simple and structural conflation maximize for concurrency through avoiding distributing soon to be stale data. The higher the rate of change, the higher the value extracted. On other words, where clients or servers are running near to saturation based on available connectivity Diffusion can automatically adapt to this load by minimizing the data distributed.

In addition to the load-adaptive nature of conflation the effect is fair for clients connected to the same topic. The frequency of distributed changes is evenly amortized across clients.

Conflation favors currency and reduces (but cannot entirely eliminate) the delivery of stale data.

Consistency
Structural conflation synthesizes complex event processing techniques in a highly efficient (lock-free wait-free concurrency) form inside the server. The specific knowledge of data structures and the semantic concerns for distributing data for a given topic in a given system allows consistent views of the data to be delivered in a way that is not possible with messaging technologies that treat messages as opaque.

Considerations when using conflation

  • Do not use conflation if there are relationships or dependencies between topics. Conflation alters the order of updates. If a conflated topic is temporally or causally related to another topic, conflation can cause unwanted behavior.
  • Do not use conflation if individual updates carry forensic storage or audit trail requirements.
  • Delta updates are conflated. Snapshots are not conflated.
  • Normal priority updates are conflated. High or low priority updates are not conflated.
  • Messages that require acknowledgment are not conflated.