Earlier this week we announced that TipOff users can now automatically discover which market data streams they are consuming. By automating the discovery of market data protocols we’ve overcome what is a well-known challenge with many monitoring solutions. Clients want to understand their market data utilisation and detect sequence gaps, but the complexity and effort involved in working out which feeds they are consuming in the initial configuration stage, and then keeping this information up-to-date, can prove very resource intensive.
We actually have clients who are using TipOff in environments where they are consuming a large number of direct exchange feeds that need to be decoded, and have asked us to manage this element for them. So as a team, we really appreciate the management overhead that can be involved in building the mapping of packet level filters and the week-in, week-out administration required to maintain this information.
Some of our clients want to manage these tasks themselves but have recently expressed concerns about the time and cost implications of doing so. Following discussion with both existing clients and prospects we realised that if we could build an automatic decoder that could be pointed at a fire hose of market data to identify the feed and then check it for sequence number gaps, it could prove hugely beneficial all round.
What we’ve delivered is a really high performance solution able to reliably keep up with 1.2 million packets per second, per feed. In doing so, TipOff is able to identify what the market data packets are, decode the packets and conduct real-time sequence gap detection, whilst also generating message rate statistics.
We’re continuously expanding our library of market data protocol decoders, and all protocols for which we have decoders for can now be automatically discovered. At the moment we offer coverage for all major North American trading venues and the vast majority of European and Asia Pacific markets. Every month the list gets longer, as client requests for additional decoders are delivered.
We now have clients actively using the new automatic protocol discovery feature and its enabling them to quickly get up and running so they can perform all sorts of analysis. For instance we have clients who are interested in detecting gaps in the market data they are receiving as missing ticks could indicate quality issues. Consolidated feed providers want to ensure they are passing on the full picture to their subscribers and data consumers want to ensure they are not basing trading decisions on incomplete information. With this new functionality users can independently assess if they are loosing packets on any of their feeds.
Additionally, it can be used for trend analysis and capacity planning. Users can gain a breakdown of the bandwidth of different feeds, calculate message counts, classify the message type and determine whether it’s the original packet or a retransmission request that was sent.
Others are keen to use the solution for point-to-point market data latency analysis. Using the automatic discovery feature to extract sequence numbers and feed information, they can then measure latency as market data propagates through their infrastructure.
This feature can also be used by clients to determine exactly which market data feeds their application groups are subscribing to. Understanding what is being consumed offers not only a cost benefit, but its also important from an overall infrastructure management perspective. If you don’t know a particular feed is being accessed through your infrastructure how can you measure and manage its performance to avoid problems emerging.
This new feature can prove beneficial to any firm with multiple market data protocols coming onto their network that they want to better understand and control. Therefore, offering significant potential to both consolidated feed providers and investment banks with multiple market data groups or applications consuming traffic.
