Our motto at Beeks is simple: Build. Connect. Analyse. In this article, I talk more about the ‘Analyse’ part of what we do.
You’ll have understood from reading about Beeks how we offer bare metal servers, virtual servers, and managed networking for clients. We specialise in offering these systems at particular locations where there is a lot of financial market liquidity. We also specialise in the security and high performance that financial clients need.
It’s easy for some people to picture a computer running in a datacentre, and you can think of connectivity as being about enabling computers to talk to each other. But the analyse bit of what we do is a bit harder to picture.
Why do clients need the analysis part of what we do? Well, the simple answers are performance and service.
People using Beeks servers typically want to be the fastest — in financial markets, speed often means profit. So measuring your performance (often, it’s as simple as how fast you are) is critical.
Service is incredibly important, because in financial markets you frequently have firms using Beeks servers who are offering a service to their end customers. For example, you might be a software firm providing a trading platform. If your clients get a poor experience of your platform, they are likely to move away to a competitor, so we need to ensure that the service we provide is exemplary.
Beeks Analytics (previously known as Velocimetrics) allows our users to quantify and report on both performance and service, and it does this using wire-level data.
The truth is on the wire
What is wire-level data?
Well, imagine you are using an electronic trading platform and you see a promising price to buy US Dollars so you send an order to a bank. If you don’t get the price that you had hoped for, the bank will say that your order was slow to reach them. The provider of the electronic trading platform swears to you that the order left their system on time. This can happen again and again. How do you know who to believe?
Wire-level data avoids this issue. It allows you to get a copy of the data that is being sent from the wire that connects your trading system to the bank (don’t worry too much for this article how it comes from the wire). Critically, it doesn’t just give you a copy of your messages as they travel through this wire, but it records them for you with highly accurate timestamps.
Our hypothetical electronic trading platform user can now use this wire-level data to solve the argument between the bank and his trading platform.
If you are especially mistrustful of the trading platform provider, you might choose to insist that the time source for the wire data is independent (as much as possible) from the time source that they use for their trading platform monitoring.
From packet capture to advanced trading analytics
What we’ve described in the example above is a good example of packet capture. This is where you’re storing packets (in modern computing, messages are sent across networks in one or more ‘packets’), and looking at the packets when you’ve got a particular problem that you want to investigate.
Beeks Analytics provides this packet capture and timestamping capability, and it can cope with extremely high message volumes when doing so — as much as 200Gbps capture in a single appliance.
But for packet capture to be useful, you need to suspect that there’s a problem in the first place. In the bad old days, that would be from a user reporting a problem — not great service.
Where Beeks Analytics comes into its own is that it saves the amount of time you need to spend digging into packet captures by storing statistics that show you when a problem occurred. More importantly, it also allows you to anticipate and prevent where the next problem is coming from. Let’s start to talk about how it does that.
Decoding and analysing packets from the wire
Moving beyond just storing the packets and the bandwidth information associated with them, Beeks Analytics decodes them as well. What do we mean by decoding? Well, each packet contains a hierarchy of one or more different network protocols. To take one example, the widely-used (in finance) FIX protocol operates on top of the TCP protocol, which itself operates on top of the IP protocol, which itself normally operates on top of the Ethernet protocol. Different information needed for analytics will exist at each level of the protocol hierarchy. For example, to know the symbol on the exchange that is being traded we would need to look at the FIX protocol, but to understand which computer on the local network it originated from, we would need to look at the addresses in the Ethernet protocol.
If we can decode information from the message, then we can provide value-added information such as:
- Is a host that I’m communicating with struggling to process the volume of data being sent to it? (this would be indicated by a reducing TCP window size)
- Is there a problem where packets are not being delivered first time, and need to be retransmitted? (this is indicated by TCP retransmissions or duplicate acknowledgments)
- Are there UDP messages that have gone missing, and where are they going missing? (this is indicated by gaps in the sequence numbers of the UDP packets that are observed — this is known as gap detection and is particularly important where you are consuming market data)
- Are there delays occurring to any FIX messages that you are sending to or receiving from your trading venues or counterparties?
By monitoring gaps and latency at multiple points in the network, we can be definitive about where the issues are occurring, and we allow you to be clear about the service level that you’re offering to your clients.
All of this monitoring is made possible by the wide range of decoders that we make available as part of the Beeks Analytics offering – Beeks Analytics has over 100 decoders for different financial markets protocols
These decoders also cover underlying network protocols and enterprise middleware used within the financial markets.
We also make it easy to decode your own custom protocols, without necessarily needing to write your own decoder code
- If your protocol uses Simple Binary Encoding, our standard SBE decoder can use the message schema file as its definition of the fields that need to be decoded.
- If your protocol is UDP based, then our AutoMD™ decoder will automatically detect the sequence number to allow gap detection to work out of the box, without knowing any other details of the message specification. More on gap detection in a future article.
- If neither of the above are suitable, the Advanced Configurable Decoder (ACD)™ allows you to define decoders in JSON configuration. The advantage of ACD is that, once a protocol is implemented using this configuration, there is no release, retest, and reinstall cycle for minor changes in the protocol’s specs. Rather than needing development (sometimes by third-parties) against an SDK to respond to internal messaging changes, you can simply update the JSON configuration.
We’ve barely scratched the surface of what Beeks Analytics is capable of — we’ve not had the time to explore in depth the different types of latency detection, or the complex correlation between different messages in a flow. We’ll cover these in future articles. In the meantime, check out our technical documents, and the Analytics Concepts Guide in particular, to understand how you can gain better observability through Beeks Analytics.
