Beeks Analytics (formerly Velocimetrics) started out as a bespoke solution built in response to a client brief, that quickly became an ambitious idea when we realised there wasn’t anything else out there that met what was a relatively common business need. That was back in the late 2000’s and since then I’ve seen the product come a long way in a relatively short number of years.
Through a process of iterative improvements, Beeks Analytics’ functionality has really grown organically. Many of the product’s enhancements have been the result of a client saying ‘I really want to do…. can Beeks Analytics help?’ to which we have invariably gone away into an engineering huddle and, more often than not, come back with a ‘Yes, and here’s how’.
Over-time these improvements have been added to the core product, along with those on our long-term roadmap, and I pretty much feel that what we have just launched with version 6 is what I really wanted to ultimately achieve when we first formed the company – by which I mean a solution that bursts out of the constraints so often associated with monitoring and performance analysis tools.
With Beeks Analytics 6.0 we wanted to be able to fully overcome the challenges of monitoring extremely large volumes of data across highly complex trading environments and processes. And we also wanted to be able to do so for long-running processes, as well as high volume “flash” trading for example, tracking the lifecycle of orders that may take a day to fill versus just those that complete in a matter of minutes or seconds, and track those trades into middle and back office processes.
We’ve met teams with this combination of challenges and most solutions available to them either monitor infrastructure rather than process, only allowing relatively simple chains of events to be monitored in large volumes or more complex chains to be monitored with lower volume, and typically only if specific application technologies are in play.
We also wanted users to be able to generate and design reports based on exactly what they want to see, in a timeframe that meets their business needs – should these be for real-time operational decision-making, to be posted as end-of-day reports, at any other chosen time or a mixture of these approaches.
Additionally, we wanted to empower the business user. Beeks Analytics enables users to rapidly quantify how issues at the infrastructure level have impacted the business and its clients and we wanted to ensure generating queries and analysing aggregated data could be done as simply as possible.
We put in a lot of the ground work to offer these pieces of enhanced functionality in previous versions, but this release takes Beeks Analytics to the next level of product maturity – delivering uncompromised end-to-end monitoring regardless of volume or complexity, with comprehensive on-demand reporting, advanced querying and aggregation capabilities.
Here is a short summary of the enhancements Beeks Analytics 6 offers, but I would be happy to explain further if you have any questions or would like greater detail:
Meeting the volume challenge
With version 5.0, Beeks Analytics started to hike up the volume mountain introducing the clustering concept – however, with version 6 we’ve seriously upped the ante.
The whole idea behind clustering is that rather than having a single server that has technical limits as to how many monitoring events it can analyse a second, Beeks Analytics’ users now can take their entire universe of events and process them via an array of interconnected servers in real-time – each of these servers is a partition within the cluster.
One of most significant problems with partitioning the correlation problem is that until you’ve correlated the data it is difficult to partition. In Beeks Analytics 6, partitions are in constant communication with each other, enabling the relationships of data being processed by any given partition to be accurately correlated with data being held by another partition. This could be for instance linking together a market data tick being processed by one partition that resulted in the decision to execute a trade being processed by a different partition. In particular we wanted the partitioning problem to be transparent to end users and those that configure Velocimetrics – that is, the system just “figures it out”, without users having to configure complex routing and correlation rules to deal with the ugly reality of distributed correlation.
By dividing up complex correlation and tracking workloads we have achieved near-linear scaling as volumes increase whilst continuing to deliver an aggregated view across the entire trading operation. Using proprietary distributed calculation schemes, Velocimetrics 6 allows aggregated and statistical reporting to be delivered effectively at high volume in real-time without introducing the computing bottlenecks that often bedevil aggregation across large data sets.
In addition to statistics, clustered Velocimetrics still enables individual item tracking and alerting (such as alerts generated because an item has gone missing) to be raised, distribution latencies across all processed trades in a period to be quickly formulated, highlighting outliers and allows full order and trade life cycles to be reconstructed.
Improved persistence
With Beeks Analytics 6 we have improved our persistence ability by an order of magnitude. One of the problems designers of high volume monitoring systems often come up against is once the data has been processed, they struggle to then push the database technology fast enough to avoid the act of transferring the data to disk, becoming a bottleneck in the overall operation.
We have overcome this problem by enabling users to scale persistence as they add more partitions to their cluster, because every partition is now writing to its own independent physical disk and database instance. Individual item and statistical data can now be retrieved by issuing cluster-wide queries. In fact, our query API hasn’t changed in Beeks Analytics 6 – the caller is unaware that there is a cluster behind the API.
Persistence improvements now allow us to record full transaction detail for the world’s largest institutions with a relatively modest cluster, reducing rack space requirements and ultimately total cost of ownership (TCO).
Augmented on-demand and scheduled reporting capabilities
Beeks Analytics comes from a real-time background showing what’s happening inside trading systems now, however, it was becoming increasingly apparent our clients wanted to be able to grab this real-time insight to generate reports for their managers and colleagues.
So with Beeks Analytics 6 we wanted to more effectively meet this requirement whilst providing clients with maximum flexibility. Therefore, we enabled users to define exactly what they wanted to create reports on, choose the selection criteria and what the report would look like. Reports can be generated in PDF format, or Excel-friendly CSV format; the former suitable to be pushed to email distribution lists on an automated schedule, whilst the latter provides a simple means of data extraction for customer-specific further analysis and presentation.
Finally, as we live in an on-demand world, Velocimetrics 6 users can also create reports intra-day by pushing a button.
Enhanced issue query and data aggregation functionality
We’ve always wanted to make using Beeks Analytics as easy as possible especially for non-technical users and I feel this version has helped to achieve this goal.
With Beeks Analytics 6 a query can be set up to find for example the location of a particular trade traversing a system or to generate a list of the worst performing trades within a period, that can be saved and then shared with other users who can activate the query whenever they want in a single click.
Velocimetrics aggregators have always been able to deal with large volumes of data, but sometimes finding the numbers you wanted in a deep or wide aggregation could be time-consuming. In Beeks Analytics 6, we’ve added two new features to help.
Firstly, a filter bar has been added to let you quickly focus the aggregator in the area you are looking for. Simply type a filter string, maybe including wildcards, and the aggregator view removes all nodes that don’t match the filter.
Secondly, you can set up an aggregator so that it only delivers the “top” rows. This feature is probably best explained with a brief example. Let’s say you had an aggregator that was rolling up computed TCP statistics across a very large number of TCP-level conversations. Let’s then further say that you are interested in the worst 20 performing conversations based on a particular statistic – for example, retransmission rates, a classic example of network-level problems. In Beeks Analytics 6, you would simply configure the aggregator to deliver data in blocks of twenty rows and sort by the retransmission column. Job done!
All of this enhanced functionality we have included in Beeks Analytics 6 and I am pretty pleased with what we have achieved. However, none of this would have been possible if it wasn’t for the great feedback we received from our clients and their willingness to work with us to test the beta version to ensure we had got everything right for them. So finally thank you all very much!
(By the way, for those of you still shocked about my opening comment mentioning that we launched in the late 2000’s yes that’s right we really did launch right in the midst one of the most financially challenging periods in recent history! I was clearly feeling either very brave or mildly crazy – either way I’m pleased to say we’re still here to tell the tale).
