drwex | What it is that I (used to) do - or, one really wicked distributed system problem (Reply)

The video shows half a second's worth of activity on the open (*) markets in the stock JNJ (Johnson & Johnson). The half-second of activity is stretched out to almost 6 minutes so you can comprehend it at human speeds.

Each node in the graph represents a marketplace (New York Stock Exchage, NASDAQ, BATS, etc). The shapes coming into the nodes from outside represent order activity: someone offering to sell or buy (commonly called 'bid' and 'ask') shares of JNJ on that marketplace, or cancelling/modifying their existing orders.

As each piece of order information hits the marketplace it then must update all its peers about this change in status. This is the distributed system part of the problem. Markets maintain what is called a "book" of bids and asks and will try to fill orders from their own books because they make money that way. The market operates with the acknowledged fiction that there is a unified picture of the book and something called the NBBO (National Best Bid/Offer). The NBBO is shown as the bottom box (called SIP) in this video.

The reason this is done is because if Market A has a better price (the NBBO) than you have on your book then you're supposed to route the order over to that market to be filled.

For fun, freeze the video at any random point and look at the differences in price. Because these are real data traveling over real wires between real computers there are always speed variances and thus each node has a different view of "reality" (and the NBBO) at any given moment in time.

There are 'fast' markets (good hardware, the best networking) and 'slow' markets. People try to go to the good markets for the best prices, but this real-world difference is an opportunity for arbitrage. If you know you can get something for a couple pennies less on a fast market and then turn around and dump it onto a slow market before that slow market's book catches up then you profit those pennies. A typical trade size is 100 shares, so if you did it right you just made $1-2 on that pair of trades. Do that hundreds of times per second and presto you're a high-frequency trader (HFT).

We also know that reality isn't smooth. What's a "fast" market at 9:37 might not be fast at 9:38. Some markets are fast in some symbols - well, faster - but not in all. So what you do is sample a lot of data and build statistical models that let you predict what's going to happen, with a certain likelihood. Statistical models don't work all the time but the laws of large numbers work in your favor. Do the same thing tens or hundreds of thousands of times a day and you'll average out ahead, assuming your model is good. Presto, you're a stat-arb (statistical arbitrage) trader.

This stuff gets treated like mystical mumbo-jumbo by the press, but it's really just the intersection of math and networking. Many of the best stat-arb traders in the world know nothing about stocks and markets, but they know a lot about Matlab and how to draw meaningful conclusions from noisy data.

The rest of us can just have fun watching the pretty lights.

(For absolutely full disclosure here it should be noted that although I no longer work in this industry I am still a tiny minority shareholder in the company that owns BATS. I get no financial compensation for publicizing this or any related informational activity.)