Statistics for engineers (a few rules of thumb)

Statistics is fake. Hash it at 50%. Do a count by for each group. Plot the 50th, 75th, 90th, 95th, and 99th percentiles for each group for every hour, and then day. Good enough.

That is most of the statistics that you will ever need.

Rule: you are measuring exactly what you are measuring

When you run A/B experiments in large systems, most of the time, the actual data you're looking at is wrong in name. Though, the data itself is always correct. Your job is figuring out what it actually means. People have a bad habit of measuring a signal, and then claiming that it implies its corollary.

Reproduce the signal. Actually carry out the action yourself. Watch the data get populated. Now; you know what that data means. Exactly what you did to reproduce the signal popping up.

Rule: data that can't be wrong tends to be less wrong

Some data actually cannot be wrong. For example, anything that would require functioning of the application. By necessisty, the data *has* to exist. Find out what this data is, and rely on it primarily for your decision making.

Data created as a dependency of a transaction is higher quality than hand wavey analytics data.

Rule: Outliers will fuck your shit up. Cap

Take a sample of each outliers. Cap your data. Measure effects at the slice of the distribution that it operates in. Averages are for chuds, look at percentiles or use rank-sum tests to test for significance.

Rule: Just because you're capping doesn't mean you shouldn't look at your outliers. Your outliers are where your outlier results come from! Look!

Plot and zoom in on the tails, look at how far it goes, take a sample of each and observe them. Actually look at your data!

Rule: Your important entities are usually the ones that live at the tail

Request success rates make a good example here. The ratio of your failed requests isn't the ratio of the value that you're dropping, because the high value requests will have out of distribution inputs, and will not fail similarly to the distribution.

Imagine your a salesman. You fail 10% of your sales. But all of your successful sales are selling potatoes, and all of your failed sales are selling 200k$ tran-zams to wealthy businessmen

Rule: distributions vary by time bucket (and pretty much every dimension)

If you look at the median amount of miles driven by someone over the entire day, it will be different than the median amount of miles driven by someone at 5-6 PM, multiplied by 24 hours to make up for the differences in time.

Rule: trust your gut over your actual data

I have made decisions even though the data was clearly against it. Just because I caught a vibe. And every time, the vibe was more right than the data. The data tends to be wrong more often than your gut!