Metrics

X-Ray your app

Morten Siebuhr

@msiebuhr / Github

Questions!

METRICS?

Grab a dictionary

№1

Metric: Greek metrikē, from feminine of metrikos in meter, by measure, from metron measure — more at measure

Miriam Webster

№2

Metric: A standard of measurement

Miriam Webster

№3

No metric exists that can be applied directly to happiness

Scientific Monthly / Miriam Webster

Bullshit!

In the beginning

Kennedy

Let´s go to the moon...

Today

Boeing 787

~ 500 GB / flight

Meanwhile in our little world

Typically

• Uptime
• Number of served requests
• Apache threads
• CPU load
• ...

We already have that

• Munin
• CollectD
• Nagios
• ...

Stuff slips through the cracks

Your
stuff slips though the cracks

Costs $$$

(Did for us! And still does...)

Built on librrd

• Complex API
• Can´t combine graphs dynamically

Quick and Dirty

log.info()

printf()

+

grep

The Good

• Pin-pointing exact problems

The Bad

• Inefficient data format
• Difficult to correlate events

The Ugly

• Filter important/unimportant
• Difficult to see long-term trends

Process all webserver logs:

15+ min

SLOOOOOOW

Logging still needed

We have no idea why these filesystems began to act up.

But we do know when and where to look in the logs.

KISS

StatsD

(Originally built by Etsy, inspired by Flickr)

Basic idea:

Send UDP with identifier, value & type

Collect & generate regular statistics

Store, Graph and act

Basic setup:

Your application + some client

StatsD

Graphite / Hosted / Other

Easy metrics

client = new StatsDClient({server: 'stats.your.tld'}) start_time = Date.now() client.increment('rpc.connection_opened') … client.increment('rpc.response_time', start_time)

Client in a tweet

$fp = fsockopen("udp://127.0.0.1",8125); fwrite($fp,"request.time:1234|ms"); fclose($fp)

Send over UDP

FAST

Small overhead

Doesn`t add to failure domain

StatsD

• Node.js
• No dependencies
• Pluggable backends
• Types: counters/gauges/sets/timings

(~2500 LOC, including test-suite & packaging)

Types

Counters

Gives you absolute reported value + normalized "per second"-numbers

Types

Gauges

Stay the same until you tell it otherwise

Types

Sets

Outputs number of different values seen

Types

Timings

Min/mean/ang/std.dev/max + 90th percentile for all reported values

Pluggable backends

amqp datadog ganglia Graphite

librato OpenTSDB socket.i0 statsd zabbix

...

Node.js

Don´t worry - clones in pretty much all languages.

Ends up in Graphite

Play with it

Correlate odd stuff

Objective:

Challenge your assumptions

Three things happen

№ 1

You will be surprised

OMG! WTF?

We have a hit-rate <50%

Assumption:

Varnish outputs more data than it ingests.

Data says:

Varnish outputs less data than it ingests!

№2

Bug / Configuration error

(Go fix it!)

№3

Make it a KPI

(Put it on a dashboard, send e-mails, rotating lights, klaxons, &c!)

Rinse, repeat

Do It Yourself

• Write a statsd-client (it´s embarassingly easy!)
• Set up virtual machine / account
• Push data over UDP.
• Verify your assumptions about how your stats should look like

Took me a few hours

(Mostly because Graphite is a PITA to set up)

Boss day 1:

Don`t waste too much time on it!

Boss day 90:

How about 1000x data?

TL;DL

• Distributed printf()´s
• Ultra-simple logging
• Stay on top of things
• Archeology
• Forecasting

Questions / Discussion

Thanks

Images

Haribo, NASA, One.com