Observe &&
Understand
RabbitMQ

INTRO

RabbitMQ Core Engineer

Gerhard Lazu

gerhardlazu   gerhard

RabbitMQ for Kubernetes
Product Manager

Michal Kuratczyk

MichalKuratczyk   mkuratczyk

Slides on your device

MANAGEMENT LIMITATIONS

RABBITMQ MANAGEMENT LIMITATIONS

New metrics system in RabbitMQ 3.8

Understand like never before

More CLI power to you

What happens next?

Management not loading

Management API response

rabbitmq-plugins enable rabbitmq_prometheus

New built-in plugin

Enable built-in plugin

Faster responses

New RabbitMQ-Overview

Spot queue imbalances

Rebalance all queues

rabbitmq-queues rebalance all

Evenly spread queues

See node links

Data sent to peer

Data in port driver buffer

Perf-Test dashboard - v1

Problem

Info

Snapshot

FIRST 3 TAKE AWAYS

Upgrade to RabbitMQ 3.8

Discover all the new features

😱 Metricless Management 🤯

Enable & visualise new metrics

Start using new metrics

First step towards better conversations

Share metrics when asking for help

Share snapshots

Help us help you!

HOW TO UNDERSTAND

RabbitMQ Management limitations

New metrics system in RabbitMQ 3.8

UNDERSTAND LIKE NEVER BEFORE

More CLI power to you

What happens next?

What happens in the
Erlang Distribution?

What happens in inet_dist?

Mirrored Classic Queue
vs Quorum Queue

Mirrored Classic Queue

• Master +2 Mirrors
• 100 msg/s @ 10KiB
• 1 Publisher → 1 Consumer
• 1 MiB / s constant message flow

MCQ - Management

MCQ - Erlang Distribution

MCQ - Data sent to peer

1 Mebibyte of msg bodies

Mirrored Classic Queue: Master +2 Slaves

Erlang Distribution Link	Traffic
Master 0 → Mirror 1	2MB / s
Mirror 1 → Mirror 2	1MB / s
Mirror 2 → Master 0	1MB / s
Master 0 → Mirror 2	1MB / s

Why does
Erlang Distribution traffic
matter?

Erlang Distribution traffic

470 MB/s link max @ 10KB in OTP 22

inet_tcp link limit @ 10KB

Erlang/OTP PR 2293

Quorum Queue

• Leader +2 Followers
• 100 msg/s @ 10KiB
• 1 Publisher → 1 Consumer
• 1 MiB / s constant message flow

QQ - Inter-node traffic

Quorum Queue 2.5x less pressure

on Erlang Distribution than Mirrored Classic Queue QQ 2.5x less pressure

MCQ Link	MCQ Traffic	QQ Link	QQ Traffic
M0 → M1	2MB / s	L0 → F1	1MB / s
M1 → M2	1MB / s	L0 → F2	1MB / s
M2 → M0	1MB / s
M0 → M2	1MB / s

Help me understand
Quorum Queue internals

Understand QQ Raft

RabbitMQ-Quorum-Queues-Raft

Log entries committed / s

Log entry commit latency

Raft members with >5k entries

Hey Rabbit!
Where is my memory?

Where is my memory?

Erlang-Memory-Allocators

Erlang-Memory-Allocators

binary_alloc

eheap_alloc

ets_alloc

All memory allocators

Discover other dashboards

Make dashboards

MORE CLI POWER TO YOU

RabbitMQ Management limitations

New metrics system in RabbitMQ 3.8

Understand like never before

MORE CLI POWER TO YOU

What happens next?

Remember htop?

Observe Erlang VM

RabbitMQ v3.8.0

rabbitmq-diagnostics observer

observer_cli home

observer_cli home reductions

observer_cli inspect process

observer_cli ETS

See RabbitMQ events

RabbitMQ v3.8.0

rabbitmq-diagnostics consume_event_stream

consume_event_stream

rabbitmq-diagnostics help

WHAT HAPPENS NEXT?

RabbitMQ Management limitations

New metrics system in RabbitMQ 3.8

Understand like never before

More CLI power to you

WHAT HAPPENS NEXT?

Expose more metrics

and address high cardinality

Build new dashboards

Erlang-Microstate-Accounting

See logs together with metrics

I meant Events

Try OpenTelemetry

LAST 3 TAKE AWAYS

Have different conversations

Observe, snapshot & share

Aim to understand and improve

Understand and improve

This is an infinite game

Ask questions in the webinar

12th of December, 2019

@RabbitMQ