k6 with Grafana and Prometheus

The problem with CLI-only

Don’t get me wrong, k6’s terminal output is clean. But it’s gone the second you close the terminal. I wanted to compare runs, see response times change over the duration of a test, and spot if things were getting worse over weeks. Basically I wanted dashboards. And since Grafana Labs literally makes k6, the path was obvious.

The stack

Prometheus

Already had Docker Compose running a bunch of stuff, so Prometheus was just another service:

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'

volumes:
  prometheus_data:

And the Prometheus config:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'k6'
    static_configs:
      - targets: ['host.docker.internal:5656']

Scrape every 5 seconds, keep data for 30 days.

Getting k6 to talk to Prometheus

This tripped me up for a bit. There’s a few ways to do it but the cleanest one is using the experimental Prometheus remote write output:

K6_PROMETHEUS_RW_SERVER_URL=http://localhost:9090/api/v1/write \
k6 run --out experimental-prometheus-rw script.js

That’s it. k6 pushes metrics directly to Prometheus. No extra exporters, no sidecars.

Updated command:

command:
  - '--config.file=/etc/prometheus/prometheus.yml'
  - '--storage.tsdb.retention.time=30d'
  - '--web.enable-remote-write-receiver'

Grafana

I’d used Grafana briefly before for some server monitoring so it wasn’t completely new. Adding it to the compose:

grafana:
  image: grafana/grafana:latest
  container_name: grafana
  restart: unless-stopped
  volumes:
    - grafana_data:/var/lib/grafana
  ports:
    - "3000:3000"
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}

Point it at Prometheus as a data source, HTTP URL http://prometheus:9090, done.

Dashboard

Started with the official k6 dashboard (ID 18030 from grafana.com) and tweaked it. What I’m looking at:

• Request rate - requests per second
• Response time percentiles - p50, p90, p95, p99 over time
• Error rate - percentage of failed requests
• VU count - active virtual users at any point
• Response time per endpoint - because not all routes are the same

Watching the p95 climb as VUs increase is something else. Like a heart rate monitor for your API.

A proper test

With all of this in place I wrote something more realistic:

import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

const errorRate = new Rate('errors');
const voteTrend = new Trend('vote_response_time');

export const options = {
  scenarios: {
    readers: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '2m', target: 30 },
        { duration: '5m', target: 30 },
        { duration: '2m', target: 0 },
      ],
      exec: 'readVotes',
    },
    voters: {
      executor: 'constant-arrival-rate',
      rate: 5,
      timeUnit: '1s',
      duration: '9m',
      preAllocatedVUs: 10,
      maxVUs: 20,
      exec: 'castVote',
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<600'],
    errors: ['rate<0.05'],
  },
};

export function readVotes() {
  const articleId = Math.floor(Math.random() * 6) + 1;
  const res = http.get(`https://api.patwos.dev/api/v1/votes/${articleId}`);

  check(res, { 'read status 200': (r) => r.status === 200 });
  errorRate.add(res.status !== 200);
  sleep(1);
}

export function castVote() {
  const params = {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${__ENV.TEST_TOKEN}`,
    },
  };

  const payload = JSON.stringify({
    article_id: Math.floor(Math.random() * 6) + 1,
    vote_type: Math.random() > 0.5 ? 'like' : 'dislike',
  });

  const res = http.post('https://api.patwos.dev/api/v1/votes', payload, params);
  voteTrend.add(res.timings.duration);
  errorRate.add(res.status !== 200 && res.status !== 201);
}

Two scenarios running at the same time. Readers fetching vote counts, voters submitting votes. The constant-arrival-rate executor is great because it keeps a fixed request rate no matter how slow responses get.

What I found

Without the dashboard I would’ve missed the rate limiter thing entirely. In the CLI it just showed “some requests failed.” In Grafana I could see the exact second the error rate spiked and match it with the request rate crossing 100 req/s. Obvious in hindsight, but you need the graphs to connect those dots fast.

Full compose

For anyone curious, here’s the observability bit of my compose file:

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-remote-write-receiver'
    networks:
      - default

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
    depends_on:
      - prometheus
    networks:
      - default

volumes:
  prometheus_data:
  grafana_data:

What’s next

Want to hook k6 into CI/CD. Quick smoke test on every deploy, 10 VUs for 30 seconds, just to catch regressions. k6 thresholds already exit non-zero on failure so plugging it into GitHub Actions should be easy enough.

Also curious about k6 browser testing for the frontend but that’s for another time.