kshark Documentation

A powerful command-line diagnostic tool for Apache Kafka and connector target connectivity.

kshark acts like a network sniffer for Kafka, providing comprehensive health checks of your entire client-to-broker communication path – and also validates connectivity from Kafka Connect workers to their source and sink database targets (MongoDB, PostgreSQL, DB2).

Document Description
Architecture System architecture, components, and design patterns
Features Complete feature list and configuration reference
Deployment Docker, Kubernetes, and CI/CD deployment guides
Security Security audit, OWASP analysis, and hardening
Release Notes Release history and upgrade notes
Connector Probe Probing MongoDB, PostgreSQL, and DB2 targets
Contributing How to contribute to kshark
Changelog Detailed change log

Layered Diagnostics

kshark systematically tests every layer of connectivity:

L3 (Network)      DNS resolution and latency
L4 (Transport)    TCP connection establishment
L5-6 (Security)   TLS handshake, certificate validation, expiry monitoring
L7 (Application)  Kafka protocol, SASL auth, metadata, produce/consume
L7 (HTTP)         Schema Registry and REST Proxy connectivity
Diagnostics       Traceroute, MTU discovery, network path analysis

Highlights (RELEASE-1a)

  • Signal handling – SIGINT/SIGTERM gracefully cancel an in-progress scan
  • Environment variable expansion – use ${VAR} in properties files
  • File permission warnings – warns when properties files are group/other readable
  • Fuzz testing – 4 fuzz targets covering security-critical parsers
  • 478 test cases with 47.8% total coverage
  • Control flow refactor – runScan() replaces goto endScan, all phases guarded by ctx.Done()

Quick Start 

# Download and run
./kshark -props client.properties -topic test-topic

# With Confluent Cloud preset
./kshark --preset cc-plain -props client.properties

# Non-interactive (CI/CD)
./kshark -props client.properties -y -timeout 120s

# Connector probe
./kshark --connect-url http://connect:8083 --connector-name my-sink

kshark-core is also an example of how agents can use our tools to debug Kafka clusters: the JSON reports are built for scripts and AI agents to read.

  • KafClaw: agent runtime in Go that connects AI agents through Kafka topics. It ships a kafclaw kshark command for Kafka diagnostics.
  • KafScale: Kafka-compatible streaming that keeps the log in S3.
  • kaf-mirror: Kafka-to-Kafka replication with a web dashboard, REST API and CLI.

Source code: github.com/scalytics/kshark-core. All Scalytics open source projects: scalytics.io/open-source.

Built with care for the Kafka community by Scalytics.