How to Self-Host WireGuard VPN on a VPS with Docker Compose

WireGuard is a modern, high-performance VPN protocol that runs inside the Linux kernel space. When compared to legacy protocols like OpenVPN or IPSec, WireGuard is significantly faster, more secure, and less resource-intensive.

This guide details how to self-host a WireGuard VPN server on a Linux VPS using Docker Compose. We will cover kernel module requirements, host network configuration, container setup, and iptables routing.

1. Prerequisites and Linux Kernel Requirements

Because WireGuard is designed to run directly in kernel space, it requires kernel-level support on the host system. While modern Linux distributions (Ubuntu 20.04+, Debian 11+, CentOS Stream 9+) include the WireGuard kernel module by default, older kernels or specialized virtualization environments (like OpenVZ or older LXC templates) might not.

Verify Kernel Support

To check your Linux kernel version, run:

uname -r

WireGuard is merged into the mainline Linux kernel as of version 5.6. If your kernel is older than 5.6, you must install the module out-of-tree or run a userspace implementation (like wireguard-go), which carries a performance penalty.

Load the Kernel Module

Verify if the module is loaded or available:

sudo modprobe wireguard
lsmod | grep wireguard

If the output shows the wireguard module, your host is ready. If not, install the headers and module for your distribution: - Ubuntu/Debian: sudo apt update && sudo apt install -y linux-headers-$(uname -r) wireguard - RHEL/Rocky Linux: sudo dnf install -y epel-release elrepo-release && sudo dnf install -y kmod-wireguard wireguard-tools

2. Host Network Configuration

To allow WireGuard clients to route their internet traffic through the VPS, the host operating system must act as a router. This requires enabling IPv4 packet forwarding.

Enable IPv4 Packet Forwarding

Temporarily enable forwarding to test configuration:

sudo sysctl -w net.ipv4.ip_forward=1

To make packet forwarding permanent across system reboots, edit /etc/sysctl.conf or create /etc/sysctl.d/99-wireguard.conf and add the following line:

net.ipv4.ip_forward=1

Apply the changes:

sudo sysctl --system

3. Firewall and Port Forwarding

WireGuard operates exclusively over the User Datagram Protocol (UDP). The default port for WireGuard is 51820. You must configure your host firewall (such as UFW, firewalld, or raw iptables) to allow incoming UDP traffic on this port.

Using UFW (Uncomplicated Firewall)

If you run UFW on Ubuntu/Debian:

sudo ufw allow 51820/udp
sudo ufw reload

Using Firewalld

On Rocky/AlmaLinux/CentOS:

sudo firewall-cmd --add-port=51820/udp --permanent
sudo firewall-cmd --reload

4. Docker Compose Deployment

We will use the widely trusted and maintained linuxserver/wireguard Docker image. This container manages the creation of peers, generates QR codes for mobile setup, and dynamically manages internal routing rules.

Directory Structure

Create a dedicated project directory:

mkdir -p ~/wireguard && cd ~/wireguard

Writing docker-compose.yml

Create a docker-compose.yml file with the following configuration:

version: '3.8'

services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
      - SERVERURL=vps_public_ip_or_domain # Replace with your VPS public IP or DDNS
      - SERVERPORT=51820
      - PEERS=3 # Number of client configuration files to generate
      - PEERDNS=1.1.1.1 # DNS server used by clients (Cloudflare DNS)
      - INTERNAL_SUBNET=10.13.13.0/24 # WireGuard internal network
      - ALLOWEDIPS=0.0.0.0/0 # Route all traffic through the VPN
      - LOG_CONFS=true
    volumes:
      - ./config:/config
      - /lib/modules:/lib/modules:ro # Allows mounting host modules if wireguard needs compilation/loading
    ports:
      - 51820:51820/udp
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped

Key Parameters Explained

• cap_add: - NET_ADMIN - SYS_MODULE: Grants the container permission to modify network interfaces (create wg0) and load kernel modules.
• volumes: - /lib/modules:/lib/modules:ro: Mounts host kernel modules into the container so that it can interact with the host's wireguard kernel driver.
• sysctls: - net.ipv4.conf.all.src_valid_mark=1: Necessary for routing table marking and to prevent routing loops within Docker's bridge network.
• PEERS: The script inside the container will automatically create configuration directories under /config/peer_<name_or_number>.

5. Understanding Routing and iptables

Once a client connects, the container needs to route the client's traffic out to the internet via the host's physical network interface. The linuxserver/wireguard container automates this by executing iptables rules when the wg0 interface is initialized.

How NAT (Network Address Translation) is Handled

When the container starts, it creates a virtual interface named wg0. In order for packets coming from the client subnet (10.13.13.0/24) to reach the outer internet, the source IP addresses must be rewritten to match the container's output interface IP.

Behind the scenes, the entrypoint script runs:

# PostUp rules
iptables -A FORWARD -i wg0 -j ACCEPT
iptables -A FORWARD -o wg0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

And upon teardown:

# PostDown rules
iptables -D FORWARD -i wg0 -j ACCEPT
iptables -D FORWARD -o wg0 -j ACCEPT
iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

Note: If your container's default output network interface is not eth0, the script automatically detects the active interface (often eth0 or wlan0 inside the container namespace).

6. Starting the VPN and Managing Peers

Deploy the Docker Compose stack:

docker compose up -d

Verify that the service is running and inspect the logs:

docker compose logs -f wireguard

Retrieving Client Configurations

The configurations and QR codes are generated dynamically in the ./config directory: - To find raw configuration files (.conf): Inspect the files under ./config/peer_1/peer_1.conf. - To scan the QR code via terminal (useful for mobile clients): bash docker compose logs wireguard | grep -A 15 "peer_1" Or read the PNG image generated directly inside the respective peer configuration subdirectory.

7. Troubleshooting and Verification

If you can connect but have no internet access, check the following points:

1. IP Forwarding Disabled on Host: Ensure cat /proc/sys/net/ipv4/ip_forward returns 1.
2. Firewall Blocking UDP: Double-check your VPS hosting provider's cloud firewall (e.g., AWS Security Groups, DigitalOcean Firewalls) to ensure port 51820 UDP is explicitly opened to the public internet.
3. Docker Bridge Route conflicts: If your local network uses the same subnet as the INTERNAL_SUBNET (e.g., 10.13.13.0/24), choose a different subnet like 10.252.1.0/24 in docker-compose.yml.