RustDesk Remote Desktop Server: Deploy Your Own Server in Docker

Self-hosting a private RustDesk infrastructure allows you to establish secure, high-performance remote desktop connections without relying on third-party cloud coordinators. This deployment utilizes two primary components: hbbs (RustDesk ID/Rendezvous Server) which coordinates connection handshakes, and hbbr (RustDesk Relay Server) which proxies session traffic when direct peer-to-peer (P2P) connections are blocked by restrictive firewalls.

Network Requirements and Port Mapping

To establish communication between remote clients and your self-hosted server, the following ports must be exposed on your VPS and allowed through any host or cloud firewalls:

Firewall Configuration (UFW)

Execute the following commands to configure UFW on a standard Ubuntu or Debian VPS:

sudo ufw allow 21115/tcp
sudo ufw allow 21116/tcp
sudo ufw allow 21116/udp
sudo ufw allow 21117/tcp
sudo ufw allow 21118/tcp
sudo ufw allow 21119/tcp
sudo ufw reload

Docker Compose Configuration

Create a dedicated directory for the deployment and configure a docker-compose.yml file. This setup isolates the data directory to persist the generated cryptographic keypair.

1. Create the Directory Structure

mkdir -p /opt/rustdesk-server/data
cd /opt/rustdesk-server

2. Define the docker-compose.yml File

Choose one of the following two networking approaches depending on your VPS network configuration.

Option A: Bridge Networking (Recommended for Multi-service VPS)

This mode maps container ports to specific host ports, allowing you to run other applications on the same host. Replace rustdesk.example.com with your VPS public IP address or FQDN.

version: '3.8'

services:
  hbbs:
    container_name: rustdesk-hbbs
    image: rustdesk/rustdesk-server:latest
    command: hbbs -r rustdesk.example.com:21117 -k _
    volumes:
      - ./data:/root
    ports:
      - "21115:21115"
      - "21116:21116"
      - "21116:21116/udp"
      - "21118:21118"
    restart: unless-stopped
    depends_on:
      - hbbr

  hbbr:
    container_name: rustdesk-hbbr
    image: rustdesk/rustdesk-server:latest
    command: hbbr -k _
    volumes:
      - ./data:/root
    ports:
      - "21117:21117"
      - "21119:21119"
    restart: unless-stopped

Option B: Host Networking (Optimized for Throughput)

Host networking removes the network isolation between the container and the host, reducing packet overhead and potential UDP packet loss.

version: '3.8'

services:
  hbbs:
    container_name: rustdesk-hbbs
    image: rustdesk/rustdesk-server:latest
    command: hbbs -r rustdesk.example.com:21117 -k _
    volumes:
      - ./data:/root
    network_mode: host
    restart: unless-stopped
    depends_on:
      - hbbr

  hbbr:
    container_name: rustdesk-hbbr
    image: rustdesk/rustdesk-server:latest
    command: hbbr -k _
    volumes:
      - ./data:/root
    network_mode: host
    restart: unless-stopped

Cryptographic Security and Mandatory Key Authentication

The inclusion of the -k _ command-line argument enforces mandatory key authentication. Without this flag, your server acts as an open relay, allowing unauthorized users to proxy connections through your bandwidth.

When the container starts for the first time, hbbs automatically generates a public/private keypair inside the ./data directory: * id_ed25519: The private key used by the server to sign responses. * id_ed25519.pub: The public key required by clients to establish authentication.

Retrieve the Public Key

To retrieve the public key needed for client configurations, display the contents of the generated .pub file:

cat /opt/rustdesk-server/data/id_ed25519.pub

Copy the printed alphanumeric string exactly. It will look similar to this: M21kYjFzZDJhM2RmYWQz...

Key Rotation

To rotate keys, stop the containers, delete the old key pair, and restart the services:

docker compose down
rm -f /opt/rustdesk-server/data/id_ed25519*
docker compose up -d

Note: Key rotation will invalidate client access until the new public key is distributed to all devices.

Client Configuration

To route remote sessions through your private infrastructure, both the controlling device and the controlled device must be configured to target your VPS.

Manual Settings Profile

1. Open the RustDesk client.
2. Click the three-dot menu next to your ID and navigate to Settings > Network.
3. Enable Unlock network settings.
4. Configure the following parameters:
5. ID Server: rustdesk.example.com (or your VPS IP address)
6. Relay Server: rustdesk.example.com:21117
7. Key: [PASTE_CONTENTS_OF_ID_ED25519.PUB]
8. Click Apply. The network indicator status at the bottom of the client window should shift to Ready.

Automated Deployment via Executable Renaming (Windows Client)

You can distribute a pre-configured client to end-users on Windows by renaming the installer executable. RustDesk parses the filename upon launch to import network parameters.

Rename the downloaded .exe installer using the following pattern: rustdesk-host=<host>,key=<key>.exe

Example:

rustdesk-host=rustdesk.example.com,key=M21kYjFzZDJhM2RmYWQz....exe

[!WARNING] If your public key contains characters that are illegal in filesystem names (such as /), you must URL-encode them within the filename, or distribute a customized RustDesk.toml configuration file instead.

Service Management and Verification

Initialize the Containers

Run the following command within the directory containing your docker-compose.yml to launch the services in detached mode:

docker compose up -d

Inspect Container Logs

Verify that the services are listening correctly and generating keys:

docker compose logs hbbs
docker compose logs hbbr

You should see log entries confirming that hbbs has loaded the keys and is listening on ports 21115 and 21116.

Enable Automatic Boot Persistence

Docker's restart: unless-stopped directive ensures the containers launch when the daemon starts. Ensure the Docker daemon itself is configured to boot on system startup:

sudo systemctl enable docker