How to Self-Host RustDesk with Docker and NGINX Reverse Proxy

In this tutorial, we’ll walk through deploying your own RustDesk server using Docker, with a secure NGINX reverse proxy on another VM. This setup mimics tools like Bomgar or TeamViewer for remote support, using your own domain: remote.yourdomain.com.

🔧 Requirements

• A Docker-compatible VM (e.g., Debian/Ubuntu) for hosting RustDesk
• A separate VM running NGINX with Certbot (your reverse proxy)
• A domain/subdomain pointing to your reverse proxy (e.g., remote.yourdomain.com)

📦 Step 1: Deploy RustDesk Server via Docker

1. SSH into your Docker host VM.

2. Create a project folder for RustDesk:

   mkdir -p ~/rustdesk && cd ~/rustdesk

3. Create a Docker Compose file:

   nano docker-compose.yml

   Paste the following:

   version: '3.8'
   
   services:
     rustdesk-hbbs:
       image: rustdesk/rustdesk-server:latest
       container_name: rustdesk-hbbs
       restart: unless-stopped
       command: hbbs -r rustdesk-hbbr:21117
       ports:
         - "21115:21115"
         - "21116:21116"
         - "21116:21116/udp"
       volumes:
         - ./data:/root
       environment:
         - ENCRYPTED_ONLY=1
   
     rustdesk-hbbr:
       image: rustdesk/rustdesk-server:latest
       container_name: rustdesk-hbbr
       restart: unless-stopped
       command: hbbr
       ports:
         - "21117:21117"
       volumes:
         - ./data:/root
       environment:
         - ENCRYPTED_ONLY=1
   

4. Start the containers:

   docker compose up -d

🌐 Step 2: Configure Your Domain

Log into your DNS provider and point an A record to your NGINX reverse proxy VM:

remote.yourdomain.com → [Public IP of reverse proxy VM]

🚪 Step 3: Set Up NGINX Reverse Proxy with TLS

1. SSH into your NGINX VM.

2. Install NGINX and Certbot if not already done:

   sudo apt update
   sudo apt install nginx certbot python3-certbot-nginx

3. Create a new NGINX site config:

   sudo nano /etc/nginx/sites-available/rustdesk

   Paste:

   server {
       listen 80;
       server_name remote.yourdomain.com;
   
       location / {
           proxy_pass http://[RUSTDESK-INTERNAL-IP]:21115;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       }
   }

   Replace [RUSTDESK-INTERNAL-IP] with the private IP of your Docker VM.

4. Enable the site:

   sudo ln -s /etc/nginx/sites-available/rustdesk /etc/nginx/sites-enabled/
   sudo nginx -t
   sudo systemctl reload nginx

5. Request a Let’s Encrypt SSL certificate:

   sudo certbot --nginx -d remote.yourdomain.com

   Follow the prompts and choose the redirect option for HTTPS.

🔐 Step 4: Secure and Verify Your RustDesk Server

1. After the containers start, get your RustDesk public key (used for secure connections):

   cat ~/rustdesk/data/id_ed25519.pub

2. Optionally embed this key into your RustDesk clients for extra security.
3. Visit https://remote.yourdomain.com — you should see a 404 or blank response (this is expected — the API is not for browsers).

💻 Step 5: Connect with RustDesk Client

1. Download the RustDesk client from rustdesk.com.
2. Open the client and go to Settings > ID/Relay Server.
3. Enter:
   • Server: remote.yourdomain.com
   • Relay Port: 21115
   • Enable UDP
4. On another machine, enter the 9-digit code and connect.

✅ Optional: Build Custom Client for Zero-Config Support

To offer a branded, no-setup client like Bomgar:

1. Use RustDesk’s client customization tool.
2. Embed your server, encryption key, and branding.
3. Distribute a prebuilt .exe or .pkg to clients for instant support with no configuration needed.

🎯 Summary

• ✅ Fully self-hosted RustDesk server
• 🔐 TLS-secured with NGINX reverse proxy
• 🧰 Remote support over your own branded domain
• 💻 Optional zero-config portable clients for MSP-style workflow

With this setup, you're no longer dependent on 3rd-party cloud relays. You're in full control of your remote support infrastructure, ready to serve clients securely and reliably.