Create an extra config file for your vhost in /etc/nginx/sites-available/btcpayserver and create a symlink for this file at /etc/nginx/sites-enabled/btcpayserver

The contents of this vhost file should look like this:

server {
	listen 80;

	root /var/www/html;
	index index.html index.htm index.nginx-debian.html;

	# Put your domain name here
	server_name btcpay.domain.com;

	# Needed for Let's Encrypt verification
	location ~ /.well-known {
		allow all;
	}

	# Force HTTP to HTTPS
	location / {
		return 301 https://$http_host$request_uri;
	}
}

server {
	listen 443 ssl http2;

	ssl on;

	# SSL certificate by Let's Encrypt in this Nginx (not using Let's Encyrpt that came with BTCPay Server Docker)
	ssl_certificate      /etc/letsencrypt/live/btcpay.domain.com/fullchain.pem;
	ssl_certificate_key  /etc/letsencrypt/live/btcpay.domain.com/privkey.pem;

	root /var/www/html;
	index index.html index.htm index.nginx-debian.html;

	# Put your domain name here
	server_name btcpay.domain.com;

	# Route everything to the real BTCPay server
	location / {
		# URL of BTCPay Server (i.e. a Docker installation with REVERSEPROXY_HTTP_PORT set to 10080)
		proxy_pass http://127.0.0.1:10080;

		proxy_set_header Host $http_host;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

		# For websockets (used by Ledger hardware wallets)
		proxy_set_header Upgrade $http_upgrade;
	}

	# Needed for Let's Encrypt verification
	location ~ /.well-known {
		allow all;
	}
}

Also, put the following in your main Nginx config file at /etc/nginx/nginx.conf:

http {

	# ... # Existing stuff

	# Needed to allow very long URLs to prevent issues while signing PSBTs
	server_names_hash_bucket_size 128;
	proxy_buffer_size          128k;
	proxy_buffers              4 256k;
	proxy_busy_buffers_size    256k;
	client_header_buffer_size 500k;
	large_client_header_buffers 4 500k;
	http2_max_field_size       500k;
	http2_max_header_size      500k;

	# Needed websocket support (used by Ledger hardware wallets)
	map $http_upgrade $connection_upgrade {
    	default upgrade;
    	''      close;
	}

}

Now test your Nginx config with service nginx configtest and reload the config with service nginx reload.

Then, you need to make sure that BTCPayServer does not try to handle HTTPS on its side, you can do this by disabling it on your BTCPayServer instance.

BTCPAYGEN_EXCLUDE_FRAGMENTS="$BTCPAYGEN_EXCLUDE_FRAGMENTS;nginx-https"
. btcpay-setup.sh -i

Notice: If your BTCPay Server install has more than one domain (for example WOOCOMMERCE_HOST or BTCPAY_ADDITIONAL_HOSTS) you will need to modify your config for each domain name. The example above only covers 1 domain name called btcpay.domain.com.