New Member
Posts: 13
Registered: ‎06-30-2017
Kudos: 1
Solutions: 1
Accepted Solution

UNMS Reverse Proxy : HaProxy

[ Edited ]

Hello there,

 

I'm currently trying to setup UNMS in order to manage my differents Ubnt equipements.

I currently have 3 ER-X, 1 AirMax NanoStationM2 and an UAP-AC-Lite (but as I understood, I need Unifi to have a centralized mangement for the last one).

 

My network setup consist of a main hypervisor with a bunch of VMs and virtual networks. I have a dedicated VM for UNMS (v0.13.3) in there.

I have also a reverse proxy (HaProxy), that I use to redispatch http/tcp requests from the outside to my differents services, and manage my ssl certificates/SSL Termination.

Then, I have three remote sites with each a ER-X, and one with a AirMax NanoStationM2.

 

My first issue was HaProxy. I did not want to install NGINX or Apache, as I already had one reverse proxy, but there is no official documentation for it. But I managed to makes it works (almost completely).

 

First of all, my UNMS install line :

 

bash /tmp/unms_inst.sh --unattended --public-https-port 443 --http-port 80 --https-port 443

Then the working haproxy config (this is not my full config file, just the necessary stuff) :

 

###############################
### STANDARD CONFIGURATION ###
###############################
global
  log /dev/log local0 info
  log /dev/log local1 notice
  log-tag haproxy
  stats socket /var/lib/haproxy/stats level admin
  stats timeout 30s
  pidfile /var/run/haproxy/haproxy.pid
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
  maxconn 100
  spread-checks 8

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  # Default ciphers to use on SSL-enabled listening sockets.
  # https://mozilla.github.io/server-side-tls/ssl-config-generator/
  ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

defaults
  log global
  mode http
  maxconn 100
  option httplog
  option dontlognull
  option log-health-checks
  option forwardfor
  option http-server-close
  retries 3
  timeout connect       30s
  timeout client        30s
  timeout http-request  30s
  timeout server        30s

#################
### FRONTEND ###
#################
frontend public
  bind *:80
  bind *:443 ssl crt /etc/haproxy/certs/

  # Optional : Allowed sources IP for additional security
  acl allowed_ip src 10.0.0.0/24

## https://www.haproxy.com/fr/blog/websockets-load-balancing-with-haproxy/ ## UNMS Websocketrouting based on Host header acl unms_ws hdr_beg(Host) -i ws. ## UNMS Websocket routing based on websocket protocol header acl unms_connection_upgrade hdr(Connection) -i upgrade acl unms_upgrade_websocket hdr(Upgrade) -i websocket # Let's Encrypt acl acl_letsencrypt path_beg /.well-known/acme-challenge/ ## Secure headers https://blog.devcloud.hosting/securing-haproxy-and-nginx-via-http-headers-54020d460283 http-response set-header Strict-Transport-Security max-age=15768000;includeSubDomains;preload http-response set-header X-XSS-Protection: 1;mode=block http-response set-header X-Content-Type-Options: nosniff http-response set-header Referrer-Policy no-referrer-when-downgrade http-response set-header X-Frame-Options SAMEORIGIN http-response del-header X-Powered-By http-response del-header Server http-response del-header X-Apache-Server-ID ## Force HTTPS http-request set-header X-Forwarded-Port %[dst_port] http-request add-header X-Forwarded-Proto https if { ssl_fc } !acl_letsencrypt http-request redirect scheme https code 301 if !{ ssl_fc } !acl_letsencrypt # UNMS use_backend unms if unms_ws allowed_ip use_backend unms if unms_connection_upgrade unms_upgrade_websocket allowed_ip ############### ### BACKEND ### ###############
backend letsencrypt server vm-haproxy localhost:8088
# UNMS
# "ssl verify none" is mandatory as UNMS itself have use a ssl certificate
backend unms
server vm-unms 192.168.10.13 ssl verify non

 

 

I successfully managed to link my three ER-X to UNMS without problem. But I encoutered an issue while linking my AirMax NanoStationM2.

 

Looking at /var/log/unms.log I first had a lws_ssl_client_connect2 failed , then I noticed I could upgrade to 6.1.9 XM (I was in 6.1.8) and the release note had a note about UNMS agent upgrade to v0.5.2

After the upgrade the error message changed to Timed out waiting SSL

The error displayed by my HaProxy was always the same : SSL handshake failure

I tried to see what sudo udapi-bridge -k -v was telling

 

2019-03-12 10:24:50 WARN  Using deprecated option allowSelfSignedCertificate, please replace it with allowUntrustedCertificate
2019-03-12 10:24:50 INFO  udapi-bridge 0.5.2-no-git
2019-03-12 10:24:50 DEBUG ca_cert_load_all_file: directory /etc/persistent/udapi-bridge doesn't exist
2019-03-12 10:24:50 DEBUG Creating Vhost 'default' (serving disabled), 1 protocols, IPv6 off
2019-03-12 10:24:50 DEBUG Unable to load SSL Client certs file from /tmp/udapi-bridge-ca.pem.nsmhhO -- client ssl isn't going to work
2019-03-12 10:24:50 DEBUG created client ssl context for default
2019-03-12 10:24:50 INFO  unms: connecting to unms.mydomain.com:443
2019-03-12 10:24:50 ERROR INTERNALAPI not intialized
2019-03-12 10:24:50 ERROR INTERNALAPI not intialized
2019-03-12 10:24:51 DEBUG new session: INTERNALAPI-014f005
2019-03-12 10:24:51 DEBUG LLDP: Driver for eth0 is `AG7240 v3.14`
2019-03-12 10:24:51 DEBUG LLDP: Driver for eth1 is `AG7240 v3.14`
2019-03-12 10:24:51 DEBUG LLDP: Driver for wifi0 not found
2019-03-12 10:24:51 DEBUG LLDP: Driver for ath0 not found
2019-03-12 10:24:51 INFO  LLDP: Skipping interface ath0 (Wireless)
2019-03-12 10:24:51 DEBUG LLDP: Driver for br0 is `bridge`
2019-03-12 10:24:51 INFO  LLDP: Skipping interface br0 (bridge)
2019-03-12 10:24:51 DEBUG LLDP: Found new Ethernet port 1 (name: eth0)
2019-03-12 10:24:51 DEBUG LLDP: Found new Ethernet port 2 (name: eth1)
2019-03-12 10:24:51 DEBUG LLDP: Found new Ethernet port 3 (name: wifi0)
2019-03-12 10:24:51 DEBUG LLDP: Found 3 ports
2019-03-12 10:24:51 DEBUG LLDP: Sending LLDP packet on port 1 (eth0)
2019-03-12 10:24:51 DEBUG LLDP: Sending LLDP packet on port 2 (eth1)
2019-03-12 10:24:51 DEBUG LLDP: Sending LLDP packet on port 3 (wifi0)
2019-03-12 10:25:12 ERROR connection error (unms.mydom.com:443): Timed out waiting SSL
2019-03-12 10:25:12 DEBUG session_destroy: session UNMS(0x4920d8) not found
2019-03-12 10:25:12 DEBUG unms rx: name=unms-status socket=internal id=fake session=014f005 payload={
  "ws_log_time":"2019-03-12 10:25:12 ",
  "ws_log_line":"connection error (unms.mydom.com:443): Timed out waiting SSL",
  "ws_log_lvl":3,
  "unms_status_desc":"Connection Error: SSL not present on dest. port",
  "unms_status":4
}
2019-03-12 10:25:12 INFO  unms: connecting to unms.mydom.com:443
2019-03-12 10:25:12 DEBUG unms rx: name=unms-status socket=internal id=fake session=014f005 payload={
  "ws_log_time":"2019-03-12 10:25:12 ",
  "ws_log_line":"unms: connecting to unms.mydom.com:443",
  "ws_log_lvl":1
}
2019-03-12 10:25:12 DEBUG unms rx: name=unms-startPing socket=internal id=fake session=014f005 payload={
  "pingHost":"unms.mydom.com",
  "pingIntervalWhenUp":30000,
  "pingIntervalWhenDown":5000
}
2019-03-12 10:25:13 DEBUG file_reader_perform: waitpid: 0
2019-03-12 10:25:13 DEBUG ping unms.mydom.com: 42.553000
2019-03-12 10:25:34 ERROR connection error (unms.mydom.com:443): Timed out waiting SSL
2019-03-12 10:25:34 DEBUG session_destroy: session UNMS(0x492098) not found
2019-03-12 10:25:34 DEBUG unms rx: name=unms-status socket=internal id=fake session=014f005 payload={
  "ws_log_time":"2019-03-12 10:25:34 ",
  "ws_log_line":"connection error (unms.mydom.com:443): Timed out waiting SSL",
  "ws_log_lvl":3,
  "unms_status_desc":"Connection Error: SSL not present on dest. port",
  "unms_status":4
}

After searching for Unable to load SSL Client certs file from /tmp/udapi-bridge-ca.pem.nsmhhO -- client ssl isn't going to work in the forums I found the very recent post

https://community.ubnt.com/t5/UNMS-Beta/quot-DEBUG-Unable-to-load-SSL-Client-certs-file-from-tmp-uda...

Which is still without response.

 

I found posts talking about some of theses errors on ER-X, talking about MTU or low speed network. The AirMax machine is indeed at a low speed network site, so I tried to replace the FQDN part of my unms key by the IP of my UNMS machine directly (the FQDN is aiming at my HaProxy). This then worked fine because my subnetworks are linked through VPN.  I can then dismiss the low speed network idea as my HaProxy and UNMS are on the same network at the same site, my AirMax machine would take an equal time to get through HaProxy or to go directly to the UNMS machine (or at a few ms close).

 

Things are then now working, but I would like to make my setup work while using my HaProxy with the AirMax machine. Every machine centralized in UNMS should have the same setup if I want something clean and if I ever want to link a new machine not in my VPN networks.

I am then requesting help about this topic as I have reached my limits in term of technical skills.

 

I also choose to make a new topic (and not answer on the 2710488) because the setup, machines are very differents, but also because I think theses informations can be very useful to anyone trying to use HaProxy instead of NGINX/Apache, and once the stuff is entirely working, it may have its place on https://help.ubnt.com/hc/en-us/articles/115015690207-UNMS-Reverse-Proxy

 

 

I will also be pleased to give detailed logs if asked.

 

Thanks for reading.


Accepted Solutions
New Member
Posts: 13
Registered: ‎06-30-2017
Kudos: 1
Solutions: 1

Re: UNMS Reverse Proxy : HaProxy

[ Edited ]

Hello again @UBNT-Radek !

I managed to find my missing ciphers : )

 

DHE-RSA-AES128-SHA
DHE-RSA-AES256-SHA
DHE-RSA-AES128-SHA256
DHE-RSA-AES256-SHA256

One of theses four was missing, only one is required. It doesnt bother the UNMS connection to have all four, but I used only the latter DHE-RSA-AES128-SHA256 for security reason.

 

To be certain to not have cipher issue, I could suggest to use the "intermediate configuration" as generated by mozilla (https://mozilla.github.io/server-side-tls/ssl-config-generator/), but I could only recommands to tune this up, aka use the modern configuration (like mine in the main post) and add missing ciphers for old systems.

 

Idk what you think about this thread at ubnt but it could be very nice to have an exemple configuration on the official documentations. I would be glad to support it if necessary.

 

Cheers.

View solution in original post


All Replies
Ubiquiti Employee
Posts: 3,430
Registered: ‎09-08-2017
Kudos: 1320
Solutions: 260

Re: UNMS Reverse Proxy : HaProxy

Hello @albanosdes. Thank you for a very well written report. Please let me talk about the built-in proxy first. I would like to assure you, that we understand some users want to use their own custom configured proxy server, but at the same time, we need to offer a solution for those who do need completely configured, easily installed NGINX. Having the proxy server installed and configured internally allows us to generate Let's Encrypt certificate which is then compatible with all devices and contains many settings which may be difficult to replicate on a different proxy. In the upcoming release 0.14.0, UCRM will be integrated and Suspend feature enabled so the proxy configuration complexity will increase significantly.

Now back to the issue on hand. In your log, there are two major clues:
"connection error (unms.mydom.com:443): Timed out waiting SSL"
"Connection Error: SSL not present on dest. Port"

 

That means your devices are having issues when connecting through SSL to port from UNMS key, which in your case has value 443.

If you have another device which works all right on this address and port, and if you are certain that the affected device really see UNMS server on this address and port then it will be necessary to check you have support for all necessary ciphers on your proxy:
ciphers.png


Also if your UNMS server is for some reason overloaded, it can have issues finishing the SSL connections. 

UBNT_Alternate_Logo.png
UNMS Support - If you want to report an issue please use this guide.

Check out our ever-evolving Help Center for answers to many common questions!

Highlighted
New Member
Posts: 13
Registered: ‎06-30-2017
Kudos: 1
Solutions: 1

Re: UNMS Reverse Proxy : HaProxy

Hello @UBNT-Radek, thanks you for taking time to answer. I completely understand your position with NGINX (or Apache), there is absolutely no reproach. Idk if your familiar with haproxy but the configuration is easy as well. I have a dedicated backend for let's encrypt certificates, but the necessary configuration for let's encrypt to work on UNMS side would not be long to write.

Do you have additional information about " In the upcoming release 0.14.0, UCRM will be integrated and Suspend feature enabled so the proxy configuration complexity will increase significantly." ? Idk what's UCRM.

But let's get back to our subject. So yes I do have all my ER-X working fine with the UNMS server using FQDN key. I did though about the ciphers, as the AirMax NanoStationM2 is AirOS6 which is rather old now, may need older ciphers that I do not have in my haproxy.
I will look into it, thanks for the screens.

As for the overload, my VM, cpu graphs are flat as the horizon, track definitely dismissed.
New Member
Posts: 13
Registered: ‎06-30-2017
Kudos: 1
Solutions: 1

Re: UNMS Reverse Proxy : HaProxy

[ Edited ]

Hello again @UBNT-Radek !

I managed to find my missing ciphers : )

 

DHE-RSA-AES128-SHA
DHE-RSA-AES256-SHA
DHE-RSA-AES128-SHA256
DHE-RSA-AES256-SHA256

One of theses four was missing, only one is required. It doesnt bother the UNMS connection to have all four, but I used only the latter DHE-RSA-AES128-SHA256 for security reason.

 

To be certain to not have cipher issue, I could suggest to use the "intermediate configuration" as generated by mozilla (https://mozilla.github.io/server-side-tls/ssl-config-generator/), but I could only recommands to tune this up, aka use the modern configuration (like mine in the main post) and add missing ciphers for old systems.

 

Idk what you think about this thread at ubnt but it could be very nice to have an exemple configuration on the official documentations. I would be glad to support it if necessary.

 

Cheers.