New Member
Posts: 40
Registered: ‎08-26-2018
Kudos: 2
Solutions: 4

LetsEncrypt fails

Hi,

 

I'm trying to enable LetsEncrypt on a new UCRM server and LetsEncrypt can not retrive a certificate.

 

I may be way off here but, I think what is happening is that when certbot tries to get a certificate it is expecting to use port 80 and I think the UCRM webserver is redirecting the traffic to port 443 and stoping LetsEncrypt form getting the responce it expects.

 

 

 

Sanitized terminal output follows.

 

root@ucrm:/home/ucrm/data/ucrm/ucrm/data/webroot# sudo letsencrypt certonly --webroot -w /home/ucrm/data/ucrm/ucrm/data/webroot -d ucrm.example.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Plugins selected: Authenticator webroot, Installer None

Obtaining a new certificate

Performing the following challenges:

http-01 challenge for ucrm.example.com

Using the webroot path /home/ucrm/data/ucrm/ucrm/data/webroot for all unmatched domains.

Waiting for verification...

Cleaning up challenges

Failed authorization procedure. ucrm.example.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://ucrm.example.com/.well-known/acme-challenge/GI8w3G1I2pwzO1f6lfKWr55nURuwRbi0H5u3g57ZZMI: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"UTF-8\"/>\n        <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n"

 

IMPORTANT NOTES:

 - The following errors were reported by the server:

 

   Domain: ucrm.example.com

   Type:   unauthorized

   Detail: Invalid response from

   http://ucrm.example.com/.well-known/acme-challenge/GI8w3G1I2pwzO1f6lfKWr55nURuwRbi0H5u3g57ZZMI:

   "<!DOCTYPE html>\n<html>\n    <head>\n        <meta

   charset=\"UTF-8\"/>\n        <meta http-equiv=\"x-ua-compatible\"

   content=\"ie=edge\">\n"

 

   To fix these errors, please make sure that your domain name was

   entered correctly and the DNS A/AAAA record(s) for that domain

   contain(s) the right IP address.

root@ucrm:/home/ucrm/data/ucrm/ucrm/data/webroot# 

Ubiquiti Employee
Posts: 1,476
Registered: ‎03-21-2016
Kudos: 244
Solutions: 161

Re: LetsEncrypt fails

Hello @User789876, yes UCRM does redirect plain http requests from port 80 to https on port 443, however that should be absolutely fine for Let's Encrypt verification and we've never had any problem with this before.

I assume you just changed your domain to ucrm.example.com in the logs for the post, but if you did not, please make sure you have correct domain set up in System -> Settings -> Application -> "Server domain name".

Another thing to check is if port 80 actually targets UCRM on your system, what do you see when you try to go to the challenge file URL in your browser?

New Member
Posts: 13
Registered: ‎09-22-2016

Re: LetsEncrypt fails

I'm seeing the same thing.  It was working just over 60 days ago when it last replaced the cert but now for some reason, it gets the wrong URL and tries to do the query against the /login under my domain instead of in the .well-known folder.

Veteran Member
Posts: 4,739
Registered: ‎05-19-2009
Kudos: 902
Solutions: 27

Re: LetsEncrypt fails

are you guys on 2.15.1??

 

its working fine for us but you guys are giving me a bit of a scare

New Member
Posts: 13
Registered: ‎09-22-2016

Re: LetsEncrypt fails

Confirmed, I am on 2.15.1.
Ubiquiti Employee
Posts: 4,170
Registered: ‎12-10-2015
Kudos: 1477
Solutions: 315

Re: LetsEncrypt fails

We will check it out, thanks for reporting.
Ubiquiti Employee
Posts: 1,476
Registered: ‎03-21-2016
Kudos: 244
Solutions: 161

Re: LetsEncrypt fails

Hello @snowtr, the Let's Encrypt integration is working properly in our test environment.

Please double check you have correct "Server domain name" configured in System -> Settings -> Application and that your UCRM is accessible via plain HTTP (port 80).

If this is all correct, please send us the Let's Encrypt log, you can find this in System -> Tools -> SSL certificate.

New Member
Posts: 13
Registered: ‎09-22-2016

Re: LetsEncrypt fails

I've confirmed the server domain name setting is correct. My firewall allows port 443 directly to the server and port 80 goes to a Kemp VIP which provides a 302 redirect to the same URL but under HTTPS.

 

Here is the log:

2019-04-12 13:34:03,224:DEBUG:certbot.main:certbot version: 0.25.1
2019-04-12 13:34:03,225:DEBUG:certbot.main:Arguments: ['--config', '/data/ssl/certbot.ini', '-d', 'ucrm.contoso.com']
2019-04-12 13:34:03,225:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-04-12 13:34:03,303:DEBUG:certbot.log:Root logging level set at 30
2019-04-12 13:34:03,304:INFO:certbot.log:Saving debug log to /data/log/ucrm/letsencrypt/letsencrypt.log
2019-04-12 13:34:03,305:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2019-04-12 13:34:03,313:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7fde2376d1d0>
Prep: True
2019-04-12 13:34:03,314:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7fde2376d1d0> and installer None
2019-04-12 13:34:03,315:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2019-04-12 13:34:03,320:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=u'valid', terms_of_service_agreed=None, contact=(u'mailto:snowtr@gmail.com',), agreement=u'https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf', key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7fde2376d610>)>)), uri=u'https://acme-v01.api.letsencrypt.org/acme/reg/54565380', new_authzr_uri=u'https://acme-v01.api.letsencrypt.org/acme/new-authz', terms_of_service=u'https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf'), c9047b4bf48f57bba33c7535ab8c7157, Meta(creation_host=u'463f20fc70ea', creation_dt=datetime.datetime(2019, 4, 4, 6, 42, 4, tzinfo=<UTC>)))>
2019-04-12 13:34:03,322:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2019-04-12 13:34:03,324:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2019-04-12 13:34:03,510:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2019-04-12 13:34:03,512:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 658
Replay-Nonce: WYr0FvfQeLIOSib3InbsbdIKnYnctVsT9n6p_r4sHFQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Fri, 12 Apr 2019 13:34:03 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 12 Apr 2019 13:34:03 GMT
Connection: keep-alive

{
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert",
  "zKHUgILkaHs": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
}
2019-04-12 13:34:03,512:INFO:certbot.main:Obtaining a new certificate
2019-04-12 13:34:03,798:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0034_key-certbot.pem
2019-04-12 13:34:03,803:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0034_csr-certbot.pem
2019-04-12 13:34:03,805:DEBUG:acme.client:Requesting fresh nonce
2019-04-12 13:34:03,805:DEBUG:acme.client:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz.
2019-04-12 13:34:03,893:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "HEAD /acme/new-authz HTTP/1.1" 405 0
2019-04-12 13:34:03,894:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 91
Allow: POST
Replay-Nonce: gmN86VUIzUoeqD6tOdr2FxOhkzv2xAbqFLhU6Pa-g2k
Expires: Fri, 12 Apr 2019 13:34:03 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 12 Apr 2019 13:34:03 GMT
Connection: keep-alive


2019-04-12 13:34:03,894:DEBUG:acme.client:Storing nonce: gmN86VUIzUoeqD6tOdr2FxOhkzv2xAbqFLhU6Pa-g2k
2019-04-12 13:34:03,895:DEBUG:acme.client:JWS payload:
{
  "identifier": {
    "type": "dns", 
    "value": "ucrm.contoso.com"
  }, 
  "resource": "new-authz"
}
2019-04-12 13:34:03,900:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
{
  "protected": "eyJub25jZSI6ICJnbU44NlZVSXpVb2VxRDZ0T2RyMkZ4T2hrenYyeEFicUZMaFU2UGEtZzJrIiwgImFsZyI6ICJSUzI1NiIsICJqd2siOiB7ImUiOiAiQVFBQiIsICJrdHkiOiAiUlNBIiwgIm4iOiAiMjNTTFhETjFKb29UV3JiejJ5UlNsZ0pTRHlVc1BiRGNLOUVQdjZxeHJENzU3clkwV0VOTVVyNlV3VV8xcm9hUTNJVHdfSm5URU1aaVhnc29jQVJQOEJRTjdDTFRTcVhCMlFUZGVfQnlLakdBdnZyTFBnRm5wc0dJWmdnUk9xcGM2NkpKenVXamJxS0JxUDJTWkNrdEJzN0RhLVpnUlZCLXRZN2V0MFJMLTk2VWRFeDNDSU4xeHN6YWpGbXdCanlVVlNROVV1d3VBYngxRUhpS0w5UVhuaTBiTklWejZ4RVpSTmFZeXNROEpUT0dSZk9WZWRyOC1yblBjNWdvUlRRU254eTFzeE1NOENjS2RWQlpIU3JPVEh1Nm1PMnV2c1VxbTkwcXpXOG80UDBGV0pFUnZMM25LeUNVWXVmZ2ZRaVN6TldKNEtpUmEzQVVZNDQwNG1TT2F3In19", 
  "payload": "ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwgCiAgICAidmFsdWUiOiAidWNybS5pdnRlY2guY2EiCiAgfSwgCiAgInJlc291cmNlIjogIm5ldy1hdXRoeiIKfQ", 
  "signature": "DMhYrgwNQu8LrGaqc4AalKbXKWYQ5dOlBjZqcsQP0ukXuy877pqNOy_8-kh2QxnFxr2ImMbZQQFCKx7PH0BIWCkVIXsACrkF6cMIlVGNt6cDAjQ70YCsT0huPtOWTiTDTZdlK10YZ4pXHVTZc-3QmEIdiFNkStlpRXBcfsXC_vO_RzfftLKkcVjnTCq-3_NaIewuuuq7BV5zYNYdxPVJtaQZXjvDaK446_-qj_6zXd_Bd-nYBAxSoJMBhNXIAFJn68pxkDL7QlCtjmvajW8tBKKngMrwFi9EeL1I-VW6aLeu_ISd-reIXdV098RyrSCeaJc6egyMOnoL_ljQ_R-ymQ"
}
2019-04-12 13:34:04,040:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/new-authz HTTP/1.1" 201 996
2019-04-12 13:34:04,042:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 996
Boulder-Requester: 54565380
Link: <https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"
Location: https://acme-v01.api.letsencrypt.org/acme/authz/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE
Replay-Nonce: WtFN8PiUqMLeo6TBfWkq8VFp4nmZCGVw6oiP-gocUNI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Fri, 12 Apr 2019 13:34:04 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 12 Apr 2019 13:34:04 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "ucrm.contoso.com"
  },
  "status": "pending",
  "expires": "2019-04-19T13:34:03Z",
  "challenges": [
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004307",
      "token": "lq27HcL_A-mHu4Yrmygc0egrvzGKcLJcJX6iiHdTfOA"
    },
    {
      "type": "http-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004309",
      "token": "zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004314",
      "token": "tNQt2jhWr5Ju4HhI6PuJpY7mH7Pr02x8ftgq97PedT0"
    }
  ],
  "combinations": [
    [
      0
    ],
    [
      1
    ],
    [
      2
    ]
  ]
}
2019-04-12 13:34:04,043:DEBUG:acme.client:Storing nonce: WtFN8PiUqMLeo6TBfWkq8VFp4nmZCGVw6oiP-gocUNI
2019-04-12 13:34:04,043:DEBUG:acme.challenges:tls-alpn-01 was not recognized, full message: {u'status': u'pending', u'token': u'lq27HcL_A-mHu4Yrmygc0egrvzGKcLJcJX6iiHdTfOA', u'type': u'tls-alpn-01', u'uri': u'https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004307'}
2019-04-12 13:34:04,044:INFO:certbot.auth_handler:Performing the following challenges:
2019-04-12 13:34:04,044:INFO:certbot.auth_handler:http-01 challenge for ucrm.contoso.com
2019-04-12 13:34:04,045:INFO:certbot.plugins.webroot:Using the webroot path /usr/src/ucrm/web for all unmatched domains.
2019-04-12 13:34:04,045:DEBUG:certbot.plugins.webroot:Creating root challenges validation dir at /usr/src/ucrm/web/.well-known/acme-challenge
2019-04-12 13:34:04,053:DEBUG:certbot.plugins.webroot:Attempting to save validation to /usr/src/ucrm/web/.well-known/acme-challenge/zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U
2019-04-12 13:34:04,054:INFO:certbot.auth_handler:Waiting for verification...
2019-04-12 13:34:04,054:DEBUG:acme.client:JWS payload:
{
  "keyAuthorization": "zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U.mBn93egE2aF2DcqRlz8KTJ3t77ziSzNSWxWFm_ciczA", 
  "type": "http-01", 
  "resource": "challenge"
}
2019-04-12 13:34:04,060:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004309:
{
  "protected": "eyJub25jZSI6ICJXdEZOOFBpVXFNTGVvNlRCZldrcThWRnA0bm1aQ0dWdzZvaVAtZ29jVU5JIiwgImFsZyI6ICJSUzI1NiIsICJqd2siOiB7ImUiOiAiQVFBQiIsICJrdHkiOiAiUlNBIiwgIm4iOiAiMjNTTFhETjFKb29UV3JiejJ5UlNsZ0pTRHlVc1BiRGNLOUVQdjZxeHJENzU3clkwV0VOTVVyNlV3VV8xcm9hUTNJVHdfSm5URU1aaVhnc29jQVJQOEJRTjdDTFRTcVhCMlFUZGVfQnlLakdBdnZyTFBnRm5wc0dJWmdnUk9xcGM2NkpKenVXamJxS0JxUDJTWkNrdEJzN0RhLVpnUlZCLXRZN2V0MFJMLTk2VWRFeDNDSU4xeHN6YWpGbXdCanlVVlNROVV1d3VBYngxRUhpS0w5UVhuaTBiTklWejZ4RVpSTmFZeXNROEpUT0dSZk9WZWRyOC1yblBjNWdvUlRRU254eTFzeE1NOENjS2RWQlpIU3JPVEh1Nm1PMnV2c1VxbTkwcXpXOG80UDBGV0pFUnZMM25LeUNVWXVmZ2ZRaVN6TldKNEtpUmEzQVVZNDQwNG1TT2F3In19", 
  "payload": "ewogICJrZXlBdXRob3JpemF0aW9uIjogInp5TjZwNXhzNlpCR1FsSzZ4bU51OVF6TGNRY3oxVnUwUzZpTUhfeG8tOVUubUJuOTNlZ0UyYUYyRGNxUmx6OEtUSjN0Nzd6aVN6TlNXeFdGbV9jaWN6QSIsIAogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9", 
  "signature": "PKMim8Asg53ozq6fQMTIKdhg6MYVqcaLSE5mWDY1vMNUqhxyQvgA1DsCOOtlxpiBcB5u9V_zSca-rIIymH2Ul9Bu3-YzWADZ6AcKk3u3UNlr7q-BZuuThXz-mioHlkxsW1JvvupU78_ScSO0bmfgTLi-nYGUdSR5GklkPIukAPXtgiZjVL6nojAQ4Na5Qe5i6PXarMzozmHr0IIi_pz83SkjnBslM8PnCb7hGnnWAIcIaNobN7Khlensik3Co7XzaL39ZCu2rD4PbZYgpeBDELOPGuG2yFmo9KY3QqYRoUFOotz1RXH7i617s1q-J6zrIYEeh16OEkotfeg2YzOUwg"
}
2019-04-12 13:34:04,166:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004309 HTTP/1.1" 202 337
2019-04-12 13:34:04,167:DEBUG:acme.client:Received response:
HTTP 202
Server: nginx
Content-Type: application/json
Content-Length: 337
Boulder-Requester: 54565380
Link: <https://acme-v01.api.letsencrypt.org/acme/authz/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE>;rel="up"
Location: https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004309
Replay-Nonce: _l7jBhu34AkVK_JB_hFdQxIB9gCh9ekupUY5tEvdUx8
Expires: Fri, 12 Apr 2019 13:34:04 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 12 Apr 2019 13:34:04 GMT
Connection: keep-alive

{
  "type": "http-01",
  "status": "pending",
  "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004309",
  "token": "zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U",
  "keyAuthorization": "zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U.mBn93egE2aF2DcqRlz8KTJ3t77ziSzNSWxWFm_ciczA"
}
2019-04-12 13:34:04,168:DEBUG:acme.client:Storing nonce: _l7jBhu34AkVK_JB_hFdQxIB9gCh9ekupUY5tEvdUx8
2019-04-12 13:34:07,172:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/acme/authz/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE.
2019-04-12 13:34:07,272:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "GET /acme/authz/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE HTTP/1.1" 200 2249
2019-04-12 13:34:07,273:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Link: <https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"
Replay-Nonce: I5C2g1LYs3Jez38VvfWxyDdk48k_n2sDZjWTqa0HBNM
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Fri, 12 Apr 2019 13:34:07 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 12 Apr 2019 13:34:07 GMT
Content-Length: 2249
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "ucrm.contoso.com"
  },
  "status": "invalid",
  "expires": "2019-04-19T13:34:03Z",
  "challenges": [
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004307",
      "token": "lq27HcL_A-mHu4Yrmygc0egrvzGKcLJcJX6iiHdTfOA"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:acme:error:unauthorized",
        "detail": "Invalid response from https://ucrm.contoso.com/login [<redacted public IP>]: \"\u003c!DOCTYPE html\u003e\\n\u003chtml\u003e\\n    \u003chead\u003e\\n        \u003cmeta charset=\\\"UTF-8\\\"/\u003e\\n        \u003cmeta http-equiv=\\\"x-ua-compatible\\\" content=\\\"ie=edge\\\"\u003e\\n\"",
        "status": 403
      },
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004309",
      "token": "zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U",
      "validationRecord": [
        {
          "url": "http://ucrm.contoso.com/.well-known/acme-challenge/zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U",
          "hostname": "ucrm.contoso.com",
          "port": "80",
          "addressesResolved": [
            "<redacted public IP>"
          ],
          "addressUsed": "<redacted public IP>"
        },
        {
          "url": "https://ucrm.contoso.com/",
          "hostname": "ucrm.contoso.com",
          "port": "443",
          "addressesResolved": [
            "<redacted public IP>"
          ],
          "addressUsed": "<redacted public IP>"
        },
        {
          "url": "https://ucrm.contoso.com/login",
          "hostname": "ucrm.contoso.com",
          "port": "443",
          "addressesResolved": [
            "<redacted public IP>"
          ],
          "addressUsed": "<redacted public IP>"
        }
      ]
    },
    {
      "type": "dns-01",
      "status": "invalid",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004314",
      "token": "tNQt2jhWr5Ju4HhI6PuJpY7mH7Pr02x8ftgq97PedT0"
    }
  ],
  "combinations": [
    [
      0
    ],
    [
      1
    ],
    [
      2
    ]
  ]
}
2019-04-12 13:34:07,274:DEBUG:acme.challenges:tls-alpn-01 was not recognized, full message: {u'status': u'invalid', u'token': u'lq27HcL_A-mHu4Yrmygc0egrvzGKcLJcJX6iiHdTfOA', u'type': u'tls-alpn-01', u'uri': u'https://acme-v01.api.letsencrypt.org/acme/challenge/q8WFhhGY2dOFM-chNglLqcAB-fN83oFagWuXG20WBqE/14691004307'}
2019-04-12 13:34:07,276:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: ucrm.contoso.com
Type:   unauthorized
Detail: Invalid response from https://ucrm.contoso.com/login [<redacted public IP>]: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"UTF-8\"/>\n        <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2019-04-12 13:34:07,276:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 155, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 226, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. ucrm.contoso.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ucrm.contoso.com/login [<redacted public IP>]: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"UTF-8\"/>\n        <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n"

2019-04-12 13:34:07,277:DEBUG:certbot.error_handler:Calling registered functions
2019-04-12 13:34:07,277:INFO:certbot.auth_handler:Cleaning up challenges
2019-04-12 13:34:07,277:DEBUG:certbot.plugins.webroot:Removing /usr/src/ucrm/web/.well-known/acme-challenge/zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U
2019-04-12 13:34:07,278:DEBUG:certbot.plugins.webroot:All challenges cleaned up
2019-04-12 13:34:07,278:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.25.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1323, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1213, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 120, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 383, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 326, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 362, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 155, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 226, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. ucrm.contoso.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ucrm.contoso.com/login [<redacted public IP>]: "<!DOCTYPE html>\n<html>\n    <head>\n        <meta charset=\"UTF-8\"/>\n        <meta http-equiv=\"x-ua-compatible\" content=\"ie=edge\">\n"
Highlighted
Ubiquiti Employee
Posts: 1,476
Registered: ‎03-21-2016
Kudos: 244
Solutions: 161

Re: LetsEncrypt fails

Hello @snowtr, I think the problem is related to the "and port 80 goes to a Kemp VIP which provides a 302 redirect to the same URL but under HTTPS"

 

Basically Let's Encrypt verification requires, that the challenge file is publicly accessible at the given URL, i.e. "http://ucrm.contoso.com/.well-known/acme-challenge/zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U" in your log.

 

I think you might have some problem in the redirect configuration.

New Member
Posts: 13
Registered: ‎09-22-2016

Re: LetsEncrypt fails

I've resolved the issue for my setup.  I still beleive it was caused by the update but here is what I did.

 

After my last reply and including the statement about the Kemp providing the 302 redirect I checked on that it confirmed that it was directly redirecting to the root of the URL, so instead of variables, it was sending to the hostname alone, which cuts out anything from the URL after it.

 

Until now, this worked.  I'd observices plenty of times, on plenty of my own servers, that the ACME protocol would try port 80 as a URL first, get the redirection, then try the full URL down into the directory.  I guessed that this was no longer happening after the update and changed my 302 redirect to go to https://%h%s instead of https://ucrm.contoso.com/ which means that the %s preserves any parts of the URL after the hostname during the initial HTTP connection.

 

Now it's working again.  I guess that there was an upgrade to the version of ACME being used in uCRM going up to 2.15.1 and that this ACME version change includes a difference in how the URL testing is handled, which no longer assumes that the initial request to HTTP may get redirected.

 

I can suggest that Ubiquiti adds the Acme version change to the Change Log or Release Notes on some place where people will see it and hopefully dig deeper into the Acme changes before upgrading.

Ubiquiti Employee
Posts: 1,476
Registered: ‎03-21-2016
Kudos: 244
Solutions: 161

Re: LetsEncrypt fails

@snowtr The problem is not with redirect, you can actually see in the log, that the redirect was processed correctly by Let's Encrypt:

"validationRecord": [
        {
          "url": "http://ucrm.contoso.com/.well-known/acme-challenge/zyN6p5xs6ZBGQlK6xmNu9QzLcQcz1Vu0S6iMH_xo-9U",
          "hostname": "ucrm.contoso.com",
          "port": "80",
          "addressesResolved": [
            "<redacted public IP>"
          ],
          "addressUsed": "<redacted public IP>"
        },
        {
          "url": "https://ucrm.contoso.com/",
          "hostname": "ucrm.contoso.com",
          "port": "443",
          "addressesResolved": [
            "<redacted public IP>"
          ],
          "addressUsed": "<redacted public IP>"
        },
        {
          "url": "https://ucrm.contoso.com/login",
          "hostname": "ucrm.contoso.com",
          "port": "443",
          "addressesResolved": [
            "<redacted public IP>"
          ],
          "addressUsed": "<redacted public IP>"
        }
      ]

The challenge URL was tried first, it got redirected to "https://ucrm.contoso.com/" and since there is no user logged in, UCRM then further redirected the URL to "https://ucrm.contoso.com/login"

If you always had the redirect configured to strip the rest of the URL, I think it could have never worked. Perhaps you used different mode of authentication for your other projects?

New Member
Posts: 13
Registered: ‎09-22-2016

Re: LetsEncrypt fails

Nothing was changed since the last two cert updates except this patching of uCRM, so yes, the redirection was broken by the update.