Ran across a fun issue with Let’s Encrypt and certbot today.
The issue at hand seemed straightforward at first: we were attempting to create a new SSL certificate via Let’s Encrypt but kept hitting a wall with the http validation process. For those unfamiliar, Let’s Encrypt utilizes a simple challenge-response mechanism to verify domain ownership. Essentially, it sends a set of http requests to the domain and expects responses to confirm control over the domain.
After much head-scratching and log-digging, I observed three successful http requests from Let’s Encrypt’s API hitting the server. Yet, the validation process refused to complete with the error “Secondary Validation Failed”. The puzzle was why? The logs seemed to indicate success.
The “Secondary validation” portion of the error did make me think that there must be a “primary validation” that was working. I finally discovered after much googling and forum post reading that Let’s Encrypt actually makes not three, but four successful http requests during its validation process.
The discrepancy led me down a rabbit hole that ended with a realization about the customer’s server configuration: geoblocking. In an attempt to bolster security, the server was configured to block non-US IP addresses. As it turns out, one of Let’s Encrypt’s validation requests originated from outside the US, thereby being blocked and causing the entire validation process to fail.
This incident underscores a crucial lesson that often goes overlooked in the documentation and troubleshooting guides: the devil is in the details. It took a deep dive into forum discussions and a considerable amount of googling to piece together the puzzle. Documenting the exact HTTP process that Let’s Encrypt goes through for verification would have been helpful.
Edit: on further digging found more details on the process https://letsencrypt.org/2020/02/19/multi-perspective-validation.html
