Introduction

Certificate Transparency is an experimental framework for publicly auditing the TLS/SSL certificates issued by certificate authorities. It does this by requiring an increasing number of issued certificates to be added to publicly-accessible repositories, called certificate transparency logs, in order for them to be trusted by the user's browser. The goal is two-fold:

  1. Easy detection of mis-issued certificates. Because certificates must be submitted to several certificate transparency logs to be trusted in-browser, website owners can also search through these same logs and get an authoritative list of all certificates that have been issued for their website.
  2. Better oversight and understanding of the entire TLS/SSL ecosystem. Crawling certificate transparency logs gives deep insight into the types of certificates being issued, and the scale of different certificate authorities.

In addition to CT logs, there need to be certificate transparency monitors. The primary function of a monitor is to ensure that CT logs behave correctly, e.g., by ensuring that they continue to store and serve all certificates that have been submitted to them. However, it is also common for monitors to use the data they collect from CT logs to offer other services such as computing ecosystem metrics, searching known certificates, and triggering alerts when new certificates are issued for certain domain names.

Methodology

This monitor fetches each log's current Signed Tree Head (STH) and set of trusted roots once per minute. New entries in a log are crawled as appropriate, and we will occasionally submit strictly-valid unexpired certificates or pre-certificates to a log. If a log is sharded by year, we will only submit certificates expiring in that year.

All requests to a log count towards our estimate of its uptime and response time. An endpoint's (get-sth, get-entries, etc.) uptime is computed as the percentage of requests to that endpoint that succeed (that is, return a 200 status code with a valid response) out of a 90 day rolling window. An endpoint's response time is computed as the average amount of time from when a request is sent until the response is fully received and parsed, in a 90 day rolling window. A log's uptime and response time is computed as a fair average of each endpoint's uptime and response time. Requests are made with the HTTP client in Golang's standard library. We do not contact endpoints that are not listed on a log's informational page, such as get-proof-by-hash and get-sth-consistency.

Some parts of the monitor may enforce different timeouts or will retry in the event that a request fails while other parts will not, which means that downtime may be penalized unfairly relative to a more intuitive interpretation of 'downtime'. Also, the response time of the get-entries endpoint is strongly correlated with the number of entries it returns, but this information is currently ignored by our monitor.

Fullness and New/Old Chains. On each log's informational page, we display an estimate of the number of unsubmitted certificates. This is computed by counting all the certificates in the CT ecosystem that we expect a log would accept, based on its trusted root CAs and whether the log is sharded by year, and subtracting the number of unique certificates the log contains. The log's 'fullness' is computed as the percentage of certificates it contains out of the set of certificates we expect it would accept.

Similarly, we break uptime and response time estimates for the add-(pre-)chain endpoints into 'new' and 'old' categories, corresponding to whether or not the submitted chain is already in the log (old) or is not (new). It is common for logs to de-duplicate submitted chains, returning the same SCT when the same chain is submitted multiple times, so we do this to measure the cases where the de-duplication mechanism succeeds and where it fails separately. If the de-duplication mechanism succeeds, the add-(pre-)chain operation is read-only but the old SCT must be reconstructed. If the de-duplication mechanism succeeds, the add-(pre-)chain operation must block until the submitted data is persisted to disk.

Cryptographic Validation. No information about cryptographic validation is propagated to the monitor's UI, but it is done. We validate that STHs are issued frequently enough, and that the hash in each STH corresponds to the hash of the leaves we've crawled. We also store all SCTs returned from a log's add-chain/add-pre-chain endpoints until we can verify that the log honored the SCT within its maximum merge delay.

Contact

Feel free to contact us at ct-logs@cloudflare.com.