On Fri, Nov 2, 2018 at 10:24 AM Wiedenhorst, Matthias via dev-security-policy <[email protected]> wrote:
> Auditor and Reviewer, as stated on > https://www.tuvit.de/fileadmin/Content/TUV_IT/zertifikate/en/AA2018072001_Audit_Attestation_E_Deutsche-Telekom-Root-CA-2_20180718_s.pdf > - the parties tasked with ensuring that the audit is meaningfully able to > ensure the criteria were met and the testing procedures were able to meet > those requirements. > > Auditors and reviewers need to be distinguished: ISO/IEC 17065 §7.5.1 > forbids that the person(s) performing the review is involved in the audit > process. > Indeed - and yet do you agree that the Reviewer is tasked with reviewing the audit methodology and artifacts to ensure it appropriately meets the objectives expected and required? A multi-party failure to assure the necessary assurance is just that - a multi-party failure. >Issue A) As part of their initial response to my complaint, TUVIT, by way > >of Matthias Wiedenhorst (Head of Certification Division TSP) stated "As a > >very first, quick cross check with our audit evidence record, I can only > >say that we did check issued certificates from within the declared period > >and that they did contain the proper qcStatement-6.3 / id-etsi-qcs-QcType > >3". However, this statement was in direct conflict with the TSP's own > >investigation and incident report, provided at > >https://bugzilla.mozilla.org/show_bug.cgi?id=1498463#c3 , which states > the > >mistake was introduced during the development of support - that is, no > such > >properly issued certificates were issued. > > We do not understand why the important fact that Matthias was not in > office and replied what he remembered from an audit that was month ago is > not mentioned here. In addition, he replied that he would verify and come > back as soon as possible (when he is in office again). That actually > happened, see below. > > Wrong or misleading information - which was only corrected upon specific > questioning and a request for proof or evidence of the claim - has been > used to disqualify CAs in the past. This statement was made after the TSP > had themselves already investigated and confirmed this was not possible. > > The same standard being applied to CA incident reports is being proposed > here - that incomplete and improper investigations raise serious questions. > > Let’s stay with present case (not with the past): In his email, Matthias > said that he is not in his office and will begin to investigate the > situation as soon as possible. We think that it is clear, that further > information contained in the email cannot be based on the result of the > investigation. Back in his office after looking into the audit logs and > verifying the qcStatement he gave the proper answer. > I appreciate the attempt to narrowly scope the issue, but that is equally an attempt to deflect or ignore the ample set of past precedent and expectations. As such, I reject the premise that this should be considered without regard to past failures by CAs or other auditors - as an auditor being entrusted to report truthfully and faithfully to the community about a CAs compliance with its own CP, CPS, and the appropriate supervisory framework, auditors are expected to consider the best practices and precedents in their activities and actions. With respect to your suggestion that the information can not be relied upon, I'm more than happy to provide the full e-mail chain if there's some consideration given to misinterpretation. However, I do not believe the reply at all indicates that the information contained was not reliable or may be counter-factual and to be corrected later. Yes, Matthias stated they were out of office - but then immediately began with a remark that "As a very first, quick cross check with our audit evidence record, I can only say that we did check issued certificates". I can understand the argument being made here that, on a more detailed examination of your audit evidence record, you discovered that it was not, in fact, checked. However, that raises significant concerns that a quick check can lead to a completely opposite conclusion, particularly for a supposedly-skilled practitioner. > As said before, we are using tools in the audit process. The sentence > about lint tools should be seen as additional information and nothing else. > Yet you've failed to describe what these tools encompass, beyond what is readily available off the shelf. Further, in your description of the methodology used, it was clear that human visual inspection without regard to the actual specification was performed. While you may have used a tool to, say, dump the DER-encoded contents into a structural representation, the procedures for examining that structural representation against the profile are clearly and significantly deficient. > Considering the significance of misencoding of profiles - which has lead > to critical misissuance and security risk (see, for example, Turktrust) - > the question about whether or not TUVIT's testing procedures are adequate > to ensure compliance are extremely relevant and appropriate. TUVIT could, > for example, attempt to resolve these concerns by providing documentation > about their approach to assessing compliance with the relevant certificate > profiles, including methodologies for sampling (not normatively specified > by the relevant documents), their testing and training procedures, and > other evidence that would feel suitable to highlight that this was, indeed, > a 'one off' approach. > > Let’s stay with the present case: We have not seen any security breach due > to the miscoding of the qcStatement and have not seen any arguments how > certificates with wrong qcStatement could be used to breach the security in > future. That is the reason why believe the non-conformity is a non-critical > non-conformity with respect to the performed assessment. In addition, a > correct qcStatement is not required by the Baseline Requirements (but by > ETSI EN 319 412-5). Having this in mind, we expect that Browser Ecosystem > should not be directly affected. > This response most perfectly fundamentally why I believe that TUVIT should no longer be recognized as a suitable auditor. The community has, for the better part of a decade now, recognized that the failure to appropriately encode certificates minimally represents a significant barrier to interoperability. The need to support and interoperate with such broken certificates has resulted in multiple security vulnerabilities being introduced into the ecosystem, such as the processing of public key encodings that were improperly DER encoded or the improper encoding of negative serial numbers. Similarly, the failure to adhere to the specified profile has resulted in a number of high-profile CA security events. For example, the inclusion of a "cA:True" within the basicConstraints of a certificate intended for subscribers, or the misuse of the EV certificate profile for testing, have lead to the dissolution of CA services due to the critical blows to trust in those providers. One of the most core, and fundamental, expectations of auditors is that they have the necessary technical skill to review the configured profiles, software, and systems to ensure the correct production of certificates. That these produced certificates align with the stated CP and CPS, and any relevant specifications incorporated therein. That they have the necessary technical judgement to recognize the critical nature of these functions, and its fundamental impact on the trust ecosystem. One would expect the degree of auditor competency to review the produced certificates on an aggressive basis, to ensure that the level of confidence in the certification is appropriate. Any sampling methodology used - even to the degree of just a single certificate - would and should have revealed this critical misconfiguration. That no such detection or discovery was made fundamentally calls into question the level of supervision and oversight by the CAB, which is a critical function to ensure those relying on the certification do not face security or compatibility risks. Furthermore, that within the framework of the nominally more rigorous, more critical trust objectives of qualified certificates, one would expect that the level of scrutiny and supervision is beyond reproach, so as not to call into disrepute the entire concept and framework. TUVIT has not met these expectations, and in doing so, fundamentally calls into question the necessary competencies of the auditors and the methodology employed. This is fundamentally no different than the rejection of those WebTrust auditors who offer letters ascribing high confidence that the principles and criteria are met, while readily available public information actively contradicts this, and even simple sampling would have revealed these flaws. This is not singling out ETSI-using auditors; this is recognizing an auditor that is failing to meet the requirements of the community. > 1) TUVIT's assessment methodologies are fundamentally flawed and do not > meet the level of assurance expected of the community > > This is an exaggerated conclusion based on the only fact that a wrong > coded qcStatement was not detected. > TUVIT - and Matthias Wiedenhorst - have been the auditor of note for T-Systems for some time, as demonstrated through https://www.t-systems.com/blob/776674/ec9fa6212c7a1c5537cfb57ad33af6bb/DL_Trust-Center_ETSI_Audit.pdf During those same periods, T-Systems had multiple issues of material non-compliance, as demonstrated through https://bug1391074.bmoattachments.org/attachment.cgi?id=8915934 This is not an exaggerated conclusion by any means, but rather, a systematic failure to apply sound audit methodology and best practices. > 2) TUVIT's and T-Systems' incident handling approach does not meet the > level of expectations of the community. This includes both the oversight of > revocation, but also the approach to auditing for when such a critical > non-conformity has been found. A resolution of "They promise to fix it" > does not provide a reasonable level of assurance, given that at fundamental > question is what other issues TUVIT failed to consider or assess. The lack > of introspection, by TUVIT, as to its auditing process and procedures is > concerning. > > How one single person (with his private hat on) can speak for the > community? I do not speak to the community decisions in the future. However, I am more than happy to detail amply the community expectations, captured in bugs and in discussions, about the handling of revocation and errors. In this regard, consider this a summary of the past discussions setting forth the expectations. > We explained before why we believe that the wrong qcStatement coding is a > non-critical nonconformity (the security of ecosystems is not affected). > Where did you copy "They promise to fix it" from? We have not used this > expression. > With respect to your previous messages: "We apply this requirement in the context of a certificate decision (for the issuance of a ETSI certificate) also for this case of a non-conformity found for an issued ETSI certificate: Correction of the bug and revocation of all concerned website certificates till end of 2018 will fulfill the requirement." and "Failure to comply with the Baseline Requirements due to the non-revocation of the concerned website certificates within the required time line is analogously regarded as minor non-conformity, that shall be fixed within three month." Both of these assert that remediation within three months time is suitable to satisfy the needs. While that is certainly the minimum required, that does not align with the expectations for resolution and/or revocation. _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
