On Tuesday, August 8, 2017 at 5:27:13 AM UTC+9, Jakob Bohm wrote: > On 07/08/2017 22:12, Alex Gaynor wrote: > > You seem to be suggesting that the thoroughness of testing is somehow > > related to how long it takes. > > > > I'd expect any serious (or even not particularly serious...) to have a > > comprehensive automated test suite that can verify that the software is > > regression free and correct in minutes or hours. If you can't deploy > > changes of any size with confidence in less than several months, I think > > you have some serious process problems. > > > > For non-essential changes,
I think this may be the first, and most serious, flaw in your argument. Compliance with Mozilla policy and standards that are, at this point, over twenty years old, are essential Ganges. Full stop, they need to be made, made quickly, and should never have reached production. > it may be a good idea to supplement the fast > automated tests by tests that take a lot longer. This could be manual > tests, or it could be tests that verify expiry procedures in real time > (e.g. issue a cert at the start of the test and verify that the OCSP > component acts as intended near the end of the test). Your examples are things you can and should write automated tests for. > > The need to deploy some changes quickly inevitably represents a > compromise between speed and quality, both in testing and coding. So > not using the rushed procedures for non-urgent changes is good general > practice. Creating a taxonomy between such is otherwise attempting to legitimize poor software development practices and poor business practices. It is a perspective that is no doubt shared by legacy software development firms, but much has been done in the past 20 years to support models of continuous development and continuous deployment. This is aided by rigorous test driven development, which is the corner stone of having the objective confidence, rather than the subjective unease. > > Consider that most end-users are not encouraged to run Firefox nightlies > and that enterprise users tend to use special ESR releases with longer > release cycles than end users. These practices represent the same > fundamental speed/quality trade-off. > While is may sound like compelling support for your argument, it does represent several unsupported or inaccurate claims. For example, Firefox users are encouraged to run other channels (like Aurora or Beta) precisely because the thorough esss of the automated testing represents a higher degree of confidence - which enables such updated versions to ship every six weeks. Similarly, Enterprise users who run ESR generally hold back the Web Platform and face greater risks, despite the considerable effort to support ESR, by virtue of running perpetually outdated software. Again, this is an area where the past twenty years have shown notions such as shipping "long term stable" software - whether it be a browser, an OS, or CA software - is actively detrimental to the ecosystem. I realize much of your message is expressing a philosophy on software development, and while I've responded to point out that alternative philosophies exist - and with more catcher in modern software development - it is likely that our entire philosophical musings are moot. Whatever the approach to development, participants in the Mozilla Root CA Program have an obligation to comply with the program requirements, and for the safety and security of Mozilla users, that compliance needs to be timely. If certain, outdated and insecure approaches to that development may be used, then that is the CAs fault and risk, and not something the community should be asked to bear. _______________________________________________ dev-security-policy mailing list [email protected] https://lists.mozilla.org/listinfo/dev-security-policy
