Hi Petr, Hehe, I missed this mail before I started summarizing today. I'll have a review for sure :)
Best wishes, Yifan On Tue, Nov 15, 2011 at 07:44:59PM +0100, Petr Mladek wrote: > Hi, > > I am going to summarize the ideas from the mail "Test case naming". > > I am sorry for the long mail. The problem is not easy. There are many > ideas. I tried to sort them from different point of view. I hope that I > did not miss anything important and it will help to find the final > conclusion. > > ---------------------------------------------------------- > > Let's start by a short summary. > > Litmus allows to split test cases by: > > + product > + branch > + testgroup > + subgroup > > > We want to split test cases by: > > + priority (P1,P2,P3,P4) > + component (General,Base,Calc,Draw,Impress,Math,Writer) > + meaning (functional, l10n, feature) > + l10n tests in languages (en,de,fr,it,ja,.............) > + platform (Linux,Windows,MAC) > > We need to take care of the following scenarios: > > + maintenance: > + creating test cases > + updating test cases > + localization > + branch for new release > + test run for new release/build > + doing tests > + analyzing results > > ----------------------------------- > > Let's look at some conditions: > > 1. priority: > > + why > + reduced test run for bugfix releases > + overal prioritization of the work > > + main requirements: > > + allow to create test run for given priority > + easily run tests according to priority > + easily see test results by priority > > > 2. component: > > + why > > + reduced test run if only writer is modified > + helps to split work by expertize > > + main requirements: > > + allow to create test run for given component > + allow to run tests by component > > 3. meaning: > > + why > > + functional tests do not depend on localization > + the list of language dependent tests might be > different for each language > + feature tests are not well described; are done only > once by one or few dedicated people > > + requirements: > > + function test appears only once in test run; can be > approved by any user with any locale and any platform > + language dependent tests must be approved more times > in all locales > + test results must clearly show what tests were > done; functional tests are important by priority; > l10n tests are important by priority and language; > e.g. German is more important than Czech because > there are more active users > > > 4. platform: > > + why: > > + some tests might depend on platform; for example the > open/save dialogs look different way on Linux,Windows, > and MAX; also KDE vs. GNOME > > + main requirements: > > + these tests must be done on all affected platforms > + test results must clearly show where it was tested > > > 5. Maintenance: > > + problems: > > + searching for a particular test case > + put test cases into the right position > + comparing en-US template with localized test cases > + localizing the whole bunch of test cases into new > language > > + requirements: > > + balance the number of branches, testgroups, > subgroups, and test cases in subgroup > + must be easy to copy test cases for new language > + often switched criteria must be in high level; > for example, do people switch more often between > priority or category or language when they work > with the test cases? > > > 6. Branch for new release: > > + requirements: > > + must be easy to do the branch > + must be easy to see all tests for the given release > > > 7. Test run creation: > > + requirements: > > + 1 or very small number of active test runs because it > is compilcated to switch between them and analyze > results from many test runs > + create test run for a given priority (basic vs. full > test) > + create test run for a given component (only Writer > modified) > > 8. Doing tests: > > + requirements: > > + must be easy to search what needs testing > + must be possible to sort testing by priority and > component > + priority is more important than component > because it does not help to have perfectly tested > Writer when Base is not tested at all > > 9. Test analyze: > > + requirements: > > + must be easy to see what was tested by priority, > component, and language > + less important language tests should not fuzz > test results by priority. e.g. it would be nice to see > that 100% of P1 tests are finished on en-US, de, fr, > it, and ja. > > > ---------------------------------- > > Grrr, are you lost? I am a bit. What to do now? :-) > > I still do not see the best solution. Let's compare some stuff by > importance: > > a) priority is more important than component > > + we do not have time to organize a test run because of a single > fix > => most of the time we will test more fixes => all components > > b) platform is not much important > > + there will be only few platform dependent test cases > + I am not aware about any test that would be platform > and language dependent > => there will only platform dependent functional tests > > We could mention the requirement in the summary or test > description. > > e) locale specific tests are questionable > > + we might limit the test cases to certain locales in summary or > test description > > c) doing test is the most important activity because it will be done by > many people most of the time > > d) test results analyze must be easy: because we want to often check > where we are > > e) creating test run must take only few minutes > > f) it must be reasonable to maintain localized test cases; it would be a > nightmare otherwise > > g) the balance of the number of branches, testgroups, subgroups, > testcases in a group is important; long lists are always hard to cope > with > > > --------------------------------------- > > > Grrr, I am even more lost now. How are you? Maybe, I should sleep to > sort the ideas and see a better picture :-) > > Ok, let's try it from another point of view. There are not that many > reasonable solutions after all. > > > i) Product is "LibreOffice" > > ii) Priority must be only in "Branch" or "TestGroup" because we want to > create test run by priority. > > Branch is ugly because it would force many test runs => makes more > actions complicated > > => Priority must by in "TestGroup" name > > > iii) We do not want "Component in "Branch" name because it would create > too many test runs. > > "Component" is not super important. > > => "Component" can be in "Testgroup", "Subgroup" or "Test case > summary" or nowhere. > > > iv) We want to clearly split language-dependent, > language-independent and feature tests. Because they need different > approach when creating and testing. > > We do not want "Language" in "Branch name" because we do not want > 100 branches for different languages. It would be nightmare to > create test run, do testing, analyze it. > > We do not want Language in "Test case summary" because it would be > hard to search "en-US" test cases inside the flood of other test > cases. > > => Language must be in "TestGroup" or "Subgroup" name > > v) "Platform" affects only few tescases > > => mention platform only in test case name > > > > --------------------------------------------------------------------------- > > > So, where are we? Still in a tunel or do we see a light? > > ok, let's do some counting: > > I) Criteria has variants: > > + priority => 4 variants > + category => 7 variants > + meaning => 3 variants > + language => 100 variants > + platform => 3 variants > > > II) Criteria and number of test cases: > > The expected number of test cases is in brackets. > > + priority: P1(10), P2(50), P3(150), P4(?) > + category: General(100), Writer(100), Calc(100),Math(20) > + meaning: function(500), l10n(20 per language), feature(100) > + platform: Linux(10), MAC(10), Windows(10) > > > III) Litmus levels and numbers: > > We want the following numbers in Litmus: > > + product: 1 > + branch: 1-5 per LO version > + testgroup: 3-20 (100 per language would be a nightmare) > + subgroup: 3-20 (max 100 if we want to put language there) > > > --------------------------------------------------- > > I think that we come somewhere. I am sure that we will need a > compromise: > > > A. 3 branches, 4 test groups, 7/100 subgroups: > ============================================== > > + LibreOffice product (three branches per release): > > + master function test branch (4 testgroups): > + P1 testgroup (7 subgroups) > + General > + Base > + Calc > + Draw > + Impress > + Math > + Writer > + P2 testgroup (7 subgroups) > + General > + Base > + ... > + P3 ... > + P4 ... > > + master l10n test branch (4 testgroups) > + P1 test group (100 subgroups) > + en > + de > + fr > + ... > + P2 test group (100 subgroups) > + en > + ... > + P3 ... > + P4 ... > + master feature test > + ??? > > + 3.5 function test branch > + 3.5 l10n ... > + 3.5 feature ... > + 3.6 function ... > + .. > > + ... > > > B. 1 branch, 9-12 test groups, 7/100 subgroups > ============================================== > > LibreOffice product (one branch per release): > > + master branch (9-12 test groups): > + P1 - function testgroup (7 subgroups) > + General > + Base > + Calc > + Draw > + Impress > + Math > + Writer > + P1 - l10n testgroup (100 subgroups) > + en > + de > + fr > + ... > + P2 - function tesgroup(7 subgroups) > + General > + Base > + ... > + P2 - l10n testgroup (100 subgroups) > + en > + de > + P3 - function tesgroup(7 subgroups) > + P3 - l10n testgroup (100 subgroups) > + P4 - function tesgroup(7 subgroups) > + P4 - l10n testgroup (100 subgroups) > + P4 - feature tesgroup (7 subgroups) (only P4?) > + General > + Base > + Calc > > > + 3.5 branch (9-12 test groups) > + P1 - function > + P1 - l10n > + ... > + 3.6 branch (9-12 test groups) > + ... > > > > C. 1 branch, 28 test groups, 101 subgroups > ========================================== > > + LibreOffice product (one branch per release) > > + master branch (28 testgroups) > + P1 - General (101 subgroups) > + function tests > + l10n tests - en > + l10n tests - de > + l10n tests - fr > + ... > + P1 - Base(101 subgroups) > + function tests > + l10n tests - en > + l10n tests - de > + ... > + P1 - Calc(101 subgroups) > + ... > + P2 -General(101 subgroups) > + P2 - Base(101 subgroups) > + ... > > + 3.5 branch (28 testgoups) > + 3.5 branch (28 testgroups) > + ... > > > ---------------------------------------------------------------------------- > > Do you see any other reasonable variant. > > Anyway,I prefer the variant B because: > > + A does not have any real advantage, it just forces more test > runs and makes things more complicated > + B is a nice compromise for any action > + C does not split well function and l10n tests. It has too many > testgroups, so it would be a nightmare to check test results > and localize the l10n tests. The only advantage is that it > allows to create test run for Writer but I am not sure it if > is worth the effort > > ----------------------------------------------------------------------------- > > Sigh, one more important thing. We might need to create the feature > branch earlier. Features are different for every release. We do not want > them in the master branch. We want them only in the release branch. We > want to branch the regression tests later from the master branch, > though. > > I suggest to create a variant of B. > > > D. 2 branches, 8 testgroups, 7/100 subgroups: > > + LibreOffice product (two branches per release): > > + master branch (8 test groups): > + P1 - function testgroup (7 subgroups) > + General > + Base > + Calc > + Draw > + Impress > + Math > + Writer > + P1 - l10n testgroup (100 subgroups) > + en > + de > + fr > + ... > + P2 - function tesgroup(7 subgroups) > + General > + Base > + ... > + P2 - l10n testgroup (100 subgroups) > + en > + de > + P3 - function tesgroup(7 subgroups) > + P3 - l10n testgroup (100 subgroups) > + P4 - function tesgroup(7 subgroups) > + P4 - l10n testgroup (100 subgroups) > > + 3.5 regression branch (9-12 test groups) > + P1 - function > + P1 - l10n > + ... > + 3.5 feature test branch > + ??? > + 3.6 regression branch (8 testgroups) > + 3.6 feature test branch > + ... > > > Does anyone reach this point? > > Did I miss something any interesting criteria or condition? > Is the preferred solution really good in all actions? > Could we solve the localization stuff a better way? How? > > Do you have a power to tell me what you think about it? :-) > > > Best Regards, > Petr > _______________________________________________ List Name: Libreoffice-qa mailing list Mail address: [email protected] Change settings: http://lists.freedesktop.org/mailman/listinfo/libreoffice-qa Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/ Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette List archive: http://lists.freedesktop.org/archives/libreoffice-qa/
