Re: [swift-corelibs-dev] [xctest] Removing outliers from performance tests

2015-12-13 Thread Mike Ferris via swift-corelibs-dev 
I think that there’s a lot of room for improvement in how we measure and 
analyze perf test results. And I like some of your ideas of adaptive numbers of 
runs and what to do with outlier results, etc… that you presented below.

We would very much like to not diverge the API of the CoreLibs XCTest and 
Xcode’s.

There may be a bit more room for some variation in the behavior. So talking 
about doing something where the number of runs varies dynamically not via API 
but through smarter execution mechanisms, or doing better statistical analysis 
such as removing outlier results, this would introduce some change in the 
results that might be reported between the two implementations, but it would 
not give rise to people writing tests that would not work cross-platform.

Further, there’s always room for discussion of taking such ideas, and even API 
additions and supporting them in Xcode’s XCTest as well. But that sort of 
discussion will have to include the practicalities of schedule and resource on 
the Xcode side (and, to some unavoidable extent, there will be aspects to that 
discussion that will not be as transparent since we’ll be weighing that work 
against other work for the testing team that cannot be discussed as freely.)

This year, our primary goal for the core libraries is to broaden the 
implementation of the existing APIs in the OS X versions of the frameworks. For 
XCTest, we would also love to come up with some better answer for test 
discovery and potentially elicit help from the community in achieving that. 
Beyond that, the Xcode team’s bandwidth for incorporating other things that 
would necessaitate change to the Xcode XCTest is going to be limited.

Mike


> On Dec 12, 2015, at 6:10 PM, Drew Crawford via swift-corelibs-dev 
>  wrote:
> 
>> Unfortunately our corelibs implementation of XCTest isn’t ready yet for 
>> performance testing.
> 
> That's why I'm here; I'm taking the temperature on implementing it.  I'm at 
> the pain level where I need a solution in the next several months, even if 
> the solution is to code it up myself.  My tests have failed 10x over this so 
> far today.
> 
> I think the real question is, if I did implement basic performance testing, 
> and I did implement a variable-sized window of runs, would that departure 
> from the Old XCTest behavior (which uses 10 runs) disqualify the PR?  It's a 
> basic compatibility question about how close we need to follow the Old XCTest 
> behavior.
> 
> e.g. if XCS wanted to migrate to corelibs-xctest and it used variable #s of 
> runs, presumably that would be an undertaking for the XCS team.  But I don't 
> know whether those concerns (if they exist) play a role in what this project 
> decides to do.
> 
> I'm going to do something on this problem eventually, unless someone else 
> solves it first.  I'm just trying to work out whether I can do something 
> upstream or whether this is a better candidate for an independent effort.
> 
> 
> 
> 
> 
> 
> 
> ___
> swift-corelibs-dev mailing list
> [email protected]
> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

___
swift-corelibs-dev mailing list
[email protected]
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Re: [swift-corelibs-dev] Upcoming holiday schedule

2015-12-18 Thread Mike Ferris via swift-corelibs-dev 
I will also be out starting tomorrow through the end of the month which means 
XCTest will get little coverage during that time.

Mike

> On Dec 17, 2015, at 9:18 AM, Philippe Hausler via swift-corelibs-dev 
>  wrote:
> 
> I will probably be around a bit more than Tony over the break; I will try to 
> keep up with the pull requests.
> 
> There are a few things that I think will speed up the process so that we can 
> keep things running well:
> 
> Discrete commits that implement a specific thing; reviewing two different 
> spots sometimes is a bit hard to follow it all.
> Review feedback from the community helps a ton
> Rebasing commits from master so that there are no merge conflicts makes my 
> job easier.
> Unit tests are awesome!
> 
> These are rules of thumb that actually can apply easily beyond the break.
> 
>> On Dec 17, 2015, at 9:11 AM, Tony Parker via swift-corelibs-dev 
>> mailto:[email protected]>> wrote:
>> 
>> Indeed! I hope the mismatch doesn’t cause us to fall too far behind. ;)
>> 
>> - Tony
>> 
>>> On Dec 17, 2015, at 9:10 AM, Brian Gesiak >> > wrote:
>>> 
>>> Thanks for the head's up! Conversely, I imagine that for many people, 
>>> holidays from work are precisely the time to contribute! :)
>>> 
>>> - Brian Gesiak
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Thu, Dec 17, 2015 at 9:03 AM -0800, "Tony Parker via swift-corelibs-dev" 
>>> mailto:[email protected]>> wrote:
>>> 
>>> Hello fellow contributors,
>>> 
>>> As you are aware, we are fast approaching the year-end holiday season. Here 
>>> at Apple, many of us are taking some time off starting next week through 
>>> the new year. As a result, it is likely that some pull requests will take 
>>> more time than expected to review and merge. Even if some of us are on 
>>> vacation, this is a great opportunity for anyone to jump in and help review 
>>> PRs!
>>> 
>>> Thanks and happy new year,
>>> - Tony
>>> 
>>>  
>>> ___
>>> swift-corelibs-dev mailing list
>>> [email protected] 
>>> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev 
>>> 
>> 
>>  ___
>> swift-corelibs-dev mailing list
>> [email protected] 
>> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev 
>> 
>  ___
> swift-corelibs-dev mailing list
> [email protected] 
> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev 
> 
___
swift-corelibs-dev mailing list
[email protected]
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

[swift-corelibs-dev] Swift3 rewriting for XCTest APIs

2016-04-15 Thread Mike Ferris via swift-corelibs-dev 
Swift3 rewriting for XCTest APIs

For the most part, the automatic heuristics seem to be doing the right things 
for Xcode’s XCTest.  We have identified a few cases where we should perhaps 
discuss doing a little extra. Almost everything seems like it imports 
reasonably according to the API guidelines 
(https://swift.org/documentation/api-design-guidelines/ 
). The few cases that 
we think need tweaking are detailed here.

These are the changes from how the API imports automatically that we are 
considering. I would love to hear any comments or suggestions.

addTest[Run]()

There’s a glitch in the importer that is making XCTestSuioteRun’s addTestRun: 
import as addTest. We’re tracking this as a bug in the importer. In the 
meantime, I would fix this in Xcode’s XCTest explicitly.

Existing imported method
XCTestSuiteRun.addTest(testRun: XCTestRun)  

Suggested new name
XCTestSuiteRun.addTestRun(testRun: XCTestRun)

Bound(by: vs. (boundBy:

Here, the importer just isn't smart enough to detect that the prepositional 
phrase that really identifies the argument is "boundBy", not "by”. We’re also 
tracking this as a possible thing to fix in the importer. In the meantime, I 
would fix this in Xcode’s XCTest explicitly.

Existing imported method
XCUIElementQuery.elementBound(by index: UInt) -> XCUIElement

Suggested new name
XCUIElementQuery.element(boundBy index: UInt) -> XCUIElement

Query "matching..."

This next little family is a bit more subtle. The first two have a noun which 
just restates the type, so by the guidelines we would remove them. Technically 
the last one's noun is a role, not a type and so what the importer is doing is 
correct. But it feels inconsistent if we remove the type names but leave that 
one alone. To make it a little more consistent, the suggestion is to move 
identifier inside the parens as a required label so all these related functions 
have the same "outside the parens" part of their names.

Existing imported method
XCUIElementQuery.matchingPredicate(predicate: NSPredicate) -> 
XCUIElementQuery
XCUIElementQuery.matchingType(elementType: XCUIElementType, identifier: 
String?) -> XCUIElementQuery
XCUIElementQuery.matchingIdentifier(identifier: String) -> 
XCUIElementQuery

Suggested new name
XCUIElementQuery.matching(predicate: NSPredicate) -> XCUIElementQuery
XCUIElementQuery.matching(elementType: XCUIElementType, identifier: 
String?) -> XCUIElementQuery
XCUIElementQuery.matching(identifier identifier: String) -> 
XCUIElementQuery

___
swift-corelibs-dev mailing list
[email protected]
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev