The risk of the current CD via Jenkins is known and was accepted as part of adopting Jenkins. The solution for the initial issue - no longer publishing to pypi - is to add a step to the existing CD pipeline which publishes the package to the s3 bucket instead of pypi.
-Marco Pedro Larroy <[email protected]> schrieb am Mi., 8. Jan. 2020, 21:55: > I understand your point. But you don't provide an alternative, and building > binary releases from the CI jenkins as it is today is a very bad idea since > it's an unsafe environment. I think it's fair to ask if you are vetoing > using codebuild for nightly releases you could provide an alternative > solution (for example Apache hosted Jenkins) or anything else. As you are > well aware non-committers can't communicate with Apache Infra or make > requests, so the onus is on you or other Apache person to provide a > solution that aligns with Apache values. > > So far I see Sam trying to help with codebuild managed binary releases and > this is taken as a tinfoil hat corporate conspiracy. It's a pity that you > claim to endorse Apache values but not support what's best for the project, > which is to have things clean and in working order. I don't think users > care where the binary releases are hosted. > > Pedro. > > On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <[email protected]> > wrote: > > > Apache only cares about source releases as far as official releases are > > concerned. But Apache also cares about it's brand and image. You are > right > > that anybody can compile an Apache project and distribute it, but it's > > under the PMCs control what can be advertised as official. This includes > > the following examples: > > > > - The official MXNet pypi, dockerhub, maven, etc account > > - The MXNet website > > - anything advertising to be MXNet > > > > If you publish a binary release and call it "AwesomeSpaghettiBolognese" > > while it's MXNet under the hood, that's totally in line with the Apache > > license. But if you decide to publish an MXNet branded package, then > that's > > covered by the brand protection. I won't go into much more detail about > > legal reasons since that's not helping this discussion. > > > > I personally am vetoing a company-owned distribution channel to be > > advertised on the MXNet website or any official documentation. Also, I'd > > like to make sure that users do not mistake it for being a release that > is > > affiliated or endorsed by Apache MXNet. > > > > We are taking a step back here and it's a pity to see that some people > are > > still not endorsing the Apache values. This will be my last email > regarding > > that topic and I will only follow up with actions after the 15th of > January > > has been reached. > > > > Best regards > > Marco > > > > > > Pedro Larroy <[email protected]> schrieb am Sa., 4. Jan. > 2020, > > 02:38: > > > > > Hey Marco. > > > > > > As far as I have learned from other Apache mailing lists while lurking > is > > > that Apache only cares about making source releases, binaries are a > > > courtesy to users that some projects decide to do, but I'm not sure I > > > understand your concerns regarding the PMC and what exactly are you > > vetoing > > > here, since everyone can compile, build and package our project as per > > the > > > open source license. I would suggest to have a constructive approach > and > > > see how we can make this happen for the best of the project, specially > > > since somebody is volunteering to help with this and dedicate valuable > > > compute resources and people's time. > > > > > > Regarding manual changes I don't see any need to have access to a code > > > build control plane for *anybody*, for several reasons, first is that > > > manual access to production account is a discouraged practice and are > > best > > > managed through pipeline deployments, second is that Code build is a > > hosted > > > service which is basically just using a build description file to do > the > > > work, there's no need to do any manual fiddling or triggering. If all > the > > > CD and description files are in the apache repository you can use your > > own > > > account or compute resources to do your own build flavor if you so > > desire. > > > > > > Is your proposal to host this in Apache infrastructure? Maybe I'm > > missing > > > something on this conversation > > > > > > Pedro. > > > > > > > > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <[email protected] > > > > > wrote: > > > > > > > Sam, while I understand that this solution was developed out of > > > necessity, > > > > my question why a new system has been developed instead of fixing the > > > > existing one or adapting the solution. CodeBuild is a scheduler in > the > > > same > > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins > > > without > > > > much hassle. > > > > > > > > I'm not volunteering for this - why should I? The role of a PMC > member > > is > > > > to steer the direction of the project. Just because a manager points > > > > towards a certain direction, if doesn't mean that they're going to do > > it. > > > > > > > > Apparently there was enough time at some point to develop a new > > solution > > > > from scratch. It might have been a solution for your internal team > and > > > > that's fine, but upgrading it "temporarily" to be the advertised way > on > > > the > > > > official website is something different. > > > > > > > > I won't argue about how the veto can be enforced. I think it's in the > > > best > > > > interest of the project if we try working on a solution instead of > > > spending > > > > time on trying to figure out the power of the PMC. > > > > > > > > Pedro, that's certainly a step towards the right direction. But > > > committers > > > > would also need access to the control plane of the system - to > trigger, > > > > stop and audit builds. We could go down that road, but i think the > > fewer > > > > systems, the better - also for the sake of maintainability. > > > > > > > > Best regards, > > > > Marco > > > > > > > > > > > > > > > > Pedro Larroy <[email protected]> schrieb am Fr., 3. Jan. > > > 2020, > > > > 20:55: > > > > > > > > > I'm not involved in such efforts, but one possibility is to have > the > > > yaml > > > > > files that describe the pipelines for CD in the Apache > repositories, > > > > would > > > > > that be acceptable from the Apache POV? In the end they should be > > very > > > > thin > > > > > and calling the scripts that are part of the CD packages. > > > > > > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Agree, but the question how a non Amazonian is able to maintain > and > > > > > access > > > > > > the system is still open. As it stands right now, the community > has > > > > > taken a > > > > > > step back and loses some control if we continue down that road. > > > > > > > > > > > > I personally am disapproving of that approach since committers > are > > no > > > > > > longer in control of that process. So far it seems like my > > questions > > > > were > > > > > > skipped and further actions have been taken. As openness and the > > > > > community > > > > > > having control are part of our graduation criteria, I'm putting > in > > my > > > > > veto > > > > > > with a grace period until 15th of January. Please bring the > system > > > > into a > > > > > > state that aligns with Apache values or revert the changes. > > > > > > > > > > > > -Marco > > > > > > > > > > > > Pedro Larroy <[email protected]> schrieb am Fr., 3. > > Jan. > > > > > 2020, > > > > > > 03:33: > > > > > > > > > > > > > CD should be separate from CI for security reasons in any case. > > > > > > > > > > > > > > > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Could you elaborate how a non-Amazonian is able to access, > > > maintain > > > > > and > > > > > > > > review the CodeBuild pipeline? How come we've diverted from > the > > > > > > community > > > > > > > > agreed-on standard where the public Jenkins serves for the > > > purpose > > > > of > > > > > > > > testing and releasing MXNet? I'd be curious about the issues > > > you're > > > > > > > > encountering with Jenkins CI that led to a non-standard > > solution. > > > > > > > > > > > > > > > > -Marco > > > > > > > > > > > > > > > > > > > > > > > > Skalicky, Sam <[email protected]> schrieb am Sa., > 7. > > > Dez. > > > > > > 2019, > > > > > > > > 18:39: > > > > > > > > > > > > > > > > > Hi MXNet Community, > > > > > > > > > > > > > > > > > > We have been working on getting nightly builds fixed and > made > > > > > > available > > > > > > > > > again. We’ve made another system using AWS CodeBuild & S3 > to > > > work > > > > > > > around > > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is currently > > > building > > > > > all > > > > > > > the > > > > > > > > > flavors and publishing to an S3 bucket here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > > > > > > > > > > > > > > > > > > There are folders for each set of nightly builds, try out > the > > > > > wheels > > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT > (9:30am > > > GMT) > > > > > and > > > > > > > > > arrive in the bucket 30min-2hours later. Inside each folder > > are > > > > the > > > > > > > > wheels > > > > > > > > > for each flavor of MXNet. Currently we’re only building for > > > > linux, > > > > > > > builds > > > > > > > > > for windows/Mac will come later. > > > > > > > > > > > > > > > > > > If you want to download the wheels easily you can use a URL > > in > > > > the > > > > > > form > > > > > > > > of: > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > Heres a set of links for today’s builds > > > > > > > > > > > > > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > (mxnet-mkl > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > > > > > > > > > > > > > > > > > ) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > (mxnet-cuXXX > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > > > > > > > > > > > > > > > > > > ) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > (mxnet-cuXXXmkl > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl > > > > > > > > > > > > > > > > > > ) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > You can easily install these pip wheels in your system > either > > > by > > > > > > > > > downloading them to your machine first and then installing > by > > > > > doing: > > > > > > > > > > > > > > > > > > pip install /path/to/downloaded/wheel.whl > > > > > > > > > > > > > > > > > > Or you can install directly by just giving the link to pip > > like > > > > > this: > > > > > > > > > > > > > > > > > > pip install > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > Credit goes to everyone involved (in no particular order) > > > > > > > > > Rakesh Vasudevan > > > > > > > > > Zach Kimberg > > > > > > > > > Manu Seth > > > > > > > > > Sheng Zha > > > > > > > > > Jun Wu > > > > > > > > > Pedro Larroy > > > > > > > > > Chaitanya Bapat > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > Sam > > > > > > > > > > > > > > > > > > > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard > > > > > > <[email protected] > > > > > > > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > > > > > We don't loose pip by hosting on S3. We just don't host > > nightly > > > > > > > releases > > > > > > > > > on Pypi > > > > > > > > > servers and mirror them to several hundred mirrors > > immediately > > > > > after > > > > > > > each > > > > > > > > > build > > > > > > > > > is published which is very expensive for the Pypi project.. > > > > People > > > > > > can > > > > > > > > > still > > > > > > > > > install the nightly builds with pip by specifying the -f > > > option. > > > > > > > > > > > > > > > > > > Uploading weekly releases to Pypi will reduce the cost for > > Pypi > > > > by > > > > > > ~75% > > > > > > > > > [1]. It > > > > > > > > > may be acceptable to Pypi, but does it make sense for us? > I'm > > > not > > > > > > > > convinced > > > > > > > > > weekly release on Pypi is a good idea. Consider one release > > is > > > > > buggy, > > > > > > > > > users will > > > > > > > > > need to wait for 7 days for a fix. It doesn't provide good > > user > > > > > > > > experience. > > > > > > > > > If someone has a stronger conviction about the value of > > weekly > > > > > > releases > > > > > > > > on > > > > > > > > > Pypi, > > > > > > > > > that person shall please go ahead and propose it in a > > separate > > > > > > > discussion > > > > > > > > > thread. > > > > > > > > > > > > > > > > > > Currently we don't have generally working nightly builds to > > > Pypi > > > > > and > > > > > > > as a > > > > > > > > > matter > > > > > > > > > of fact we know that we can't have them due to Pypi's > policy > > > and > > > > > our > > > > > > > > > apparent > > > > > > > > > need for large binaries. Given this fact and that no > > objection > > > > was > > > > > > > raised > > > > > > > > > by > > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus > on > > > > > > stopping > > > > > > > > > upload > > > > > > > > > attempts of nightly builds to Pypi. > > > > > > > > > > > > > > > > > > With consensus established, we can change the CI job to > stop > > > > trying > > > > > > to > > > > > > > > > upload > > > > > > > > > the nightly builds and then request Pypi to increase the > > limit. > > > > > Then > > > > > > we > > > > > > > > > have one > > > > > > > > > less blocker for the 1.6 release. > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > [1]: Lower cost due to less releases, but higher cost due > to > > > > 500MB > > > > > -> > > > > > > > > 800MB > > > > > > > > > limit increase. Assuming that the limit increase translates > > > into > > > > > > > actually > > > > > > > > > larger > > > > > > > > > binaries. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote: > > > > > > > > > Are weekly releases an option? It was brought up as concern > > > that > > > > we > > > > > > > might > > > > > > > > > lose pip as a pretty common distribution channel where > people > > > > > consume > > > > > > > > > nightly builds. I don't feel like that concern has been > > > properly > > > > > > > > addressed > > > > > > > > > so far. > > > > > > > > > > > > > > > > > > -Marco > > > > > > > > > > > > > > > > > > Lausen, Leonard <[email protected]<mailto: > > > > > > > > > [email protected]>> schrieb am Mi., 4. Dez. 2019, > > > > > > > > > 04:09: > > > > > > > > > > > > > > > > > > As a simple POC to test distribution, you can try > installing > > > > MXNet > > > > > > > based > > > > > > > > on > > > > > > > > > these 3 URLs: > > > > > > > > > > > > > > > > > > pip install --no-cache-dir > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > pip install --no-cache-dir > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > pip install --no-cache-dir > > > > https://d19zq12jzu4w95.cloudfront.net/ > > > > > > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > > > > > > > > > > > > > > > > > > > > where --no-cache-dir prevents caching the downloaded file, > > for > > > > the > > > > > > > > purpose > > > > > > > > > of > > > > > > > > > testing. (cu101 chosen based on large size) > > > > > > > > > > > > > > > > > > The first URL uses standard S3 bucket in US. The second > uses > > S3 > > > > > > > > Accelerate > > > > > > > > > based > > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm > > > adding > > > > > the > > > > > > > > third > > > > > > > > > URL, > > > > > > > > > as S3 Accelerate may or may not use all new CloudFront > > > endpoints > > > > > yet. > > > > > > > > > > > > > > > > > > Regarding voting: Uploading to Pypi is currently > impossible, > > > > which > > > > > > is a > > > > > > > > > reality > > > > > > > > > (so there is no option to continue as we do currently). > Pypi > > > > folks > > > > > > > > > indicated > > > > > > > > > they will unblock our uploads to Pypi once we stop > uploading > > > > > nightly > > > > > > > > > releases > > > > > > > > > and taking up 20% of their ressources [1]. > > > > > > > > > > > > > > > > > > If there are any shortcomings or problems identified with > > > > uploading > > > > > > to > > > > > > > > S3, > > > > > > > > > we > > > > > > > > > can work to address them. But for now, status quo is broken > > and > > > > > this > > > > > > > > seems > > > > > > > > > the > > > > > > > > > only solution addressing Pypi's problem. > > > > > > > > > > > > > > > > > > I don't mind if you state that you object to lazy consensus > > and > > > > > > start a > > > > > > > > > vote. If > > > > > > > > > your "maybe [...] start a proper vote" was supposed to be > an > > > > > > objection > > > > > > > to > > > > > > > > > lazy > > > > > > > > > consensus, please state so clearly (I'm not sure if "maybe" > > > > > qualifies > > > > > > > as > > > > > > > > > objection). Though I think it only makes sense with at > least > > 2 > > > > > > options > > > > > > > to > > > > > > > > > vote > > > > > > > > > on. Status quo is not a meaningful option, as it is already > > > > broken. > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706 > > > > > > > > > > > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote: > > > > > > > > > Excellent! Could we maybe come up with a POC and a quick > > > writeup > > > > > and > > > > > > > then > > > > > > > > > start a proper vote after everyone verified that it covers > > > their > > > > > > > > > use-cases? > > > > > > > > > -Marco > > > > > > > > > > > > > > > > > > Sheng Zha <[email protected]> schrieb am Di., 3. Dez. > > 2019, > > > > > 19:24: > > > > > > > > > > > > > > > > > > Yes, there is. We can also make it easier to access by > using > > a > > > > > > > > > geo-location based DNS server so that China users are > > directed > > > to > > > > > > that > > > > > > > > > local mirror. The rest of the world is already covered by > the > > > > > global > > > > > > > > > cloudfront. > > > > > > > > > > > > > > > > > > -sz > > > > > > > > > > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu < > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > Isn't there an s3 endpoint in Beijing? > > > > > > > > > > > > > > > > > > It seems like this topic still warrants some discussion and > > > thus > > > > > I'd > > > > > > > > > > > > > > > > > > prefer > > > > > > > > > if we don't move forward with lazy consensus. > > > > > > > > > > > > > > > > > > -Marco > > > > > > > > > > > > > > > > > > Tao Lv <[email protected]> schrieb am Di., 3. Dez. 2019, > > > 14:31: > > > > > > > > > > > > > > > > > > * For pypi, we can use mirrors. > > > > > > > > > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > > As we have many users in China, I'm considering the > > > > > > > > > accessibility of > > > > > > > > > S3. > > > > > > > > > For pip, we can mirrors. > > > > > > > > > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > > > > > > > > > > > <[email protected] > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > I would like to remind everyone that lazy consensus is > > assumed > > > > > > > > > if no > > > > > > > > > objections > > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been > > some > > > > > > > > > > > > > > > > > > discussion > > > > > > > > > about > > > > > > > > > the proposal, but to my understanding no objections were > > > > > > > > > raised. > > > > > > > > > If the proposal is accepted, MXNet releases would be > > installed > > > > > > > > > via > > > > > > > > > pip install mxnet > > > > > > > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > > > > > > > pip install --pre mxnet > > > > > > > > > > > > > > > > > > (or with the respective cuda version specifier appended > etc.) > > > > > > > > > > > > > > > > > > To obtain releases built automatically from the master > > branch, > > > > > > > > > users > > > > > > > > > would need > > > > > > > > > to specify something like "-f > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option > > to > > > > > > > > > pip. > > > > > > > > > Best regards > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote: > > > > > > > > > Hi MXNet Community, > > > > > > > > > > > > > > > > > > since more than 2 months our binary Python nightly releases > > > > > > > > > > > > > > > > > > published > > > > > > > > > on Pypi > > > > > > > > > are broken. The problem is that our binaries exceed Pypi's > > > > > > > > > size > > > > > > > > > limit. > > > > > > > > > Decreasing the binary size by adding compression breaks > > > > > > > > > > > > > > > > > > third-party > > > > > > > > > libraries > > > > > > > > > loading libmxnet.so > > > > > > > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193 > > > > > > > > > Sheng requested Pypi to increase their size limit: > > > > > > > > > https://github.com/pypa/pypi-support/issues/50 > > > > > > > > > > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet > > > > > > > > > binaries > > > > > > > > > with > > > > > > > > > nightly > > > > > > > > > release to Pypi] is the bandwidth consumed when several > > > > > > > > > hundred > > > > > > > > > mirrors > > > > > > > > > attempt > > > > > > > > > to mirror each release immediately after it's published". > So > > > > > > > > > Pypi > > > > > > > > > is > > > > > > > > > not > > > > > > > > > inclined to allow us to upload even larger binaries on a > > > > > > > > > nightly > > > > > > > > > schedule. > > > > > > > > > Their compromise is to allow it on a weekly cadence. > > > > > > > > > > > > > > > > > > However, I would like the community to revisit the > necessity > > > > > > > > > of > > > > > > > > > releasing pre- > > > > > > > > > release binaries to Pypi on a nightly (or weekly) cadence. > > > > > > > > > > > > > > > > > > Instead, we > > > > > > > > > can > > > > > > > > > release nightly binaries ONLY to a public S3 bucket and > > > > > > > > > instruct > > > > > > > > > users > > > > > > > > > to > > > > > > > > > install from there. On our side, we only need to prepare a > > > > > > > > > html > > > > > > > > > document that > > > > > > > > > contains links to all released nightly binaries. > > > > > > > > > Finally users will install the nightly releases via > > > > > > > > > > > > > > > > > > pip install --pre mxnet-cu101 -f > > > > > > > > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > > > > > > nightly.html > > > > > > > > > > > > > > > > > > Instead of > > > > > > > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > > > > > > > Of course proper releases and release candidates should > > > > > > > > > still be > > > > > > > > > made > > > > > > > > > available > > > > > > > > > via Pypi. Thus releases would be installed via > > > > > > > > > > > > > > > > > > pip install mxnet-cu101 > > > > > > > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > > > > > > > This will substantially reduce the costs of the Pypi > project > > > > > > > > > and > > > > > > > > > in > > > > > > > > > fact > > > > > > > > > matches > > > > > > > > > the installation experience provided by PyTorch. I don't > > > > > > > > > think the > > > > > > > > > benefit of > > > > > > > > > not including "-f > > > > > > > > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > > > > > > > > > matches the costs we currently externalize to the Pypi > team. > > > > > > > > > > > > > > > > > > This suggestion seems uncontroversial to me. Thus I would > > > > > > > > > like to > > > > > > > > > start > > > > > > > > > lazy > > > > > > > > > consensus. If there are no objections, I will assume lazy > > > > > > > > > > > > > > > > > > consensus on > > > > > > > > > stopping > > > > > > > > > nightly releases to Pypi in 72hrs. > > > > > > > > > > > > > > > > > > Best regards > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
