(pre)dexing is a known issue. I'm not sure how much faster it's going to be in 0.8 with parallel dexing since you have a lot of projects and you can use parallelization there already.
18secs for a manifest merge seems like a crazy amount of time. do you have a lot of manifest data to merge? I'm going to build a fake multi-project setup with 100s of project to look at scaling issues. Can you tell me a bit more about your inter-project dependencies? Basically do you have 1 apps and 139 flat libs that don't depend on each other, or do you have a fairly deep dependency graph with lots of transitive dependencies? On Thu, Jan 16, 2014 at 12:09 PM, Ashton Cummings <[email protected]>wrote: > *"clean build" with no changes = 3h1m53.88s* > > Top 15 tasks (roughly the same for each project) > :projectA:preDexRelease 1m26.29s > :projectA:preDexDebug 1m25.14s > :projectA:processDebugManifest 18.452s > :projectA:processReleaseManifest 17.705s > :projectA:lint 10.699s > :projectA:mergeReleaseResources 8.264s > :projectA:dexDebug 7.742s > :projectA:dexRelease 7.449s > :projectA:mergeDebugResources 4.303s > :projectA:compileDebugJava 1.672s > :projectA:processReleaseResources 1.662s > :projectA:processDebugResources 1.325s > :projectA:compileReleaseJava 1.219s > :projectA:packageDebug 1.179s > :projectA:packageRelease 0.923s > > > *clean assembleDebug with pre-dex turned off = 58m58.29s* > > Top 15 tasks (roughly the same for each project) > :projectA:dexDebug 56.567s > :projectA:mergeDebugResources 5.135s > :projectA:processDebugManifest 3.517s > :projectA:compileDebugJava 1.579s > :projectA:packageDebug 1.174s > :projectA:processDebugResources 0.972s > :projectA:processDebugJavaRes 0.373s UP-TO-DATE > :projectA:prepareLibrary1UnspecifiedLibrary 0.340s > :projectA:prepareLibrary2UnspecifiedLibrary 0.218s > :projectA:compileDebugAidl 0.057s > :projectA:compileDebugRenderscript 0.046s > :projectA:prepareLibrary3UnspecifiedLibrary 0.045s > :projectA:prepareLibrary4UnspecifiedLibrary 0.043s > :projectA:prepareLibrary5UnspecifiedLibrary 0.029s > :projectA:prepareLibrary6UnspecifiedLibrary 0.028s > > I can not find my data for the parallel run. I will do another one of > those runs and post the data. Let me know if this is helpful for you. > > Thanks > > > On Thursday, January 16, 2014 1:46:45 PM UTC-6, Ashton Cummings wrote: >> >> I have been using --profile to figure out the bottlenecks. However, due >> to the nature of our project, i can not share that information. =/ >> >> I can give you times and stuff, just not project names and details. So, >> unless i modify the data substantially, i can not share the profile info. >> Sorry. >> >> However, i will give you the build time improvements and where the >> bottlenecks are. If you need any specifics, let me know. I will try and get >> you details that i can. >> >> I will get the info to you as soon as i can. I have so much profile data >> now, i am going to have to run through it and see if i can find the correct >> data. Might just do the runs again to make sure to get you the correct data. >> >> On Thursday, January 16, 2014 12:21:46 PM UTC-6, Xavier Ducrohet wrote: >>> >>> Do you remember the time gain you had when disabling pre-dexing? >>> >>> If you wouldn't mind. I'd love to get the output of running gradle with >>> --profile. It will create a report in the build folder of the main project >>> which will show me build time for all projects and all tasks in all >>> projects. >>> >>> >>> On Thu, Jan 16, 2014 at 10:06 AM, Ashton Cummings <[email protected]>wrote: >>> >>>> I hope that we can. Our alternatives that i am testing are Maven and >>>> ANT. Right now our maven approach is showing times that are comparable to >>>> our ANT. I personally want to stick with gradle, however, the decision is >>>> not up to me. I just give the higher ups my results and conclusions. =) >>>> >>>> I will keep an eye out for those performance improvements and will keep >>>> our people informed. >>>> >>>> Thank you for your time and assistance. If you figure out anything else >>>> or have any other suggestions, please continue to contact me. I really hope >>>> to convince them to move to gradle. >>>> >>>> >>>> On Thursday, January 16, 2014 11:13:32 AM UTC-6, Xavier Ducrohet wrote: >>>> >>>>> BTW, we are still a ways off from 1.0. I totally understand not >>>>> wanting to switch to Gradle right now (especially given your build times), >>>>> but if you decide to not switch right now, please don't plan to stick with >>>>> Ant for 2+ years (or some other random long duration). >>>>> The Gradle build system will see a lot of changes in the next 6 >>>>> months. I would recommend revisiting it mid-year at least :) >>>>> >>>>> >>>>> On Thu, Jan 16, 2014 at 9:10 AM, Xavier Ducrohet <[email protected]>wrote: >>>>> >>>>>> http://tools.android.com/tech-docs/new-build-system contains our >>>>>> changelog. >>>>>> >>>>>> We generally don't include Gradle specific changes in it even when we >>>>>> start requiring newer versions of Gradle, but performance fixes will >>>>>> probably be indicated in it because we know it's a pain point. >>>>>> >>>>>> >>>>>> On Thu, Jan 16, 2014 at 8:54 AM, Ashton Cummings >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> Xavier, >>>>>>> >>>>>>> Thanks for the information. We are coming to a point now where the >>>>>>> decision to switch our build process is coming to an end. We would >>>>>>> really >>>>>>> like to move to gradle, hopefully you guys are able to get these >>>>>>> optimizations in soon. Once we make the decision, we are going to be >>>>>>> stuck >>>>>>> with it for a while. >>>>>>> >>>>>>> Where can i keep up to date with the exact changes that are going >>>>>>> into each build of the gradle plugin? I need to know when these changes >>>>>>> are >>>>>>> in place as soon as possible. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> On Thursday, January 16, 2014 10:48:52 AM UTC-6, Xavier Ducrohet >>>>>>> wrote: >>>>>>> >>>>>>>> Right now each project will pre-dex its dependencies on its own. >>>>>>>> This means 2 components depending on the same library will both run >>>>>>>> pre-dex >>>>>>>> on that library's classes.jar which is silly. We're looking at fixing >>>>>>>> this. >>>>>>>> >>>>>>>> Another thing we are looking at is parallelization at the task >>>>>>>> level. Right now Gradle only supports it at the projects level. It'll >>>>>>>> build >>>>>>>> 2 sub projects in parallel (if there are no dependencies between the >>>>>>>> two), >>>>>>>> but in a single project it won't parallelize its task. We are working >>>>>>>> with >>>>>>>> the Gradle developers to change this. >>>>>>>> >>>>>>>> Of course the tasks themselves can do their work in parallel, and >>>>>>>> most of our tasks do already. Pre-dexing doesn't though, so if a >>>>>>>> project >>>>>>>> has more than one dependencies it'll run pre-dex one by one. This is >>>>>>>> fixed >>>>>>>> in the next Gradle version. >>>>>>>> >>>>>>>> We do take performance very seriously, but we also wanted to >>>>>>>> provide a lot of features and flexibility in our build. This means we >>>>>>>> do >>>>>>>> some extra work. For instance our resource merger is extra work that >>>>>>>> aapt >>>>>>>> does in a way but differently (more efficiently in some ways, less in >>>>>>>> others, but with much less flexibility). We are working to do more to >>>>>>>> optimize each task and, as I mentioned above, provide generic >>>>>>>> performance >>>>>>>> improvements as well. >>>>>>>> >>>>>>>> I'm a bit surprised your build went from 1:20 to only 0:58 when >>>>>>>> running assembleDebug vs build. 'build' does both debug and release but >>>>>>>> also runs lint. It should be a lot more work than just assembleDebug. >>>>>>>> Also, >>>>>>>> do you have flavors? >>>>>>>> >>>>>>>> One other thing to consider is incremental build. Pre-dexing for >>>>>>>> instance makes clean build clearly slower but incremental builds >>>>>>>> faster. >>>>>>>> The rest of the build is also very good at only doing what needs to be >>>>>>>> done >>>>>>>> and nothing extra, which Ant wasn't very good at. >>>>>>>> >>>>>>>> Disabling pre-dexing on all projects without touching each >>>>>>>> project's build.gradle doesn't seem to be easy. The issue is that >>>>>>>> doing (in >>>>>>>> the main build.gradle): >>>>>>>> >>>>>>>> subprojects { >>>>>>>> /// some config >>>>>>>> } >>>>>>>> >>>>>>>> this gets applied before all the other projects' own build.gradle >>>>>>>> are applied. So unless you can apply the 'android' or >>>>>>>> 'android-library' in >>>>>>>> there, you won't be able to use our DSL. I need to look into this. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 16, 2014 at 6:51 AM, Ashton Cummings < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> My Stack post <http://stackoverflow.com/q/21125302/427763> >>>>>>>>> >>>>>>>>> Using gradle 1.9-rc-3 >>>>>>>>> When building with gradle on a multi-project setup containing >>>>>>>>> roughly 140 projects/libraries, the build time took 1 hour and 22 >>>>>>>>> minutes. >>>>>>>>> And i was using "--parallel". And our ANT build takes less than >>>>>>>>> 20 minutes without parallel building. >>>>>>>>> >>>>>>>>> Here is what i was initially doing: >>>>>>>>> >>>>>>>>> ./gradlew clean >>>>>>>>>> ./gradlew build --parallel >>>>>>>>>> >>>>>>>>> >>>>>>>>> This took 1 hour and 22 minutes, roughly. I did a little testing >>>>>>>>> it seems like the dexing is taking the longest amount of time. Is >>>>>>>>> there a >>>>>>>>> way to get the gradle process to re-use the stuff it has already >>>>>>>>> dexed? If >>>>>>>>> the libraries have already been built, it should re-use the already >>>>>>>>> dexed >>>>>>>>> libraries. >>>>>>>>> >>>>>>>>> I saw the option --no-rebuild, but when i run with that option it >>>>>>>>> says the following: >>>>>>>>> >>>>>>>>>> File '/path/to/project/build/libs/project.aar' specified for >>>>>>>>>> property 'bundle' does not exist. >>>>>>>>>> >>>>>>>>>> I replaced the path and the project name with generic stuff. >>>>>>>>> >>>>>>>>> *** >>>>>>>>> >>>>>>>>> After some assistance i added "preDexLibraries = false" to all of >>>>>>>>> my "build.gradle" files. However, i still would like to know a >>>>>>>>> centralize >>>>>>>>> place that i can put that entry and it affect all the other >>>>>>>>> "build.gradle" >>>>>>>>> files. Because having to add that to all of the "build.gradle" files >>>>>>>>> is >>>>>>>>> annoying and if i ever have to remove it, it will be more annoying. >>>>>>>>> This >>>>>>>>> did not reduce the time to what we are needing. >>>>>>>>> >>>>>>>>> After some more assistance, i ran "assembleDebug" instead of >>>>>>>>> "build". This way it will only do debug builds instead of both >>>>>>>>> release and >>>>>>>>> debug. This still did not bring us down to the time we are getting >>>>>>>>> with >>>>>>>>> ANT. With all these changes we are only down to 58 minutes. And our >>>>>>>>> ANT >>>>>>>>> process is less than 20 minutes. Our ANT process is what android >>>>>>>>> provides >>>>>>>>> only modified to support large multi-project setups and to re-use >>>>>>>>> dexed >>>>>>>>> libraries. The dex process in the gradle process is taking 1+ minute >>>>>>>>> for >>>>>>>>> each project. It is by far the largest time consumer. >>>>>>>>> >>>>>>>>> *** >>>>>>>>> >>>>>>>>> So, how can we speed the process up more? Is gradle re-using the >>>>>>>>> dexed libraries after they have been dexed? What are your >>>>>>>>> suggestions? For >>>>>>>>> us to move over to gradle, it has to be able to compete with the ANT >>>>>>>>> process. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "adt-dev" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Xavier Ducrohet >>>>>>>> Android SDK Tech Lead >>>>>>>> Google Inc. >>>>>>>> http://developer.android.com | http://tools.android.com >>>>>>>> >>>>>>>> Please do not send me questions directly. Thanks! >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "adt-dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Xavier Ducrohet >>>>>> Android SDK Tech Lead >>>>>> Google Inc. >>>>>> http://developer.android.com | http://tools.android.com >>>>>> >>>>>> Please do not send me questions directly. Thanks! >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Xavier Ducrohet >>>>> Android SDK Tech Lead >>>>> Google Inc. >>>>> http://developer.android.com | http://tools.android.com >>>>> >>>>> Please do not send me questions directly. Thanks! >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "adt-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> >>> >>> -- >>> Xavier Ducrohet >>> Android SDK Tech Lead >>> Google Inc. >>> http://developer.android.com | http://tools.android.com >>> >>> Please do not send me questions directly. Thanks! >>> >> -- > You received this message because you are subscribed to the Google Groups > "adt-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- Xavier Ducrohet Android SDK Tech Lead Google Inc. http://developer.android.com | http://tools.android.com Please do not send me questions directly. Thanks! -- You received this message because you are subscribed to the Google Groups "adt-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
