Yeah, I would start by investigating what is different about raw git performance as Ashwanth alludes to. Just to clarify, you don't run agents on the same machine as the server, do you?
These are forked processes from GoCD, and the speed at which a given git operation runs isn't really affected by GoCD in any relevant way I can think of. If there is a difference in the performance of the location where GoCD server is storing its temporary artifacts <https://docs.gocd.org/current/installation/install/server/linux.html> between new and old (/var/lib-go-server mount, normally), that might be something to look at. The thread dumps you post are generally just indicative of the material subsystem being very slow and not being able to get through the number requiring material checks/polls in the allotted time. There are many posts and such about slow performance around the place e.g, here <https://github.com/gocd/gocd/issues/10480>, here <https://github.com/gocd/gocd/issues/9588> and elsewhere in GitHub going back a well - perhaps can mine the existing resources for ideas. As to why things appear to have degraded since your previous server version, it's a bit difficult to tell. What was your rebuild methodology? Did you follow https://docs.gocd.org/current/advanced_usage/one_click_backup.html#restoring-gocd-using-backup? Are you sure you restored the config folder (wrapper-properties.conf and similar) and any overrides to GoCD memory allocation, system properties etc? It's conceptually possible that your previous configuration had tuned the material subsystem to either do more in parallel or to poll less frequently and such, or that you had some global git config which somehow affected git performance - but without more detail, it's probably just idle speculation. -Chad On Wed, Jan 11, 2023 at 7:27 PM 'Ashwanth Kumar' via go-cd < [email protected]> wrote: > Can you check the avg. disk io on the machine? Something like: > > $ iostat -k 2 > > Also are you using SSDs or HDDs on the GoCD Server? If you're using HDDs > see if you can run the same setup on SSDs to see if there is a difference. > > Thanks, > > > On Wed, 11 Jan 2023 at 12:28, Brandon V <[email protected]> wrote: > >> Hi all, >> >> Our GoCD server is acting slow. We have >1000 pipelines that trigger >> automatically on updates to various branches of a Git repository. For each >> branch that needs to be picked up by GoCD, we have a distinct git material >> with shallow clone set to true. The time to trigger from Git updates is >> quite slow. The UI is also a bit sluggish in general. >> >> Environment info: >> Linux >> GoCD server 21.1.0 >> PostgreSQL 12.8 database >> >> The server was recently recreated updating the OS from Ubuntu 16.04 to >> Ubuntu 20.04. The most important pieces of configuration were restored from >> the previous server. However, it's possible some configuration was lost, or >> some package version was changed, or something like that. I'd appreciate >> any pointers. >> >> While trying to debug this, I noticed a few things: >> >> CPU utilization on the machine is hovering around 60%, when there are no >> jobs running. >> >> The top consumers of CPU on the machine are git commands run by the GoCD >> server. At any given time there are about 8 instances of git processes like >> "git branch -r --contains <commit-sha>". Each of these git commands can be >> using a whole CPU. >> >> Looking at things related to git, I noticed: >> >> The GoCD server logs in /var/log/go-server/go-server.log have a lot these >> messages (referencing the different deployment branches): >> >> 2023-01-11 06:44:04,168 WARN [ThreadPoolTaskScheduler-7] >> MaterialUpdateService:204 - [Material Update] Skipping update of material >> GitMaterial{[email protected]:repo/app.git, branch='abranch', >> shallowClone=true, submoduleFolder='null'} which has been in-progress since >> Wed Jan 11 06:43:04 UTC 2023 >> >> Using jstack to get thread dumps from the server, this seems to be the >> java stack trace where those git processes are launched: >> >> "130@MessageListener for MaterialUpdateListener" #130 daemon prio=5 >> os_prio=0 cpu=14618.25ms elapsed=100474.11s tid=0x00007fa2ded9dca0 >> nid=0xea6f in Object.wait() [0x00007fa20c5ad000] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait([email protected]/Native Method) >> - waiting on <no object reference available> >> at java.lang.Object.wait([email protected]/Unknown Source) >> at java.lang.ProcessImpl.waitFor([email protected]/Unknown Source) >> - locked <0x00000005370e58e0> (a java.lang.ProcessImpl) >> at >> com.thoughtworks.go.util.ProcessWrapper.waitForExit(ProcessWrapper.java:54) >> at >> com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:354) >> at >> com.thoughtworks.go.util.command.CommandLine.runOrBomb(CommandLine.java:378) >> at >> com.thoughtworks.go.domain.materials.SCMCommand.runOrBomb(SCMCommand.java:38) >> at >> com.thoughtworks.go.domain.materials.git.GitCommand.containsRevisionInBranch(GitCommand.java:364) >> at >> com.thoughtworks.go.config.materials.git.GitMaterial.modificationsSince(GitMaterial.java:132) >> at >> com.thoughtworks.go.server.service.materials.GitPoller.modificationsSince(GitPoller.java:35) >> at >> com.thoughtworks.go.server.service.materials.GitPoller.modificationsSince(GitPoller.java:26) >> at >> com.thoughtworks.go.server.service.MaterialService.modificationsSince(MaterialService.java:134) >> at >> com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:56) >> at >> com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.insertLatestOrNewModifications(MaterialDatabaseUpdater.java:157) >> at >> com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterialWithNewRevisions(MaterialDatabaseUpdater.java:149) >> at >> com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$2.doInTransaction(MaterialDatabaseUpdater.java:108) >> at >> com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:23) >> at >> com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$executeWithExceptionHandling$2(TransactionTemplate.java:43) >> at >> com.thoughtworks.go.server.transaction.TransactionTemplate$$Lambda$1842/0x00000008045df9c8.doInTransaction(Unknown >> Source) >> at >> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) >> at >> com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:40) >> at >> com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:105) >> - locked <0x0000000537f34488> (a java.lang.String) >> at >> com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:64) >> at >> com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:32) >> at >> com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83) >> at >> com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63) >> at java.lang.Thread.run([email protected]/Unknown Source) >> >> >> >> Any help is appreciated, thanks! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/3c3fe87e-5639-4316-9dfa-bc3dcf68d901n%40googlegroups.com >> <https://groups.google.com/d/msgid/go-cd/3c3fe87e-5639-4316-9dfa-bc3dcf68d901n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > > Ashwanth Kumar / ashwanthkumar.in > > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CAD9m7CyPznfmbZ5wSd7yDSBEAibS9uuFexoL3DTjfbdj-Dm%3DDQ%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CAD9m7CyPznfmbZ5wSd7yDSBEAibS9uuFexoL3DTjfbdj-Dm%3DDQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8qPOHNmrQ0M0Cv2%3DwBhmY0d98AdQSRsOjshshpWeAnjQ%40mail.gmail.com.
