Performance issues after update to 2.42.1

SCM-Manager
1

Hi.

We have a server with scm-manager.

Two weeks ago, I updated from v2.41 to v2.42. Since then, whenever someone logs in via the web interface java uses lots of resources. We have 4 available processors and the CPU is around 80% on the 4 of them (around 300%-380% total CPU usage according to top). The host runs (upon uname -a):
Linux 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Repository usage works fine from git/svn clients. The issue only affects the web interface, which is extremely slow.

We have around 1K repositories, mainly git. We have run scm-server without issues for a long time with near 1.5K repos. I did a cleanup last week after the problems arose.

Reviewing the log, I can see very frequent INFO entries for

  • sonia.scm.repository.spi.GitCatCommand
  • sonia.scm.user.DefaultUserManager, with message “modify user XXXX of type null”

Additionally, I see frequent ERROR entries for

  • sonia.scm.work.UnitOfWork - task sonia.scm.search.LuceneSimpleIndexTask@91831f failed after XX.XX ms
  • ERROR com.github.legman.EventBus - ScmEventBus-1 - could not dispatch event: RepositoryHookEvent{repository=XXXXXXXXXX (AuTW1dlob6SK), type=POST_RECEIVE} to handler SynchronizedEventHandler{null, public void sonia.scm.statistic.StatisticListener.onHookEvent(sonia.scm.repository.PostReceiveRepositoryHookEvent), true}
  • Errors related to missing paths in some svn repositories.
    • While browsing a repo via the web: ERROR sonia.scm.repository.spi.SvnBrowseCommand - could not fetch file properties
    • As a result of internal operation: sonia.scm.NotFoundException: could not find path with id VP01/untitled/untitled.vpp in revision with id 302 in repository with id XXXXXXXXX

This is the information I found most relevant reviewing the logs.

I’d appreciate any help to diagnose the problem.

Best regards

Ignacio

2

Hi @diversion

welcome to our community and thank you for your report. We already have a look at it and will come back soon with a first feedback.

Best,
Christoph

3

Hey @diversion ,

we just released version 2.42.2 of SCM-Manager. This should fix your performance issues. We would be glad to hear from you whether this works for you, too.

Thank you very much for this quick report!

Best
René

4

Hi!

Thank you very much for your support. It worked!

Yesterday we spent a while debugging system statistics, and we found some info that I still consider interesting. Perhaps it is what you found. But, maybe, it could give info about some other issued that only appear under this situation.

Essentially, what we found was:

  • After service start, it spends a few minutes vary busy and, then, it throws an error indicating that “search.LuceneSimpleIndexTask failed after 8min” and that stored field ‘content’ is too large (800000000 characters)".
  • We observed that any scm-manager load in the client side forced a lot of CPU load and lots of disk read/write transactions.
  • Also, we suspect that, when loading the page, the complete list of repositories was sent, although only the first 10 were shown.
  • Panels such as “My tasks”, “Favourites”, etc also seemed to cause traffic or, at least, waited a lot to have a response, even though they were disabled in the user UI options.

Probably all this is consistent with what you found. However, if you want any further information about any of the things we found, let us know.

Thank you again!

Best,

Ignacio @diversion

5

Thank you for the interesting analysis results. We will check each point and try to improve it.

Feel free to share more findings if you stumble upon something. Feedback is also always welcome. :slight_smile: