Openboxes Running slowly

Hey all,

Just reaching out as I’ve noticed the last few times that I’ve been logging into Openboxes that the serve seems to be running really slow. Pages don’t seem to load or just take a long time.

Is there any maintenance going on that might have affected this, as didn’t seem to be like this before February? Main reason for the concern is that I am getting ready to upload a lot of stock to the system and concerned that it will struggle to load smoothly when doing this.

Kind regards,

John

Hey @Joda84

Which OpenBoxes server are you referring to?

Justin

If you’re referring to one of our servers, let me know the server URL and I’ll investigate.

If you’re noticing slowness on one of your servers, here are some common causes:

  • not enough heap allocated to Tomcat (default is small, so you might be running out of memory)
  • VM is too small (not enough memory or CPUs for )
  • MySQL is not tuned (generally not a problem for smaller instances though)
  • latency on retrieving certain resource requests (CSS or JS files hosted externally)
  • latency on requests (your browser is in a different country from the server)
  • lots of Quartz background jobs running (triggered by an inventory import maybe)

If you can answer the following questions I might be able to pinpoint the problem more easily:

  • what are your server specs (RAM, CPU, instance type)?
  • where is the server located?
  • where are you located (in relation to the server - i don’t need specifics)?
  • are you seeing slowness across all pages or specific ones (e.g. dashboard)?
  • are there any errors in the server logs around the time of slowness?
  • for the pages that are slow to load can you confirm that the server logs (catalina.out) are reporting longer response times as well (check the footer of the page as well)?
  • how many transactions have been created?

Justin

Hey Justin,

Sorry for the delay - I’ve attached a screenshot of what is happening.

@Joda84 Open a terminal window and run the following command.

ps -ef | grep java

Post the response here. It should look something like this.

tomcat   2387795       1  1 02:00 ?        00:12:02 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dsun.io.useCanonCaches=false -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms2048m -Xmx2048m -XX:MaxPermSize=128m -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start

Once I have confirmed the issue we’ll work on increasing memory for the Tomcat configuration. We’ll need to know how much memory is available so run free -h

free -h
               total        used        free      shared  buff/cache   available
Mem:           7.8Gi       2.9Gi       359Mi       4.0Mi       4.5Gi       4.5Gi
Swap:             0B          0B          0B

Also if you could include the Apache error logs, that would also be helpful

tail -100 /var/log/apache2/error.log

Hey Justin,

See details of the error code that we are experiencing.

ype Exception Report

Message org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.codehaus.groovy.grails.exceptions.GrailsRuntimeException: javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.codehaus.groovy.grails.exceptions.GrailsRuntimeException: javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded

net.sf.ehcache.constructs.web.filter.Filter.logThrowable(Filter.java:143)

net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:91)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:255)

org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:183)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(GrailsPageFilter.java:245)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:134)

org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

Root Cause

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.codehaus.groovy.grails.exceptions.GrailsRuntimeException: javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:656)

org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)

javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

javax.servlet.FilterChain$doFilter$0.call(Unknown Source)

org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)

org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)

org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)

grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)

net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:255)

org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:183)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(GrailsPageFilter.java:245)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:134)

org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

Root Cause

org.codehaus.groovy.grails.exceptions.GrailsRuntimeException: javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded

org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver.resolveException(GrailsExceptionResolver.java:133)

org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:987)

org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:319)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)

org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)

javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

javax.servlet.FilterChain$doFilter$0.call(Unknown Source)

org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)

org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)

org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)

grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)

net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:255)

org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:183)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(GrailsPageFilter.java:245)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:134)

org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

Root Cause

javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded

net.sf.ehcache.constructs.web.filter.Filter.logThrowable(Filter.java:143)

net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:91)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)

org.codehaus.groovy.grails.web.errors.GrailsExceptionResolver.resolveException(GrailsExceptionResolver.java:120)

org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:987)

org.codehaus.groovy.grails.web.servlet.GrailsDispatcherServlet.doDispatch(GrailsDispatcherServlet.java:319)

org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)

org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)

org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)

javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

javax.servlet.FilterChain$doFilter$0.call(Unknown Source)

org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)

org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)

org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)

grails.plugin.springcache.web.GrailsFragmentCachingFilter.doFilter(GrailsFragmentCachingFilter.groovy:66)

net.sf.ehcache.constructs.web.filter.Filter.doFilter(Filter.java:86)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:298)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:264)

org.codehaus.groovy.grails.web.util.WebUtils.forwardRequestForUrlMappingInfo(WebUtils.java:255)

org.codehaus.groovy.grails.web.mapping.filter.UrlMappingsFilter.doFilterInternal(UrlMappingsFilter.java:183)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.obtainContent(GrailsPageFilter.java:245)

org.codehaus.groovy.grails.web.sitemesh.GrailsPageFilter.doFilter(GrailsPageFilter.java:134)

org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.codehaus.groovy.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:69)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)

org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)

org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

Root Cause

java.lang.OutOfMemoryError: GC overhead limit exceeded

Note The full stack trace of the root cause is available in the server logs.

@Joda84 Can you send the output of the two commands free -m and ps -ef | grep java. Those are vital because it will tell me how much memory is available vs how much is currently allocated to the Tomcat server.

Once we determine the available vs allocated you will follow the instructions in the docs re: setting memory. Please read the Note under the Create tomcat.service unit file section of the docs for more details.

And, lastly … this error just means that the Tomcat (really Java) process is spending most of its time garbage collecting (reclaiming memory) and recovering very little, which leads Tomcat to spin on certain requests (or possibly all requests once the OutOfMemoryErrors start) which causes Apache to return a 502 Bad Gateway errors because it’s not receiving a response from Tomcat in a reasonable amount of time.

Morning Justin,

Thanks for getting back to me. I am not use to coding, but wanted to know how I can access the Openboxes system to make the changes as I use the system through the web browser???

That is a bit of a loaded question.

I would start by creating an issue in Github to define what you want to do and then we can discuss how you would go about implementing those changes.

Note: Unfortunately, OpenBoxes isn’t like a basic PHP application that allows you to make changes to the script files and restart the server.