Openboxes webUI slow loading on LAN with no WAN/Internet connection

Hello everyone,

I have encountered an issue with the Openboxes webUI, which is loading extremely slowly every time I disconnect my on-premise server from the Internet. Originally, my goal was to install Openboxes as an on-premise server that would be disconnected from the Internet and accessed locally via LAN (in this scheme: Server-Router-Client/Tablet). When I first installed Openboxes on my server, it was connected to the router, which had a connection to the Internet. When I accessed Openboxes using the IP:8080/openboxes, the login page loaded almost instantly. Also, browsing the rest of the webUI was with no issues. A few days ago, I unplugged the server from the Internet to test it out in the LAN setting with no WAN/Internet connection. When I accessed the login page the same way as before, the webpage loaded really slowly (almost 5 minutes) before it finally appeared. Sometimes it just timed out. And when I logged in and browsed the webUI, the loading took a few minutes, and sometimes the webpages appeared a bit messed up. So, my question is whether this is some kind of networking issue or if Openboxes (NGINX or Tomcat) uses some kind of dependencies that are downloaded every time from the Internet? Just to clarify, I’m using Linux server with v0.9.0-hotfix1 (Grails 3.3.16) version of Openboxes with NGINX web server configured to run on localhost:8080 and I was accessing it via a wireless router using the server IP (I´m checking my server IP everytime with the network scanner, haven´t set it to static yet). I also tried using multiple routers with the same results. I also tried disabling the firewall in my router with no success.

I think the issue is that some bootstrap content like Jquery is loaded off side, like:

https://cdn.jsdelivr.net/npm/bootstrap@4.3.1/dist/js/bootstrap.min.js
https://cdnjs.cloudflare.com/ajax/libs/datatables/1.9.4/jquery.dataTables.js
https://use.fontawesome.com/releases/v5.14.0/js/all.js

This is a great question and something we need to discuss internally to see if there’s a decent solution. I would like OpenBoxes to be usable on a LAN but we have not made any effort to ensure that it can.

At the moment, you probably won’t be able to run OpenBoxes via the browser on computers that have never had access to the Internet since there are resources (javascript, CSS, fonts, etc) that are being pulled from CDNs when pages are loaded for the first time (as @David_Douma suggested).

You are currently in a position where OpenBoxes on your computer mostly works for you because you were previously connected to the Internet. In other words, those resources were downloaded and cached by your browser. As soon as you clear your cache you’re going to start seeing problems.

We can definitely pull all resources down into the WAR file and forgo the need for CDNs. We just haven’t prioritized that.

The particular issue regarding slowness is related, but probably a slightly different problem which you should be able to confirm easily by using the browser’s Inspect feature.

The first time you load a page you’ll get all resources downloaded from their respective sources (local server, CDNs, etc)

On subsequent page loads, you’ll notice a bunch of resources under the Network tab with “(memory cache)” or “(disk cache)” in the Size column. These are the ones that’ll be fine for a period of time since they’ll continue to be loaded from the browser cache. But once you clear the cache, then you’ll need the internet again for anything that has been downloaded from a CDN.

In addition, you’ll also notice a bunch of resources that continue to be downloaded each time (nr-spa-1.252.1.min.js) along with requests that are made to services we use for application monitoring and .

For example, the POST request to NRJS-643a83ee6a48748926e is sending metric data to NewRelic to help us identify performance issues.

POST https://bam.nr-data.net/events/1/NRJS-643a83ee6a48748926e

a    = 600299440
v    = 1.252.1
to   = MwAGY0tXWkQFUUdaWwpKS1hJU1pVC0pWQBsFEBBfFlpbUA1c
rst  = 828
ck   = 0
s    = e94758a5c21d5205
ref  = https://obdev3.pih-emr.org/openboxes/auth/login
ptid = 5ff39916-0001-bfee-cd38-018e33ce095e
hr   = 0

As far as I recall, most of these features (HotJar) are disabled by default, so they should not be causing you any issues. That is, except for the New Relic requests. But, I was told by a developer that these requests would not impact performance as they should be happening asynchronously.

I’ll try to add this topic to our tech huddle for tomorrow and see if we can come up with a strategy and perhaps a short-term fix. If you can pinpoint the slow-loading requests then we might be able to get a fix in before the next release.

Here’s how to take a screenshot showing all requests being loaded for a particular page.

  1. Ensure Internet is not enabled on your local machine
  2. Open your browser
  3. Navigate to OpenBoxes login page
  4. Right click anywhere in the page
  5. Select the Inspect option
  6. Click on the Network tab
  7. Refresh the login page (in order to download the resources for that page)
  8. Take a screenshot

The most important data to capture (besides the Name) will be Time, Size, and Waterfall, so please make sure those columns and their values are visible.

You can also download the full log by clicking the download arrow (Export HAR…) in the top bar of the Network tab.

image

Thank you, gentlemen, for your responses. I will try to simulate the same conditions under which the loading slowdown occurred (i.e. without internet and with a clear cache) and will add screenshots from the browser network monitor here along with the HAR log.

Here are the results after I disconnected from Internet, went to the Login page in Mozilla, cleared cache and then refreshed the page with network monitor running. It was loading as slow as the first time when I encountered this issue. I hope it contains all the necessary data.



This is a mashup from all those three screenshots for better overview

Yeah, the logs are showing part of what I expected … that you’re encountering request timeouts for each of those CDN resources. But I don’t see the New Relic requests so I assume that feature is disabled by default.

/me goes off to look at our production server

Yup, looks like New Relic is only enabled if you set all or some of these command line arguments.

jmiranda@obnav:~$ ps -ef | grep tomcat | grep newrelic
...
-javaagent:/var/lib/tomcat85/newrelic/newrelic.jar 
-Dnewrelic.config.application_logging.forwarding.context_data.enabled=true 
-XX:StartFlightRecording 
-Dnewrelic.config.jfr.enabled=true 
...
org.apache.catalina.startup.Bootstrap start

So, in short, the New Relic agent is not the problem. The CDN resources are the problem. And without refactoring our code to include all resources within the WAR file, there’s probably no chance you’ll be able to get your implementation working without Internet.

I brought this issue up during our Tech Huddle this morning and was told that it’s not a priority for the organization funding development of OpenBoxes and there’s a lot of tech debt ahead of it in the backlog, so this probably won’t be prioritized for any of the upcoming releases (v0.9.1, v0.9.2, etc).

So you have one of three or four options

  1. Wait until our development team can prioritize it (which might be months)

  2. Hire your own developer to contribute a patch for this improvement

  3. Fund the development through us (I’ll either work on it myself or find a developer to work on it)

  4. Abandon OpenBoxes altogether

If you want to look into options (2) or (3) I can investigate the issue further and try to estimate the hours I think it might take. That’ll give us a ballpark cost estimate.

As I see it, there are a few issues that need to be addressed (potentially more once you get to requirement 3).

  1. Should require all resources (js, CSS, fonts) to be included in the WAR file and not served from remote CDNs

  2. Should allow disabling of all dependencies / features that require Internet (NewRelic APM, barcode label generation). We might actually want to add a configuration property to make this easier.

    openboxes.security.lockdown.enabled = true
    
  3. Should investigate what else might be causing slowness once resources are pulled in locally

  4. [optional] Should allow failover to a configured endpoint when requests timeout

    • Load resources locally instead of CDNs
    • If this works, we could either add a small improvement to the application to load a custom JavaScript file from the filesystem on your serve. That would allow you to experiment with a failover feature like the one in that article.
    • Unfortunately, that would be the easy part of the solution. The difficult part would be building your own local CDN server (see below).
<aside>

Aside: I just thought of one (somewhat painful) path you can take without code changes. You could create your own CDN-like server locally that can serve all of these resources and provide mock responses to requests for external APIs that we might be calling.

In a nutshell, you’d need to do something like this

  1. Enable Internet

  2. Download each of the resources required by the application (or fetch them from the browser cache)

  3. Create a local CDN server using NGinx or Apache (or an open-source CDN software)

  4. Configure local DNS to resolve all external hosts to the local CDN server

      Oh my, I'm sweating just thinking about how horrible this is ...
    
  5. Configure virtual hostnames for all of the external CDNs

  6. Load content into directories

  7. Ensure paths in the request URL are maintained (should be configurable within Nginx or Apache or you can create the same directory structure under each CDN’s document root)

NOTE: There might be a more straightforward / automated way of doing this, but that is outside the scope of my expertise.

</aside>

I have stumbled upon this topic while looking for the answer waht could be the reason that alomost every page (being even just a list of units of measure) takes like 5-6 seconds to load when accessed for the first time and only like 0,015 seconds afterwards. I was quite surprised to see, that despite being “on-premise”, the appliation relies on the access to the external resources to get info vital for its functionality.

Since the topic is quite old now i`d like to ask if anything has changed in this regard over time?

PS: did i get it right that some features (like bar code generation) are fully dependent on the external service?

Thanks

Thanks for the bump on this topic.

Since the topic is quite old now i`d like to ask if anything has changed in this regard over time?

Nothing has been done yet unfortunately. But it’s been added to our tech debt opportunity backlog which is reviewed periodically by our development team for tasks to take one. This is one of those tech debt tasks that is low-hanging fruit (high-value-low-cost) so I’m hoping we can get this prioritized soon.

PS: did i get it right that some features (like bar code generation) are fully dependent on the external service?

Generating a barcode label is just a matter of taking a string template (mix of ZPL code and variable placeholders) and performing string interpolation with a map of data. There’s a dependency (Zebra ZPL library GitHub - w3blogfr/zebra-zpl: Library used to generated commons ZPL command to print label on Zebra printer with java) that is used to send ZPL to a local printer, but I believe the only feature that is outsourced to an external service is the “ZPL to image preview” which is just a convenience feature. You’ll most likely design your barcode labels (ZPL) using another system so you probably won’t need to preview it in OpenBoxes.

Thanks for the reply.
I think i have misread some portion of the thread above. does this “At the moment, you probably won’t be able to run OpenBoxes via the browser on computers that have never had access to the Interne” means that the client machine must have internet connection to download some files (or post some performance data) from CDN, not the server?
So this issue will be critical when:

  1. running both the server and client inside locked network without any means to access internet from the client pc
  2. either CDN experiences some issues (or is not available due to access restrictions from providers, authorities or CDN blacklisting some ip ranges for example)

correct?

Yes, most web resources (CSS, javascript) used within the web application are downloaded from CDNs. We are aware of the issue but just haven’t been able to work on it yet.

We’ll need to do a discovery to make we’ve addressed all components that require Internet access. As far as I recall these are the primary capabilities that need to be supported in order to run the application on a LAN.

  • Disable observability by default (Sentry, New Relic)
  • Disable analytics by default (Google)
  • Disable APIs that require Internet access (Currency API)
  • Disable or locally configure SMTP server somewhere on the LAN (if desired, possible)
  • Locally host logos and images (stored in database, hosted by app)
  • Verify barcode printing works on the LAN (send ZPL to local IP address)
  • Disable barcode preview when running on a LAN (labelary.com)
  • Locally host web resources as internal depedencies (CSS, Javascript)

The only one that is not yet complete is the last one. The work of moving from CDN-provided to internal resources is not a heavy lift. However, testing the entire application to make sure we don’t break anything probably is a heavy lift.

If anyone from the community would like to take a stab at this one I can create a branch to get us started, deploy it to a test server, and we can work on it over time.

So this issue will be critical when:

  1. running both the server and client inside locked network without any means to access internet from the client pc
  2. either CDN experiences some issues (or is not available due to access restrictions from providers, authorities or CDN blacklisting some ip ranges for example)
    correct?

Correct.