The Core Technologies Blog

Professional Software for Windows Services / 24×7 Operation


AlwaysUp 12.5: Dropbox fixes, Email improvements, AutoStartDelay and More

What's new in AlwaysUp 12.5

We’re closing out the tumultuous 2020 with version 12.5 of AlwaysUp — our “run anything as a Windows Service” utility used by tens of thousands of savvy computer administrators across the globe.

This release delivers a few major enhancements:

File synchronization fixes for Dropbox 105+ running as a service

Dropbox on Windows

Dropbox threw us a curve-ball in September. With the advent of version 105, Dropbox suddenly stopped synchronizing files to the cloud when it was running as a service in Session 0.

This issue affected many of our customers, and as the Dropbox automatic updates started rolling out, the emails and complaints started coming in!

After a few days of frantic experimentation, our developers arrived at an effective and simple solution. Apparently setting the QT_OPENGL system variable to software restores all functionality. Dropbox copies files again.

We decided to incorporate the fix into AlwaysUp, to save new Dropbox customers from having to manually set QT_OPENGL. AlwaysUp Version 12.5 and later will automatically set the QT_OPENGL environment variable so that you don’t have to.

One word of caution: If it turns out that setting the QT_OPENGL variable causes problems in a future version of Dropbox, you can neutralize the fix through another environment variable. Set the ALWAYSUP_NO_DROPBOX_OPENGL_FIX system variable to any value and AlwaysUp will not alter the QT_OPENGL value.

Emails include recent service/application activity

Customers who have configured email alerts will notice that messages now contain the service’s last five events from the Windows Event Logs. The idea is to provide helpful context when something unusual happens, to avoid you having to log on and interrogate the event logs yourself.

Here is an email showing the new Recent Activity section:

Emails include recent service/application activity

Specify when Windows launches “Automatic (Delayed Start)” services

By default, services with startup type set to Automatic (Delayed Start) start 120 seconds after the last automatic service is launched.

That two minute gap is fine for most, but what if your server is blazing fast? In that case, one minute may be better.

Or worse, suppose your legacy server is painfully slow to boot? A five minute delay may be more appropriate there.

Fortunately you can adjust the delay in the registry. And now you can do the same in AlwaysUp too!

Simply select Set Auto Start Delay from the Tools menu to bring up the new feature. Set an appropriate delay value and click OK to save:

Set Auto Start Delay in AlwaysUp

As noted in the window, the setting applies to all services set to Automatic (Delayed Start). It’s not just for services created by AlwaysUp.

Full compatibility with Windows 10 20H2

Microsoft published Windows 10, version 20H2 in October 2020.

Looking at the release notes, 20H2 doesn’t include significant changes to the Windows Services infrastructure. The update focused mostly on end-user improvements for the Edge browser, task tray notifications and the like.

Nevertheless, our team tested AlwaysUp 12.5 extensively on the new version of Windows 10. We’re pleased to report that no problems were detected and AlwaysUp remains fully compatible with all versions of Windows 10.

AlwaysUp is compatible with Windows 10 20H2

As usual, please review the release notes for the full list of features, fixes and improvements included in AlwaysUp 12.5.

Upgrading to AlwaysUp 12.5

If you purchased AlwaysUp version 11 (after June 2018), you can upgrade to version 12.5 for free. Simply download and install “over the top” to preserve your existing applications and all settings. Your registration code will continue to work as well.

If you bought AlwaysUp version 10 or earlier (before June 2018), you will need to upgrade to use version 12.5. Please purchase upgrades here — at a 50% discount.

See the full upgrade policy for additional details.

Enjoy!

Posted in AlwaysUp | Tagged , , , , , , , | Leave a comment

Q&A: Why does AlwaysUp Restart my Application During Windows Shutdown?

Q&A: Why does AlwaysUp Restart my Application During Windows Shutdown?
This seems like strange behavior, but when you are using autologon to launch an application in session 1 and reboot the server, during the reboot AlwaysUp briefly relaunches the application in session 0. Here’s an example from the activity log:

AlwaysUp System Shutdown Log

Why does that happen? How do I prevent it from happening?

— Kevan

Hi Kevan. We have seen this before…

You’re encountering a Race Condition as Windows shuts down

Unfortunately you are experiencing the effects of the chaotic and poorly documented Windows shutdown process.

From the timeline you have provided, you initiated the reboot at about 10:26 AM, right? Within a few seconds of that, Windows logs off the user signed in to Session 1.

That system-initiated logoff terminates Session 1. Windows stops all programs running in Session 1 — forcibly if necessary. And since your application is running in Session 1, it’s terminated too. That happens at about 10:26:08.

AlwaysUp, which is constantly monitoring your application in the background, immediately notices that the Session is closed and that your application has stopped. AlwaysUp writes an informative log entry at 10:26:08:

Session 1 closed

Next, AlwaysUp springs into action to ensure that your application runs 24/7. It knows your preference is to run in Session 1, but that’s no longer possible. The only reliable place to restart your application is in the session that is always available to Windows Services — the isolated Session 0.

So AlwaysUp starts your application in Session 0. The log line at 10:26:09 confirms that action:

Application started in Session 0

However, during the process of rapidly restarting your program in Session 0, AlwaysUp is notified of another important event. Windows is shutting down. This is reported at 10:26:09 too:

Windows shutdown initiated

And finally, a mere two seconds later, both the application and service stop.

So the only hiccup — as you mentioned — is the unnecessary, “phantom” restart that happens at 10:26:08. This is because AlwaysUp is not aware of the system shutdown at the time when it detects the application outage. AlwaysUp is just doing it’s job, trying to make sure that your application is always running, 24×7.

This is a classic race condition, where two events (“session closed” and “system shutting down” notifications) occur in an unpredictable order. When AlwaysUp is notified of the shutdown first, it won’t restart your application. However, when AlwaysUp learns of the session close first, it will restart your application. And unfortunately you never know which sequence/behavior you’re going to get. 🙂

How to avoid application restarts during Windows Shutdown

To overcome the problem, configure AlwaysUp to wait for a few seconds before restarting your application. Make that change on the Restart tab.

For example, here we have instructed AlwaysUp to delay for 15 seconds:

AlwaysUp: Delay restart

That short pause should provide enough time for AlwaysUp to receive the Windows shutdown notification and exit gracefully — without a phantom restart.

Of course, this adjustment means that AlwaysUp will not restart your program immediately if it crashes. If that’s a problem (i.e. your application must achieve close to 100% uptime), you may want to reduce the delay. Even 5 seconds should be sufficient to sidestep the dreaded race condition.

Posted in AlwaysUp | Tagged , , , , | Leave a comment

4 Affordable Tools to Monitor & Bulletproof your Web Application/Server

Monitor Web Application/Server

Does your application accept HTTP requests?

Are you responsible for a mission-critical application that features a HTML, XML or JSON web interface?

And can you visit a URL (such as “http://yoursite.com:3232/status”) to confirm that your web application is working well?

If so, website monitoring tools — which periodically “ping” your site and confirm that requests are being accepted — can significantly improve your application’s reliability. If you’re not already leveraging such tools, do yourself a favor and grab one today!

Improving uptime: Detection vs. Remediation

Some tools focus on detection. Like the mythical canary in the coal mine, their primary job is to recognize that there is a problem and quickly inform you. They can give you a meaningful head start on resolving the problem — before your important customers start complaining.

But while detection is clearly an advantage, the gold standard is remediation. A quick, automatic remedy minimizes downtime and reduces (or eliminates) stress for the humans involved. Let the machines fix the problem while you eat dinner with your family! 🙂

With that important distinction in mind, let’s review a few inexpensive tools/services that we have used in our professional lives, and a couple that our company has developed for Windows.

1. Pingdom (Detection only)

Starting at $10/month, Pingdom offers a basic monitoring service that checks your URL every minute. If your site is unreachable, it will immediately email you to let you know.

Here are the options when adding a new URL check:

Pingdom URL Check: Settings

Notice that you can configure the check to be performed from multiple geographies. That is a welcome feature if you have customers accessing your site from around the globe.

The URL check’s alerting settings are equally intuitive:

Pingdom URL Check: Alerting

And alerts can be sent via email or SMS/text:

Pingdom User Alert Settings

We use Pingdom to babysit a handful of customer sites/applications. All in all, it’s a reliable detection tool that quickly shouts whenever there is a problem.

2. SiteUptime (Detection only)

SiteUptime is another excellent monitoring tool. It works much like Pingdom, but with a few extra capabilities.

Adding a URL to check is straightforward. Only the basics are required:

SiteUptime Monitor Settings

Things get interesting when you delve into the advanced settings. For example, you can choose the city from where the checks should originate:

SiteUptime Monitor Advanced Settings

Alerts can be sent by email, sms/text or by a phone call:

SiteUptime Monitor Advanced Settings

PagerDuty is also supported.

You can use SiteUptime entirely free (with 1 monitor, 30 minute check interval) but the professional-grade service starts at $10/month. Check it out!

3. AlwaysUp (Detection & Remediation for Windows Applications)

While Pingdom and SiteUptime are affordable, easy to setup services, their effect is limited to failure detection. Once a problem has been identified, they cannot help you take action to resolve the interruption.

That shortcoming is understandable. After all, those services work from the “outside”; they do not know anything about your web application (other than the URL you supplied).

On the other hand, our AlwaysUp utility — designed to run any application in the background as a Windows Service — can do more because it has control of your web application’s process. And at a one-time cost of $50 per server, it is a bargain.

When configured with the check web server sanity check, AlwaysUp will improve reliability by:

  1. Checking the URL every few seconds or minutes (configurable)

  2. Automatically restarting the application (or rebooting the machine) if the URL does nor respond or returns a 5XX error

  3. Sending you an email to let you know what happened

AlwaysUp: Check web server sanity check

The result is that your web application will be down only for a few seconds or minutes.

Note, however, that AlwaysUp works on Windows only. It is not an option for Linux web applications.

4. Service Protector (Detection & Remediation for Windows Services)

If your application runs as a Windows Service, then Service Protector should be your tool of choice for improved reliability.

Like AlwaysUp, it will leverage the check web server sanity check to:

  1. Check the URL periodically

  2. Automatically restart the service (or rebooting the machine) if the URL times out or returns an error

  3. Email you about the issue, to keep you in the loop

You can protect an unlimited number of services on a single server for $70. If you use the product for just a single year, that works out to $0.20 per day. Peace of mind, on the cheap!

Honorable mention (Detection only)

We’ve heard great things about another three site monitoring tools but we have never tried them. If you not on Windows and are interested in detection alone, they are definitely worth a look:

Enjoy!

Posted in Software | Tagged , , , , , | Leave a comment

Q&A: How do I Remove an AlwaysUp “At-Login” Startup Task?

  At some point I must have setup a command line to start my application in AlwaysUp when I log in. But I do not need to do that any longer. I deleted the configuration in AlwaysUp but I do not know how to review and delete what I did in the command line. So this error comes up every time I log in:

AlwaysUp Command Line Error

Can you help me please?

— Nic

Hi Nic. The attempt to launch an obsolete AlwaysUp service is coming from one of several places. Please check the following:

Is there a shortcut in your Windows Startup folder?

Your Windows Startup folder holds shortcuts to applications that should be started when you log in. You can find it here:

C:\Users\YOUR-USER-NAME\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

(You can find your user name by running whoami at a command prompt.)

Navigate to that folder in File Explorer. Do you have any shortcuts that launch an AlwaysUp application? Look for the AlwaysUp icon:

For example, here is a shortcut that restarts Dropbox in the current session sitting in the startup folder:

Dropbox Service Shortcut in the Startup Folder

If you find a shortcut for your bogus application, please delete it. That will eliminate your problem.

If you don’t see an appropriate shortcut, note that on Windows 10, there is another startup folder for all users:

C:\ProgramData\Microsoft\Windows\Start Menu\Programs\StartUp

Please check for an AlwaysUp shortcut there as well.

If you still don’t find any shortcuts, let’s move on to the more obscure locations.

Is there a scheduled task?

The Task Scheduler is a powerful Windows component that can launch applications in response to specific trigger events. It is possible that someone created a task that launches your application/service whenever you log in. Let’s see if we can find such a task.

  1. Start the Windows Task Scheduler by running taskschd.msc. (You can also find it in Control Panel by searching for “Schedule tasks”).

  2. In the Task Scheduler window that comes up, select Task Scheduler (Local) on the left and scroll down to the Active tasks section in the center panel:

    Task Scheduler Active Tasks
  3. Browse the list of active tasks. Look for any mention of your application, AlwaysUp or anything else relevant to your situation. The Triggers column should say At log on, perhaps for your user account.

    For example, here we have identified an AlwaysUp task that restarts Plex Media Server in the current session (i.e. visibly on your desktop) whenever you log on:

    Plex Media Server Task
  4. Let’s confirm that this is the offending task.

    Double-click on the task to move to its location. And double-click the entry to open its properties.

    Switch to the Actions tab and examine the actions. An AlwaysUp task will point to the AlwaysUp executable and reference the application name in the optional arguments, as shown here for Plex:

    AlwaysUp Scheduled Task Action

    Also look for a batch file with a single command launching AlwaysUp.exe.

  5. Once you are comfortable that you have identified the obsolete task, dismiss all sub-windows and return to the main Task Scheduler window. Right-click the task and select Delete to remove it from your system:

    Delete Plex Scheduled Task
  6. Still nothing? OK, let’s bring out the big guns…

What does Autoruns show?

Autoruns is a free administrative tool created by Microsoft. Its purpose is to identify every single application that Windows launches automatically — either when the computer boots or when you log in.

To check for an AlwaysUp task with Autoruns:

  1. Download Autoruns. Extract the zip file to a folder on your hard drive (e.g. C:\Apps\Autoruns).

  2. Start Autoruns as an administrator. (The tool needs admin rights to help you eliminate unwanted startup programs.)

  3. Wait a few seconds for Autoruns to collect its information.

  4. The Everything tab lists all applications, services and scripts launched at startup. Scroll through the list and see if you see any AlwaysUp tasks. You can also use the search function.

    Autoruns: All Startup Programs

    In any case, be sure to omit services. Those will be listed in the HKLM\System\CurrentControlSet\Services section. Those are not the problem.

  5. If you find an AlwaysUp task, you can delete (or disable) it from Autoruns.

    But please proceed with caution. Make 100% sure that you are taking the right action! You don’t want to inadvertently eliminate a vital operation.

Still stumped? Please get in touch for a few more tips

For additional guidance, please send an email to our support team detailing everything you have tried. Please include an export of your Autoruns data (created by selecting File > Save from the menu) so we can see what is automatically loaded onto your system. We’ll help you get the job done!

Posted in AlwaysUp | Tagged , , , , , | Leave a comment

Q&A: Why did Dropbox Stop Copying/Synchronizing my Files?

Q&A: Dropbox stopped copying my files
  My client who has AlwaysUp for Dropbox running on his server informed me that his files stopped syncing.

I created a test file in their folder that is supposed to be syncing and sure enough, it was not working. I checked the service in AlwaysUp and it was running. I tried restarting it and still nothing. I stopped the service and opened Dropbox and it is now syncing.

Please inform me of what I need to do from here. Thanks.

— Kurt

Hi Kurt, sorry to hear of the trouble!

In response, our technical team took the latest version of Dropbox (Stable Build 105.4.651) for a test drive on our Windows Server 2019 machine. Sure enough, file synchronization stalled when running as a Windows Service in Session 0.

Our first instinct was to apply the time-tested advice from this troubleshooting article. We carefully ensured that:

  1. Dropbox.exe wasn’t updating itself (because frequent automatic updates can thwart 24/7 operation)

  2. The Dropbox sync folder was accessible

  3. Dropbox was not trying to get our attention (by showing a notification or confirmation prompt)

Yet after doing all that the problem remained!

So, we rolled up our technical sleeves, unplugged the phone, paused the email and dived into Dropbox. Here’s what we observed.

Dropbox synchronized files at first but stalled after a few minutes

Our testers employ a simple test plan to validate basic file synchronization functionality. It goes like this:

  1. Start Dropbox from AlwaysUp (in Session 0)

  2. Wait 1-2 minutes, to allow the program to initialize and get its bearings

  3. Copy a new file into the PC’s Dropbox folder

  4. Wait up to 1 minute, for Dropbox.exe to recognize the new file and copy it up to the cloud

  5. Check that the new file is listed on Dropbox.com

When we followed the process, step 5 failed. The new file did not appear on the website.

So, we tried again. This time, we noticed something interesting. Dropbox synchronized all pending files when it started, but it would not copy the new file we added. Apparently, synchronization worked at the outset but “broke” after a few minutes. Bizarre!

Upon closer examination — using Microsoft’s excellent Process Explorer — we observed the Dropbox application’s network connections behaving strangely. Let me explain.

A minute after Dropbox starts, the application opens several connections back to “home base”. Here you can see six sockets in the ESTABLISHED state linked to multiple cloud servers on the 162.125.*.* network:

Dropbox TCP/IP connections

A few minutes later, five of the connections have vanished and the final link is dead on its feet (in the terminal CLOSE_WAIT state):

Dropbox TCP/IP connections - Closed

At this point, Dropbox is unable to synchronize any files because it doesn’t have a functioning connection to the back-end servers.

Applying the “QT_OPENGL” fix restored synchronization

About a year ago, we experienced a puzzling situation where Dropbox would hang when started as a Windows Service. This post documents the problem and this post describes the fortuitous fix.

Back then, Dropbox wouldn’t process any files at all — very different from today’s situation, where the program starts strong but quickly goes AWOL.

Nevertheless, a very similar fix works!

Once we set the “QT_OPENGL” system environment variable to “software” (as described below), Dropbox seemed to maintain its server connections indefinitely.

To verify, we wrote a simple script that adds a new file to the Dropbox sync folder every 30 seconds. We let it run for six hours. During that time, Dropbox was able to upload each file to the cloud and every one showed up properly on the website.

Here is a screenshot of the test while in progress:

Dropbox copy test in progress

How to apply the “QT_OPENGL” fix

Please perform this step-by-step process to resume your client’s 24/7 service:

  1. Start a Windows command prompt as an Administrator (an elevated command prompt)

  2. Run this command:

    SETX /M QT_OPENGL software

    SETX /M QT_OPENGL software
  3. Close the command prompt.

  4. Stop and start your Dropbox entry in AlwaysUp.

You can confirm that the fix was applied by looking for the QT_OPENGL value in your system environment variables (available through the Control Panel):

QT_OPENGL System Environment Variable

Please be sure to get in touch if you notice any other misbehavior!

Posted in AlwaysUp | Tagged , , , , | 1 Comment