The Core Technologies Blog

Professional Software for Windows Services / 24×7 Operation


Q&A: Why does AlwaysUp Restart my Application During Windows Shutdown?

Q&A: Why does AlwaysUp Restart my Application During Windows Shutdown?
This seems like strange behavior, but when you are using autologon to launch an application in session 1 and reboot the server, during the reboot AlwaysUp briefly relaunches the application in session 0. Here’s an example from the activity log:

AlwaysUp System Shutdown Log

Why does that happen? How do I prevent it from happening?

— Kevan

Hi Kevan. We have seen this before…

You’re encountering a Race Condition as Windows shuts down

Unfortunately you are experiencing the effects of the chaotic and poorly documented Windows shutdown process.

From the timeline you have provided, you initiated the reboot at about 10:26 AM, right? Within a few seconds of that, Windows logs off the user signed in to Session 1.

That system-initiated logoff terminates Session 1. Windows stops all programs running in Session 1 — forcibly if necessary. And since your application is running in Session 1, it’s terminated too. That happens at about 10:26:08.

AlwaysUp, which is constantly monitoring your application in the background, immediately notices that the Session is closed and that your application has stopped. AlwaysUp writes an informative log entry at 10:26:08:

Session 1 closed

Next, AlwaysUp springs into action to ensure that your application runs 24/7. It knows your preference is to run in Session 1, but that’s no longer possible. The only reliable place to restart your application is in the session that is always available to Windows Services — the isolated Session 0.

So AlwaysUp starts your application in Session 0. The log line at 10:26:09 confirms that action:

Application started in Session 0

However, during the process of rapidly restarting your program in Session 0, AlwaysUp is notified of another important event. Windows is shutting down. This is reported at 10:26:09 too:

Windows shutdown initiated

And finally, a mere two seconds later, both the application and service stop.

So the only hiccup — as you mentioned — is the unnecessary, “phantom” restart that happens at 10:26:08. This is because AlwaysUp is not aware of the system shutdown at the time when it detects the application outage. AlwaysUp is just doing it’s job, trying to make sure that your application is always running, 24×7.

This is a classic race condition, where two events (“session closed” and “system shutting down” notifications) occur in an unpredictable order. When AlwaysUp is notified of the shutdown first, it won’t restart your application. However, when AlwaysUp learns of the session close first, it will restart your application. And unfortunately you never know which sequence/behavior you’re going to get. 🙂

How to avoid application restarts during Windows Shutdown

To overcome the problem, configure AlwaysUp to wait for a few seconds before restarting your application. Make that change on the Restart tab.

For example, here we have instructed AlwaysUp to delay for 15 seconds:

AlwaysUp: Delay restart

That short pause should provide enough time for AlwaysUp to receive the Windows shutdown notification and exit gracefully — without a phantom restart.

Of course, this adjustment means that AlwaysUp will not restart your program immediately if it crashes. If that’s a problem (i.e. your application must achieve close to 100% uptime), you may want to reduce the delay. Even 5 seconds should be sufficient to sidestep the dreaded race condition.

Posted in AlwaysUp | Tagged , , , , | Leave a comment

4 Affordable Tools to Monitor & Bulletproof your Web Application/Server

Monitor Web Application/Server

Does your application accept HTTP requests?

Are you responsible for a mission-critical application that features a HTML, XML or JSON web interface?

And can you visit a URL (such as “http://yoursite.com:3232/status”) to confirm that your web application is working well?

If so, website monitoring tools — which periodically “ping” your site and confirm that requests are being accepted — can significantly improve your application’s reliability. If you’re not already leveraging such tools, do yourself a favor and grab one today!

Improving uptime: Detection vs. Remediation

Some tools focus on detection. Like the mythical canary in the coal mine, their primary job is to recognize that there is a problem and quickly inform you. They can give you a meaningful head start on resolving the problem — before your important customers start complaining.

But while detection is clearly an advantage, the gold standard is remediation. A quick, automatic remedy minimizes downtime and reduces (or eliminates) stress for the humans involved. Let the machines fix the problem while you eat dinner with your family! 🙂

With that important distinction in mind, let’s review a few inexpensive tools/services that we have used in our professional lives, and a couple that our company has developed for Windows.

1. Pingdom (Detection only)

Starting at $10/month, Pingdom offers a basic monitoring service that checks your URL every minute. If your site is unreachable, it will immediately email you to let you know.

Here are the options when adding a new URL check:

Pingdom URL Check: Settings

Notice that you can configure the check to be performed from multiple geographies. That is a welcome feature if you have customers accessing your site from around the globe.

The URL check’s alerting settings are equally intuitive:

Pingdom URL Check: Alerting

And alerts can be sent via email or SMS/text:

Pingdom User Alert Settings

We use Pingdom to babysit a handful of customer sites/applications. All in all, it’s a reliable detection tool that quickly shouts whenever there is a problem.

2. SiteUptime (Detection only)

SiteUptime is another excellent monitoring tool. It works much like Pingdom, but with a few extra capabilities.

Adding a URL to check is straightforward. Only the basics are required:

SiteUptime Monitor Settings

Things get interesting when you delve into the advanced settings. For example, you can choose the city from where the checks should originate:

SiteUptime Monitor Advanced Settings

Alerts can be sent by email, sms/text or by a phone call:

SiteUptime Monitor Advanced Settings

PagerDuty is also supported.

You can use SiteUptime entirely free (with 1 monitor, 30 minute check interval) but the professional-grade service starts at $10/month. Check it out!

3. AlwaysUp (Detection & Remediation for Windows Applications)

While Pingdom and SiteUptime are affordable, easy to setup services, their effect is limited to failure detection. Once a problem has been identified, they cannot help you take action to resolve the interruption.

That shortcoming is understandable. After all, those services work from the “outside”; they do not know anything about your web application (other than the URL you supplied).

On the other hand, our AlwaysUp utility — designed to run any application in the background as a Windows Service — can do more because it has control of your web application’s process. And at a one-time cost of $50 per server, it is a bargain.

When configured with the check web server sanity check, AlwaysUp will improve reliability by:

  1. Checking the URL every few seconds or minutes (configurable)

  2. Automatically restarting the application (or rebooting the machine) if the URL does nor respond or returns a 5XX error

  3. Sending you an email to let you know what happened

AlwaysUp: Check web server sanity check

The result is that your web application will be down only for a few seconds or minutes.

Note, however, that AlwaysUp works on Windows only. It is not an option for Linux web applications.

4. Service Protector (Detection & Remediation for Windows Services)

If your application runs as a Windows Service, then Service Protector should be your tool of choice for improved reliability.

Like AlwaysUp, it will leverage the check web server sanity check to:

  1. Check the URL periodically

  2. Automatically restart the service (or rebooting the machine) if the URL times out or returns an error

  3. Email you about the issue, to keep you in the loop

You can protect an unlimited number of services on a single server for $70. If you use the product for just a single year, that works out to $0.20 per day. Peace of mind, on the cheap!

Honorable mention (Detection only)

We’ve heard great things about another three site monitoring tools but we have never tried them. If you not on Windows and are interested in detection alone, they are definitely worth a look:

Enjoy!

Posted in Software | Tagged , , , , , | Leave a comment

Q&A: How do I Remove an AlwaysUp “At-Login” Startup Task?

  At some point I must have setup a command line to start my application in AlwaysUp when I log in. But I do not need to do that any longer. I deleted the configuration in AlwaysUp but I do not know how to review and delete what I did in the command line. So this error comes up every time I log in:

AlwaysUp Command Line Error

Can you help me please?

— Nic

Hi Nic. The attempt to launch an obsolete AlwaysUp service is coming from one of several places. Please check the following:

Is there a shortcut in your Windows Startup folder?

Your Windows Startup folder holds shortcuts to applications that should be started when you log in. You can find it here:

C:\Users\YOUR-USER-NAME\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

(You can find your user name by running whoami at a command prompt.)

Navigate to that folder in File Explorer. Do you have any shortcuts that launch an AlwaysUp application? Look for the AlwaysUp icon:

For example, here is a shortcut that restarts Dropbox in the current session sitting in the startup folder:

Dropbox Service Shortcut in the Startup Folder

If you find a shortcut for your bogus application, please delete it. That will eliminate your problem.

If you don’t see an appropriate shortcut, note that on Windows 10, there is another startup folder for all users:

C:\ProgramData\Microsoft\Windows\Start Menu\Programs\StartUp

Please check for an AlwaysUp shortcut there as well.

If you still don’t find any shortcuts, let’s move on to the more obscure locations.

Is there a scheduled task?

The Task Scheduler is a powerful Windows component that can launch applications in response to specific trigger events. It is possible that someone created a task that launches your application/service whenever you log in. Let’s see if we can find such a task.

  1. Start the Windows Task Scheduler by running taskschd.msc. (You can also find it in Control Panel by searching for “Schedule tasks”).

  2. In the Task Scheduler window that comes up, select Task Scheduler (Local) on the left and scroll down to the Active tasks section in the center panel:

    Task Scheduler Active Tasks
  3. Browse the list of active tasks. Look for any mention of your application, AlwaysUp or anything else relevant to your situation. The Triggers column should say At log on, perhaps for your user account.

    For example, here we have identified an AlwaysUp task that restarts Plex Media Server in the current session (i.e. visibly on your desktop) whenever you log on:

    Plex Media Server Task
  4. Let’s confirm that this is the offending task.

    Double-click on the task to move to its location. And double-click the entry to open its properties.

    Switch to the Actions tab and examine the actions. An AlwaysUp task will point to the AlwaysUp executable and reference the application name in the optional arguments, as shown here for Plex:

    AlwaysUp Scheduled Task Action

    Also look for a batch file with a single command launching AlwaysUp.exe.

  5. Once you are comfortable that you have identified the obsolete task, dismiss all sub-windows and return to the main Task Scheduler window. Right-click the task and select Delete to remove it from your system:

    Delete Plex Scheduled Task
  6. Still nothing? OK, let’s bring out the big guns…

What does Autoruns show?

Autoruns is a free administrative tool created by Microsoft. Its purpose is to identify every single application that Windows launches automatically — either when the computer boots or when you log in.

To check for an AlwaysUp task with Autoruns:

  1. Download Autoruns. Extract the zip file to a folder on your hard drive (e.g. C:\Apps\Autoruns).

  2. Start Autoruns as an administrator. (The tool needs admin rights to help you eliminate unwanted startup programs.)

  3. Wait a few seconds for Autoruns to collect its information.

  4. The Everything tab lists all applications, services and scripts launched at startup. Scroll through the list and see if you see any AlwaysUp tasks. You can also use the search function.

    Autoruns: All Startup Programs

    In any case, be sure to omit services. Those will be listed in the HKLM\System\CurrentControlSet\Services section. Those are not the problem.

  5. If you find an AlwaysUp task, you can delete (or disable) it from Autoruns.

    But please proceed with caution. Make 100% sure that you are taking the right action! You don’t want to inadvertently eliminate a vital operation.

Still stumped? Please get in touch for a few more tips

For additional guidance, please send an email to our support team detailing everything you have tried. Please include an export of your Autoruns data (created by selecting File > Save from the menu) so we can see what is automatically loaded onto your system. We’ll help you get the job done!

Posted in AlwaysUp | Tagged , , , , , | Leave a comment

Q&A: Why did Dropbox Stop Copying/Synchronizing my Files?

Q&A: Dropbox stopped copying my files
  My client who has AlwaysUp for Dropbox running on his server informed me that his files stopped syncing.

I created a test file in their folder that is supposed to be syncing and sure enough, it was not working. I checked the service in AlwaysUp and it was running. I tried restarting it and still nothing. I stopped the service and opened Dropbox and it is now syncing.

Please inform me of what I need to do from here. Thanks.

— Kurt

Hi Kurt, sorry to hear of the trouble!

In response, our technical team took the latest version of Dropbox (Stable Build 105.4.651) for a test drive on our Windows Server 2019 machine. Sure enough, file synchronization stalled when running as a Windows Service in Session 0.

Our first instinct was to apply the time-tested advice from this troubleshooting article. We carefully ensured that:

  1. Dropbox.exe wasn’t updating itself (because frequent automatic updates can thwart 24/7 operation)

  2. The Dropbox sync folder was accessible

  3. Dropbox was not trying to get our attention (by showing a notification or confirmation prompt)

Yet after doing all that the problem remained!

So, we rolled up our technical sleeves, unplugged the phone, paused the email and dived into Dropbox. Here’s what we observed.

Dropbox synchronized files at first but stalled after a few minutes

Our testers employ a simple test plan to validate basic file synchronization functionality. It goes like this:

  1. Start Dropbox from AlwaysUp (in Session 0)

  2. Wait 1-2 minutes, to allow the program to initialize and get its bearings

  3. Copy a new file into the PC’s Dropbox folder

  4. Wait up to 1 minute, for Dropbox.exe to recognize the new file and copy it up to the cloud

  5. Check that the new file is listed on Dropbox.com

When we followed the process, step 5 failed. The new file did not appear on the website.

So, we tried again. This time, we noticed something interesting. Dropbox synchronized all pending files when it started, but it would not copy the new file we added. Apparently, synchronization worked at the outset but “broke” after a few minutes. Bizarre!

Upon closer examination — using Microsoft’s excellent Process Explorer — we observed the Dropbox application’s network connections behaving strangely. Let me explain.

A minute after Dropbox starts, the application opens several connections back to “home base”. Here you can see six sockets in the ESTABLISHED state linked to multiple cloud servers on the 162.125.*.* network:

Dropbox TCP/IP connections

A few minutes later, five of the connections have vanished and the final link is dead on its feet (in the terminal CLOSE_WAIT state):

Dropbox TCP/IP connections - Closed

At this point, Dropbox is unable to synchronize any files because it doesn’t have a functioning connection to the back-end servers.

Applying the “QT_OPENGL” fix restored synchronization

About a year ago, we experienced a puzzling situation where Dropbox would hang when started as a Windows Service. This post documents the problem and this post describes the fortuitous fix.

Back then, Dropbox wouldn’t process any files at all — very different from today’s situation, where the program starts strong but quickly goes AWOL.

Nevertheless, a very similar fix works!

Once we set the “QT_OPENGL” system environment variable to “software” (as described below), Dropbox seemed to maintain its server connections indefinitely.

To verify, we wrote a simple script that adds a new file to the Dropbox sync folder every 30 seconds. We let it run for six hours. During that time, Dropbox was able to upload each file to the cloud and every one showed up properly on the website.

Here is a screenshot of the test while in progress:

Dropbox copy test in progress

How to apply the “QT_OPENGL” fix

Please perform this step-by-step process to resume your client’s 24/7 service:

  1. Start a Windows command prompt as an Administrator (an elevated command prompt)

  2. Run this command:

    SETX /M QT_OPENGL software

    SETX /M QT_OPENGL software
  3. Close the command prompt.

  4. Stop and start your Dropbox entry in AlwaysUp.

You can confirm that the fix was applied by looking for the QT_OPENGL value in your system environment variables (available through the Control Panel):

QT_OPENGL System Environment Variable

Please be sure to get in touch if you notice any other misbehavior!

Posted in AlwaysUp | Tagged , , , , | 1 Comment

Essential Windows Services: LanmanServer / Server

Server Service

What is the LanmanServer (Server) service?

The LanmanServer service allows your computer to share files and printers with other devices on your network.

The service’s display name is Server and it runs inside the service host process, svchost.exe. By default, the service is set to start automatically when your computer boots:

LanmanServer Windows Service

What happens if I stop the LanmanServer service?

There will be a couple of potentially significant consequences if you stop the Server service.

1. File sharing will not work

Is your computer hosting a shared folder? If so, that folder will no longer be available to network computers once you stop LanmanServer.

Mapped drives pointing to your computer’s shared folders will be inaccessible and remote users will experience abrupt errors like this:

Mapped drive not accessible

In addition, all sharing controls will disappear from your computer’s File Explorer windows.

For example, the drive properties window has a Sharing tab while LanmanServer is running:

The Sharing tab is accessible

But once the service stops, the Sharing tab vanishes:

The Sharing tab has vanished

Of course, only remote users will be impacted. The underlying files and folders will still be available on the computer hosting the drive.

2. Remote printing will not work

Is your computer sharing its printer with other computers? If so, the service being down will affect folks printing from those other computers.

Specifically, Windows will tell them that the remote printer is “Offline”. All their print requests will enter the print queue, patiently waiting for the printer to come back online:

The printer is offline and jobs are queued

Note that local printing (from the computer with the printer attached) will still work. Only remote users/computers will be affected.

Is it safe to disable LanmanServer?

Yes, it should be safe to disable the service — provided that your computer is not sharing files or printers. As mentioned above, those services won’t operate without the Server service.

Note that Internet-based networking features (for example, accessing web sites from your browser) will still be available when the service is disabled. In theory, many non-networked or home computers will do just fine without LanmanServer always running.

And because LanmanServer doesn’t have any dependencies, disabling it shouldn’t prevent any other services from starting up:

LanmanServer has no dependencies

However, despite the above evidence to the contrary, Microsoft recommends keeping the Server service enabled. Unfortunately they don’t provide a reason or explanation!

Please proceed with caution if you decide to disable the LanmanServer service.

Questions? Problems?

If you would like to know more about the Server service, or you have a specific problem, please feel free to get in touch. We will do our best to help you!

Posted in Windows Services | Tagged , , , | 9 Comments