The Core Technologies Blog
Our Software // Windows Services // 24×7 Operation
If you are responsible for keeping a CPU-hogging Windows Service running all the time, then we have some great news! Service Protector, our time-saving administrative tool that helps any Windows Service achieve 100% uptime, is now able to monitor processor use across all your server’s CPUs.
Previous versions of Service Protector would only monitor a single CPU/Core. This approach fit with the vast majority of today’s popular windows services that run entirely on one CPU, but it was inadequate for newer, performance-hungry services designed to make use of all a server’s CPUs. This release addresses that deficiency by detecting “runaway” services consuming too many cycles across all the CPUs.
How to Activate Windows Service “CPU Hog” Detection Across All Processors
To identify and automatically restart a misbehaving windows service that ties up multiple CPUs, simply check the Average over all CPUs (instead of only one) box when configuring Service Protector’s Monitor tab:
With the above setting, a service running on a 4-CPU machine that consumes over 95% of all processing power would be automatically restarted by Service Protector.
Our free ServiceTray utility manages any Windows Service from a convenient tray icon. Most folks that use the software configure it to start at login by placing a shortcut in the Windows Startup Folder, but unfortunately that method is not effective on the latest versions of Windows.
Why Doesn’t the Startup Folder Work for ServiceTray?
Microsoft introduced a set of security features called User Account Control (UAC) in Windows Vista and Server 2008. The key concept behind UAC is that of “least privilege” — where all applications run with normal, non-administrative rights until someone explicitly allows elevation to the more powerful context. Yet while UAC has certainly made PCs more secure from malicious viruses and other hostile actions, it can occasionally foil legitimate use.
For example, UAC doesn’t play nicely with programs configured to start automatically when you log in. Indeed, if an application or shortcut in the startup folder requires administrative rights, Windows will not start it! Such is the case with ServiceTray, which must run as an administrator to start, stop and interrogate your Windows Service.
So how do you get ServiceTray to start when you login? We have identified a couple of solutions:
Solution #1: Disable UAC
Perhaps the simplest way to get Windows to automatically launch ServiceTray is to turn off UAC. Indeed, a quick Google search will turn up many articles showing how to disable UAC. Apparently many people have found UAC very frustrating and end up turning it off. However as this post points out there are many security implications to consider if you go that route. So caveat emptor!
Solution #2: Create an “At Login” Scheduled Task
You may have heard that the Windows Task Scheduler is the ideal choice for running background tasks at scheduled times, but did you know that it can also fire up an application on your desktop when you log in? Here is how to do that for ServiceTray:
Start Task Scheduler. This is best done by running taskschd.msc from a command prompt, or by opening the Control Panel, searching for “schedule” and clicking the Schedule tasks link.
From the Task Scheduler window, click the Create Basic Task… action on the right:
You should now be looking at the Create Basic Task Wizard window where you can enter a name for the new task we are creating. Something like “Start ServiceTray on Login” would be appropriate, but you can enter anything you like.
Click Next > to move on.
In the Trigger section, select the When I log on option and click Next >.
Next, choose the Start a program option. As usual, click Next > to proceed to the next screen.
Now it’s time to tell the Task Scheduler how to start your ServiceTray shortcut. We’ll need to find that information from the shortcut itself.
Find the shortcut you created with ServiceTray. Right-click on it and select Properties. The window that comes up should look like this:
The Target field contains the full command line used to launch SerivceTray with your selected Windows Service. From that field, copy the path to the ServiceTray executable (probably “C:\Program Files\ServiceTray\ServiceTray.exe” — don’t forget the quotes) and place it in the Switch back to the Create Basic Task Wizard’s Program/script field. Put everything else into the Add arguments (optional) area.
For example, our ServiceTray shortcut command was:
“C:\Program Files\ServiceTray\ServiceTray.exe” “Spooler” -icon 1
Here is what the Create Basic Task Wizard window looks like after that command has been entered:
Click Next > to move on.
The basic configuration is now done and you should now see a summary of the scheduled task to be created.
But before clicking the Finish button, make sure to check the Open the Properties dialog… box because we’d like to make one more modification.
And finally, in the Properties window, check the Run with highest privileges box. This magical option will ensure that ServiceTray starts with full admin rights, so that it can manage your Windows Service.
Click OK to dismiss the dialog. Please close the Task Scheduler as well as your task is now in place.
While deactivating UAC is certainly quicker and easier, we recommend going with solution #2 as it doesn’t require any compromises in security/protection.
To transfer your AlwaysUp Windows Services to a new computer, please follow this 4-step process:
1. Export your AlwaysUp Application(s) from your Existing Server
If you don’t want to move any applications/services from your old server, go straight to step 2.
Otherwise, to export each of your AlwaysUp applications to a XML file:
Select Application > Export All… to summon the Browse For Folder selector:
Choose the folder where you would like the XML files to be saved.
Copy the XML files to a flash drive, or make them available on a network share that is accessible from your new PC. You will need them in step 3.
2. Download & Install AlwaysUp on your New Server
On your new PC, download AlwaysUp from our web site and follow these step-by-step instructions to install it.
3. Import your AlwaysUp Application(s) into your New Installation
If you exported applications in step 1, now is the time to import each of them into your new installation. On your new PC:
Select Application > Import…
Next, select the XML file representing the service you wish to restore. This will launch the Add Application window, already populated with the details of your service.
Review your application’s settings. Please pay special attention to the path to your main executable or script on the General tab (or any other file locations configured for your service) which may be different on this new PC.
Note that you may have to re-enter your Windows password on the Logon tab because that password was not exported along with your other settings (to enforce security standards).
Click the Save>> button to record this service.
Please repeat these steps for each of the applications/services you wish to restore.
4. Register AlwaysUp on your New Server
Chances are that registration code used on your old PC will not work on the new one. To obtain a new code, please send an email to support@CoreTechnologies.com with:
We will respond with the registration code for your new PC, along with instructions for where to enter it.
That’s it! Best of luck with your new machine!
Tod Daniels of d’innovative reported a strange problem. His JScript file (run with Microsoft’s CSCRIPT.EXE) started fine from his desktop but failed when run as a windows service with AlwaysUp.
By troubleshooting the situation from the command line, Tod was able to narrow the problem down to a problem when reading from the registry. Specifically:
regedit /e F:\test1.reg “HKEY_LOCAL_MACHINE\Software\Broadcom\BACS” run outside AlwaysUp produces a file containing the registry export.
regedit /e F:\test2.reg “HKEY_LOCAL_MACHINE\Software\Broadcom\BACS” run inside an AlwaysUp CMD session does not produce a file.
Both commands were run in the same user account, so why the different results?
The discrepancy is due to the different views of the registry presented to 32-bit and 64-bit applications.
Most modern versions of Windows are 64-bit. All the major applications and supporting DLLs distributed with the OS are 64-bit. To ensure that older 32-bit applications continue to run fine on these new operating systems, Microsoft engages in some “creative trickery”:
32-bit applications see a 32-bit version of the System32 Folder
Even though the Windows System32 folder (typically, C:\Windows\System32) is stocked with 64-bit applications, a 32-bit application has that folder “mapped” to a counterpart (C:\Windows\System32\Wow64) filled with 32-bit versions instead. So when a 32-bit application runs the “DIR” or “REGEDIT” commands, it is actually invoking the 32-bit version in the Wow64 folder. This silent mapping ensures compatibility when a 32-bit application invokes one of those standard Windows utilities.
AlwaysUp is a 32-bit application and is constrained by this behavior. When we’re troubleshooting, the command prompt is launching the 32-bit version of regedit.
64-bit applications work with a (slightly) different view of the Registry
In the 64-bit operating systems, some registry keys actually have a 32-bit version and a 64-bit version! One such key is HKLM\Software. 32-bit applications can write to this key normally, however the changes show up under HKLM\Software\Wow64 instead. A 64-bit application can see both versions of the keys and can choose which version to access.
Now Tod is using Windows Server 2012 R2 which is 64-bit. Our “a-ha” moment came when we noticed that this key exists:
but the corresponding 32-bit key does not:
This discrepancy means that 64-bit applications can access HKEY_LOCAL_MACHINE\Software\Broadcom\BACS while 32-bit applications cannot see a registry entry with that same name.
Tod was able to start the 64-bit version of regedit from AlwaysUp by exploiting another bit of Microsoft trickery — the Sysnative folder. This is the full path that enabled regedit to find the Broadcom key:
C:\Windows\Sysnative\regedit /e F:\test1.reg “HKEY_LOCAL_MACHINE\Software\Broadcom\BACS”
Ultimately he was able to launch the 64-bit version of CSCRIPT from the same path and his application is now functioning as expected as a Windows Service!
Windows Services promise 24/7 operation and it can be very frustrating when they don’t start as expected! Here are the top five reasons why a service may fail to launch when your server boots:
Your Windows Service isn’t set to Start Automatically
A window service can be configured NOT to start when your computer reboots. Indeed, a service can be set to startup only on demand, or entirely disabled so it cannot run at all.
To check that your service is properly configured:
Start the Services Control Panel application.
Find your service in the list and double-click it to show its properties.
Ensure that the Startup type field is set to Automatic.
Note that Automatic (Delayed Start), where your service starts 1-2 minutes after all Automatic services have been launched, may also be acceptable.
A Dependent Service Failed to Start
Some windows services depend on other services to support their work. Windows enforces these dependencies when booting. For example, suppose there are two services, A and B, both set to start automatically at boot. If A depends on B, then Windows will launch service B before starting service A. If service B fails to start, Windows will not start service A.
Open your service in services.msc and switch to the Dependencies tab to see if your service relies on others. If so, the problem may be with one of those dependent services.
The Service Account’s Password has Changed
If your service runs in a specific user account, was the password for that account changed recently? If so, you should edit your service and enter the new password (on the Log On tab):
There is a Problem with the Domain Account’s Group Policy
If your service is running in a domain account, ensure that the domain account’s group policy has the “Log on as a service” right. As described in this article, the service may run fine at first but suddenly stop working when the local policy (which has the right) is overwritten by the global policy (which does NOT have the right).
Look for the telltale “The service did not start due to a login failure” message in the Event Logs to identify this situation.
There is another problem, reported in the Event Logs
Your service may be shutting down because it has encountered a fatal error. The devil will be in the details so be sure to scour the Event Logs for any helpful messages from your service.
Your service may also maintain its own log files, separate from what is available in the Event Viewer. Please be sure to consult those as well!
Why Windows Services Should Avoid Showing Popups/Dialog Boxes
Unlike a regular application (like Microsoft Word) that you launch from a desktop icon and actively work with to accomplish a specific task, a Windows Service is designed to start when your computer boots and run entirely in the background — even if you never log on.
Yet despite this design philosophy, there is no restriction that actually prevents a Windows Service from displaying windows and trying to interact with users logged on to the PC. Indeed, such “Interactive Services” were encouraged in Windows XP and Windows Server 2003, but Microsoft changed their tune in Windows Vista. In our opinion, a couple of glaring problems caused the folks in Redmond to reverse course and actively discourage Windows Services from trying to get the desktop user’s attention:
What happens when no one is logged on?
It is unclear what should happen when a Windows Service shows a prompt and no user is there to answer it. Holding up the action while waiting for someone to log on and click a button can be problematic for tasks designed to perform important tasks, 24/7.
What happens when several users are logged on?
When a service throws up a prompt, which of the multiple users logged on should be allowed to respond to that window? One? All?
These thorny issues should discourage anyone from writing an interactive service today!
But what is you have inherited a legacy service implemented under the more permissive Windows XP or Server 2003 rules? What if commercial realities force you to run an interactive application as a windows service with a service wrapper like AlwaysUp? How do you keep those services running without interruption, getting around annoying prompts that would otherwise stall an important process?
Three Ways to Automatically Dismiss Dialog Boxes and Popups
Write a Custom AutoIt Script
AutoIt is a free scripting language designed for automating the Windows GUI. With this powerful tool, you can easily write scripts to activate windows, click buttons, check boxes, enter text and much more.
For example, this AutoIt script clicks the “OK” button on a window named “OCR Error”:
; Try to bring the window to the top, to
; prepare it for input.
; Click the OK button in the window.
ControlClick("OCR Error", "OK", "");
Note that an AutoIt script can be compiled into an executable to run on any machine — even those without AutoIt installed.
Buy Buzof, the “Annoying Windows Eliminator”
Buzof is a commercial utility that makes it easy for non-programmers to automatically answer Windows prompts. Its point and click interface enables you to “train” the software to take the appropriate action on those annoying popups. Here is what Buzof looks like when configured to watch for four popups, clicking the “Yes” or “No” buttons:
Note that Buzof is really meant for clicking buttons and other simple actions. More complex situations, such as checking a box and subsequently clicking a button or typing in a user name, are beyond its abilities.
Develop your own Windows GUI Utility
If you are an experienced Windows programmer, then it won’t be too difficult to create your own solution. This may be a good investment of your time if using AutoIt or purchasing Buzof are not viable options.
How to Run your Automated Solution on the Isolated Session 0 Desktop
Once you have settled on a method to automatically dispatch the dialog boxes and popups, you must now set it up to run on the desktop where Windows Services run — in Session 0. There are a couple of ways to achieve this:
Run your Utility as a Windows Service
We recommend using our AlwaysUp application to run your chosen utility as a Windows Service. If you are using an AutoIt script, set it up to run periodically, perhaps once every minute. For Buzof, which should be started once and kept running continuously, any service wrapper (such as Microsoft’s Srvany) will perform the basic job.
Run your Utility with the Task Scheduler
Setup a basic scheduled task to start your solution. It will run entirely in Session 0.
Note that you can also launch your utility manually in Session 0 using Microsoft’s PsExec command line tool.
Best of luck keeping your Windows Services operating 24×7, without annoying interruptions!
Backups are an essential part of any professional system. Schedule them regularly or risk losing valuable data (and precious time) when a component fails.
But backing up a live system can often be tricky. The backup software must be “lock” each file to capture a consistent snapshot, and that exclusive access, though temporary, can cause another application to panic and throw up its hands in failure. Or the reverse can happen — an application can lock a file and prevent the backup software from capturing it. And what good is a backup if it doesn’t capture all your important data?
So it’s always best to have a “quiet” system when performing a backup. Close all non-essential applications in advance. But what about Windows Services? Unlike regular desktop programs, those shouldn’t be stopped indefinitely.
This is the dilemma faced by one of our customers. Backups run every weekend constantly fail because his windows service prevents the backup from accessing key files. He tried stopping the service before he left work on Friday evening and restarting it on Monday morning, but users complained loudly about the extended downtime.
The obvious solution is to have a much smaller downtime window for the windows service — only for the duration of the backup. Here is how to do so in an automated fashion, with a couple of well-timed scheduled tasks that leverage the useful NET command.
Part 1: Create a Scheduled Task to Stop your Windows Service before the Backup Starts
Start the Windows Task Scheduler by running taskschd.msc from a command prompt. (You can also find it in Control Panel by searching for “Schedule tasks”).
In the window that comes up, click Create Basic Task… on the right to launch the Create Basic Task Wizard:
Enter a name for your new task. Something like Stop service before backup starts should work nicely. Click Next > when you are done.
In the next couple of steps, we’ll specify when to stop the windows service. Since our backup occurs every Saturday at 10 PM, we’ll setup the task to shut down the service a minute before, at 9:59 PM:
Click Next > to proceed.
On the subsequent screen, select Start a program and click Next >:
Here is where we must specify the program to shut down the service. We’ll use the NET STOP command.
- Enter NET.EXE in the Program/Script line.
- And in the arguments field, enter
where <Service-Name> is the name of the service. We have used “Spooler” in this tutorial but you will obviously use your own service instead. Be sure to enclose that name in quotes if it contains a space!
Click Next > to continue.
We’re almost done! You should see a screen confirming your settings. Click Finish to record this new scheduled task.
Part 2: Create a Scheduled Task to Restart your Windows Service after the Backup Completes
Basically, repeat Part 1 but with the following changes:
When naming the task in Step 3, call it something like Restart service after the backup is done instead.
In Step 4, set the time so that the task will run AFTER the backup completes.
In Step 6, run the NET START command instead. That is, set the arguments field to
Here is what the ending summary screen should look like once you are done:
And that’s it. Since implementing this tactic a few weeks ago, our customer’s backups have been problem-free. We’re confident that the same solution will work for you!
We’re big fans of Oracle’s VirtualBox software. It’s powerful and free, and it enables our small company to efficiently test and validate our software on various versions of Windows without having to invest in costly, dedicated hardware.
However, we noticed a strange problem with VirtualBox last week. After following our tutorial to setup a VirtualBox Virtual Machine (VM) as a Windows Service with AlwaysUp, we noticed that even though the machine was working normally, the VirtualBox Manager application incorrectly listed the VM as “Powered Off”. But it had been on for more than 10 days!
And it wasn’t just the VirtualBox Manager. Even the powerful VBoxManage command line utility thought that our VM wasn’t running:
However, when we ran a command prompt in Session 0, VBoxManage saw the VM there:
So it seems that the VirtualBox tools can only see virtual machines running in the same session that they are in.
Therefore, we simply can’t rely on the state shown in the VirtualBox Manager application when running our VM in the background (in Session 0) with AlwaysUp.
Customer Theodore Bogucki recently reported strange behavior when running Dropbox as a windows service with AlwaysUp:
If I run Dropbox as a normal application, the CPU will run at about 10%. Following the directions to set up Dropbox in AlwaysUp as a service, I will get Dropbox running at 100% CPU and the file upload is extremely slow.
I am using the latest version of Dropbox and the latest version of AlwaysUp on a Windows 2012 R2 server.
What can be causing Dropbox to run amok like this when running as a windows service?
After some back and forth to investigate, Theodore determined that his antivirus program (Symantec Endpoint Protection) was conflicting with Dropbox — but only when running as a service! He said:
The Dropbox folders currently stores SQL backup files from my SQL server. I was able to fix the issue by adding an exclusion in SEP for the file extension used for the SQL backup files. I just as easily could have specified the folder to exclude.
Once this was done I stopped Dropbox on the desktop and started it using AlwaysUp and the CPU only spiked for minute or so while Dropbox started up and then settled down to its normal level. I also observed that network activity was the same as when Dropbox was running on the desktop.
If you are experiencing a similar problem, here are the instructions for excluding files and folders from scans in Symantec Endpoint Protection. A simple Google search should turn up the same information for other popular antivirus packages.
Finally, thanks to Theodore for bringing this problem (and the solution) to our attention!
The mystery: “The service did not start due to a login failure”
One of our customers reported a very strange problem last week. After about a day of running flawlessly, their windows service would suddenly fail to start after a reboot. The error reported by the Event Viewer hinted at a problem with the service user’s account:
Trying to start the service directly from the Windows Services Control Panel application produced the same unsatisfying result:
The service account’s password had not changed, and the user had no problem logging into the server interactively. Why was the windows service failing to login?
Luckily we were able to get the service going again by re-entering the user’s password:
When doing so, we noticed that the “Log on as a service” right had to be granted again. Very suspicious…
But a mere 24 hours later, the problem resurfaced! Once again, the service failed to start after a reboot.
The problem: Group Policy overwriting Local Policy
The message about the “Log on as a service” right lead us to the root of the problem.
Entering the password in services.msc updated the user’s rights in the machine’s Local Group Policy — a collection of settings that define how the system will behave for the PC’s users. However, since the user and server were part of a domain, those local settings were periodically overwritten by the domain’s group policy, which had not been updated with the new permission. And because the necessary permission “disappeared” on the machine, the service failed the next time it tried to start.
The solution: Modify the Domain Group Policy
To fix the problem, we must update the domain group policy and explicitly give the service user the “Log on as a service” right. To do so:
Start the Group Policy Management application.
Open Control Panel, navigate to System and Security > Administrative Tools, and double-click Group Policy Management on the left.
(Note: Don’t search for “group” in Control Panel. That will lead you to the “Edit group policy” link, which opens the local group policy!)
Find your default domain policy on the left. Right-click it and select Edit to bring up the Group Policy Management Editor window.
On the left, navigate to Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > User Rights Assignment and select
the Log on as a service entry on the right.
Double-click Log on as a service to bring up its Properties window.
Add the user running your windows service to the list and click OK to record the change.
Next time your domain policy is copied to your server, it will bring along the Log on as a service right for the user. You shouldn’t encounter the “logon failure” error again!
A closing note for the folks at Microsoft: A better error message please!
Instead of reporting the generic “logon failure”, why not be more precise and say something like “The user doesn’t have the necessary rights to start the service”? You could provide even more guidance by listing the missing rights.
The bottom line is a that a helpful error message highlighting the true problem would have led us straight to the solution and avoided a few anxious days for us and our client.