The Core Technologies Blog

Professional Software for Windows Services / 24×7 Operation


Why you should Disable Smart Sync when Running Dropbox as a Windows Service

Dropbox Smart Sync

What is Dropbox Smart Sync?

The team at Dropbox says it best:

 Smart Sync is a Dropbox feature that helps you save space on your hard drive. Access every file and folder in your Dropbox account from your computer, using virtually no hard drive space. Smart Sync is available for Dropbox Plus and Professional customers, and members of Dropbox Business teams.

And this short video discusses the considerable benefits of Smart Sync:

Indeed, Smart Sync has been a very popular feature since its debut in 2017.

How Smart Sync works: Supporting technologies

Smart Sync leverages a trio of Windows capabilities:

1. NTFS Sparse Files technology

NTFS Sparse File functionality allows Dropbox to create an “empty shell” for any file — with all the usual information (name, size, etc.) but with none of the actual content.

So when you create a file on Dropbox.com, the file’s information (its “metadata”) is immediately downloaded to your computer but none of the file’s contents are transferred in that operation. As a result, you can see the file on your system but it consumes almost no space on your hard drive. Very efficient!

But what happens when you try to access the zero-content file? That’s where the next technology comes in…

2. Windows file system Minifilter driver technology

When you access a file in your Dropbox folder, demanding to see its content, a Windows Minifilter driver ensures that the file’s contents are quickly fetched from Dropbox.com. The arrangement is fairly technical, but the following example illustrates the basic concept.

Suppose you have an online-only file called “notes.txt” in your Dropbox folder. Its contents have not yet been downloaded to your computer. When you double-click on the file:

  1. Windows starts Notepad (the program associated with .TXT files), passing it the full path to notes.txt.
  2. Notepad calls the Windows API ReadFile function to grab the contents of notes.txt
  3. But before invoking ReadFile, Windows intercepts the operation and notifies the Dropbox Minifilter that a program would like to read the contents of notes.txt.
  4. The Minifilter, seeing that the file’s contents have not yet been downloaded, arranges for the file’s contents to be retrieved from Dropbox.com and saved on the local PC.
  5. Windows next calls ReadFile, which returns the contents of the file.
  6. Notepad displays the contents of the file.

Most of the magic happens in step 4. The next section examines how that works.

3. Windows Interprocess Communication (IPC)

At first, we thought that the Minifilter component did all the heavy lifting, downloading content as necessary. However Dropbox says otherwise (the Minifilter is a “system extension”):

We don’t use system extensions to make network requests
We don’t use system extensions to parse any data in the filesystem
We don’t use system extensions to read or write files

Clearly the Minifilter isn’t downloading the files from Dropbox.com. So what’s doing it?

The answer: Dropbox.exe — the process run when you launch Dropbox on your desktop.

So when you request an “online-only” file not yet on your hard drive:

  1. The Minifilter receives the request.
  2. The Minifilter contacts the Dropbox.exe process and asks it to get the file.
  3. Dropbox.exe goes out to the Internet and downloads the file from Dropbox.com.
  4. Windows makes the file available to the user.

The Minifilter and Dropbox.exe interact using Windows Interprocess Communication — a collection of technologies supporting communication between different programs.

Now that we understand how Smart Sync works, let’s highlight three implications of the technical architecture.

Implication #1: Files cannot be downloaded without an Internet connection

This shouldn’t be a surprise.

Since files are fetched from the cloud as needed, you must be connected to the Internet to download files from Dropbox.com.

Dropbox confirms this limitation in their FAQ as well:

Can I access online-only content when I’m not connected to the internet?
No, online-only content isn’t stored locally on your computer. Connect to the internet to access online-only content.

Implication #2: Files cannot be downloaded if Dropbox isn’t running

As mentioned before, the Dropbox process (Dropbox.exe) plays a key role in downloading “incomplete” Smart Sync files. No file can be downloaded if Dropbox is not running.

To be clear, the Minifilter is always notified when you try to access an “online-only” file in the Dropbox folder. But if Dropbox isn’t running, there is no way for the Minifilter to download the file. The operation fails and you get a wonderful “Unspecified error” message (pictured here from the Dropbox forums):

Unspecified Error when opening a file

Note that this behavior only occurs when Smart Sync is on. Without Smart Sync — where Dropbox synchronizes all files to your hard drive — each document you see on your computer is readily available whether Dropbox is running or not.

Implication #3: Files cannot be downloaded if Dropbox is running as a Windows Service

Perhaps the most subtle consequence of the Smart Sync architecture has to do with background operation. Specifically, the Minifilter and Dropbox must operate in the same Windows Session for the two to communicate.

This is not a problem when you use Dropbox interactively on the desktop. All components run in the single logon session and Dropbox downloads files on demand, as expected.

However, customers running Dropbox as a background Windows Service face a problem. Dropbox will be running in Session 0 (the home for all Windows Services), while the Minifilter will be operating in the user’s interactive session (for example Session 1). The components will not be able to communicate and the files cannot be downloaded or opened. The “Unspecified Error” will abound.

So if you’re running Dropbox as a Windows Service (perhaps with our AlwaysUp utility), you should definitely turn Smart Sync off. Read on to find out how to do that.

How to Disable Smart Sync

You have a couple of options. You can either disable Smart Sync for specific folders, or turn off the feature entirely.

Disable Smart Sync for a single folder

This video illustrates how to adjust your Smart Sync folder settings:

And here are the step-by-step instructions to disable Smart Sync for a specific folder only:

  1. Click the Dropbox tray icon (Dropbox tray icon) to open the Dropbox menu
  2. Click the folder icon in the upper right:
    Open Dropbox Folder
  3. Find the folder that you would like to change. Right-click that folder and select Smart Sync > Local from the menu:
    Set Smart Sync folder to Local

That’s it. In a minute or two, the Dropbox process will notice the change and will download the entire folder to your hard drive.

Turn off Smart Sync for your Dropbox Plus, Professional, or Business account

To turn off the Smart Sync feature entirely, please follow these instructions to opt out of the system extension on Dropbox.com.

Dropbox Business team administrators can deactivate Smart Sync by opting out here (sign in required).

Questions about Smart Sync? Need Help?

Please get in touch and we’ll be happy to help — with this or any Dropbox feature.

Posted in Dropbox | Tagged , , , , | 5 Comments

Moving your AlwaysUp Applications/Services? Easily Import Them All at Once

Easily import applications/services into AlwaysUp

Do you manage multiple applications in AlwaysUp?

Across multiple machines?

If so, AlwaysUp’s new “bulk import” feature might make your life a bit easier. 🙂

Instead of adding applications one-by-one (as described in the “How to move your applications” article), you’re now able to import up to 50 applications/services together. This has the potential to save you a ton of time when you have lots of services to move.

Let’s run through the new import process, to give you a feel for how it works.

How to import applications & create new Windows Services

  1. First, choose “Import” from the “Application” menu:

    AlwaysUp Import Menu
  2. In the file selector that comes up, choose the XML files you wish to import. You can select up to 50.

  3. AlwaysUp reviews your request for a few seconds:

    Windows Service Applications Import: Preparing
  4. Next, AlwaysUp examines each file, checking for common errors. Here you can see the Dropbox service being validated:

    Windows Service Applications Import: Validating Dropbox
  5. Depending on the results of validation, you may encounter one or more of the following “problem” screens.

    For example, if an XML file is unreadable, corrupt or does not contain valid XML, you may see this error:

    Import Applications: Bad XML Error

    Your only option will be to skip the file.

    If the Windows account that should run the application is not available, you will receive the following notice:

    Import Applications: Unknown Windows Account

    To fix the problem, click the Edit & Fix button to open the application’s properties, switch to the Logon tab and specify a valid account.

    Similarly, if no password appears in the XML file — a common occurrence since AlwaysUp strips away passwords when exporting — a password will be requested:

    Import Applications: Enter Windows Password

    Other errors and warnings may come up as well. For example, the password for your mail server may be missing or the path to your executable may be different on the new server:

    Import Applications: Plex Errors/Warnings

    Once again, you will have to invoke Edit & Fix to continue.

  6. Once validation is complete, you will be presented with a summary of the results:

    Import Applications: Ready to Create Services

    So far no applications have been added to AlwaysUp and no new services have been created. If you cancel now, importing would have done nothing to your computer.

  7. When you are ready to proceed, click Continue to start the addition/creation phase. Here we see the Plex Windows Service being created:

    Import Applications: Adding Plex Windows Service

    Note: Even though the files & applications were already validated, you may still run into trouble while services are being created. As before, you will have the opportunity to fix the problem (or skip the application).

  8. And finally, the process concludes with a summary of the number of applications created:

    Import Applications: Done

All done. Wasn’t that easy?

Posted in AlwaysUp | Tagged , , , | Leave a comment

AlwaysUp 12: Bulk Import & Improvements for Dropbox

New Version of AlwaysUp

AlwaysUp version 12 is out!

Here are the most notable advancements in our popular “run anything as a Windows Service” software:

Easily import application XML files “in bulk” — instead of one at a time

The best way to copy (or move) an AlwaysUp application to a different computer is to:

  1. Export the application to an XML file (from the existing installation), and then

  2. Import the XML file to create a new application (with the new installation)

This works beautifully if you have one or two services, but importing files one by one creates significant overhead if you have 20 applications to restore!

The new “bulk import” capability aims to reduce that overhead. With the new feature, you simply select the files you wish to import and walk through the step-by step wizard to create your new applications.

Take a look at some of the screens:

Easily import your Windows Service applications into AlwaysUp

In the coming weeks, we’ll write an article that digs into this new process in greater detail. But in the meantime, please don’t hesitate to try it!

Smoother shutdowns for your Dropbox Windows Service

Customers running the popular Dropbox file synchronization software may notice that the latest version launches multiple copies of Dropbox.exe. One instance is responsible for file synchronization, while another two are there for crash protection and stability. The whole curious arrangement is described in our “Why are there 3 copies of Dropbox Running on my PC?” post from December 2019.

Dropbox processes in Task Manager

AlwaysUp version 12 understands how Dropbox works and will shut down the three processes in the optimal order. Doing so avoids orphaned/leftover processes whenever the Dropbox service stops.

Other fixes & improvements

  • Our team spent considerable time bulletproofing of the code for Windows Server 2019. The changes are the result of thousands of hours of testing in our rigorous quality assurance simulator.

  • The program’s Help menu now links directly to the most common FAQ entries, to provide expert guidance when customers need it most:

    AlwaysUp Help/FAQ Menu

  • Licensing problems — though rare — are handled much more gracefully now. (We aim to eliminate the problems entirely in a future release.)

  • Our development team managed to sneak in a couple of under-the-hood tweaks for Windows 10 Insider Preview Build 19577, released last week. (Thankfully there don’t seem to be any significant changes to Windows Services in Microsoft’s latest revision of the forever OS.)

As usual, please review the release notes for the full list of features, fixes and improvements included in AlwaysUp 12.

Upgrading to AlwaysUp 12

If you purchased AlwaysUp version 11 (after June 2018), you can upgrade to version 12 for free. Simply download and install “over the top” to preserve your existing applications and all settings. Your registration code will continue to work as well.

If you bought AlwaysUp version 10 or earlier (before June 2018), you will need to upgrade to use version 12. Please purchase upgrades here — at a 50% discount.

See the full upgrade policy for additional details.

Enjoy!

Posted in AlwaysUp | Tagged , , , , | Leave a comment

Q&A: Why doesn’t OneDrive Delete my Files when Running as a Windows Service? [RESOLVED]

Q&A - OneDrive Not Deleting Files
  Our client has AlwaysUp 11.8.3.74 installed on a Windows Server 2019 platform and uses it to deliver OneDrive as a service.

This was set up as per your guide back in August, however, recently we have noticed some issues with OneDrive sync functionality.

My problem is, if I delete a file in OneDrive via the web interface, OneDrive started by AlwaysUp (in this session or background, it doesn’t make any difference) will not delete the file in the local PC folder. If I stop the service and start OneDrive manually via the Start Menu, the file will be deleted immediately.

Everything is working fine, except deleting file via web interface and syncing this deletion to the PC.

Looking forward to your help.

— Bernd

June 11 2021: This issue has been resolved (jump to the details).

Hi Bernd, sorry to hear of the problem! We investigated and here is what we found out:

The problem exists in OneDrive for Windows Version 19.192 (November 13, 2019)

We were able to reproduce the issue described on two Windows 10 test machines — one with version 1809 and the other running version 1903. We performed the following steps:

  1. Downloaded and installed OneDrive Version 19.192.0926.0012.

  2. Followed our step-by-step our tutorial showing how to setup OneDrive as a Windows Service.

  3. Started OneDrive as a Windows Service (in the background, in Session 0).

  4. Logged in to OneDrive online.

  5. Deleted a file from the website.

  6. On the local PC, opened File Explorer to the OneDrive folder holding the file just deleted.

  7. Waited two minutes for the file to disappear from the local folder.

Unfortunately the file was never removed!

When we stopped the OneDrive service and restarted OneDrive normally on the desktop, the file was quickly removed from the local PC — just as you experienced.

Additions, renames and other file operations work fine

It seems that the problem is indeed limited to the propagation of online deletions.

We confirmed that these operations worked flawlessly, with changes synchronized between the OneDrive website and the PC in a few seconds:

  • Adding a file to the OneDrive website

  • Adding a file to the OneDrive folder on the local computer

  • Deleting a file from the OneDrive folder on the local computer

Deletions work properly with Files On Demand

Curiously, online file deletions are reflected on the local PC in one scenario — when Files On Demand (FoD) is engaged and the file has not yet been downloaded to the local computer.

When we executed the test above with FoD on, the file (or rather the “stub” representing the file) was quickly removed after its counterpart was deleted online.

Awaiting a fix from Microsoft

We’ve posted a message to the OneDrive forum asking for help but so far there has been no word from the folks in Redmond.

Hopefully Microsoft will provide a fix soon. We will be sure to let you know if we come up with a workaround before then.


UPDATE: Fixed in AlwaysUp version 12.8 (June 2021)

If you are encountering this problem, please:

  • Upgrade to the latest version of AlwaysUp.

  • Edit OneDrive in AlwaysUp and check the Launch the application without admin rights option on the Logon tab:

    OneDrive Windows Service: Run Restricted
  • Save your changes.

When you next start OneDrive from AlwaysUp, all operations will work properly — including deletions.

Posted in OneDrive | Tagged , , , | 24 Comments

Q&A: How do I Test the “Check Web Server” Sanity Check?

Q&A - Test Web Server
  I’ve already started the trial of the Service Protector and I’m comfortable with the way it is working. I manually stopped my web service and Service Protector started it up right away.

I’m specifically interested in detecting some nasty 502 errors and downloaded the check website BAT file.

Any suggestions on how to simulate or force a 502? My test environments rarely crash (of course) and I need to gain a little bit of confidence before applying to one of our big customers.

— Greg

Hi Greg, thanks for trying Service Protector.

After discussing with our team, here are three ways you can test:

1. Specify a non-working URL

To make the sanity check work, you had to specify your server’s URL in the batch file (in the SET url line).

For example:

      SET url=http://localhost:8080

To confirm that your service will be promptly restarted when the sanity check fails, simply specify a bogus URL in the batch file.

An invalid URL will cause the URL ping to fail and signal Service Protector to restart your Windows Service — every time the sanity check runs.

To perform this test:

  1. Open the batch file in your favorite text editor

  2. Update the SET url line with a URL that does not point to an actual site

  3. Save the batch file

  4. Launch Service Protector

  5. Highlight your web server service and select Application > Start Protector

  6. Wait for the sanity check to be run (as per the frequency you provided when you setup the sanity check)

  7. Validate that your service is stopped and restarted, as expected

2. Update your web application to return a 502

The previous test applies to all failures, not only 502 errors.

If you have control of your web site code — in Laravel, Django, ASP.NET or another dynamic framework — you can create a URL endpoint that simply returns 502.

Put that URL that into the check website batch file (in the SET url line) and your service should be restarted whenever Service Protector runs the sanity check.

3. Use the free webhook service

If you don’t want to mess with your web site, we recommend use the free webhook service to create a URL that always returns a 502. If you place that URL in your batch file, your service should be restarted every time the sanity check is run.

To create the endpoint:

  1. Open this page in your browser: https://webhook.site/

    A unique URL will be created automatically for you:

    Create a Webhook URL
  2. This URL is immediately usable, but it will return HTTP Status code 200 by default. Click the Edit button and change the Default status code to 502. You might as well put in some response text, to explicitly signal what the URL does too:

    Set the Webhook default status code to 502

Visit the URL to ensure that it is responding as expected:

Check that the Webhook URL returns 502

Note that the URL won’t be available forever. From the Webhook.site Terms of use:

  For non-Premium URLs, data may be automatically deleted after a maximum of 7 days.

So you’ll have to purchase a premium offering if you want the URL to persist.

Happy testing!

Posted in Service Protector | Tagged , , , , | Leave a comment