DirectoryPulse - Keyliner Backup Software

DirectoryPulse - keyliner Backup Software

keyliner has written a free Windows backup program.
-2024.08 - Version 2.00 - Updated with new features

Backup large swaths of directories to a separate backup drive.  Files can be saved in dated folders, as compressed files, or as generational backups (multiple versions of files).  Jobs can be saved and re-loaded, and scheduled for automatic runs. 

Backup are non-proprietary and fast!

Write to local drives, SAN drives, and cloud drives, organized by project, by date, etc.

(Click for larger view)

Features:

* Backup to a local disk, SAN, One-drive, GDrive, etc.
* Non-proprietary backup - straight-forward file-copies

* Backup to optional dated-folders (YYYY-MMDD)
* Supports Generational backups
   (multiple versions of same file, V1, V2, etc.)

* Optional ZIP backups
   (reduce upload times to OneDrive)

* Can skip non-changed files
   Automatically skip cache folders, temp files, and hibernation files
   Automatically skips Windows System Files
   Skips One-drive virtual file links (saving re-downloading)

* Manages its own backup inventory, erasing old backups
* Configurations can be saved for easy retrieve and re-run
* Automate and schedule with Windows Scheduler
* Occupies zero RAM or resources when not in use; no background tasks

* Includes a nifty Directory Report that displays a sortable report, showing file-sizes and counts in a way that file explorer does not.  Use this to find your biggest disk hogs.

DirectoryPulse was originally written as a Student and Instructor exercise in Volume 6 of the Computer Programming Book, War and Peace Programming in C#  -- Amazon Kindle, written and published by keyliner.  It grew into this program.

Try it:

See below for download and installation instructions.

1.  Launch program, "DirectoryPulse.exe" from the Start Menu tile or by double-clicking the .exe.

The program was written with C#, .DotNet 8.0. When first launched, you may be prompted to install Microsoft's .DotNet 8.0 libraries.

2.  On first-launch, you are prompted for default backup destinations.  These are one-time settings. 

Most users setup these destinations:

D:\drive (local secondary drive, or USB disk)
SAN  (local network disk, or assign to a USB disk)
Cloud  (Onedrive or GDrive backup folder)

For each destination, type or browse to a top-level "backup" folder name -- such as "Backups"; all backups, of any type, will live in this folder.  (Optionally, browse to each destination.  Use the word "none" to disable the SAN or CLOUD destinations.) 

Click for larger view

Save the changes and return to the main panel. 
The program builds the top-level destination directory automatically when you close the panel. 

3.  On the main panel, select which directories to backup.    

In the top-most field, type the path, browse, or use File Explorer to drag-n-drop a folder onto the target. 
(DirectoryPulse! only cares about folder-backups; you cannot select an individual file.)

Optionally, right-mouse-click the field and choose "Add User Profile Documents".
+Multiple folders can be stacked into the backup job.

When done adding folders,

4.  Click Button "Refresh DOS Directory List"

This inventories the folders and shows a nifty report, showing all directories and subdirectories, with file counts and sizes. Sort by column-headings; right-mouse-click for a context menu.  This list is used by the backup.

5.  Session Settings: 

Set a backup destination and other preferences by clicking the Gear icon:

Click the "Gear" icon (-- this job's configuration settings)
Name this configuration file (e.g. "MyFavoriteBackup" or "Local-DataFolder")
Type a short description
Choose the type of backup ("Dated Folder [1-10]" - recommended)

Click for larger view

Set a pre-named sub-folder (which helps to build-out a destination-path).  Subfolders, such as "Projects", "Monthly", "Adhoc" are offered.

Once the subfolder is selected, a suggested backup destination path is constructed.  This is a recommended value, but can be changed manually to any other path (destination).  Destination folders are built automatically when the backup starts. 

The default subfolder names and other paths are adjustable.  See "System Defaults", this panel.

See the next article for Detailed Switch documentation.

6.  "Save" these preferences or click "Apply" to only use for this session.

7.  From the main screen, click bottom-row button "Backup"

The backup runs.  The backup re-creates the source-path (backup folders) in the destination path, creating each directory, one-for-one.  For example, if backing-up C:\data\subfolder1, the backup destination might be D:\Backups\adhoc\data\subfolder1.

When done, a report displays, along with a log file.

You are done!

If the backup type is "Dated Folder [1-10]", the backup destination has a subfolder: 
For example:  D:\Backups\Adhoc\2024.0804.  In this folder is a duplicate of the source drive's directory structure, with all eligible files.

Each time this configuration is run, it creates a new folder, keeping 10 generations of backups.  On the 11th backup (the 11th date), the oldest date is automatically deleted -- a self-cleaning backup.

Details and additional program documentation can be found here:
Detailed Switch documentation

Installation Steps:

DirectoryPulse is free to download and use for personal and commercial use. 
No registration, no login, no email. No advertisements, no nags, no spying.
Keyliner does not (and cannot) track who downloads or runs this program.

Installation is easy:
Download the .exe (.zip) and its supporting files into any folder on your hard disk.
Double-click the .exe to run - no installation required. 

Using the .exe from a download folder, or copying to a (my Documents) folder is a quick workaround for various Windows security concerns.  Some vendors recommend this, but these folders are inappropriate for executable software.  Instead, the program should be copied to Program Files so it gains the protection of other Windows security features. Total time: about a minute.

Follow these steps for a more professional installation:

A.  Download the .exe and support files (as a .zip) to a Download or Temp folder:

From Keyliner's public GDrive, click this link and download to a local temp or download directory.  

Download Link
https://drive.google.com/file/d/1WrvEVFQUPBWpxuIRMI5vINBkSesuGCd4/view?usp=drive_link

When downloading, different browsers behave differently.
Select "Save-As"

Windows security will not let you download directly into Program Files -- you must save to a temp directory (technically, you will not be able to remove the "mark of the web" if downloaded directly into Program Files).

Since keyliner cannot afford a signing certificate, you will be prompted that the file is not safe (being downloaded from the internet).  Click "more information" and allow the program to download/run.

B.  Mark the download as safe-to-run:

Using File Explorer,

Right-mouse-click the downloaded DirectoryPulse.zip
Select "Properties"
Check [x] Unblock.  (This removes the "mark of the web.")

                 Click for larger view

* Only do this if you trust keyliner *and* only if downloaded from keyliner's public GDrive. 

If "Unblock" is not visible, it has already been unlocked (by Microsoft Edge).
Once [x] Unblocked is clicked, this security menu disappears.

C.  Create a Program folder to hold the program:

Using File Explorer, open folder C:\Program Files,
Create a new utility folder:  C:\Program Files\Util

D.  Copy the .exe and two support files to ProgramFiles\Util:

Using File Explorer,
Double-click to open the .zip folder

Copy/paste three files from the zip,
Paste to C:\Program Files\Util

DirectoryPulse.exe
DirectoryPulse.dll
DirectoryPulse.runtimeconfig.json

Do this copy as a two-step, copying from the temp/download folder, then into Program Files.  

E.  Create a Start Menu Tile:

Using File Explorer,
Tunnel to C:\Program Files\Util
Right-mouse-click the DirectoryPulse.exe and "Pin to Start"
The program is ready to run.  See icon on Start Menu.

On first-time launch, you are prompted for default backup destinations -- this is where you choose where you want the backups to write to.  See details earlier in this article, with additional program documentation here: Detailed Switch documentation



Version history:
2.00 - 2024.0810  Numerous design changes. 
          Improved Configuration selection and prompts
          Simplified Backup-type and option settings
          Added subfolder-cosmetic tagging
          Added drag-n-drop support
          Added eyecandy to help guide decisions
          Improved backend INI file settings

Note: Version 1.x INI config files not compatible with this version; sorry.  You will have to rebuild the config files.

1.05 - 2023.0430 Moved the Config/INI lookup button to top of panel
          Added cosmetic "Destination:" during the backup
          Added a "Delete" button (in addition to the Context menu) for INI Delete
1.04 - Fixed bug where sometimes root drive not inventorying: "Unexpected error in DOS directory"
1.03 - Minor changes to the report Log file, making it easier to find skipped directories
1.02 - Not released
1.01   2021.0610 Initial Release

Thank you to my Beta-testers: DLW, of Boise.

DirectoryPulse - Other documentation

 
DirectoryPulse! is a backup program for Windows computers and was written and tested with Windows 11.  The program should work with older versions of Windows.  

See this article for a general description and download instructions: DirectoryPulse Introduction

keyliner's DirectoryPulse program is free to download.  No registration.  No spyware.  No installation. 

DirectoryPulse works like this. 

• From the main landing page, add one or more top-level folders to backup
• Click "Refresh Directory Listing" to get a report
  
• Click the Gear Icon
• Use the Presets to assemble a destination path, and to pick the type of backup.
  
  
• Click Apply
  Optionally, click "Save" and save the preferences for later re-use
  
  
• On the main panel, click "Backup"

Saved Preferences can be re-retrieved, making a repeatable backup.  This can be scheduled and automated.  

First-Time/One-Time Recommended Setup:

On first-time launch, a "SystemDefaults" panel is displayed. 

The program was written with C#, .DotNet 8.0. When first launched, you may be prompted to install Microsoft's .DotNet 8.0 libraries.

A.  Change the default PRESET destinations to match your local drives.  For example, if you have a D:\ Drive, you can use it for quick, local backups.  The destinations are optional but recommended

There are three major types of destinations:

1.  DefaultLocal (typically Drive D:; can be an external USB drive)

2.  DefaultSAN (local Network Drive, Synology, etc.  Can be a USB drive. Disable with the word "none" -- or just ignore.)

3.  DefaultCloud (typically OneDrive or GDrive)

Type, browse, or drag-n-drop a root path for each destination -- pointing to a pre-built top-level folder, such as "D:\Backups".  Details below.

 

- Change DefaultLocal 

Set to a local backup drive.
Typically a "D:\" or USB drive. 
A root-drive plus a subfolder is required: for example:  D:\backup 

If a second drive is not available, build a dedicated sub-folder on the C: drive  (C:\backups\). 
(This is not ideal because it does not protect you from drive failures.) 

- Change DefaultSAN

Set to a Network-aware SAN drive, an external USB, or other resource for longer-termed backups.  Even though this choice says "SAN", any path can be used.  When pointing to this device, typically use a UNC\share-name or a mapped drive.  External USB drives can also be used. 

Examples:

DefaultSANBackupDestination  = \\SynologyNAS\Data\Backups   or
DefaultSANBackupDestination  = E:\Backups   or
DefaultSANBackupDestination  = \\PC-2\share\Backups  or
DefaultSANBackupDestination  = NONE to disable

The destination requires a root drive and subfolder (e.g. \Data, or Data\Backups).  You cannot write backups to the root of a drive. 

Note:  if you are not authenticated to this drive, or the drive is offline, setting this field is slow as the program attempts to navigate and test the path.  Be patient.  The offline drive will time-out.  You can continue to build a non-existent paths, on the chance they are available later.

- Change Default Cloud  

Typically point to a OneDrive or GDrive local folder

Examples:

DefaultCloudBackupDestination =  C:\Users\<username>\OneDrive\Backup
"None" to disable

Gdrive's location is:  
DefaultCloudBackupDestination = C:\Users\<username>\AppData\Local\Google\DriveFS

Replace <username> with your Windows 10 username or browse the Users folder (or leave text as a literal "<username>" and let the program determine the folder).  An active OneDrive (or GDrive) account is required.  

Use the Cloud destination for offsite backups. When the backup job is built, the backup can (perhaps "should be") set with the [x] Zip option.  This improves file-transfer times and reduces disk quotas.  Typically, this does not use dated-folders or generations.

Continue with these other System Preferences:

d.  On the same System Default panel, the "Default Subfolder" field shows a comma-delimited list of recommended subfolder names.  These are cosmetic names that help build the assembled destination path.  Review the list, adding or subtracting entries. 

This is a comma-separated list, defaulting to:  Projects, Daily, Weekly, Monthly, Adhoc, Test.  Selecting "Projects, Daily, Weekly...." does nothing more than appends a common, predicable name to the already-typed destination path.  The assembled path can be overrode and changed by re-typing the destination's path. 

Click SAVE and close the SystemDefault panel.


During the Actual Backup
-------------------------------------------------------------------

Naming the Configuration Settings:

During the actual backup (the "Gear" icon -- see previous article for how this looks), the job can be named and saved. This way the same backup, with the same settings, can be launched at any time. 

There is an art to naming the configuration files -- name the files in a way that makes sense to you when looking at them a month later. 

Here are the names I am using:

Local-AllData-Dated.ini
Local-Util-Dated.ini
OneDrive-Projects.ini
OneDrive-Projects-Compressed.ini
OneDrive-ProgrammingBook-Compressed.ini
SAN-AllData-Dated.ini
Test.ini

When named and saved, the configuration files are saved as clear-text in
C:\Users\<you>\Prefs\DirectoryPulse\iniFiles

These are simple ascii text files and can be edited, deleted, and renamed at will from within File Explorer.  There is nothing magic here.  (The Config-file select icon also has a context-menu for manipulating the files within the program.)

Gear Details:
-------------------------------------------

Backup Types:

From the configuration panels (the gear icon), choose the backup type, where I typically use "Dated Folder":

 

Dated Folders:


Dated-folder makes a full backup of every file and subdirectory in the selected paths.  The entire backup's path and structure is re-created within the dated-folder. 

D:\Backups\myProject\2024-0801\(C:\)Data\subfolder\deeper-folder\way-down-deep   etc.

This type of backup allows you to go back in time and recover older versions of the file -- or entire subdirectories, but the backup is relatively slow -- having to make a copy of each file, each time the backup runs. 

The option, [x]Limit Dated Backups keeps 10 dated generations of backups -- ten versions of the entire disk structure.  On the 11th generation, the oldest folder is auto-deleted, making for a self-cleaning backup.  This is the type of backup I use the most often.  Be aware this type of backup can occupy considerable space on the backup drive.

Click for larger view

To preserve a backup (to keep it from self-cleaning), use File Explorer to rename the dated-folder to a different, non-date-like name.  For example:  Hold-2024-0801


Generational Backups:

This is the fastest backup and is similar to traditional "Differential Backups".  Only one folder structure is built.  The first-time backup is slow -- having to backup every file in the structure.  Subsequent backups only backup changed files.

When a changed file is detected, the older version of the file is renamed and the new version arrives -- first in the stack:

For example: 

myFile.xlsx      -- the most current version
myFile-#01.xlsx  -- the previous version (from the last backup)

myFile-#02.xlsx  -- the next oldest version

On the 11th version (the 11th edit), the oldest is discarded.

With this type of backup, a backup-subdirectory might contain 10 versions of the same file; each dated.

Because this backup only backs-up 'changed' files, it is a faster backup.  But because the generational-files "intermingle" in the same folder, recovering an entire folder is messier.  This is really meant for small-volume recovery.  Use this for "transactional backups" -- small data-folders with volatile files -- word processing documents, spreadsheets, and the like.

This type of backup can take significantly less space than the Dated-Folder backups.

Added benefits:  If a file is deleted from the original (Source) drive, older versions of the file remain in the backup-set and can be manually recovered.  In other words, older generations of the file remain "forever" in the backup. 

If the folder has 'lots-of' delete activity, the deleted files continue to occupy space in the backup.  The deleted files can be recovered by manually deleting the entire backup structure and starting over -- at the risk of losing other generational backups.  If you do this, make an immediate new backup as soon as possible.  Perhaps, some day I'll write an "orphan cleanup routine."

Zipped Backups:

Zipped backups are full-backups (no #generations), but the entire folder is backed-up as a ZIP file.  The Zip backup is typically used for OneDrive/GDrive/AWS backups and the zip-files take obviously less space and less time to transmit. 

For example:

C:\Data\subfolder1\50-files
becomes one zip file:  D:\backups\deepstorage\Data\subfolder1\subfolder1.zip

C:\Data\subfolder2\400-files

becomes  D:\backups\deepstorage\Data\subfolder2\subfolder2.zip

 

This is meant for archival backups.  In a disaster, recovery is by folder, one folder at-a-time.  This is not ideal if an entire disk is lost, but is useful for recovering relatively small number of individual files or folders.

-------------------------------------------------------------
Other switches:

[x] "Auto-Run"

Auto-Run is intended for command-line or Windows Task Scheduling, and is required for tasks called by Windows Scheduler.  This switch allows the backup to run unattended.  This switch means "run unattended" -- with no prompting, and the program auto-closes when done. 

Scheduling Backups:

Use the Windows built-in "Scheduler" to schedule periodic backups: See Microsoft documentation for details.  In summary:  Click Start, Run, "Task Scheduler".  Pass command-line parameters when launching, see next. 

Command Line Parameters:
DirectoryPulse can be launched from a command line with optional parameters to automate backups.  Or a desktop (shortcut) icon can be built with these same parameters.

By passing a previously-built preference file (ini file) from either a Shortcut, Windows Start Menu Tile, or from Windows Task Scheduler, tasks can be fully automated.  For example:

C:\Program Files\Util\DirectoryPulse.exe ini=MyFavoritebackup.ini

where: 

MyFavoriteBackup.ini is the name of a previously-saved DirectoryPulse Preference file
[x]Auto-Run should be (must be) flagged in the INI file

When launched with a Preference/INI file, the program loads, retrieves the preferences, scans the directories, and launches the backup.  When the backup completes, the program auto-closes.  Log files record the transaction.  Automation requires [x]Auto-Run as one of the switches inside the preference file. 


[x] Delete Path before Backup

This switch deletes the entire destination folder prior to running the new backup.  This is a brute-force switch and can only be used with "SIMPLE" backups.   Other backups, such as "Dated" backups, self-clean and do not need this setting.  With Generational backups, this switch is illogical and cannot be selected.

[x] Discard Cache Directories

Recommended.  Discards obvious cache directories in Firefox, WordPerfect, and other such programs.  The list of discardable directories and keywords is adjustable -- see the SystemConfig.ini file.

[  ] Allow System Files

Allows backups in directories such as C:\Program Files, C:\Program Files\Common Files.  This is not particularly recommended.  Using this switch will backup the files, if they are in the "SourcePath" list, but it does not backup Registry or System DLL's that might be required.  This is a typical restriction for installed programs.

[  ] Use last  DOS Inventory

Mostly used for diagnostics and this switch should not be used.  Under the hood, when clicking "Refresh Directory List", the program writes a DOS DIR - an ascii DOS Directory /s listing which shows the source paths.  A report is written as a temporary file (see users\prefs).  This checkbox says to keep the previously-built report and do not re-inventory.

Other files:

When first launched, DirectoryPulse.exe creates several files in
C:\Users\<username>\prefs. 

In here, find small control (.ini) files, optional reports, and log files.  In this same area, note the file "dirCreate.txt" -- this is created when "Refresh DOS Directory Listing" is clicked.  Of interest, this is nothing more than a DOS DIR listing, which uses this command:

dir (a directory name/*.*) /-C /N /oG /S >C:\Users\...\dirCreate.txt

This is the input file used by the backup program.  Basically, if DOS can see the file, DirectoryPulse can work with it.

SystemConfig.INI
Contains global settings, including destination paths, and discard directories.
This is a simple ascii text file and can be edited with care.
If you screw-up this file, see the nearby "controlFileBackups" folder or simply delete the file to start over.

Log Files:

ASCII-text log files are found in
C:\Users\(your name)\Prefs\DirectoryPulse\Logs 

The (15) most-recent logs are kept.  This count is not adjustable.

Easter Egg:  Hover the mouse here.

Restores:
 
DirectoryPulse uses simple file copies and the backups live in a folder-by-folder reconstruction on your source files.  To restore a file or folder, use File Explorer to copy that file/folder back to the source drive.  This program does not help in the restores.  Perhaps a later version.

To recover a generational file (e.g. testfile-#01.xlsx", copy the numbered version and then manually rename, removing the -#00 appendage.  Because some files may have more than one version, updated on different dates, restores are not automated.  

Other Comments:

DirectoryPulse is not an image backup and cannot be used to recover a crashed hard disk.  Instead, it is meant to keep operational copies of data, which can help recover from spreadsheet-blunders, ransomware, viruses, etc.

DirectoryPulse is not meant to backup Windows System folders, or ProgramFiles and it actively avoids these areas.  In any case, restores from program folders would be flawed because of Registry and other concerns. 

However, from the Settings panel, "[x] Allow System Files" will attempt to backup C:\Program Files, and similar folders -- if they are in the backup-list (Source Process Paths) -- but in no case will it fiddle in the C:\Windows folders.  This is by design.

OneDrive files:

DirectoryPulse sees and inventories OneDrive files -- especially if "MyDocuments" is in the backup path -- but this program avoids off-site pointers.  In other words, if OneDrive has a pointer to an off-line file, DirectoryPulse skips that file, even if it is in the backup path, and even if the file were marked as "changed."  (See Users\YourName\OneDrive).

The reason:  A backup of an off-site OneDrive file would cause it download and expand locally on your PC.  This could be gigabytes of network and disk activity.  DirectoryPulse assumes Onedrive is adequate and it will ignore the file.  

But, if the OneDrive file has already downloaded and expanded locally, and it is in the backup path, DirectoryPulse will back it up -- assuming the file has had a recent change.  This does not incur any extra overhead from the network or disk gods.

Backup Thoughts:

With any backup strategy, it is wise to make Offline or Near-Offline backups -- backing up to an external USB drive -- then disconnect the drive.  In the event of a ransomware attack (where every file, every drive, every SAN, is encrypted, that drive would be safe.  

Consider offsite backups.  Consider .ZIP backups.


Slow Backups to USB devices:

DirectoryPulse does nothing special when writing to the backup drive -- the technique is similar to an xcopy or robocopy. 

I have found some USB disks -- particularly "thumb drives" -- are painfully slow -- especially if the backup set is large.  The slowness is regardless if DirectoryPulse is being used or not.  This is a gnarly problem and seems best resolved by not using that type of device.

See this article for hints on how to improve this.
https://keyliner.blogspot.com/2010/07/acronis-2010-usb-drive-backup-speed.html

Preference File Location:
C:\Users\<login>\prefs\DirectoryPulse

Log File Location:
C:\Users\<login>\prefs\DirectoryPulse\Logs

--------------------
This program was a blast to write, and I use it daily.  I hope you enjoy using it too.  Your comments are welcome. 

Related Article:
DirectoryPulse Introduction
USB Drive Backup Speed Slow