One of the projects in Keyliner's "War and Peace Programming in C#" Student and Instructor Workbook was a disk directory and report program. That student project morphed into a Backup program, which is available here for download. This article documents the program's technical switches and features.
See this article for a general description: DirectoryPulse Introduction
** A new version will be released within a week or so: 2024.03.12
This documentation is under construction and may not be accurate.
Keyliner's DirectoryPulse program is free to download. No registration. No spyware. No installation.
Download Link
https://drive.google.com/file/d/1iXVog877_BVnF0qtLw7cz1tCKn6P1Qsc/view?usp=sharing
DirectoryPulse works like this.
- From the main landing page, add one or more top-level folders to backup
- Click "Refresh Directory Listing" to get a report
- Click the Gear Icon
- Use the Presets to assemble a destination path, and to pick the type of backup.
- Click Apply
Optionally, click "Save" and save the preferences for later re-use - On the main panel, click "Backup"
Saved Preferences can be re-retrieved, making a repeatable backup. This can be scheduled and automated.
Preference File Location:
C:\Users\<login>\prefs\DirectoryPulse
Log File Location:
C:\Users\<login>\prefs\DirectoryPulse\Logs
First-Time/One-Time Recommended Setup:
a. After launching DirectoryPulse, click Gear Icon
b. In form A015, on the right side, click button "System Defaults"
c. Change the default PRESET destination paths to match your local drives.
These are optional settings, but recommended.
There are three major types of destinations:
1. DefaultLocal (typically Drive D:)
2. DefaultSAN (local Network Drive, Synology, etc.)
3. DefaultCloud (typically OneDrive or GDrive)
Type a root path for each destination -- pointing to a pre-built top-level folder. Details, next
 |
| Click for larger view |
- Change DefaultLocal
Set to a local backup drive.
Typically a "D:\" or USB drive.
A root-drive plus a subfolder is required: for example: D:\backup
If a second drive is not available, build a dedicated sub-folder on the C: drive (C:\backups\).
(This is not ideal because it does not protect you from drive failures.)
A "local" destination is normally used for daily "differential" backups. This type of backup can be set to keep 10 versions of each changed file by using the [x] Generations switch.
- Change DefaultSAN
Set to a Network-aware SAN drive, an external USB, or other resource for longer-termed backups. Even though this choice says "SAN", any destination can be used. When pointing to this device, typically use a UNC\share-name or a mapped drive. External USB drives can also be used.
Examples:
DefaultSANBackupDestination = \\SynologyNAS\Data\Backups or
DefaultSANBackupDestination = E:\Backups or
DefaultSANBackupDestination = \\PC-2\share\Backups or
DefaultSANBackupDestination = NONE to disable
The destination requires a root drive and subfolder (e.g. \Data, or Data\Backups); do not write backups to the root of the drive.
Note: if you are not authenticated to this drive, or the drive is offline, setting this field is slow as the program attempts to navigate and test the path. Be patient. The offline drive will time-out. You can continue to build a non-existent paths, on the chance it is available later.
- Change Default Cloud
Typically point to a OneDrive or GDrive local folder
Examples:
DefaultCloudBackupDestination = C:\Users\<username>\OneDrive\Backup
"None" to disable
Replace <username> with your Windows 10 username or browse the Users folder (or leave text as a literal "<username>" and let the program determine the folder). An active OneDrive (or GDrive) account is required.
Use the Cloud destination for offsite backups. When the job is built the destination can (perhaps "should be" set with the [x] Zip option. This improves file-transfer times
and reduces disk quotas. Typically, this does not use dated-folders
or generations.
d. On the same System Default panel, the Default Preset selection list shows pre-built subfolder names. This helps assemble subfolders at the Destination path. This is a comma-separated list, defaulting to: Projects, Daily, Weekly, Monthly, Adhoc
For example,
If the SAN Preset is highlighted ("\\synologyNAS\Data\backups"),
choosing "Daily" appends a "\Daily" subfolder:
Selecting "Projects, Daily, Weekly...." does nothing more than appends a common, predicable name to the already-typed destination path. The folder is built automatically when the backup runs. The assembled name can be overrode and changed by re-typing the destination's path.
I now use and recommend these presets:
"Projects,Data,MyDocs,Weekly,Monthly,Generations,Util,Adhoc"
Best Practices:
-----------------------------------------------------------------------------------
(Assuming a local disk for data, and using OneDrive or a SAN for offsite backups.)
In the Gear menu, use the PRESETS to choose recommended default settings.
Although the Presets are recommendations, they can be manually changed after assembled.
Then Save these settings to a permanent configuration INI file.
1. Create a daily backup configuration.
This is the fastest backup. After the first full backup, all subsequent backups are fast, only backing up changed files.
Set the destination to a Local drive (such as D:), with a dedicated subfolder, such as "\daily".
Make this a non-dated backup*
D:\Backup\Daily
Recommend these switches:
- Do not [ ] Append Date (do not use a Dated-folder)
- Use [x] Generations
"Generations" will only backup changed files and will keep 10 generations of changes. Each day's backup will be fast, only backing up the changed files. This depends on the backup folder name to not have a [date] appendage.
For example, in a particular backup directory:
Filexyz.wpd (current backup version)
Filexyz-001.wpd (slightly older version)
Filexyz-002.wpd (an even older change)
:
Filexyz-009.wpd (the ninth changed version; the oldest)Filexyz-010.wpd (pushed off the stack and deleted from backup)
Give these preferences a name and a description.
Click "Save"
The preference can be retrieved from the main panel by clicking the 123 icon.
2. Create weekly and monthly backups to SAN or an External USB.
For these types of backups, typically I used [x] Allow Dated backups, and I check [x] Limit dated backups. These are good for expansive Data backups or for project backups.
Dedicate a subfolder for each type (e.g. \weekly, \monthly)
For example, destination path: \\SANDrive\Share\Backups\Weekly
or \\USBDrive (F:)\Share\Backups\Weekly\MyProjectBackup
Use these switches:
- [x] Append Date [x] Limit Dated Backups
- [ ] Generations [ ] Zip (unchecked)
Save this preference with a name and description.
It can be re-retrieved from the main panel by clicking the 123 icon
These are full-folder backups, all files, with 10 dated versions. The 11'th version rolls off. These backups can be disk-intensive.
Optionally, select a non-Dated backup folder destination (a project backup)
Use these switches:
- [ ] Append Date (uncheck) [ ] Limit Dated Backups (unchecked)
- [ ] Generations (uncheck) [ ] Zip (unchecked)
On first-time run, it runs a full-folder backup.
Subsequent runs only backup changed files, making for a fast backup. Each file is date-compared with the previous backup. If the file-date is the same, the file is skipped. Although this backup is fast, it only keeps the most recent changes of the files.
3. Create a (monthly) Cloud Backup
Dedicate a subfolder for each type (e.g. \monthly)
For example, destination path: \\OneDrive\Backups\Monthly
Use these switches:
- [ ] Append Date (unchecked) [ ] Limit Dated Backups
- [ ] Generations [x] Zip
If using AppendDate and Limit-DatedBackups, it can build up 10 different .zip versions. This can be expensive for disk space (especially on OneDrive, GDrive). This is not recommended. Instead, treat the .zip backups as "fire insurance."
When using Zip, you cannot use Generations. If checked, it is ignored.
4. Use Project or Adhoc backups for one-off backups.
- [x] Append Date (typical) [x] Limit Dated Backups
[ ] Generations [x] Zip (optionally)
Recommended PRESET Default Chart:
 |
| Click for larger view |
Switches and Settings:
[x] "Append Backup Date"
This is especially useful during scheduled backups or on "project backups."
Keep each project in its own subdirectory and each can have its own date-stamp folders.
Typically used with a smaller project backup, where a recommended folder structure might be:
D:\backups\projects\WarAndPeace
D:\backups\ProfileMyDocuments
\\SynologyNas\Data\Backups\Projects\WarAndPeace
The actual backup destination gets a date appended
D:\backups\projects\WarAndPeace\2021-0522 (date varies)
D:\backups\ProfileMyDocuments\2021-0522
\\SynologyNas\Data\Backups\Projects\WarAndPeace\2021-0522
If dated, each backup behaves as a full, unconditional backup.
Each time the
destination is a brand-new (dated) folder, and because of this, all files are backed
up. There is no date-checking, and no Generations, regardless of [x] Generations checkbox. Reason: Because of the folder-date, it is a new, empty folder - no dates or generations can live there.
[x] Append Backup Date "stacks" multiple copies of backups at the destination -- this can be a serious disk-hog -- but disks are cheap, use them:
D:\backups\projects\WarAndPeace\2021-0501 (a full backup)
D:\backups\projects\WarAndPeace\2021-0502 (a full backup)
D:\backups\projects\WarAndPeace\2021-0503 (a full backup)
[ ] Append Backup Date
If [ ] Append Date is unchecked, backups write to the same folder as a non-dated destination.
The first-time
backup acts as a full-backup.
Subsequent backup to this same folder backup only changed files.
Each file is date-compared with the previous backup.
If the source file-date is the same as the previously-backed up file, the file is skipped.
This makes for a fast backup, but it only keeps the most recent version of the file. You cannot fall back to older dates, or to older generations.
See also "[ ] Use Generations" -- which is slightly different -- keeping 10 versions of the same changed file(s)
[x] "Limit Dated Backups"
If "[x] Append Backup Date" -- and [x] Limit Dated Backups"
The limit switch tells DirectoryPulse to manage the backup inventory by looking at the Destination folder (with a date). Ten copies of the dated-backup destination are allowed to survive, with each new dated version pushing the others down the stack. The 11th version is discarded. This switch is date-sensitive, looking at the subdirectory name. If the name contain YYYY-MMDD, it is are subject to the eleventh-hour delete.
D:\backups\projects\WarAndPeace\2021-0501 (oldest deleted)
D:\backups\projects\WarAndPeace\2021-0502 (a full backup)
D:\backups\projects\WarAndPeace\2021-0503 (a full backup)
:
:
D:\backups\projects\WarAndPeace\2021-0510 (a full backup)
D:\backups\projects\WarAndPeace\2021-0511 (a full backup)
The 11th folder-version auto-deletes, no prompting.
* The Deletes are time-consuming and can delay the start of the next backup.
The ten-count is not adjustable.
I typically use dated-folders for project backups, or with Weekly scheduled backups. Look at the project-level and count the 10 possibilities. This way I have (up to) 10 versions, various dates, of all files -- even if a file was deleted, I can likely find an older folder with a recoverable version.
If other backup types (other projects) are intermingled in this same dated folder (why would you do this?), they will also be deleted. Thus, keep the top-level project folder different. The delete looks at the (subfolders with dates); sorts alphabetically, and keeps the ten-most recent.
To keep a dated-backup from auto-deleting, use File Explorer and manually rename the dated folder to something less-date-like: For
example ...\Projects\WarAndPeace\2021-0516Hold. Or use a different project folder.
If [x] "Limit Dated Backups" is checked but [ ] "Append Backup Date" is not checked, the Limit switch is ignored, without comment. Illogical, but no harm.
[x] "Use Generations"
Generational backups can track 10 versions of an individual file. Each older version is given a number, #01 - #10 and all ten versions are stored in the same destination folder, but each has a slightly different name. The regularly-named file is the most recent backup version.
For example, if backups are run nightly, and a file was modified on Monday, Tuesday,
and Friday (today), that file appears three times in the destination.
testfile.xlsx (today's most current backup has the non-changed name, "Friday's")
testfile-#01.xlsx (Tuesday)
testfile-#02.xlsx (Monday - the oldest file)
To make this work, the backup directory must have the same name (don't use a dated-backup folders). As the backup runs, it looks into the destination and compares each file's date-time stamp with the current file. If the date is different, it makes a new generational backup of that file. If the (modified) dates are the same, the file is skipped.
If the file is a new, first-time backup, it arrives without a #00 number. Subsequent backups will push this file's number to #01, #02, etc.
If the source file has a different date, the original file's backup name is sequentially renamed to #01, #02, ...#10, making up to 10
generations. Each version is shifted to a larger number, where larger numbers are older versions of the file. The 11th
(oldest) version is pushed off the stack and discarded.
If a (daily)-backup is missed, the next run picks
up with the same date-check, with no harm done, again shifting versions as
needed.
"Use Generations" only works if the destination folder is the same name on each backup - that is a fixed, non-dated backup
destination. -- If using "[x]Append Backup Date", the folder name changes each
day and Generations are not given the opportunity to stack.
[x] "Zip Folders"
Each destination folder is compressed into a single ZIP
file. A folder with 100 files will be compressed into one ZIP, with 100
files within. If the backup traverses 50 other folders, 50 ZIPs are
built; one for each folder (assuming files are in that folder).
This is best used when writing to OneDrive or GDrive for off-site backups. Compressed files are faster to upload and occupy less disk quota.
Recommendation: Use [x] Zip Folders for small project backups, not for big all-encompassing DATA or MyDocuments backups. Zips are not convenient to recover from because the folders have to be manually un-zipped.
Zip backups do not check file date-time-stamps. Generations cannot be used with Zip backups. Each Zip is assumed to be new and unique and each Zip always contains all files in that folder.
[x] "Auto-Run"
Auto-Run
is intended for command-line or Windows Task Scheduling, and is
required for tasks called by Windows Scheduler. This switch allows the backup to run unattended. This switch means "run unattended." If you happen to use this preference from the desktop app, it behaves slightly differently. If run from a batch or task-scheduler, it runs unattended. Details about this are too unimportant to document here.
Saving Preferences and Settings:
Multiple jobs (preferences) can be saved at the Gear icon, as documented above.
Each preference file is saved as an INI file.
See folder C:\Users\(your name)\Prefs\DirectoryPulse\INI.
Typed or selected Destinations, and all program-switches, are stored with a name and description of your choosing. Once saved, they can be re-loaded from the main landing panel by clicking the 123 icon:
When saving a backup preference, give the preference file a name and description to help remind you of the settings. This is visible in the 123 menu. Illustrated are 7 different preference files, with 7 different destinations and other settings:
Adhoc Backups:
Gear-icon and other settings can be set without saving the changes in a Preference file. From the preferences screen, click "Apply" (without "Saving"). Return to the main panel. Click Refresh, then Backup. "Adhoc" backups do not have to write to the "Adhoc" destination path.
Scheduling Backups:
Use Windows standard "scheduler" to schedule periodic backups: See Microsoft documentation for details on Windows Scheduler.
Click Start, Run, "Task Scheduler". Pass command-line parameters when launching.
Command Line Parameters:
DirectoryPulse can be launched from a command line with optional parameters to automate backups. Or a desktop (shortcut) icon can be built with these same parameters.
By passing a preference file from either a
Shortcut, Windows 10 Tile, or from Windows Task Scheduler, tasks can be fully automated. For example:
C:\Program Files\Util\DirectoryPulse.exe ini=MyFavoritebackup.ini
where:
MyFavoriteBackup.ini is the name of a previously-saved DirectoryPulse
Preference file
[x]Auto-Run must be flagged in the INI file
When
launched with a Preference/INI file, the program loads, retrieves the preferences, scans
the directories, and launches the backup. When the backup completes,
the program auto-closes. Log files record the transaction. Automation requires [x]Auto-Run as one of the switches inside the preference file.
Log Files:
ASCII-text log files are found in
C:\Users\(your name)\Prefs\DirectoryPulse.
The 15 most-recent logs are kept. This count is not adjustable.
Other files:
When first launched, DirectoryPulse.exe creates several files in
C:\Users\<username>\Prefs.
In here, find small control (.ini) files, optional reports, and log files. In
this same area, note the file "dirCreate.txt" -- this is created when "Refresh DOS Directory Listing" is clicked. Of interest, this
is nothing more than a DOS DIR listing, which uses this command:
dir (a directory name/*.*) /-C /N /oG /S >C:\Users\...\dirCreate.txt
This is the input file used by the backup program. Basically, if DOS can see the file, DirectoryPulse can work with it.
Restores:
DirectoryPulse uses simple file copies and the backups live in a folder-by-folder reconstruction on your backup drives. To restore a file or folder, use File Explorer to copy that file/folder back to the source drive. This program does not help in the restores.
To recover a generational file (e.g. testfile-#01.xlsx", copy the numbered version and then manually rename, removing the -#00 appendage. Because some files may have more than one version, updated on different dates, restores cannot be automated. Perhaps Restores can be a future enhancement.
Other Comments:
DirectoryPulse is not an image backup and cannot be used to recover a crashed hard disk. Instead, it is meant to keep operational copies of data, which an help recover from spreadsheet-blunders, ransomware, viruses, etc.
DirectoryPulse is not meant to backup Windows System folders, or ProgramFiles folders, and it actively avoids these areas. In any case, restores from program folders would be flawed because of Registry and other concerns.
However, from the Settings panel, the current-session switch "[x] Allow System Files" will attempt to backup C:\Program Files, and similar folders -- but in no case will it fiddle in the C:\Windows folders. This is by design.
Finally, inside of the settings screen "A020 System Defaults," is a link to a system-wide INI file. Obscure settings, not exposed in the configuration panels, can be found and adjusted. For example, File and Directory exclusion masks. Use care when editing this file.
OneDrive files:
DirectoryPulse sees and reports OneDrive files, but avoids off-site pointers during a backup. See Users\YourName\OneDrive.
The
reason: A backup of an off-site OneDrive file causes it to download locally to your PC and could be gigabytes of network and disk activity. DirectoryPulse assumes Onedrive is an adequate backup and it will ignore the file pointers. But, if the one-drive file has already downloaded locally, and it is in the backup path, DirectoryPulse will back it up.
Finally, with any backup strategy, it is wise to make Offline or Near-Offline backups -- backing up to a External USB drive -- and then disconnect the drive. In the event of a ransom-ware attack (where every file, every drive, every SAN, is encrypted, that drive would be safe. OneDrive backups have some protections against Ransom-ware attacks, but it may not detect the attack in time to prevent some of the damage.
This program was a blast to write and I use it daily. I hope you enjoy using it too. Your comments are welcome and I would like to know if you find the program useful.
Related Article:
DirectoryPulse Introduction
No comments:
Post a Comment
Comments are moderated and published upon review. (As an aside, not a single spam has been allowed through; why bother?)