The source code of gblk is licensed under a [AGPL3](license.qmd) Licence
## Description
## Description
The git borg linker utility (abreviated as gblk) is a tool that aims to ease the usage of [borgbackup](https://borgbackup.readthedocs.io/en/stable/) in a project using [git](https://git-scm.com/) as a version control system.
The git borg linker utility (abreviated as gblk) is a tool that aims to ease the usage of [borgbackup](https://borgbackup.readthedocs.io/en/stable/) in a project using [git](https://git-scm.com/) as a version control system.
gblk is meant to be used inside a project using git as a version control system. It will only be helpful if your projet folder contains a `.git` folder. A `results` folder must be present in your project directory as gblk will try to backup it.
You can check the [gblk's documentation here](https://lbmc.gitbiopages.ens-lyon.fr/hub/git_borg_linker/).
\ No newline at end of file
To sum up gblk must be used in a folder having this minimal structure:
```bash
project
├── .git
└── results
```
To display the help of gblk run the following command:
```console
$gblk help
gblk
A tool used to link borg and git together
USAGE:
gblk <SUBCOMMAND>
OPTIONS:
-h, --help Print help information
SUBCOMMANDS:
checkout Checkout results to the current git commit
clean This command cleans the .tmp repository of the project folder
clone Clones a repository given a destination
commit Save the results folder of a git repository in an archive
compact This command frees repository space by compacting segments
config This command can bed used to add gblk configuration
create-hooks Create github hooks to use gbl automaticaly after commit, before and after
checkout
delete This command deletes an archive from the repository or the complete
repository
delete-hooks Remove the post-checkout and the post-commit hooks
diff Show differences between two commits of the `results` folder
help Print this message or the help of the given subcommand(s)
init Initialize a borg repository inside a git project
list List the content of the .borg archive
mount Mount an old file/directory from one or multiple archive named after git
commits into the .mount folder inside the project directory
pre-co Check if a checkout can be performed without losing data
prune This command prunes the .borg repository. This can be used to keep only
archive created during a given time interval
pull This command can be used to pull a repository using a remote
push This command can be used to push a repository using a remote
remote This command can be used to add a new remote for push and pull commands
restore This command moves the borg folder .tmp/<PROJECT_DIR>_bkp folder into .borg
repository
umount Unmount everything in the folder .mount
```
You can type `gblk help <SUBCOMMAND>` or `gblk <SUBCOMMAND> --help` to display the help of any given subcommands.
Note that **`create-hooks` subcomand can be abbreviated to `ch`, and `checkout` subcommand can be abbreviated to `co`**. For example `gblk co --help` will work the same as `gblk checkout --help`
### Example usage without git hooks
Usage of gblk (init, commit, pre-co, checkout) without hooks
```console
$mkdir project
$cd project
$mkdir results src
$git init
$gblk init # creation of a .borg repository at the root of your filesystem
$exa -a--tree--level=1
.
├── .borg
├── .git
├── results
└── src
$
$# Creation of a simple script that creates a result file
$# The next command is important: it will check if your results folder doesn't contain new results compared to your archive with the current git id. If there is no errors, then you wont lose any data
$gblk pre-co
$git co 62efe302b6c2e7ab0dfd9c08ddfb0a87ea699c6d
$gblk co --mode hard # hard is used to delete file that were not present in the first commit. Otherwise only existing files at the destination commit will be updated.
$ls results
result.txt
```
Not: if gblk pre-co says that you might lose data compared to the saved version of your actual commit, then use `gblk commit --update`.
### Example usage with git hooks
Git hooks are commands that can be automatically executed before and after some git commands. They are defined in the repository `.git/hooks`.
gblk can create two hooks:
*`post-commit` hook that executes `gblk commit` after every git commit
*`post-checkout` hook that execute:
1.`git co` to revert back to the last commit as a pre-checkout hooks doesn't exits.
2.`gblk pre-co` to be sure to not lose any data before the actual chekout
3.`git checkout` do the actual chekout
4.`gblk co` to revert back to the results folder corresponding to your
target commit
As the `pre-checkout` hook doesn't exits, this is the `post-checkout` hook that
is used to cancel the first checkout and check for data loss.
Note that when gblk creates hooks it also modifies the `.git/config file` to add 3 aliases:
1. alias `co`: Performs a quiet checkout. This alias is used in step1 of the
post-checkout hooks, so **it is recommended to use it when you perform a
checkout**. It allows to have a quiet initial checkout that is then quietly
reverted so gblk can check that no data is lost.
2. alias `conh`: This alias performs a checkout without the `post-checkout`
hooks. This can be usefull when you perform a checkout to a deleted commit
on your `.borg` archive. If you want to checkout to another commit, gblk
pre-co will prevent that because it will think that the results folder was
not commited with `gblk commit`. To perform a checkout anyway you can use:
Note : basculement sur '2da3c535543fb9a216b52f29ecf598b6310c1223'.
...
HEAD est maintenant sur 2da3c53 src/script.sh: initial commit
$gblk co --mode hard # hard is used to delete file that were not present in the first commit. Otherwise only existing files at the destination commit will be updated.
$cat results/*
File: results/result.txt
result line
$git co master
$cat results/*
File: results/newresult.txt
newresult line
newline
File: results/result.txt
result line
```
## mount command
Gblk as two commands `mount` and `unmount` that can be used to respectively
mount your borg archives into the `.mount` folder of the project directory or
unmount them.
> **⚠ Important note** : When the borg archive is mounted, borg is locked ! It
> mean that you can get the following error message if you forget to unmount a
> borg acrchive when calling a **borg command** on it:
> ` Failed to create/acquire thelock ..` . To resolve it just use
> `gblkt umount`. gblk will **silently unmount** the borg archive before
> commit, checkout, pre-checkout and mount commands.
### gblk mount usage:
```text
gblk-mount
Mount an old file/directory from one or multiple archive named afert git commits into the .mount
folder inside de project directory
USAGE:
gblk mount [OPTIONS]
OPTIONS:
-c, --commit <COMMIT>
Commit name, sh: Glob is supported. This is an optional parameter: if not set then all
commit archives will be mounted into the .mount directory
-d, --diff
Displays the differences between two files mounted corresponding to the given path.
Note that if only one file is recovered then, the other is taken from the current result
folder
This option is deactivated when used with --diff
-h, --help
Print help information
-l, --last <LAST>
Consider last N archive after other filter were applied
-p, --path <PATH>
The file/directory to extract. This is an optional parameter. If not set then all files
in the archive will be displayed
-v, --versions
If set, displays the .mount directory in 'version view'.
- Normal view: The `.mount` directory contains a subfolder with the name of archives.
- Version view: The `.mount` directory contains the results folder and every file within
it becomes a directory storing every version of that file
```
### Examples:
```console
$gblk mount # mount all archives in `.borg` into `.mount` folder
$gblk mount -v# mount all archives in `.borg` into `.mount` folder in version view.
$# It is not necessary to execute "gblk umount" before the above command because its run silently before "gblk mount"
$gblk mount -c'[ab]*'# mount all archives named after commits strating with 'a' or 'b'. Note that the quotes are necessary for -a options
$gblk mount -c'ae8rt77*'-p'results/fichier.txt'# mounts all files matching 'results/fichier.txt' inside archives named after commits starting with'ae8rt77*'
$gblk mount -p'results/**/*.txt'# mounts all txt files inside every archives
$gblk mount --last 2 # mount the last two archives named after the last commits
$gblk umount # unmount the archive mounted into `.mount` directory
```
If we have two archive containing a file named `file.txt` and we want to
direclty compare them in a similar way as `git diff`, then we can enter:
```
gblk mount -p 'results/file.txt' --diff
```
The differences between the two files will be displayed with delta:
**If only one file matches the path given with the `--path` argument of the `mount` command, then, `gblk` will search if a match can be found in the current `results` folder.**
Learn how to customize delta display by going [here](https://github.com/dandavison/delta)
Note that you can also display the differences between images. To be able to do
so, [imagemagick](https://imagemagick.org/script/index.php) must be installed.
You you plan to make pdf diff, you might want to change imagemagick `/etc/ImageMagick-[VERSION]/policy.xml` and replacing the line:
available for a nimage diff: PNG, JPEG, BMP, ICO, SVG, PDF
Let's say, we have one image named im1.png inside the last commit. to compare it with the image im1.png in the result folder, you can type:
```
gblk mount --last 1 -p 'results/im1.png' --diff
```
The diff image will be created inside the `.tmp` directory of the project folder.
## borgignore file
Its possible to tell borg to ignore files in the results folder.
To do that, you can create a `.borgignore` file at the root of the project directory.
To ignore a given file named `file.txt` you can add to your `.borgignore` file the following content:
```sh
- results/file.txt
```
Note that:
1.**you have to put a `results/` prefix in front of your files**.
2. To exclude a file, the line must begin by `- `.
To ignore every files named file.txt wherever they are, use the following syntax:
```sh
- /**/file.txt
```
You can also ignore files with a given extention inside a folder (named `folder` here) with:
```sh
- results/folder/*.txt
```
To ignore all files with a given extention use:
```sh
- /**/*.txt
```
To rescue files from being ignored by another pattern, you can use a line begining by '+ ' in `.borgignore` file.
Example: If we have a `test` folder inside the `results` folder containing the files a.txt, b.txt, ..., z.txt and we want to ignore everything except the c.txt file. This can be done with:
```sh
- results/test/*.txt
+ results/test/c.txt
```
## Delete command
This command is a wrapper of the borg delete command. If you need information about the `borg delete` command, you can check [borg's documentation](https://borgbackup.readthedocs.io/en/stable/usage/delete.html)
This command can be used to delete specific archive directly by their name or by a prefix or a glob
Note that this command doesn't actually free disk space. You have to use `gblk compact` afterwards to achieve this
### gblk delete usage
To display the help of gblk delete, run the following command:
```
gblk delete -h # -h for compact help, --help for a more exhaustive help
```
To see what archive you are about to remove, enter
```
gblk delete --list --dry-run [OTHER_OPTIONS]
```
- The `dry-run` option will keep the archive unchanged
- The `--list` option will display what was deleted (without `--dry-run` option) or what would be deleted (with the `dry-run` option)
## Prune command
This command is a wrapper of the `borg prune` command. Check [borg's documentation](https://borgbackup.readthedocs.io/en/stable/usage/prune.html) for more details.
This command can be used to keep archives created during a given period of time and remove others.
Note that this command doesn't actually free disk space. You have to use `gblk compact` afterwards to achieve this.
### gblk prune usage
To display the help of `gblk prune`, run the following command:
```
gblk prune -h # -h for compact help, --help for a more exhaustive help
```
To see what archives you are about to remove, enter
```
gblk prune --list --dry-run [OTHER_OPTIONS]
```
## gblk compact
This command frees `.borg` repository space.
You can use this command after deleting one or more archives because it will really free repository space.
To use this command, you can run:
```
gblk compact # -h for compact help, --help for a more exhaustive help
```
To compact you `.borg` folder, you can run
```
gblk compact --verbose
```
If the amount of parts that need compaction is big the `.borg folder`, this command may take a while. Consider using the `--progress` option in this case.
## gblk config
Sometimes, you want to only keep a small number archives of your `results` folder to save some space. If you always want to keep all backups from last week and one backup per month for 5 month, it can tedious to always remember the prune archive doing that:
You may want to put those settings in a local or in a global configuration file to always prune a given project or all your projects in the same way.
gblk provide a way to do that by using the borg configuration file `.borg/config` as a local configuration file and the `~/.gblkconfig` file as a global configuration file.
> Note: **If both the local and global configuration files contain gblk settings used for pruning, only the local settings are used..**
### Add new settings in configuration file
To **add or update** settings in the local configuration file you can use the following command:
```
gblk config add <KEY> <VALUE> [--global]
```
Where `KEY` corresponds to a `prune` command argument. You can choose from:
And `VALUE` corresponds to the value to associate with the key
Check [borg documentation](https://borgbackup.readthedocs.io/en/stable/usage/prune.html) to know what those arguments do. You can also run the command `gblk prune --help` to see a description of those arguments.
> Note: you can also use the `KEY` by replacing the '-' by an '_'. gblk will have the same behavior.
For example, to keep **all backups from last week and one backup per month for 5 month**, you must run the following command:
```
gblk config add keep-within '7d'
gblk config add keep-monthly 5
```
You can use the flag `--global` to set those settings in the global configuration file
### Display gblk settings used for pruning
To display the current list of setting of the gblk local or global configuration file, you can use the following command:
```console
$gblk config show # add --global to see setting the the global configuration file
keep_within = '7d'
keep_monthly = 5
```
> Note: The '-' in keep_within is replaced by an underscore inside the configuration file
### Remove gblk settings used for pruning
To remove a setting previouly defined in the local configuration file you can enter the following command
```
gblk config rm <KEY> [--global]
```
For example, let's remove the setting `keep_within`:
```console
$gblk config show
keep_within = '7d'
keep_monthly = 5
$gblk config rm keep-within # 'gblk config rm keep_within' also works
$gblk config show
keep_monthly = 5
```
To do the same thing with the global configuration file just add `--global` at the end of those 3 commands.
### Pruning archives using gblk settings
Finnaly, to prune your results archives using the settings defined in the global or local configuration you can use the following command:
```
gblk config prune [OPTION]
```
> Note: **If both the local and global configuration files contain gblk settings used for pruning, only the local settings are used..**
You can see what option you can add to your command with
```console
$gblk config prune --help
gblk-config-prune
Prune using the project configuration
USAGE:
gblk config prune [OPTIONS]
OPTIONS:
-h, --help Print help information
Filtering options:
-n, --dry-run Do not change the repository
--list Output verbose list of archive
-s, --stats Print statistics for the deleted archive
--force Force deletion of corrupted archives, use `--force --force` in case `--force`
does not work
```
## sharing gblk repositories
When you produce some results for a particular project, you may want to share them with others who are working with you. With gblk you can share your `.borg` repository with other or update it with chnages done by others.
> Note: The choice of sharing the complete archive folder `.borg` was made to always benefits from borg's deduplication
> **⚠ Warning**: The way of duplicating and storing an archive folder is not the way borg was intended to be used. Indeed the replacment of a borg repository by an older one is considered suspicious and borg blocks the executions of its commands for this archive. See [this page](https://borgbackup.readthedocs.io/en/stable/faq.html?highlight=attack#this-is-either-an-attack-or-unsafe-warning) for details.
### Remotes
In order to be able to share your archived results in the `.borg` folder, you use remotes like in git. Remotes can be defined *globally* or *locally*:
-*local* remotes are only defined for a given project
-*global* remotes are available for any gblk projects. It can be usefull if you often want to store your gblk results on a same location.
> Note: If a *global* remote have the same name as a *local* remote, only the local remote can be used for this project.
The file that will store the local remotes is `.borg/.gblkconfig` inside the root of your project folder.
The file that will sotre the global remote is `~/.gblkconfig` in your `HOME` folder
A remote is defined by a *name* and a *path*. The *name* is used to identify the remotes and the *path* this the path pointed by the remote. An *path* on a remote server can be defined like this `[USER@]HOSTNAME:PATH`. Note that hostnames defined in `ssh config file` can be used here. The *path* must point to an existing directory.
Below is the help of the remote command:
```sh
gblk remote --help
```
It display the following message.
```
gblk-remote
This command can be used to add a new remote for push and pull commands
USAGE:
gblk remote <SUBCOMMAND>
OPTIONS:
-h, --help Print help information
SUBCOMMANDS:
add Add or update a remote in the configuration file
help Print this message or the help of the given subcommand(s)
rm Remove the configuration of a given key in prune
show Display globally and locally defined remotes
```
### Creation of a new remote
To add a new local remote you can use the following command:
```sh
gblk remote add KEY VALUE
```
Where `KEY` Is the name of the remote and `VALUE` The URL it points to.
To define a globally defined remote you can enter the following commands:
```sh
gblk remote add KEY VALUE --global
```
> Note: This command can also be used to uptade the value of an existing commands
### Removing a remote
To remove a remote named `KEY` you can use this command
```sh
gblk remote rm KEY
```
You can add the flag `--global` at the end of the previous command to remove a globally defined remote.
### Displaying remotes
To display remotes you can use the command:
```sh
gblk remote show
```
It will display *locally* and *globally* defined remotes. Note that only unmasked `globally` remotes are displayed (e.g the global remotes that don't share a name with any local remotes).
Here is an example of what the commands can display:
```
local: /path/to_destination_folder
psmn: psmn:/home/user/test_folder (global)
```
You can see that *globally* defined remote are tagged with `(global)`.
## gblk `clone`, `push` and `pull` commands
## Clone command
This command can be used after a `git clone` command. If a gblk archive folder is available for the project, then you can execute the following command:
```sh
gblk clone [HOSTNAME]:PATH
```
Where `PATH` is the path of a gblk archive folder. The `PATH` must point to a folder with this structure (corresponding to a borg remote archive):
```
PATH
├── archive_list
├── config
├── data/
└── README
```
> Note: clone muste be used with a *path* and not a remote name.
Here are the steps that the clone command execute:
1. Checks if `.borg` folder already exists.
2. Checks if the remote directory already exists and contains `data`, `config` and `archive_file` inside.
3. Copies with rsync the remote directory into `.borg`
4. Creates the local config file `.borg/.gblkconfig` using the name of the archive
5. Add a local remote called `origin` into the local config file
6. Creates (or append in) a `.gitignore` file containing `.borg`, `.tmp`, `.mount` inside and a `results/.gitignore` containing `*` and `!.gitignore`.
7. Create hooks if needed
Here is the help of the clone commands:
```sh
gblk-clone
Clones a repository given a destination
USAGE:
gblk clone [OPTIONS] <PATH>
ARGS:
<PATH>
The path pointing to a borg folder
OPTIONS:
-c, --compression <COMPRESSION>
The compression to use automatically at each commit if hooks are created
[default: lz4]
-h, --help
Print help information
-H, --hooks
If specified, hooks are created inside `.git/hooks repository`
-m, --mode <MODE>
The checkout mode used by gblk automatically after a git checkout: soft or hard. This
option is only used if hooks are created. The hard mode will delete every file in your
results folder and extract those corresponding to the commit targeted by the checkout.
The soft mode will only update files that existed in the targeted checkout
[default: hard]
```
### Push command
To copy the content of the `.borg` folder into another location you can use the `push` command like this:
```sh
gblkt push KEY
```
Where `KEY` is the name of a *global* or *local* remote. **Path are not supported!**.
Here are the steps that the push command execute:
1. Checks if the *remote folder* (defined by the remote) exists
2. Checks if the *remote archive folder* (folder that will contains the archives) exits. It corresponds to the *path* of the remote and the name of the folder containing the `.borg` folder.
3. If it doesn't exists:
- go to step 5
4. If it exists:
- Checks if the borg id is the same between the remote and the local borg archive. Stops the command if not.
- gblk uses the `archive_list` file on the *remote archive folder* to compare it with the current archive list saved and show to the users the differences between the remote and the local archive lists.
- Then it asks the user whether or not to continue the push.
5. Copies with rsync the content of the current `.borg` folder into the *remote archive folder*
> **⚠ Warning**: The pull command can delete s old archive inside it !
## Pull command
The pull commands allow to replace the content of your `.borg` archive into a remote folder.
It can be used with the following command:
```sh
gblk pull KEY
```
Where `KEY` is the name of a *global* or *local* remote. **Path are not supported!**.
Here are the steps that the pull command execute:
1. Checks if the *remote folder* (defined by the remote) exists
2. Checks if the *remote archive folder* (folder that will contains the archives) exits. If it doesn't, stops the pull.
3. Checks if the borg id is the same between the remote and the local borg archive folder. Stops the pull if not.
4. gblk uses the `archive_list` file on the *remote archive folder* to compare it with the current archive list saved and show to the users the differences between the remote and the local archive lists.
5. Asks the user whether or not to continue the pull
6. Saves and clean `.borg` folder. The content of the old `.borg` folder is saved inside the `.tmp` folder and can be recovered with the `gblk restore` command if something goes wrong afterward.
7. Copies with rsync the content of the *remote archive folder* into the current `.borg` folder.
8. Removes borg manifest timestamp and the cache associated to the current project archive folder.
> Note: The pull command saves the inital content of the `.borg` folder inside the `.tmp` directory. **Remember to delte it if the pull is sucessful
## Gblk restore command
This command can be used to restore your 'old' `.borg` folder the way it was before the last pull command. It can help to recover from an error that occured during the pull.
To restore your old `.borg` folder, enter the command
```sh
gblk restore
```
## Gblk clean command
After a pull or after using `gblk mount` with the `--diff` flag, temporary files are stored inside the `.tmp` directory. You can remove those with the command:
