Skip to content
Snippets Groups Projects
Select Git revision
  • b7d856b937a660b94e54b8190c04bc65d123ab6e
  • master default protected
  • doc
  • dev_doc
  • dev_push
  • dev
6 results

git_borg_linker

  • Clone with SSH
  • Clone with HTTP
    • user avatar
    src/main.rs: documentation change
    nicolas authored
    b7d856b9
    History
    user avatar b7d856b9
    History
    Name Last commit Last update
    src
    .gitignore
    Cargo.lock
    Cargo.toml
    LICENCE
    Readme.md
    Readme.md

    Git borg linker utility

    Description

    The git borg linker utility (abreviated as gblk) is a tool that aims to ease the usage of borgbackup in a project using git as a version control system.

    It helps you to track the changes in your results folder every time you commit a change in your code. For versionning your results gblk uses borgbackup a tool to create backups and using a data deduplication technique.

    Prerequisites

    To install gblk git and borgbackup must be installed on your system.

    To install borg, you can go to borg's installation page.

    As gblk is written in rust, you need to install it with:

    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

    Installation

    To install gblk run the following command:

    cargo install --git https://gitbio.ens-lyon.fr/LBMC/hub/git_borg_linker

    Usage

    gblk is meant to be used inside a project using git as a version control system. It will only be helpful if your projet folder contains a .git folder. A results folder must be present in your project directory as gblk will try to backup it.

    To sum up gblk must be used in a folder having this minimal structure:

    project
├── .git
└── results

    To display the help of gblk run the following command:

    $ gblk help
A tool used to link borg and git together

This tool was created to link borg and git together and ease the management of developpment artifact
versionning using git

USAGE:
    gblk <SUBCOMMAND>

OPTIONS:
    -h, --help
            Print help information

SUBCOMMANDS:
    checkout
            Checkout results to the current git commit
    commit
            Save the results folder of a git repository in an archive
    create-hooks
            Create github hooks to use gbl automaticaly after commit, before and afetr checkout
    help
            Print this message or the help of the given subcommand(s)
    init
            Initialize a borg repository inside a git project
    list
            List the content of the .borg archive
    pre-co
            Check if a checkout can be performed without losing data

    You can type gblk help <SUBCOMMAND> or gblk <SUBCOMMAND> --help to display the help of any given subcommands.

    Note that create-hooks subcomand can be abbreviated to ch, and checkout subcommand can be abbreviated to co. For example gblk co --help will work the same as gblk checkout --help

    Example usage without git hooks

    Usage of gbl without

    $ mkdir project
$ cd project
$ mkdir results src
$ git init
$ gblk init # creation of a .borg repository at the root of your filesystem
$ exa -a --tree --level=1
.
├── .borg
├── .git
├── results
└── src
$
$ # Creation of a simple script that creates a result file
$ echo "echo 'result line' > results/result.txt" > src/script.sh
$ bash src/script.sh
$ ls results
result.txt
$ git add src/script.sh
$ git commit -m "src/script.sh: initial commit"
$ git rev-parse --verify HEAD # Show current commit
62efe302b6c2e7ab0dfd9c08ddfb0a87ea699c6d
$ gblk commit # creation of an archive in .borg repository
Repository: /home/nicolas/Documents/project/.borg
Archive name: 62efe302b6c2e7ab0dfd9c08ddfb0a87ea699c6d
Archive fingerprint: fbb7444b0d11da22959f7611b66d8d6378b666b379237d46a0448de352fbbb62
Time (start): Thu, 2022-05-12 14:35:43
Time (end):   Thu, 2022-05-12 14:35:43
Duration: 0.00 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                  610 B                548 B                548 B
All archives:                   12 B                 15 B                735 B

                       Unique chunks         Total chunks
Chunk index:                       3                    3
------------------------------------------------------------------------------

$ gblk list # list archive in .borg repository
62efe302b6c2e7ab0dfd9c08ddfb0a87ea699c6d Thu, 2022-05-12 14:35:43 [fbb7444b0d11da22959f7611b66d8d6378b666b379237d46a0448de352fbbb62]
$
$ # New change
$ echo "echo 'newresult line' > results/newresult.txt" > src/script.sh
$ bash src/script.sh
$ ls results
newresult.txt   result.txt
$ git add src/script.sh
$ git commit -m "src/script.sh"
$ gblk commit
Repository: /home/nicolas/Documents/project/.borg
Archive name: 705b95f48fe52bf9aac4406e6d4d7eb16a75f543
Archive fingerprint: ed45c00ec2059f366f53c9c9288a72ff1c9428a16ce37614bf59a56b89fc4dee
Time (start): Thu, 2022-05-12 14:37:43
Time (end):   Thu, 2022-05-12 14:37:43
Duration: 0.00 seconds
Number of files: 2
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                  625 B                566 B                551 B
All archives:                   39 B                 48 B              1.56 kB

                       Unique chunks         Total chunks
Chunk index:                       6                    7
------------------------------------------------------------------------------
$ gblk list
62efe302b6c2e7ab0dfd9c08ddfb0a87ea699c6d Thu, 2022-05-12 14:35:43 [fbb7444b0d11da22959f7611b66d8d6378b666b379237d46a0448de352fbbb62]
705b95f48fe52bf9aac4406e6d4d7eb16a75f543 Thu, 2022-05-12 14:37:43 [ed45c00ec2059f366f53c9c9288a72ff1c9428a16ce37614bf59a56b89fc4dee]

$ # checkout
$ # The next command is important: it will check if your results folder doesn't contain new results compared to your archive with the current git id. If there is no errors, then you wont lose any data
$ gblk pre-co 
$ git co 62efe302b6c2e7ab0dfd9c08ddfb0a87ea699c6d
$ gblk co --mode hard # hard is used to delete file that were not present in the first commit. Otherwise only existing files at the destination commit will be updated.
$ ls results
result.txt

    Not: if gblk pre-co says that you might lose data compared to the saved version of your actual commit, then use gblk commit --update.

    Example usage with git hooks

    Git hooks are commands that can be automatically executed before and after some git commands. They are defined in the repository .git/hooks.

    gblk can create two hooks:

    • post-commit hook that executes gblk commit after every git commit
    • post-checkouthook that execute:
      1. git co to revert back to the last commit as a pre-checkout hooks doesn't exits.
      2. gblk pre-co to be sure to not lose any data before the actual chekout
      3. git checkout do the actual chekout
      4. gblk co to revert back to the results folder corresponding to your target commit

    Note that if gblk has created hooks then it has also modified the .git/config file to add an alias named co that performs a quiet checkout. This is this alias that is used in step1, so it is recommended to use it when you perform a checkout. It allow to have a quiet initial checkout that is then quietly reverted so gblk can check that no data is lost.

    $ mkdir project
$ cd project
$ mkdir results src
$ git init
$ gblk init --hooks # creation of a .borg repository at the root of your filesystem and add hooks to your .git/hooks folder
$ # Note: If you forgot the --hooks option you can always enable tem later with `gblk create-hook`
$ exa .git/hooks -a --tree --level=1 | grep -v sample
.git/hooks
├── post-checkout
├── post-commit
$
$ # Creation of a simple script that creates a result file
$ echo "echo 'result line' > results/result.txt" > src/script.sh
$ bash src/script.sh
$ ls results
result.txt
$ git add src/script.sh
$ git commit -m "src/script.sh: initial commit"
------------------------------------------------------------------------------  
Repository: /home/nicolas/Documents/project/.borg
Archive name: 2da3c535543fb9a216b52f29ecf598b6310c1223
Archive fingerprint: e3fc804ec86e7d372f44cdc7e8c88bcd23cecedfdf7dc6ed3ac78c86a31f375b
Time (start): Thu, 2022-05-12 17:10:40
Time (end):   Thu, 2022-05-12 17:10:40
Duration: 0.00 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                  610 B                548 B                548 B
All archives:                   12 B                 15 B                735 B

                       Unique chunks         Total chunks
Chunk index:                       3                    3
------------------------------------------------------------------------------
[master (commit racine) 2da3c53] src/script.sh: initial commit
 1 file changed, 1 insertion(+)
 create mode 100644 src/script.sh
$ git rev-parse --verify HEAD # Show current commit
2da3c535543fb9a216b52f29ecf598b6310c1223
$ gblk list # list archive in .borg repository
2da3c535543fb9a216b52f29ecf598b6310c1223 Thu, 2022-05-12 17:10:40 [e3fc804ec86e7d372f44cdc7e8c88bcd23cecedfdf7dc6ed3ac78c86a31f375b]
$
$ # New change
$ echo "echo 'newresult line' > results/newresult.txt" > src/script.sh
$ bash src/script.sh
$ git add src/script.sh
$ git commit -m "src/script.sh"
------------------------------------------------------------------------------  
Repository: /home/nicolas/Documents/project/.borg
Archive name: f47abde74c32fe570bf69ca28168120e67703754
Archive fingerprint: e48648ee971fd47c39fcaedf8aee73138334db0f4255e42b2631303df308d2ca
Time (start): Thu, 2022-05-12 17:11:52
Time (end):   Thu, 2022-05-12 17:11:52
Duration: 0.00 seconds
Number of files: 2
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                  625 B                566 B                551 B
All archives:                   39 B                 48 B              1.55 kB

                       Unique chunks         Total chunks
Chunk index:                       6                    7
------------------------------------------------------------------------------
[master f47abde] src/script.sh
 1 file changed, 1 insertion(+), 1 deletion(-)
$ # checkout
$ # Let's add a file to results.txt that was not saved by borg before
$ echo "newline" >> results/newresult.txt
$ # Let's try to make a checkout
$ git co 2da3c535543fb9a216b52f29ecf598b6310c1223
Note : basculement sur '2da3c535543fb9a216b52f29ecf598b6310c1223'
...
Basculement sur la branche 'master'
Your results folder contains unsaved changes!
Please update your current commit with:  gbl commit --update
$ # Nothing happens because we have unsaved changes. Let's update our changes
$ gbl commit --update # this rewrite the archive named after the current commit id
------------------------------------------------------------------------------
...
------------------------------------------------------------------------------
$ git co 2da3c535543fb9a216b52f29ecf598b6310c1223
Note : basculement sur '2da3c535543fb9a216b52f29ecf598b6310c1223'.
...
HEAD est maintenant sur 2da3c53 src/script.sh: initial commit
$ gblk co --mode hard # hard is used to delete file that were not present in the first commit. Otherwise only existing files at the destination commit will be updated.
$ cat results/*
File: results/result.txt
result line
$ git co master
$ cat results/*
File: results/newresult.txt
newresult line
newline

File: results/result.txt
result line