diff --git a/Readme.md b/Readme.md index e7842415a9be52c9918b0f0a369dce3b0ee0b117..828ec9122955f0064dbf01d0e711506fec944922 100644 --- a/Readme.md +++ b/Readme.md @@ -25,6 +25,7 @@ the `--diff` option of the `gblk mount` command. You can optionally install [ImageMagick](https://imagemagick.org/script/index.php). ImageMagick aims to perform various operation on images. This tool is needed if you plan to create diff of image with gblk with the `--diff` option of the `gblk mount` command. +If you want to share your borg folder whith gblk `clone`, `push` and `pull`, `rsync` must be installed on your computer and on remote machines where you want to store/share you results. `ssh` program must be installed to send the archived results to remote servers. ## Installation @@ -646,4 +647,236 @@ Filtering options: -s, --stats Print statistics for the deleted archive --force Force deletion of corrupted archives, use `--force --force` in case `--force` does not work -``` \ No newline at end of file +``` + +## sharing gblk repositories + +When you produce some results for a particular project, you may want to share them with others who are working with you. With gblk you can share your `.borg` repository with other or update it with chnages done by others. + +> Note: The choice of sharing the complete archive folder `.borg` was made to always benefits from borg's deduplication + +> **⚠ Warning**: The way of duplicating and storing an archive folder is not the way borg was intended to be used. Indeed the replacment of a borg repository by an older one is considered suspicious and borg blocks the executions of its commands for this archive. See [this page](https://borgbackup.readthedocs.io/en/stable/faq.html?highlight=attack#this-is-either-an-attack-or-unsafe-warning) for details. + +### Remotes + +In order to be able to share your archived results in the `.borg` folder, you use remotes like in git. Remotes can be defined *globally* or *locally*: +- *local* remotes are only defined for a given project +- *global* remotes are available for any gblk projects. It can be usefull if you often want to store your gblk results on a same location. + +> Note: If a *global* remote have the same name as a *local* remote, only the local remote can be used for this project. + +The file that will store the local remotes is `.borg/.gblkconfig` inside the root of your project folder. +The file that will sotre the global remote is `~/.gblkconfig` in your `HOME` folder + +A remote is defined by a *name* and a *path*. The *name* is used to identify the remotes and the *path* this the path pointed by the remote. An *path* on a remote server can be defined like this `[USER@]HOSTNAME:PATH`. Note that hostnames defined in `ssh config file` can be used here. The *path* must point to an existing directory. + +Below is the help of the remote command: + +```sh +gblk remote --help +``` + +It display the following message. + +``` +gblk-remote +This command can be used to add a new remote for push and pull commands + +USAGE: + gblk remote <SUBCOMMAND> + +OPTIONS: + -h, --help Print help information + +SUBCOMMANDS: + add Add or update a remote in the configuration file + help Print this message or the help of the given subcommand(s) + rm Remove the configuration of a given key in prune + show Display globally and locally defined remotes +``` + +### Creation of a new remote + +To add a new local remote you can use the following command: + +```sh +gblk remote add KEY VALUE +``` + +Where `KEY` Is the name of the remote and `VALUE` The URL it points to. + +To define a globally defined remote you can enter the following commands: + +```sh +gblk remote add KEY VALUE --global +``` + +> Note: This command can also be used to uptade the value of an existing commands + + +### Removing a remote + +To remove a remote named `KEY` you can use this command + +```sh +gblk remote rm KEY +``` + +You can add the flag `--global` at the end of the previous command to remove a globally defined remote. + + +### Displaying remotes + +To display remotes you can use the command: + +```sh +gblk remote show +``` + +It will display *locally* and *globally* defined remotes. Note that only unmasked `globally` remotes are displayed (e.g the global remotes that don't share a name with any local remotes). + +Here is an example of what the commands can display: + + +``` +local: /path/to_destination_folder +psmn: psmn:/home/user/test_folder (global) +``` + +You can see that *globally* defined remote are tagged with `(global)`. + + +## gblk `clone`, `push` and `pull` commands + +## Clone command + +This command can be used after a `git clone` command. If a gblk archive folder is available for the project, then you can execute the following command: + +```sh +gblk clone [HOSTNAME]:PATH +``` + +Where `PATH` is the path of a gblk archive folder. The `PATH` must point to a folder with this structure (corresponding to a borg remote archive): + +``` +PATH +├── archive_list +├── config +├── data/ +└── README +``` + +> Note: clone muste be used with a *path* and not a remote name. + +Here are the steps that the clone command execute: +1. Checks if `.borg` folder already exists. +2. Checks if the remote directory already exists and contains `data`, `config` and `archive_file` inside. +3. Copies with rsync the remote directory into `.borg` +4. Creates the local config file `.borg/.gblkconfig` using the name of the archive +5. Add a local remote called `origin` into the local config file +6. Creates (or append in) a `.gitignore` file containing `.borg`, `.tmp`, `.mount` inside and a `results/.gitignore` containing `*` and `!.gitignore`. +7. Create hooks if needed + +Here is the help of the clone commands: + +```sh +gblk-clone +Clones a repository given a destination + +USAGE: + gblk clone [OPTIONS] <PATH> + +ARGS: + <PATH> + The path pointing to a borg folder + +OPTIONS: + -c, --compression <COMPRESSION> + The compression to use automatically at each commit if hooks are created + + [default: lz4] + + -h, --help + Print help information + + -H, --hooks + If specified, hooks are created inside `.git/hooks repository` + + -m, --mode <MODE> + The checkout mode used by gblk automatically after a git checkout: soft or hard. This + option is only used if hooks are created. The hard mode will delete every file in your + results folder and extract those corresponding to the commit targeted by the checkout. + + The soft mode will only update files that existed in the targeted checkout + + [default: hard] +``` + +### Push command + +To copy the content of the `.borg` folder into another location you can use the `push` command like this: + +```sh +gblkt push KEY +``` + +Where `KEY` is the name of a *global* or *local* remote. **Path are not supported!**. + + +Here are the steps that the push command execute: +1. Checks if the *remote folder* (defined by the remote) exists +2. Checks if the *remote archive folder* (folder that will contains the archives) exits. It corresponds to the *path* of the remote and the name of the folder containing the `.borg` folder. +3. If it doesn't exists: + - go to step 5 +4. If it exists: + - Checks if the borg id is the same between the remote and the local borg archive. Stops the command if not. + - gblk uses the `archive_list` file on the *remote archive folder* to compare it with the current archive list saved and show to the users the differences between the remote and the local archive lists. + - Then it asks the user whether or not to continue the push. +5. Copies with rsync the content of the current `.borg` folder into the *remote archive folder* + +> **⚠ Warning**: The pull command can delete s old archive inside it ! + + +## Pull command + +The pull commands allow to replace the content of your `.borg` archive into a remote folder. + +It can be used with the following command: + +```sh +gblk pull KEY +``` + +Where `KEY` is the name of a *global* or *local* remote. **Path are not supported!**. + +Here are the steps that the pull command execute: +1. Checks if the *remote folder* (defined by the remote) exists +2. Checks if the *remote archive folder* (folder that will contains the archives) exits. If it doesn't, stops the pull. +3. Checks if the borg id is the same between the remote and the local borg archive folder. Stops the pull if not. +4. gblk uses the `archive_list` file on the *remote archive folder* to compare it with the current archive list saved and show to the users the differences between the remote and the local archive lists. +5. Asks the user whether or not to continue the pull +6. Saves and clean `.borg` folder. The content of the old `.borg` folder is saved inside the `.tmp` folder and can be recovered with the `gblk restore` command if something goes wrong afterward. +7. Copies with rsync the content of the *remote archive folder* into the current `.borg` folder. +8. Removes borg manifest timestamp and the cache associated to the current project archive folder. + +> Note: The pull command saves the inital content of the `.borg` folder inside the `.tmp` directory. **Remember to delte it if the pull is sucessful + + +## Gblk restore command + +This command can be used to restore your 'old' `.borg` folder the way it was before the last pull command. It can help to recover from an error that occured during the pull. + +To restore your old `.borg` folder, enter the command + +```sh +gblk restore +``` + + +## Gblk clean command + +After a pull or after using `gblk mount` with the `--diff` flag, temporary files are stored inside the `.tmp` directory. You can remove those with the command: + +```sh +gblk clean +```