Borg provide an efficient and secure way to backup data. To learn more about borg, you can visit\url{https://www.borgbackup.org/}
Borg: Efficient and secure backups.\url{https://www.borgbackup.org/}
\begin{block}{Main advantages}
\begin{block}{Main advantages}
\begin{itemize}
\begin{itemize}
\item Easy to use
\item Easy to use
\item Supports deduplication
\item Supports deduplication
\item Supports compression
\item Supports compression
\item Creates an archive folder (we can handle it like we want)
\item Creates an archive folder (we can handle it like we want)
\item No restriction in backup deletion
\item No restrictions in backups deletion
\end{itemize}
\end{itemize}
\end{block}
\end{block}
...
@@ -286,11 +288,13 @@ Borg provide an efficient and secure way to backup data. To learn more about bor
...
@@ -286,11 +288,13 @@ Borg provide an efficient and secure way to backup data. To learn more about bor
\caption*{Git Borg Linker (gblk)}
\caption*{Git Borg Linker (gblk)}
\end{figure}
\end{figure}
... Was developed to integrate borg with git.
Developed to integrate borg with git.
\begin{block}{}
\begin{block}{}
gblk will handle data versioning by using git history and uses borg to do the backups. So it has all the advantages you get by using borg with the addition of git integration
gblk handles data versioning by using git history and uses borg to do the backups.
\vspace*{0.2cm}
gblk: Borg advantages + git integration
\end{block}
\end{block}
...
@@ -299,16 +303,16 @@ gblk will handle data versioning by using git history and uses borg to do the ba
...
@@ -299,16 +303,16 @@ gblk will handle data versioning by using git history and uses borg to do the ba
Note that \alert{git}, \alert{borg} and \alert{rust} is required to use gblk.
Note that \alert{git}, \alert{borg} and \alert{rust} is required to use gblk.
\vspace*{1cm}
\vspace*{0.5cm}
To install rust and gblk, you can use the following command:
To install rust and gblk, you can use the following commands:
\begin{lstlisting}[language=bash]
\begin{lstlisting}[language=bash]
$ # Install git
$ # Install git
...
@@ -350,7 +354,7 @@ $ mkdir results; src
...
@@ -350,7 +354,7 @@ $ mkdir results; src
$ git init
$ git init
\end{lstlisting}
\end{lstlisting}
At this point you have to init gblk to start using it, this can be done with the command\texttt{gblk init}:
Initialize gblk to start using it: Done with\texttt{gblk init}:
\includegraphics[width=11cm]{image/img5.png}
\includegraphics[width=11cm]{image/img5.png}
...
@@ -362,20 +366,35 @@ Available compressions: no, lz4, zstd, zlib or lzma. See \href{https://borgbacku
...
@@ -362,20 +366,35 @@ Available compressions: no, lz4, zstd, zlib or lzma. See \href{https://borgbacku
gblk can optionally create git hooks (with \texttt{gblk init} command or \texttt{create-hooks}) to automatically execute commands:
gblk can optionally create git hooks (with \texttt{gblk init} command or \texttt{create-hooks}) to automatically execute commands:
\pause
\pause
\begin{itemize}
\begin{itemize}
\item after a commit to automatically backup you data
\item after a commit to automatically save you data
\item after a checkout to:
\item after a checkout to:
\begin{enumerate}
\begin{enumerate}
\item Revert your checkout
\item Revert the checkout
\item Check if you have unsaved data in your current \texttt{results} folder. If you have stops the checkout.
\item Check if there is unsaved/missing data in the \texttt{results} folder (compared to the archive of the current commit). If so, stops the checkout.
\item Performs git checkout
\item Perform git checkout
\item Performs a gblk checkout (restore your \alert{results} folder as it was at the destination commit).
\item Perform a gblk checkout (restore your \alert{results} folder as it was at the destination commit).
\end{enumerate}
\end{enumerate}
\end{itemize}
\end{itemize}
\pause
\pause
The hooks created by those commands can be found in \texttt{.git/hooks} folder and are name \texttt{post-commit} and \texttt{post-checkout}.
The hooks created by those commands can be found in \texttt{.git/hooks} folder and are name \texttt{post-commit} and \texttt{post-checkout}.
\vspace*{0.5cm}
\end{frame}
Note that those commands will also create a git alias: \texttt{co}\alert{for checkout} for the current project only. It allow to make the git checkout quiet (because it will be reverted)
\begin{frame}[fragile]{Git Borg Linker - init}
\texttt{gblk init ---hooks} and \texttt{create-hooks} commands will also create 3 git aliases:
\begin{itemize}
\item\texttt{co}: for \alert{checkout}. It allow to make the git checkout quiet (because it will be reverted)
\pause
\item\texttt{conh}: For \alert{checkout}. It allow to make a checkout without using the hooks. It is useful to bypass the check of unsaved/missing data in the \texttt{results} folder. \alert{You have to combine it with \texttt{gblk checkout ---mode hard}} to checkout your \texttt{results} folder to the new current commit.
\pause
\item\texttt{cnh}: For \alert{commit}. It allows to make a simple git commit without saving your result repository (may lead to data loss)
\end{itemize}
\vspace*{0.2cm}
Note: Those aliases are valid only \alert{for the current project}. They can be found in the file \texttt{.git/config}
\end{frame}
\end{frame}
...
@@ -389,14 +408,14 @@ $ gblk init --hooks
...
@@ -389,14 +408,14 @@ $ gblk init --hooks
borg repository initialised at .borg
borg repository initialised at .borg
\end{lstlisting}
\end{lstlisting}
This command create an empty \texttt{.borg} repository were the backups of your results folder will be saved.
Creates an empty \texttt{.borg} repository were the backups of your results folder will be saved.
\end{frame}
\end{frame}
\begin{frame}[fragile]{Git Borg Linker - commit}
\begin{frame}[fragile]{Git Borg Linker - commit}
Let's add a code that will produce results file.
Let's add a code that will produce a result file.
\begin{lstlisting}[language=bash]
\begin{lstlisting}[language=bash]
$ echo "echo 'result line' > results/result.txt" > src/script.sh # create a script file that produce a result file
$ echo "echo 'result line' > results/result.txt" > src/script.sh # creates a script file that produces a result file
$ bash src/script.sh # creation of a results/result.txt
$ bash src/script.sh # creation of a results/result.txt
$ gblk commit # commit the results, we have to use this command after commit if gblk hooks are not enabled
$ gblk commit # commit the results, we have to use this command after commit if gblk hooks are not enabled
\end{lstlisting}
\end{lstlisting}
\pause
\pause
A backup of our result file is created after this \texttt{git commit} command. It has the name of the current git commit. We can use \texttt{gblk list} to list the backups created:
A backup \alert{with the name current git commit} was created. We can use \texttt{gblk list} to list the backups:
\begin{lstlisting}[language=bash]
\begin{lstlisting}[language=bash]
$ gblk list
$ gblk list
...
@@ -445,7 +468,8 @@ b1da0e305c906fb242bc8...
...
@@ -445,7 +468,8 @@ b1da0e305c906fb242bc8...
\begin{frame}[fragile]{Git Borg Linker - checkout}
\begin{frame}[fragile]{Git Borg Linker - checkout}
@@ -488,23 +512,30 @@ $ git co b1da0e305c906fb242bc8... # checkout to the first commit
...
@@ -488,23 +512,30 @@ $ git co b1da0e305c906fb242bc8... # checkout to the first commit
Your results folder contains unsaved changes!
Your results folder contains unsaved changes!
Please update your current commit with: gblk commit --update
Please update your current commit with: gblk commit --update
Or revert it back to it's previous state with gblk commit --revert
\end{lstlisting}
\end{lstlisting}
To avoid losing data, gblk pre-co command will stop the checkout if new data is found in the \texttt{results} folder.
To avoid losing data, gblk pre-co command will stop the checkout if new data is found in the \texttt{results} folder.
\vspace*{0.5cm}
You can either remove the change or save the changes with the command \texttt{gblk commit --update} and then proceed to checkout.
You can either:
\begin{itemize}
\item remove the change (\texttt{gblk commit ---revert})
\item save the changes (\texttt{gblk commit ---update}).
\end{itemize}
And proceed to checkout
\end{frame}
\end{frame}
\begin{frame}[fragile]{Git Borg Linker - diff}
\begin{frame}[fragile]{Git Borg Linker - diff}
To see differences between two backups of your results folder you can use \texttt{gblk diff}.
Check for differences between backups using \texttt{gblk diff}.
\includegraphics[width=10cm]{image/img8.png}
\includegraphics[width=10cm]{image/img8.png}
Note that you can only the name of backups (that corresponds to the SHA1 of a commit) saved in \texttt{.borg} folder.
Note that you can only use the name of backups (that corresponds to the SHA1 of a commit) saved in \texttt{.borg} folder.\alert{Branch names can't be used}.
Example:
Example:
...
@@ -524,9 +555,9 @@ $ gblk diff 25fdb6808cd...
...
@@ -524,9 +555,9 @@ $ gblk diff 25fdb6808cd...
\begin{frame}[fragile]{Git Borg Linker - delete }
\begin{frame}[fragile]{Git Borg Linker - delete }
For now gblk does'nt handle backup deletion. To delete a backup, you'll have to use borg
For now, gblk does not handle backup deletion. To delete a backup, you can use borg
Use borg delete to remove specific commits or the N first or last commit
Use borg delete to remove specific commits or the N first or last commits
\begin{lstlisting}[language=bash]
\begin{lstlisting}[language=bash]
$ borg delete .borg::b1da0e305c906fb242bc8ef5699edeaa8c2a6d64 # deletion of a selected commit the disk space is not freed
$ borg delete .borg::b1da0e305c906fb242bc8ef5699edeaa8c2a6d64 # deletion of a selected commit the disk space is not freed
$ borg compact .borg
$ borg compact .borg
...
@@ -557,7 +588,7 @@ To tell gblk to don't track some files in the \texttt{results} folder, a \texttt
...
@@ -557,7 +588,7 @@ To tell gblk to don't track some files in the \texttt{results} folder, a \texttt
Example:
Example:
To avoid tracking all files in the folder \texttt{results/test} and to avoid tracking txt file in the subfolder \texttt{results/notxt} you can write the following .borignore file
To avoid tracking all files in the folder \texttt{results/test} and to avoid tracking txt file in the subfolder \texttt{results/notxt} you can write the following .borgignore file
\begin{lstlisting}[language=bash]
\begin{lstlisting}[language=bash]
results/test/*
results/test/*
...
@@ -565,10 +596,34 @@ results/notxt/*.txt
...
@@ -565,10 +596,34 @@ results/notxt/*.txt
\end{lstlisting}
\end{lstlisting}
\vspace*{0.5cm}
\vspace*{0.5cm}
When checking out with hard mode, ignored files won't be deleted.
Note : When checking out with hard mode, ignored files won't be deleted.
\vspace*{0.1cm}
\alert{Warning} : If you put a blanck line at the end of \texttt{.borgignore} file: Results file that didn't previously exists on the destination commit won't be deleted even with \texttt{--mode hard}.
\end{frame}
\end{frame}
\begin{frame}[fragile]{Git Borg Linker - Still under development }
\begin{center}
\includegraphics[height=4cm]{image/img9.png}
\end{center}
\begin{block}{}
\begin{itemize}
\item Still needs a feedback of user for:
\begin{itemize}
\item improvements/new features
\item Bug fix
\end{itemize}
\end{itemize}
\end{block}
\end{frame}
\section{Conclusion}
\section{Conclusion}
\begin{frame}[fragile]{Conclusion}
\begin{frame}[fragile]{Conclusion}
...
@@ -580,7 +635,7 @@ Git borg linker
...
@@ -580,7 +635,7 @@ Git borg linker
\begin{itemize}
\begin{itemize}
\item Integrates git and borg so it can be used for data versioning in a bioinformatic projects.
\item Integrates git and borg so it can be used for data versioning in a bioinformatic projects.
\item Is easy to use
\item Is easy to use
\item Handles data deduplication to reduce the cost in storage space when large file are generated
\item Handles data deduplication to reduce the cost in storage space when large files are generated
\end{itemize}
\end{itemize}
But \alert{it can be slow when the results folder is big}
But \alert{it can be slow when the results folder is big}