The Subtree Merge Workflow offers a relatively painless mechanism for managing shared library source code as a component of a larger project. If you have ever had a shared library, custom control, or other component under development that you also wanted to use within one or more dependent projects, you no doubt understand what I mean.
This post is a condensed version of a longer post with more examples, and aimed at those less familiar with Git. If, like me, you have a little trouble following what is going on here, check out the other post first. This one is more of a reference.
In searching for an optimal way to handle the “nested libraries” problem, I came across a few articles on the Sub-tree Merge workflow in GIT. In this workflow, we accomplish the following (this represents my understanding of what is going on – if you know differently, please let me know in the comments!):
- Maintain a tracking branch which tracks the remote shared library project, from which we can pull in changes made in mainline development of the shared library. This remote tracking branch tracks and represents the history of the shared library, distinctly form the history of the main project.
- From this remote, create a “subtree” as a subdirectory within the master branch (or whichever branch we designate) of the main project. The new sub-directory contains a copy of the shared library source code. This subtree does not bring with it the history of commits in the actual shared library. Instead, it joins and shares history with the main project.
- Once this is done, we can pull new change sets down from the shared library remote as needed, and merge them into our sub tree (there’s a trick to this, though, so keep reading). Likewise, we can also merge changes we make within our subtree directory out to our remote tracking branch, and push them up to the shared library remote.
My Mental Picture of the Subtree Workflow:
In my head, I picture the subtree relationships something like THIS:
Create a Subtree in your Main Project
In the examples below, I will use MainProject as the name of the main project, in which I want to utilize a shared library which is also under development. I will use the iTextTools library from a previous post as the example shared library.
Step 1: Point a Remote in the Main Project at the Shared Library Repository:
$ git remote add <library_remote_name> <library_remote_location>
$ git remote add itext_remote https://github.com/xivSolutions/iTextToolsExample.git
Step 2: Fetch Shared Library into the Working Tree:
$ git fetch <library_remote_name>
$ git fetch itext_remote
Step 3: Checkout a New Tracking Branch Based on the Shared Library Remote:
$ git checkout -b <new_branch_name> remote_name/branch_name
git checkout –b itext_branch itext_remote/master
Step 4: Read the Library Project Into Master as a Subtree Directory:
Before performing this step, switch back to branch master (or whatever your chosen target branch is). The read-tree command is going to read the contents of the remote library tracking branch into the branch you execute the read-tree command from.
- The --prefix option establishes the name of the subdirectory into which the contents of the read will be placed.
- The --u flag tells git to update the contents of the working tree once the read is complete. Files will be staged for commit when the read is complete.
$ git read-tree --prefix=<subdirectory_name>/ -u <shared_library_branch>
$ git read-tree -–prefix=iTextTools/ –u itext_branch
Step 5: Commit the New Files:
Commit as normal. From this point forward, the files added in the new subdirectory will be tracked as part of this branch, and share history with MainProject. However, Git also knows about the subtree directory. We can leverage this information to merge changes between the local version of our shared library and the remote library repository, without merging the separate project histories (which have nothing in common other than the use of the library project source code).
As we will see, there are a few things to keep in mind when doing so, but overall it is pretty simple.
Update the Library Sub-Project from the Shared Library Remote
If changes are made to the shared library at the source repository, we will likely want to bring them into our sub-project so that the subproject is up-to-date with the latest code. The process is a little different than the standard pull, fetch/merge, or rebase process we are accustomed to, because most of the time, we don’t want to bring the commit history from the shared library remote into our main project (as they are unrelated and share no common commits).
Before performing this update, I recommend committing any other local changes you may have introduced in MainProject so that, when you commit the imported changes retrieved for the shared library code, it will stand as its own unique commit.
Step 1: Checkout the Shared Library Remote Tracking Branch:
$ git checkout <library_branch_name>
$ git checkout itext_branch
Step 2: Pull from Remote Library Repository:
When you set up the remote tracking branch originally, you also pointed it at the the shared library remote repo, so Git understands that git pull from this branch means to fetch/merge from the proper location:
$ git pull
Step 3: Checkout Target Branch
Before we perform any merging, we obviously need to make sure we are back on the branch we want to merge the changes into:
$ git checkout <target_branch>
$ git checkout master
Step 4: Merge Changesets without Merging History
This is where things differ a little from a normal merge or rebase. We want to merge all of the changes, but we don’t want to bring any commit history with it. Also, because we are targeting our subtree directory, we need to explicitly specify the subtree merge strategy. In the below, the –s flag tells git we will be explicitly prescribing a merge strategy, followed by the strategy to use (obviously, subtree in this case):
$ git merge --squash -s subtree --no-commit <library_branch>
$ git merge --squash -s subtree --no-commit itext_branch
Step 5: Commit:
Now the sub-project version of the shared library is up-to-date with the remote. Commit as normal. I recommend using a descriptive commit message indicating that this commit is specifically related to updating the subtree project.
Update the Shared Library Remote with Changes from the Library Sub-Project
It may happen that while working in your main project, you introduce modifications or additional features to the shared library code that you decide should be integrated into the Shared Library source repository. Doing so is essentially the reverse of the preceding. Simply merge the changes from your working branch out to the shared library remote tracking branch (again, without bringing the local project history along), then push to the shared library remote (and/or submit a pull request, depending upon who maintains the shared library repository).
Obviously, prior to performing the following steps, commit your changes in the working tree on branch master (or whatever your working branch is) in your main project.
Step 1: Checkout the Shared Library Remote Tracking Branch
$ git checkout <library_branch>
$ git checkout itext_branch
Step 2: Merge Changes from the Library Sub-Project Without Merging History
As with the previous situation, in all but some very odd circumstance we don’t want to merge the two projects histories together, so we use the --squash option when we merge. Also as before, we explicitly specify the subtree merge strategy:
$ git merge --squash -s subtree --no-commit <source_branch>
$ git merge –squash –s subtree –no-commit master
Step 3: Commit the Merged Changes:
Now, assuming there were no merge conflicts, commit as normal, again using a descriptive commit message identifying the commit as a merge from a subtree in an outside project.
Step 4: Push Changes to the Shared Library Repo
When we go to push changes from our remote tracking branch to the shared library remote, the command below will actually push our changes on a new branch (named, not surprisingly, “itext_branch”). For me, this is the behavior I want. This way, I (or whoever is the maintainer of the remote library) can integrate the changes into the mainline of development from within the library remote. In other words, we are not merging directly back into master (rarely how we would want to do things under these circumstances). Obviously, if we are NOT the maintainer of the shared library, this is where a pull request would come in instead of a direct merge.
$ git push <library_remote> <library_branch>
$ git push itext_remote itext_branch
I do not claim to have the authoritative post on this topic by any means. I have written this, and the longer-form post, as much as a learning exercise and reference for myself as anything else. I have sought to present the information in a way which would have helped me understand ti when I was first learning some of Git’s more advanced features and techniques.
I am aware that there are more condensed version of some of the commands used, and quite possibly better ways of doing this. If you have read this far, and have noted some incredible stupidity on my part herein, please do point it out in the comments.
As I learned this, I referred frequently to the following, among other sources:
John on Google CodeProject
Up to this point . . .
This is the third part in a series focused on getting familiar with Git for Windows developers. The series assumes little or no experience using the command line. If you missed the first few posts, here is what we have covered so far:
In the previous post, we learned how to navigate directories within the Windows file system, create new directories, and delete directories. Now let’s look at creating files, and performing rudimentary (VERY rudimentary) text editing from the Bash command prompt.
Before We Proceed . . .
Let’s create a directory to use for the following examples. Open your Bash Command Line Interface (CLI) and make sure you are in your default Windows user folder. Create a new directory within your user folder called BashExcercises, and then move into that directory (Review the previous post if you are unsure how to do this):
Create a Directory for the Following Exercises
Also remember, when we are explaining syntax, we will often use square brackets to denote optional items, and angle brackets to denote use-provided content. In the following:
mkdir [options] <FileName>
We do not type the square brackets or angle brackets. The [options] are, well, optional, meaning there may be none or there may be many, and the <Filename> is required and provided by the user.
Bash: Create/Edit Files (touch/echo)
There are a couple of ways to create a new file using the Bash command prompt. The most basic is the touch command. The purpose of the touch command is ostensibly to change/modify the date/time stamps of files. However, it is commonly used to create new files as well, since, when used without any optional arguments, it does just that: creates a new file, with a date stamp.
Using the Touch Command to Create a New Text File
The syntax for the touch command is
Create a new file with a date stamp:
touch [options] <filename>
There are a number of options for use with the touch command, but for now, we are going to focus on simple file creation, and we won’t be using any of the options. If we want to create a simple text file, we can type:
Create a new Text File Named MyNewTextFile:
$ touch MyNewTextFile.txt
Type the above into your Bash window and hit enter:
Remember from the previous post that in many cases, when a Bash command executes successfully, we are rewarded with simply a new command prompt. However, we can use the ls command we learned from the last post to examine the contents of our directory:
What do you know? There is is. Now, you could open the new file using Notepad (or the text editor of your choosing), but you won’t see much. We have created an empty text file. To open the file using Notepad from the Bash prompt, simply type:
Open MyNewTextFile.txt in Notepad using Bash:
$ notepad MyNewTextFile.txt
Or, of course, you could open the file the old-fashioned way - by double clicking on it with your mouse!
Using the echo Command to Write Strings as Output
Before we look at using echo to create a new file, we should understand a little about what echo DOES.
The primary design purpose of echo is apparently to write strings to the standard output (usually, the Bash CLI).
The syntax for echo is:
Echo Command Syntax:
$ echo [options] <string>
Lets try that now. Type the following into Bash:
Output a line of Text to the Bash Screen:
$ echo show this line on the screen
We can also tell the echo command to write a line of text to a file, using the following syntax:
Syntax: Append text to the end of a file:
$ echo [options] <Text string> >> <FileName>
By using the >> (double right-hand angle-brackets) operator, we tell Bash to append the text to the end of the file. Type the following into the Bash window and hit enter:
$ echo This line was added to the end of the file >> MyNewTextFile.txt
Now, if we use our $ notepad MyNewTextFile.txt Command, we see the following:
Close that window, and let’s add another line. Type the following and hit enter:
Adding Another Line of Text:
$ echo This is another line of text >> MyNewTextFile.txt
The use $ notepad MyNewTextFile.txt again:
Huh. Looks like it worked, kind of. Apparently, Bash isn’t kidding about appending the new text to the end of the file. Happily, we can use the > (a single right-hand angle bracket) operator to replace the text in the current file and get rid of that mess. Type this and hit enter. Then open the file in Notepad again:
Replace the Text in a File with a New Text String:
$ echo This line replaced the old mess > MyNewTextFile.txt
Your Bash window should look like THIS:
And you should see this in your Notepad window:
Use the Echo Command to Create a New File with Text Content:
Ok, back to the original subject - creating a new text file. We can use echo to create a new file, and include the specified text string as content. In fact, it works the same as when we appended or replaced the text in an existing file, because, lacking an option argument telling it not to, Bash will assume that if the file specified in our echo command does not exist, that we want to create one. Type the following and hit enter:
$ echo This is a new line in a new file >> DifferentTextFile.txt
Then use ls -1 to display the directory contents. This is what your Bash window should look like:
Note the presence of our new file. Let’s open it in Notepad, and see what there is to see:
Well, well well. Whadd’ya know!
Bash: Remove Files (rm)
Ok, now lets get rid of our first file. To remove (delete) files using Bash, we use the rm command. The syntax for removing files is as follows;
rm [options] <FileName>
SO now, let’s delete that first file, MyNewTextFile.txt. Type the following, hit enter, then use the ls -1 command to display the directory contents:
Your Bash window should look like this:
Remove a Directory and all Contents (rm -rf)
In the previous post, we discussed how to remove an empty directory. But what if we want to remove a direcory and all of its contents? We use the rm command with the -r and -f options:
Syntax for Removing a Directory and All of its Contents:
rm -rf <FolderName>
So now we are going to remove the BashExcercises directory we created at the beginning of this post. Of course, we can’t remove a directory if we are currently IN that directory. So first, let’s return to the directory above (in this case, our home folder) using the cd .. Comand (the cd Command, followed by a single space, followed by two periods in a row. This one means “move up one level”).
Next, type the following into your Bash window, hit enter, and then use the ls -d */ command to view the directories within the current directory (which should be, at this point, your Windows User folder):
$ rm -rf BashExcercises
When you are done, your Bash window should resemble this (note that the directories are organized alphabetically in columns, and the BashExcercises directory is no longer there. Also note that for obvious reasons, your user folder will contain different folders than mine!).
Now, let’s use everything we just learned in this post, plus we’ll add one final tidbit at the end which you may find useful. We’re going to step through this right quick. We will:
- Add one more new directory
- Add a file containing a line of text to the new directory
- Remove the directory and its contents
- Have Bash tell us what was removed.
1. Add a New Directory Named OneLastFolder to our current directory:
$ mkdir OneLastFolder
2. Add a New File Named OneLastFile.txt, containing the text “One more text line” to the new folder:
$ echo One more text line >> OneLastFolder/OneMoreFile.txt
Notice in the above, I did not navigate into the new folder to create the new file, but simply used the relative path from my current location.
3. Remove the directory and all its contents, and have Bash tell us what was done:
rm -rfv OneLastFolder
Notice in the above, I added one more optional argument, the -v option (for “verbose”). This tells Bash to print the actions it took to the output window.
When you have done all of that, your window should look like this:
There you have it. You now have the most basic set of tools for using the Bash Command Line Interface. From here, we are ready to examine the basic command line operations required to start using Git to manage our code. We’ll pick that up in the next post in this series.
John on Google
This is the second installment of a multi-part series about getting your feet wet with Git for Windows Developers. If this is your first time here, you may want to review Part I of the series.
If you have been following along so far, you likely have by now installed msysgit on your development machine, and performed the initial configuration necessary to begin using git to track changes in your source code. So, what next?
Well, before we can do much with Git, we need to make sure we have at least a token familiarity with using the Bash command prompt. To do this, we will walk through some basic commands related to navigating in bash, getting information about the file system and environment, and working with directories and files. All of these are necessary in order to work with Git using the Bash command line.
Since these posts can get long (due in large part to the large number of screen shots), this may span a few posts. It’s not as bad as it looks, really. Lots of pictures.
If you are already familiar with the concepts here, and mainly just need help with the proper syntax for performing these actions in Bash, refer to my Basic Git Command Line Reference Post which contains a listing of the most common git and Bash commands without all the narrative.
Otherwise, read on, and let’s get comfortable using the Bash command line in a Windows environment.
Get Acquainted with Bash
The Bash command line comes from a Unix heritage, and even if you are familiar with the Windows command line, or Powershell, using Bash is different.
First, we should get a little comfortable moving around our file system and performing basic tasks with files and directories.
When you first open the Bash command window, you should see something like this:
In the above, the text in green is the User Name and Computer Name, displayed using the convention UserName@ComputerName. The curlicue character following means that we are in our default directory (or folder), usually our Windows User Folder.
A line which begins with the default prompt (in this case, a $ symbol) is a line on which you will enter a command. Lines without the $ symbol will represent feedback from the system. Each time you type in a command and hit enter, Bash will execute the command, return whatever output (if any) and the leave you with a new prompt.
Note that many commands simply execute, and there is no output. It is a principle amongst Linux programming that a good program executes silently, and only provides feedback if requested, or if something has gone wrong. More on this momentarily.
List the Directories within the Current Folder
We can list other directories within our current folder using the ls command. Type ls into your Bash window and hit enter:
Use Bash to list the contents of the current directory:
Huh. That’s a lot of stuff, and except for those items with file extensions, it is hard to tell what is a directory (folder) and what is a file of some sort. Wouldn’t it be nice to show just the folders, without the other stuff?
In a clear case of stating the obvious, the following will clear your Bash window. Type this and hit enter:
Clear the Bash window:
Now, let’s try to examine the folders in our current directory again, and see if we can get a more useful view of the subfolders we are working with. Type the following command into the Bash window, and hit enter again:
Print a listing of all directories within the current directory:
$ ls -d */
The result should look like this:
That’s a little more like it. Ok, so now we can see what folders exist within our current directory.
For a cleaner listing of the variations for the ls command, see the Show Directory Contents section of my Basic Git Command Line Reference Post.
Create a New Directory in the Current Directory
Next, let’s create a new folder. The command for creating a new directory is as follows:
Create a new directory within the current directory:
$ mkdir NewFolderName
Type that into your Bash window, substituting MyNewFolder for NewFolderName and hit enter. This is what you should see when you are done:
But hey - nothing happened, right? Wrong.
As mentioned above, when the mkdir command executed properly, the result is “silence.” In other words, Bash assumes YOU know that, lacking additional feedback, everything went fine. Let’s check and see. Type your new “Show me all the folders, but only the folders” command we discussed previously, and hit enter:
Well, what do you know. There is our new folder. But wait. What if we want spaces in the name of our folder? Let’s try THAT with a another new folder. Type the following and hit enter:
Create a folder named My Other New Folder:
$ mkdir My Other New Folder
When you hit enter, you should see something like this:
Hey, looks like everything worked! No complaints from Bash. Let’s see, using our “Show directory contents” command again:
Uh-oh. Looks like things didn’t go quite the way we expected. As it turns out, we confused Bash, because the actual syntax of the mkdir command is:
The full syntax for the mkdir command:
$ mkdir Folder_1 Folder_2 . . . Folder_n
So when we typed in a folder name containing spaces, Bash thought we were creating a separate new folder for each word. As it turns out, most of the time when using Bash, we need to enclose text with spaces in quotes, so that Bash will know that the quoted text represents a single object or idea. Let’s try creating our new folder with spaces in the name again. This time, type the following into the Bash window:
Create a new folder with spaces in the name:
$ mkdir "My Other New Folder"
When you are done, you should see something like this:
Again, we assume that nothing went wrong from a technical standpoint, because Bash executed the command and returned without complaint. If we check to see again, we find the following:
Yay! So now we can not only create folders, but with spaces in the names too, dammit!
For more on creating directories, see the Creating New Directories Section in my Basic Git Command Line Reference Post
Move to a Folder Within the Current Directory
Ok, so now let’s navigate ourselves into the first folder we created, MyNewFolder. First, though, let’s clear the bash window again, so we can get rid of all the clutter we have accumulated so far.
When we want to move to a new location using Bash, we use the cd command:
The correct syntax for this command is:
Change Directory Command Syntax:
$ cd [options] [<DirectoryName>]
In the above, the square brackets denote optional inputs, and the angle brackets denote user-supplied values. In both cases, we do not type the actual brackets into the command window.
Since Bash already knows the current directory we are in, we can specify a folder name which exists in the current directory, without the rest of the directory path. Type the following into the Bash window, and hit enter. You should see something similar to this:
Note in the above that the line above our prompt has changed, and appears to indicate our current path as:
Remember that Bash uses the ~ symbol to denote our default (User) directory, so this is a relative reference within the user directory. From where we are right now, there are a number of ways we could return to the user directory.
The first is to simply enter the cd command with no additional input. Doing this will return us to the default directory from wherever we happen to be in the file system:
Return to the Default Directory (“Take Me Home”)
Another option is to type cd - (that’s the cd command followed by a space, followed by a single dash). This tells bash to go back to the previous directory. In this particular case, the previous directory also happens to be our home user folder, but it really doesn’t matter what directory we came here from, cd - returns us there.
Return to the Previous Directory (“Take Me Back”)
$ cd -
The third option is to type cd .. (that is the cd command, followed by a space, followed by two periods in a row). This command tells bash to go up one level in the folder hierarchy.
Move Up One Level (“Take Me Up”)
$ cd ..
Considering that we are currently in a folder within our home directory, you can see why in the current situation, these three are all about the same in terms of returning to the home directory. At the moment, it is not only the last place we navigated from, but is also one level above our current location. And it is, indeed the home folder. So all three options work for us in this case. Try the “Take Me Up” command. Type it into your Bash window and hit enter. You should see something like this:
Note that we now appear to back in the “directory known as ~” or, our user folder.
Now type the “Take Me Back” Command (cd -) and hit enter:
Now, here we are right back in the MyNewFolder directory. Note that Bash decided to tell us where it took us in long form:
A Linux-Style Directory Path
But wait - what is with that funny directory syntax? That is the Linux directory style. In fact, when typing explicit directory paths into Bash, you will need to use that format, because Bash will not recognize the Windows directory style. It’s easy enough to follow though:
- Instead of using the forward-slash (\) character as a path delimiter, Bash (and Linux) uses the backslash (/)
- /c/ is the Linux way of expressing the familiar C:\
Move to a Specific Directory Using the Complete Directory Path
Knowing this, we can also use explicit paths for navigation. For example, let’s say I wish to move from the current directory to my Documents folder. I can type the following:
Move to a specific directory using the directory path:
$ cd /c/Users/CurrentUserFolder/Documents
Try that now, substituting your user folder, assuming that you have a standard Windows file structure, which includes a Documents folder in your user folder by default:
Ok. If all went well, you should now find yourself in your Documents folder. Note that even though you are within a subfolder of your user folder, Bash is now displaying the long form of the directory path, instead of the shorthand relative path.
Now try using the “Take Me Home” version of the cd command (simply type cd and hit enter):
Presto - back in our home directory.
For more on directory navigation, see the Navigating the File System Section of my Basic Git Command Line Reference Post
Ok, now what about those folders we “accidentally” created when we entered a multi-part directory name? You know, the folders named “My” and “Other” and “New” and “Folder”? Remember THIS:
Since these folders are all empty, we can simply use the rmdir command:
Remove One or More Empty Directories:
$ rmdir Directory_1 Directory_2 . . . Directory_n
Type the rmdir command as follows and hit enter. When you are done, you should see something like this:
Then, if we check the directory contents again, we see that those extra directories no longer exist:
Removing an empty directory or directories is a simple undertaking. The process is a little different if the directory to be removed contains files or other directories. We’ll look at this in the next post.
Next: Creating and Working with Files using Bash
John on Google