TL:DR – For a newcomer to Linux, likely starting out with Ubuntu or Linux Mint, installing applications that are not part of the Software Manager or Synaptic Package Manager catalog for the chosen distro is not always intuitive at first. I take a rather long look at how to get Sublime Text 2 properly installed on your machine, and do my best to explain what is going on, rather than simply provide terminal entries to copy and paste. I would be happy to hear from experience Linux users about how I might improve, or where I have explained something poorly.
A few months back, I decided to expand my horizons and explore Ruby and Rails. I also decided that I would do so in the more native Linux environment, rather than go the Ruby-on-Windows route. This was one of the best decisions I have made in terms of developing my skills and experience as a programmer.
The learning continues. I started with Linux Mint 13, which has a friendly enough GUI, but for most of what I am doing, I try to use the Bash CLI as much as possible. I've never been very comfortable with the command line, and so long as I am learning a new language, in a new OS environment, I figured it was time to overcome that limitation as well.
If you are an experienced Linux user, there is probably nothing here for you. This is really basic, and yet I had to look around and cull some information from a variety of sources in order to figure out how to do this.
Why Sublime Text 2
Unless you have been in a coma, you have no doubt at least heard of Sublime Text 2. Sublime Text 2 is a cross-platform (Mac/Windows/Linux), highly customizable text editor with an outstanding compliment of features. I have found it to be my preferred, go-to tool for text manipulation. While it is not an IDE, Sublime Text presents sufficient IDE-like features that for many purposes, you won't miss the extra cruft. Also, of course, in many programming communities (Ruby, JavaScript), code is more often written in a text editor.
NOTE: As of this writing, Sublime Text 3 is available to registered Sublime Text users as a beta release.
Sublime Text 2 is not currently part of the Synaptic Package Management system on Linux Mint (or Ubuntu). Therefore, there is no magical apt-get install command as you might use to install other software on your Linux system, so we have to do a little more work.
Of course, the straightforward method of installing Sublime Text 2 on your Linux Box is to download the appropriate (23 or 64-bit) .tar file from the Sublime Text site, unpack, and locate in the directory of your choice. You can do this manually by going to the Sublime Text 2 Downloads page and clicking the appropriate download link, or you can do it all from the terminal, as described below.
If you are not as familiar with Bash command line as you would like, see my previous posts. While these were written in the context of using Git for Windows developers, the basic Bash commands are explained:
This method is described on the Sublime Text Site/Support/Linux/Installation page. Simply open a terminal in the directory you use for applications, and enter the following command (use the appropriate version fro your machine):
NOTE: As of this writing, Sublime Text 2.0.1 is the most recent stable release. If the stable release is updated, the URL's in the links below will change, and you will need to copy the updated URL from the Sublime Text site.
Download the Linux 32-Bit Version of Sublime Text 2:
$ wget http://c758482.r82.cf2.rackcdn.com/Sublime%20Text%202.0.1.tar.bz2
Download the Linux 64-Bit Version of Sublime Text 2:
$ wget http://c758482.r82.cf2.rackcdn.com/Sublime%20Text%202.0.1%20x64.tar.bz2
Extract the "Sublime Text 2.0.1.tar.bz2" file (this will be "Sublime Text 2.0.1 x64.tar.bz2" for the 64 bit version):
Extract the Sublime Text .tar file:
tar vxjf "Sublime Text 2.0.1.tar.bz2"
Then you can add a sym link to the executable file with a short name for convenience (it seems to be a convention to use the alias "subl" for ease of use from the terminal. The executable file will be located in the extracted Sublime Text 2 directory. For example, if you extracted the .tar contents into a directory ~/apps then the sublime_text executable will be "home/Sublime Text 2/sublime_text" (since there are spaces in the directory name, we need to use quotes around the path).
Add a Sym link:
sudo ln -s "~apps/Sublime Text 2/sublime_text" /usr/bin/subl
The above method is easiest, but does not leave you with a convenient way to update Sublime Text in the future short of removing the current installation, re-downloading, and re-installing. There is an arguably better method, which relies on the Personal Package Archive system.
Canonical, the company which supports Ubuntu, has created the Launchpad.net site which, among other things, hosts a repository for Personal Package Archives (PPA's). Here, individuals and teams can upload their own software and installation packages, and it is possible to find deployment packages for software that is not included in the Ubuntu or Linux Mint Synaptic Package Manager for your specific distribution. It is also possible to add the PPA to your Synaptic catalog, so that you can then run apt-get install, apt-get update and the like to keep your package up to date.
Or, at least as up to date as the package maintainer at Launchpad keeps theirs.
The WebUpd8team at Launchpad has created (among other things) a PPA for Sublime Text 2 which is up to date with version 2.0.1 as of this writing. To add Sublime Text 2 to your Synaptic catalog, and install according to the install script published with the PPA, follow these steps:
Add the Sublime Text 2 Repository to your Synaptic Package Manager:
sudo add-apt-repository ppa:webupd8team/sublime-text-2
Update:
sudo apt-get update
Install Sublime Text:
sudo apt-get install sublime-text
Next, check the usr/bin directory. You should see at least one file, named sublime-text-2, and you should also see two others, named sublime-text and subl. These create aliases you can use to invoke Sublime Text 2 from the command line. If the subl and sublime-text files are not present, copy the sublime-text-2 file and make them:
Create alias files (if not present):
$ sudo cp /usr/bin/sublime-text-2 /usr/bin/sublime-text
$ sudo cp /usr/bin/sublime-text-2 /usr/bin/subl
There you have it. You can now use Sublime Text 2 from you command line. Also, you should see it available in your GUI in the applications menu.
This has been a long post about a relatively simple operation. My goal has been to explain the concepts as fully as possible, under the assumption that there are those out there, like myself, new enough to Linux to need the extra handholding. Thanks for reading!
John on Google CodeProject
TL;DR -
Webmatrix 3 is cooler than you might think. It offers a very convenient set of features which to my mind are a complement to, and not replacement for, a more robust development environment. Particularly for doing quick touch-ups to HTML or CSS for an existing site, tracking such changes in source control, and easily publishing the updates to Azure or your standard hosting provider. The addition of source control integration and Azure deployment make Webmatrix exceptionally useful in addition to your more robust development tools.
Microsoft Webmatrix 3 will never be confused with a full-fledged application development tool. Nonetheless, with the version 3 release of this lightweight, free suite of site creation and management tools, Microsoft has, in my opinion, hit something of a sweet spot in tooling for easily building, deploying, managing and maintaining basic web sites and/or sites created from various template packages.
Let’s face it, these days, if you need a simple blog, CMS, forum, or other tried-and-true site category, odds are good you will not be rolling your own, but instead using (perhaps with some customization) an existing package such as Wordpress, Joomla, Orchard CMS, or some other framework. Or it may be that you are creating a simple static site which does not require a full complement of development tools such as Visual Studio.
Webmatrix 3 provides a handy set of features for maintaining and deploying changes to a site without having to crack open a full solution in Visual Studio. Minor tweaks to HTML or CSS are quick and easy. Also, one of the highlights of this new release is some basic source control integration, notably absent from previous versions. Lastly, built-in deployment to the Windows Azure platform, either directly using Webmatrix, or by hooking your Azure-hosted site up to your project repository such that pushing to branch master automatically publishes changes to your live site.
While in some ways Webmatrix looks a little like a toy, I strongly suggest that you give it a closer look before dismissing it completely. Using Webmatrix 3, it is possible to get a basic site up and running quickly and with little fuss, particularly for those who need little more than to deploy and lightly customize templates available in the extensive Web Application Gallery, included out of the box. Also notable is that, since version 2, Webmatrix supports Node.js an PHP, in addition to the expected ASP.NET Web Pages.
Basic File Management and Code Editing
As I mentioned previously, Webmatrix is not a full IDE. However, it does provide a competent text editor with competent syntax highlighting and autocomplete, with a full complement of dictionaries for HTML and CSS:
Git Source Control Integration
The most attractive feature of the new release, for me, is the source control integration. When you click on the Source Control tab at the top of the editor window, you are presented with a choice to use Git or Microsoft’s own TFS. Personally, I have no use for TFS, so let’s see what the integrated Git offering is all about.
As soon as you click on the Git option, Webmatrix initializes a new Git repository in the local folder in which you are working, and presents you with a few GUI controls offering the commonly used Git functions:
The GUI-based Git functionality provided here is very basic. To perform any but the basic functions identified by the ribbon items available in the window shown above, one must crack open the Shell. One complaint I have is that the “Shell” icon seen in the picture above does not open a standard Bash command shell (which is what I have become accustomed to with Git) but instead a Windows Powershell CLI.
The Webmatrix 3 Powershell Window for Git:
The msysGit/Git for Windows/Bash CLI I Have Come to Know and Love:
Webmatrix Git integration appears to depend on libgit2Sharp, a .NET wrapper for the C-based libgit2 port of native Git functionality into a portable, platform-agnostic library of core Git functionality. While I am not fully-versed on the implications of this, I DO notice that the Git/Bash command line interface, as made available in Webmatrix, is not the same as I am accustomed to, either in msysGit (on Windows) or when using Git on my Linux box. While most things appear to behave in a (mostly) familiar way,I found it disorienting at first (no doubt this is a reflection on my unfamiliarity, and not the implementation). I will be doing some more exploring to see what the differences are here.
On the other hand, if I am needing much more advanced Git/Source Control functionality, no doubt I should probably be moving to a more powerful tool altogether. Another option is to use the standard Git for Windows command line when such advanced git features are needed (provided you have Git for Windows or msysGit installed). Happily, on the “Home” menu tab, there is a “Launch in Visual Studio” button. In other words, when you need more power, it’s there (provided you have Visual Studio installed).
Windows Azure Integration
If you develop for the Microsoft stack (and even if you don’t) it has been difficult to miss the growing presence of the Windows Azure platform, and specifically, Windows Azure Websites. Currently, you can create up to 10 sites on the Windows Azure platform for free. There are some trade-offs, in that under the free model, you can’t use custom domain names, and there are some resource limitations. Additionally, it looks like upgrading to the paid model might quickly become more expensive than traditional shared hosting. However, the Azure platform as a hosting provider shows increasing promise, and some very cool features.
As with most Microsoft products, integration with their Windows Azure platform has become a first-class citizen in Webmatrix 3. From the minute you install the product and run it for the first time, you are presented with opportunities to add your Azure account (if you have one) or create a new one ( if you don’t). Likewise (and more pertinent to this article), there are multiple ways to rather seamlessly take advantage of Azure as a host for sites you create in Webmatrix.
From the moment I first create or open a new website on my local machine, I am able to deploy immediately to my Azure Account:
Open a Local (existing) Website Folder:
Select the Folder Containing Your Website Files:
You will then be asked if you want to create a new Azure Website (provided you have your Azure account set up):
Create Azure Site Dialog:
Once you click “OK” your existing site will be provisioned on your Azure account, and publicly visible at the link you see in the yellow notification box at the bottom of the screen.
Site Provisioned on Your Azure Account:
At this point, I can work on my site locally, and test changes by running it in the built-in IIS development server installed with Webmatrix.
Changes Automatically Uploaded When You Publish
What has happened behind the scenes here is Webmatrix has taken care of setting up an Azure website on your account, and all of the required MS Web Deploy in order to publish changes. For example, say I have made a few changes to the CSS or HTML for my site. I can preview them by clicking the “Run” button on the Home tab, which will open the site in my default browser running in my localhost IIS Server:
If I like what I see, I can then use the “Publish” button to automatically upload my changes to the live Azure site.
Other Deployment Options
Of course, publishing directly to an Azure website is only one of the simplified deployment options available in Webmatrix 3. Also present are standard Web Deploy and FTP deployment settings so that changes to your site can be easily published to the hosting provider of your choice.
Another Tool in the Chest
Webmatrix 3 is cooler than you might think. It offers a very convenient set of features which to my mind are a complement to, and not replacement for, a more robust development environment. Particularly for doing quick touch-ups to HTML or CSS for an existing site, tracking such changes in source control, and easily publishing the updates to Azure or your standard hosting provider. The addition of source control integration and Azure deployment make Webmatrix exceptionally useful in addition to your more robust development tools.
There is a rich set of basic features for common deployment and site maintenance scenarios available here. While as I have said, Webmatrix is emphatically not a full-fledged development tool, it is a handy addition, particularly for quick and dirty site creation and maintenance, or for those who need a simple, easy to learn way of getting something published on the web.
Additional Resources:
CodeProject John on Google
TL;DR – Setting up automatic deployment to Windows Azure Websites from your source control repository is just about painless. While the free Windows Azure Websites offering is not without it’s shortcomings, and the “pay-as-you-go” model remains a little spendy for most do-it-yourselfers, the Azure websites platform brings a lot to the table. Automatic deployment from source control is very high on that list.
The Windows Azure platform has come a long, long way in the past year. While Azure is far from the only “Infrastructure-As-A-Service” platform out there, it does present a compelling feature set, and the Azure team has demonstrated an aggressive deployment cycle in pushing improvements out the door. Also, pricing for so-called “cloud services” in general have been steadily falling, making platforms such as Windows Azure, Amazon Web Services, and Linode more and more compelling.
One of the most interesting features offered by Azure of interest to the larger community are Azure Websites. Azure provides an easy to use interface for hosting one or more sites, at a variety of service levels, from a free entry level package to dedicated instance hosting.
More than One Way to Skin a Cat
There are any number of ways to manage deployments to Windows Azure Websites. This is only one. There are also numerous integration tools for Visual Studio, and for really quick and dirty solutions, check out the recently released Webmatrix 3 Preview. How you deploy will depend on your needs, and possibly your technical abilities.
Personally, whether I’m working in Visual Studio, Webmatrix, or simple text files, I find the automatic deployment from my remote source control repo the most attractive.
Deployment to Azure from Source Control
The Azure team has made it very simple to set up an Azure website, and in fact, to do so directly from source control. You can automatically publish changes to your Azure-hosted site simply by pushing to the master branch of your repo on various source control service providers, or simply from your local repository on your machine.
In this post, we’ll walk through the basics of deploying a website to Azure directly from Github, and then see how when we push changes to master on our Github repository, the changes are automatically published to our live site.
Getting Started
To follow along, you will need:
For our example, I will use a very basic site I have sitting around, consisting of an index.html file, a CSS file, and an a Jpeg image (this just happens to be a small practice project I had sitting around, so I used it).
First, I will initialize a new local repository for the site, and commit all the files. Then, after creating a corresponding remote repo on my Github account, I push the content up to the new remote:
Create a Git Repository and Push to Remote:
Now that your site files have been pushed to the Github remote, log into your Azure account, navigate to the Websites” tab, and click the Create A Website link:
From here, your should be looking at something like the picture below. Click on the “Custom Create” link:
From here, you will see some options, as below. You will need to provide a URL for your Azure site. Note that the free websites do not allow custom domain names. You can pick any name you want, (so long as it is not already taken) but it will be a sub-domain within .azurewebsites.net. In my case, I entered jattenGitDeploymentExample, and the full URL for my site will be jattenGitDeploymentExample.azurewebsites.net. Not the greatest URL, but it will do for our example.
You will also need to pick a geographical region in which your site will be hosted. This represents the primary datacenter location where your files will live. The most likely choice would be the one closes to where you live.
We don’t need a database for this simple exercise. Last, make sure the “Publish from source control” box is checked. Then click the right-pointing arrow at the bottom of the screen.
Setting Up Deployment Options for a Windows Azure Website:
Next, you need to tell Azure where your repo is. Notice there are a number of available options. In other words, while we are using Github for this example, you could just as easily publish from any of the other listed options, including ( but not visible in the picture here) your local machine.
Select Your Source Code Provider:
If you are not signed in to Github on your local machine, you will be prompted to do so, and taken to your account sign-in at Github in a separate browser window. Once you have signed in to Github, return to the Azure window and select Github again.
Once you select your source control location and click the right-pointing arrow again, the next window asks you which repository you want to publish. Select the repo for the site we just pushed up to master. Also, you can specify a different branch. Here, we are going to stick with master. However, you could just as easily create a special branch (named” production” for example). Just remember, whenever you push changes to the branch specified here, the result will be published to your site:
Select the Remote Repository to Hook Into:
Once you click the check icon in the lower right corner, Azure will hook into your Github repository and deploy your site. This might take a few seconds. When provisioning and deployment are complete, you should see this:
The Site is Deployed and Live:
Clicking on the link in the URL column will open your site in a new browser tab. Congratulations, your site is now deployed, direct from your Github repository. More importantly, your site is now hooked into branch master, and when changes are pushed or merged to that branch, those changes will be automatically deployed to your Azure site.
Opening the Example Site:
Now, if we look in the image above, we can see that there is some problem with my HTML – apparently I am missing the opening bracket on the <!DOCTYPE html> tag. Also, I want to add some text to my main content area.
In doing this, we’ll take a quick look at how changes are automatically deployed when we push to the remote repo on Github.
Pushing Changesets to Automatically Deploy
First, I want to fix that problematic DOCTYPE tag. I’ll open the file, and correct it in my text editor:
Fixing the Broken <!DOCTYPE html> Tag:
Then I will add some text content to the main content area:
Adding Text Content:
Now, I save my changes, then open my Bash window. Checking status, I can see my changes are ready to be stages and committed. So I do that, and then push to origin master:
Commit and Push to Remote:
That done, I hit “Reload” (or hit the F5 key) on my browser window where my site is currently open:
Reload the Site in the Browser Window:
Voila! My changes have been deployed, simply by pushing to the master branch of my Github repository. The broken <!DOCTYPE html> tag has been fixed, and the new text appears in the Main Content area.
Viewing Deployment History in the Azure Portal
Once your site is deployed, Azure keeps a record of the deployment history. If you click on your website in the websites list of the Azure portal:
You will be taken to THIS window. Click on the deployment history link:
Notice, the active deployment is your most recent commit. In fact, if you look closely, you will see that the deployment ID shown just to the left of the blue Github icon matches the commit hash from your repository:
Automatic Deployment to Windows Azure Websites is a slick feature of the Windows Azure platform. While Azure has come a long way, it is clearly just getting started. Hopefully, costs will continue to come down such that deploying one or more sites to this platform matches comparable shared and dedicated hosting plans.
Additional Resources
John on Google CodeProject
This post describes a rather specific use of Git’s rebase command. Rebase is a powerful and complex topic which I am not fully covering here. I will attempt a more thorough look at rebase in general in an upcoming post. If you are newer to Git, as I am, you might want to refer to a few of my previous posts covering some of the basics:
A handy feature of Git is the ability to use the interactive rebase command to combine, re-organize, and otherwise clean up a messy commit history once development of a feature or bug fix is complete. I am a firm believer in the practice of “commit early, commit often” approach. However, this can leave untold number of commits which, when merged back into mainline development, clutter up a repository.
How often do you end up creating a commit which consists of some modified whitespace after cleaning up a code file, or a commit arising from updating some comments or renaming variables?
Oftentimes, when we complete a feature or fix, we can clearly see how (hindsight being, more often than not, 20/20)we would have liked the code to come together. In fact, for myself, I often wish I could just combine all of the relevant commits into one, which I could then merge back into master. For more complex features, or those which may have undergone substantial evolution during development, it can sometimes become clear how the commit history should have occurred, and not necessarily in the order of manner in which it actually did.
For reasons of maintenance, or particularly if we plan to push our code out to a public repository, it would be nice if we could straighten things up a little, and present a commit history to the world which will:
- Allow us to rollback changes in a manner which does not cripple our codebase.
- Make sense to those who might need to maintain or perhaps branch from our repository.
- Not leave us looking like disorganized idiots (“Did you really commit the deletion of a single empty line?”).
Interactive rebase gives us a way to write that history.
A Simple Example
I will start with an overly simple example. Assume a straightforward git repository consisting of a master branch. In order to add a new feature, you create a feature branch, names (appropriately, feature). On the new feature branch, you work up the basic functionality of the new feature, knowing you will be fine-tuning things shortly. However, you commit the work to this point.
While developing the primary feature code, you noticed a bug in another part of the codebase, perhaps impacting the development of your new feature. You quickly spot a solution, fix the bug, and commit.
Once that is done, you fine-tune the new feature code until it is right where you want it. After committing those changes, you decide to do some cleanup. You remove unnecessary whitespace, rename some methods for clarity, and other housekeeping tasks. After committing this last batch of changes, you are ready to merge your work into master.
Except . . . the new feature, in this case, is not overly complex, and you sure wish you could merge it into master as a single commit. Also, that bug fix should remain separate, as that should be applied to master separately from the new feature. That way, if for whatever reason the new feature needs to be rolled back, it can be done so independently of the bug fix. As of right now, the state of your repo is something like this:
Repo After Feature Development
If we were to perform a standard merge at this point, our repo would look like this:
Repo State After Hypothetical Standard Merge:
As I stated, this is a rather simplified example. But you can see where we might want to combine our feature development (commits C, D, and E) into a single commit before merging to our master branch, while leaving our bug fix (commit C) on its own.
Use Git’s Interactive Rebase Feature to Re-Write History
I created a (very contrived) sample project with a main application on master, and a feature branch. After writing some trivial code to illustrate the commit history we are using in our example, I ran Gitk, which now looks like this:
The Feature Branch Shown in Gitk:
Similarly, the git log command, run from within our feature branch, shows us this:
Git Log Contents for Feature Branch:
If we want to re-write the commit history here such that commits B, D, and E are combined into a single new commit, we can tell git to run an interactive rebase which will allow us to provide direction to git with respect to how we would like it to handle each of the commits involved in the rebase.
What we are actually going to do here, consistent with how rebase works, is rewrite one or more commits in the same branch onto the parent commit we specify in the command below. Because we will be using the interactive flag, we will be allowed to reorder the commits we have selected for the operation, as well as provide some other direction as to how we want git to handle the process.
The Command Syntax is like this:
$ git rebase -i HEAD~n
In the above, n represents the number of commits we would like to edit. However, it is important to note that this actually points to the parent of commit n as the rebase target. So, having examined the state of my feature branch, I can see that I would like clean up commits B thru E. I want to combine Commits B, D and E into s single commit, and leave commit C as a stand-alone (the bug fix).
Here is what I will type into the command shell:
$ git rebase -i HEAD~4
After I type that in and hit enter, my editor widow opens, and looks like this:
The Editor Window After Entering the Interactive Rebase Command:
In this case, the first four lines which start with “pick” represent the commits I have chosen to edit. Note the order in which they appear. When we are finished editing, git will process our revised commits from top to bottom. This is significant because we can re-order them if we so choose. Also note the comments below. We are given some options here which we can apply to the editing process. If we replace the pick keyword with one of these flags for any specific commit, that operation will be performed on that commit.
Of particular interest to us in this case is the squash or s flag. When we replace pick with squash, we tell git to combine that commit with the previous one (remember the processing order, top to bottom). Remember what we want to do in this rather contrived example – we want to combine commits B, D, and E into one. However, the current processing order doesn’t allow us to do this, because our bug fix is sandwiched between commits B and D.
So first we will need to change the processing order by moving some commits around. In the editor, we can easily Cut/Paste the top two lines to look like this instead:
Changing the Commit Processing Order in the Editor:
Now, git will process our edited commits as ordered above, starting with C, then B, then D, then E. Next, we tell git to squash commits D and E by replacing the pick flag with s or squash, so that they both roll up into the previous commit (Commit B, in this case):
Change the Commit Processing Flags in the Editor:
Now, having made the changes above, SAVE THE EDITOR CONTENTS (I often forget this step!) and close the editor window. Your command line window will return while Git processes the commits, and then the editor reappears. Git now wants you to sort out the commit messages, so that the newly created commit history will make sense.
In our example case, the first thing that we see is this:
Making Sense of the Commit History and Messages in the Editor:
We can see that the three commit messages in question are from our original commits B, D, and E, related to our new feature. We can clear all of these (and the comments included with them) and type a new, more meaningful commit message. Note that any line prefixed with a hash symbol (“#”) will be ignored:
Modifying the Commit Message for the Combined Feature Commit in the Editor:
Now, SAVE THE EDITOR CONTENT AGAIN and close the editor window. The command line window now looks like this:
Command Window After Successful Interactive Rebase:
If we run git log now, we see the following:
Git Log After Successful Interactive Rebase:
If we run Gitk, we see this:
As we can see, in the above, we now have a single commit representing the implementation of our new feature, and the original bug fix (previously, commit C) where previously there were four distinct commits. We have re-written history, in preparation for a sensible merge into our master branch.
Notice that our bug fix commit (previously commit C) now has a different hash identifier than previously, and the combined commit is brand-new as well. This is important, because again, HISTORY HAS BEEN CHANGED. The implication here is that if the previous history has been pushed to a public repository (or otherwise cloned, perhaps locally) the history of the feature branch will no longer be the same. In other words, we might as well think of Commit C as a new commit F, and the combined commit as a new commit G.
IMPORTANT: This should not be performed on commits which have been pushed to a public repository, if other committers are working with your code!!! It is generally best to reserve this technique for code in your local working tree, before publishing.
Graphically, our repository now looks like THIS:
Repo After Interactive Rebase:
But, What About Merge Conflicts During the Interactive Rebase?
As often as not, you will run into merge conflicts during this process. I contrived the example above specifically to illustrate the ability to combine commits, and studiously made sure conflicts would not occur here.
In cases where there are conflicts, Git will stop the rebase process, and demand that you fix any conflicts before resuming. Once you have resolved any conflicts, type git rebase –continue in the command window, and the process will resume. We’ll take a look at conflict resolution in another post.
Additional Resources
John on Google CodeProject
NOTE: This post is mainly targeted at newer git users who may want a more detailed look at what is happening. If you consider yourself comfortable with git, I have also created a “reference” version that skips most of the narrative:
The Quick Reference
- Link to condensed version HERE
Also, some navigation aids, because this is a long-ish post:
Problems introduced by “Nested” library projects in a Git Repository
Often it is the case I find myself building out a project and incorporating another library I have been working on (such as a custom control) as a dependency. The library in question may, in fact, be consumed by other projects as well (that IS the purpose of creating such libraries, right?). Equally as often, the project I am referencing as a dependency in my main project will itself be in a state of evolution. A problem arises related to either or both of the following:
- The newly introduced dependency (the “sub-project” e.g. custom control or such) is also being actively developed, and I will want the option to pull in any changes as that library evolves.
- I may, in the course of working on my “main” project, make some modifications to the sub-project, some or all of which I decide should be incorporated into the main line of development of the shared library.
John on Google CodeProject 
Set a reference to the binaries of the shared library from within my main project, perform all modifications to the shared library from within that project, and re-build so that changes are reflected in the binary outputs. - Use Git “Sub modules” to refer to a specific commit within the library project, and manage the updating of these commit pointers as the library project evolves.
- Copy the dependency project source into the main project. Yuck. Isn’t the whole idea of libraries to avoid this type of thing?
I have not used Git sub modules much, but my limited experience with them indicates to me that outside of certain circumstances, they are a pain. Equally unsatisfactory under the conditions I described in the first paragraph is the managing of a standard binary reference (especially in cases where I may need to customize the shared library within my main project, while keeping it up to date with the development repo. The third option is not even worth considering.
In searching for an optimal way to handle the “nested libraries” problem, I came across a few articles on the Sub-tree Merge workflow in GIT. In this workflow, we accomplish the following (this represents my understanding of what is going on – if you know differently, please let me know in the comments!):
- Maintain a tracking branch which tracks the remote shared library project, from which we can pull in changes made in mainline development of the shared library. This remote tracking branch tracks and represents the history of the shared library, distinctly form the history of the main project.
- From this remote, create a “subtree” as a subdirectory within the master branch (or whichever branch we designate) of the main project. The new sub-directory contains a copy of the shared library source code. This subtree does not bring with it the history of commits in the actual shared library. Instead, it joins and shares history with the main project.
- Once this is done, we can pull new change sets down from the shared library remote as needed, and merge them into our sub tree (there’s a trick to this, though, so keep reading). Likewise, we can also merge changes we make within our subtree directory out to our remote tracking branch, and push them up to the shared library remote.
Essentially, we will have told Git to read the contents of the remote tracking branch into a sub-directory on branch master, and remember that this sub-directory represents a subtree based on the remote tracking branch. Graphically, I think of it a little like this:
This is How I Picture the Git Subtree Merge Workflow:
Walk Through:
Let’s walk through the process of creating a subtree within an example project, and then we’ll discuss some things to pay attention to when it comes to merging changes to and from the subtree remote.
I have created an example project as the “main” project for use in the following examples. I will be using a Pdf Merge utility project created for another post on this blog as the “shared library.” The main project is called, unsurprisingly, “MainProject.” The shared library is my oh-so-useful iTextToolsExample. We will assume I have already created MainProject, created a local repository on my development machine, and a remote repo on Github.
From the Git command line, we set up a remote that points at the repository for the shared library project. I will be using an example project as the main project, and adding a shared library I created for another post as a subtree.
Git Add Remote Syntax:
$ git add <local_remote_name> <remote_location>
e.g:
$ git remote add itext_remote https://github.com/xivSolutions/iTextToolsExample.git
Next we need to fetch down the shared library into our working directory using the remote we just created. Note, we want to use “fetch” here, as we don’t want to commit what we add to branch master:
Git Fetch Syntax:
$ git fetch <remote_name>
e.g:
$ git fetch itext_remote
Note the warning after we fetch about “no common commits.” That’s ok – we wouldn’t expect there to be an common history here, we are importing a library for that very reason!
Now, in keeping with the illustration above, we want to get the contents of the shared library tracked in their own remote tracking branch, so we can keep things in sync separate from master:
Checkout to New Branch Syntax:
$ git checkout -b <new_branch_name> remote_name/branch_name
e.g:
$ git checkout –b itext_branch itext_remote/master
Now, the new tracking branch points to the root of the shared library project. We can check this by examining the directory contents here in itext_branch:
Directory Listing in iText Branch:
And then checkout master and compare:
Directory Listing in Master:
Now is when we perform the magic. We will use git read-tree to read the contents of itext_branch into a subdirectory of master. This will be essentially the same as copying it all in, with on small exception: Git knows and remembers that this is a subtree related to the remote tracking branch itext_branch.
Git Read-Tree Syntax:
$ git read-tree --prefix=<subdirectory_name>/ –u <shared_library_branch>
e.g:
$ git read-tree –-prefix=itextTools/ –u itext_branch
In the above, <subdirectory_name> can be anything you like – it is going to be the name of a new “folder” or subdirectory within your mater branch. The --prefix flag tells git that this will be the name of a directory. I usually use the same name as the original shared library project. Note it is important to include the forward-slash after this. The –s flag tells git to go ahead and update the working tree with the new changes. Also note, we execute this command from within branch master:
Running Git Read-Tree From Branch Master:
Now, if we run git status, we can see that a bevy of new files have been added to our master branch, in a directory named (not surprisingly) “iTextTools:”
Status of Branch Master After Read-Tree Operation:
After you have performed the read-tree operation, the files have been read into our master branch as described above. The important thing here is that the history they share with the remote source repository (as referenced by the itext_branch) has NOT. We can commit this as a single commit:
Commit the New Directory and Files:
We have now added the shared iTextTools library as a subtree in our master branch. If we check the contents of our project directory on branch master now, we see the new iTextTools subdirectory:
Now, what if there are changes introduced to the main shared library source project? How can keep our sub-project in sync with mainline development on the shared library?
If we need to pull changes down from the original shared library source into our sub-project, we can simply checkout the remote tracking branch itext_branch and use git pull, which will pull down and merge any changes made since we originally fetched the project. Say we (or whoever maintains the itextTools project) added a new class named SomeNewClass and solution file to the main development repo for the itextTools library:
Pulling Changes down from the Shared Library Remote:
While the above pulls the changes down, we need to be careful with this next step – merging the new change set into our subtree in master. The changes we just pulled down bring with them all of the history from the iText remote repository. In almost all cases, we do not want to merge this history with that of our main project, so we can’t simply do a merge or rebase to get the new changes into master. Instead, we want to use the following:
Squash and No-Commit Change Sets into the Subtree (Syntax):
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge --squash –s subtree –-no-commit itext_branch
In the above, the --s flag indicates that you are going to specify which merge strategy git should use in merging changes. This is followed by the strategy itself, which in this case is the subtree strategy. We are also telling git to squash the commits from <source_branch> into a single change set, and to not automatically commit when the merge is complete.
So, now that we have pulled the changes down into our remote tracking branch, we checkout master, and execute the above command:
Merging Changes from Remote Shared Library into Subtree (from branch master):
If we check the status of master again, we see the changes from the remote iText library repo have been merged into the appropriate subdirectory, and staged for commit:
Status of Master After Subtree Merge Strategy with --Squash and --No-Commit:
From here, simply commit (with a descriptive commit message!).
If you made some changes to the shared library code within your subproject and decide you want to push them out to the main development repo, the process is essentially the same, in reverse. You merge from the subtree directory within the main project (again, explicitly specifying the Subtree Merge strategy), out to the tracking branch that points to the shared library remote. Then push the changes up to the remote.
Say we made some changes to SomeNewClass while working in MainProject and we decided they should be incorporated into mainline development for the itextTools project main repo:
- Once you have made changes to the shared library from within branch master of your main project, commit the changes as normal.
- Then, checkout the tracking branch that points to the shared library remote.
Now, we use the same basic merge command from before, in the opposite direction:
Syntax to Merge from Subtree to the Remote Tracking Branch:
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge --squash –s subtree --no-commit master
Merging the itextTools Subtree into the Remote Tracking Branch:
As in the previous example, we can check Git status and see that the changes we made in our sub-tree project have been merged into the remote tracking branch and are now staged for commit:
Now commit the changes. I usually state clearly in the commit message that these are changes from <project>/subtree so that, when added to the shared library, I can clearly identify that this set of changes came from the sub-project. Now it’s time to push the changes back up to the main shared library repo.
Pushing from Tracking Branch to Shared Library Remote Syntax:
$ git push <shared_library_remote> <library_tracking_branch>
e.g:
$ git push itext_remote itext_branch
Pushing to the Remote Repository:
Note that using the syntax above will push the changes into a new branch in my remote repo, named (not surprisingly) itext_branch. This will allow me to apply the changes to the master or production branch of the shared library using a merge or pull request (in the case where I am not the maintainer of the shared library).
There is More To It
I have brushed through the basics here. There are numerous ways to condense some of the commands I used here, and like most things, there are probably dozens of different viewpoints on how to best manage project dependencies in source control. The Subtree Strategy has been working best for me, in my working context, but I welcome your feedback, and especially, corrections where I might have it wrong, or have missed seeing a better way.
Thanks for reading.
Additional Resources
I am not going to pretend there aren’t any number of good resources from which I learned the above. Most of the information here I assimilated from elsewhere. Here I have striven to simply present my own understanding, in a way which might have helped *ME* when I was trying to figure it out. Here are some of those sources:
John on Google CodeProject