NOTE: This post is mainly targeted at newer git users who may want a more detailed look at what is happening. If you consider yourself comfortable with git, I have also created a “reference” version that skips most of the narrative:
The Quick Reference
- Link to condensed version HERE
Also, some navigation aids, because this is a long-ish post:
Problems introduced by “Nested” library projects in a Git Repository
Often it is the case I find myself building out a project and incorporating another library I have been working on (such as a custom control) as a dependency. The library in question may, in fact, be consumed by other projects as well (that IS the purpose of creating such libraries, right?). Equally as often, the project I am referencing as a dependency in my main project will itself be in a state of evolution. A problem arises related to either or both of the following:
- The newly introduced dependency (the “sub-project” e.g. custom control or such) is also being actively developed, and I will want the option to pull in any changes as that library evolves.
- I may, in the course of working on my “main” project, make some modifications to the sub-project, some or all of which I decide should be incorporated into the main line of development of the shared library.
John on Google CodeProject 
Set a reference to the binaries of the shared library from within my main project, perform all modifications to the shared library from within that project, and re-build so that changes are reflected in the binary outputs. - Use Git “Sub modules” to refer to a specific commit within the library project, and manage the updating of these commit pointers as the library project evolves.
- Copy the dependency project source into the main project. Yuck. Isn’t the whole idea of libraries to avoid this type of thing?
I have not used Git sub modules much, but my limited experience with them indicates to me that outside of certain circumstances, they are a pain. Equally unsatisfactory under the conditions I described in the first paragraph is the managing of a standard binary reference (especially in cases where I may need to customize the shared library within my main project, while keeping it up to date with the development repo. The third option is not even worth considering.
In searching for an optimal way to handle the “nested libraries” problem, I came across a few articles on the Sub-tree Merge workflow in GIT. In this workflow, we accomplish the following (this represents my understanding of what is going on – if you know differently, please let me know in the comments!):
- Maintain a tracking branch which tracks the remote shared library project, from which we can pull in changes made in mainline development of the shared library. This remote tracking branch tracks and represents the history of the shared library, distinctly form the history of the main project.
- From this remote, create a “subtree” as a subdirectory within the master branch (or whichever branch we designate) of the main project. The new sub-directory contains a copy of the shared library source code. This subtree does not bring with it the history of commits in the actual shared library. Instead, it joins and shares history with the main project.
- Once this is done, we can pull new change sets down from the shared library remote as needed, and merge them into our sub tree (there’s a trick to this, though, so keep reading). Likewise, we can also merge changes we make within our subtree directory out to our remote tracking branch, and push them up to the shared library remote.
Essentially, we will have told Git to read the contents of the remote tracking branch into a sub-directory on branch master, and remember that this sub-directory represents a subtree based on the remote tracking branch. Graphically, I think of it a little like this:
This is How I Picture the Git Subtree Merge Workflow:
Walk Through:
Let’s walk through the process of creating a subtree within an example project, and then we’ll discuss some things to pay attention to when it comes to merging changes to and from the subtree remote.
I have created an example project as the “main” project for use in the following examples. I will be using a Pdf Merge utility project created for another post on this blog as the “shared library.” The main project is called, unsurprisingly, “MainProject.” The shared library is my oh-so-useful iTextToolsExample. We will assume I have already created MainProject, created a local repository on my development machine, and a remote repo on Github.
From the Git command line, we set up a remote that points at the repository for the shared library project. I will be using an example project as the main project, and adding a shared library I created for another post as a subtree.
Git Add Remote Syntax:
$ git add <local_remote_name> <remote_location>
e.g:
$ git remote add itext_remote https://github.com/xivSolutions/iTextToolsExample.git
Next we need to fetch down the shared library into our working directory using the remote we just created. Note, we want to use “fetch” here, as we don’t want to commit what we add to branch master:
Git Fetch Syntax:
$ git fetch <remote_name>
e.g:
$ git fetch itext_remote
Note the warning after we fetch about “no common commits.” That’s ok – we wouldn’t expect there to be an common history here, we are importing a library for that very reason!
Now, in keeping with the illustration above, we want to get the contents of the shared library tracked in their own remote tracking branch, so we can keep things in sync separate from master:
Checkout to New Branch Syntax:
$ git checkout -b <new_branch_name> remote_name/branch_name
e.g:
$ git checkout –b itext_branch itext_remote/master
Now, the new tracking branch points to the root of the shared library project. We can check this by examining the directory contents here in itext_branch:
Directory Listing in iText Branch:
And then checkout master and compare:
Directory Listing in Master:
Now is when we perform the magic. We will use git read-tree to read the contents of itext_branch into a subdirectory of master. This will be essentially the same as copying it all in, with on small exception: Git knows and remembers that this is a subtree related to the remote tracking branch itext_branch.
Git Read-Tree Syntax:
$ git read-tree --prefix=<subdirectory_name>/ –u <shared_library_branch>
e.g:
$ git read-tree –-prefix=itextTools/ –u itext_branch
In the above, <subdirectory_name> can be anything you like – it is going to be the name of a new “folder” or subdirectory within your mater branch. The --prefix flag tells git that this will be the name of a directory. I usually use the same name as the original shared library project. Note it is important to include the forward-slash after this. The –s flag tells git to go ahead and update the working tree with the new changes. Also note, we execute this command from within branch master:
Running Git Read-Tree From Branch Master:
Now, if we run git status, we can see that a bevy of new files have been added to our master branch, in a directory named (not surprisingly) “iTextTools:”
Status of Branch Master After Read-Tree Operation:
After you have performed the read-tree operation, the files have been read into our master branch as described above. The important thing here is that the history they share with the remote source repository (as referenced by the itext_branch) has NOT. We can commit this as a single commit:
Commit the New Directory and Files:
We have now added the shared iTextTools library as a subtree in our master branch. If we check the contents of our project directory on branch master now, we see the new iTextTools subdirectory:
Now, what if there are changes introduced to the main shared library source project? How can keep our sub-project in sync with mainline development on the shared library?
If we need to pull changes down from the original shared library source into our sub-project, we can simply checkout the remote tracking branch itext_branch and use git pull, which will pull down and merge any changes made since we originally fetched the project. Say we (or whoever maintains the itextTools project) added a new class named SomeNewClass and solution file to the main development repo for the itextTools library:
Pulling Changes down from the Shared Library Remote:
While the above pulls the changes down, we need to be careful with this next step – merging the new change set into our subtree in master. The changes we just pulled down bring with them all of the history from the iText remote repository. In almost all cases, we do not want to merge this history with that of our main project, so we can’t simply do a merge or rebase to get the new changes into master. Instead, we want to use the following:
Squash and No-Commit Change Sets into the Subtree (Syntax):
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge --squash –s subtree –-no-commit itext_branch
In the above, the --s flag indicates that you are going to specify which merge strategy git should use in merging changes. This is followed by the strategy itself, which in this case is the subtree strategy. We are also telling git to squash the commits from <source_branch> into a single change set, and to not automatically commit when the merge is complete.
So, now that we have pulled the changes down into our remote tracking branch, we checkout master, and execute the above command:
Merging Changes from Remote Shared Library into Subtree (from branch master):
If we check the status of master again, we see the changes from the remote iText library repo have been merged into the appropriate subdirectory, and staged for commit:
Status of Master After Subtree Merge Strategy with --Squash and --No-Commit:
From here, simply commit (with a descriptive commit message!).
If you made some changes to the shared library code within your subproject and decide you want to push them out to the main development repo, the process is essentially the same, in reverse. You merge from the subtree directory within the main project (again, explicitly specifying the Subtree Merge strategy), out to the tracking branch that points to the shared library remote. Then push the changes up to the remote.
Say we made some changes to SomeNewClass while working in MainProject and we decided they should be incorporated into mainline development for the itextTools project main repo:
- Once you have made changes to the shared library from within branch master of your main project, commit the changes as normal.
- Then, checkout the tracking branch that points to the shared library remote.
Now, we use the same basic merge command from before, in the opposite direction:
Syntax to Merge from Subtree to the Remote Tracking Branch:
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge --squash –s subtree --no-commit master
Merging the itextTools Subtree into the Remote Tracking Branch:
As in the previous example, we can check Git status and see that the changes we made in our sub-tree project have been merged into the remote tracking branch and are now staged for commit:
Now commit the changes. I usually state clearly in the commit message that these are changes from <project>/subtree so that, when added to the shared library, I can clearly identify that this set of changes came from the sub-project. Now it’s time to push the changes back up to the main shared library repo.
Pushing from Tracking Branch to Shared Library Remote Syntax:
$ git push <shared_library_remote> <library_tracking_branch>
e.g:
$ git push itext_remote itext_branch
Pushing to the Remote Repository:
Note that using the syntax above will push the changes into a new branch in my remote repo, named (not surprisingly) itext_branch. This will allow me to apply the changes to the master or production branch of the shared library using a merge or pull request (in the case where I am not the maintainer of the shared library).
There is More To It
I have brushed through the basics here. There are numerous ways to condense some of the commands I used here, and like most things, there are probably dozens of different viewpoints on how to best manage project dependencies in source control. The Subtree Strategy has been working best for me, in my working context, but I welcome your feedback, and especially, corrections where I might have it wrong, or have missed seeing a better way.
Thanks for reading.
Additional Resources
I am not going to pretend there aren’t any number of good resources from which I learned the above. Most of the information here I assimilated from elsewhere. Here I have striven to simply present my own understanding, in a way which might have helped *ME* when I was trying to figure it out. Here are some of those sources:
John on Google CodeProject
The Subtree Merge Workflow offers a relatively painless mechanism for managing shared library source code as a component of a larger project. If you have ever had a shared library, custom control, or other component under development that you also wanted to use within one or more dependent projects, you no doubt understand what I mean.
This post is a condensed version of a longer post with more examples, and aimed at those less familiar with Git. If, like me, you have a little trouble following what is going on here, check out the other post first. This one is more of a reference.
In searching for an optimal way to handle the “nested libraries” problem, I came across a few articles on the Sub-tree Merge workflow in GIT. In this workflow, we accomplish the following (this represents my understanding of what is going on – if you know differently, please let me know in the comments!):
- Maintain a tracking branch which tracks the remote shared library project, from which we can pull in changes made in mainline development of the shared library. This remote tracking branch tracks and represents the history of the shared library, distinctly form the history of the main project.
- From this remote, create a “subtree” as a subdirectory within the master branch (or whichever branch we designate) of the main project. The new sub-directory contains a copy of the shared library source code. This subtree does not bring with it the history of commits in the actual shared library. Instead, it joins and shares history with the main project.
- Once this is done, we can pull new change sets down from the shared library remote as needed, and merge them into our sub tree (there’s a trick to this, though, so keep reading). Likewise, we can also merge changes we make within our subtree directory out to our remote tracking branch, and push them up to the shared library remote.
My Mental Picture of the Subtree Workflow:
In my head, I picture the subtree relationships something like THIS:
Create a Subtree in your Main Project
In the examples below, I will use MainProject as the name of the main project, in which I want to utilize a shared library which is also under development. I will use the iTextTools library from a previous post as the example shared library.
Step 1: Point a Remote in the Main Project at the Shared Library Repository:
$ git remote add <library_remote_name> <library_remote_location>
e.g.
$ git remote add itext_remote https://github.com/xivSolutions/iTextToolsExample.git
Step 2: Fetch Shared Library into the Working Tree:
$ git fetch <library_remote_name>
e.g:
$ git fetch itext_remote
Step 3: Checkout a New Tracking Branch Based on the Shared Library Remote:
$ git checkout -b <new_branch_name> remote_name/branch_name
e.g:
git checkout –b itext_branch itext_remote/master
Step 4: Read the Library Project Into Master as a Subtree Directory:
Before performing this step, switch back to branch master (or whatever your chosen target branch is). The read-tree command is going to read the contents of the remote library tracking branch into the branch you execute the read-tree command from.
- The --prefix option establishes the name of the subdirectory into which the contents of the read will be placed.
- The --u flag tells git to update the contents of the working tree once the read is complete. Files will be staged for commit when the read is complete.
$ git read-tree --prefix=<subdirectory_name>/ -u <shared_library_branch>
e.g:
$ git read-tree -–prefix=iTextTools/ –u itext_branch
Step 5: Commit the New Files:
Commit as normal. From this point forward, the files added in the new subdirectory will be tracked as part of this branch, and share history with MainProject. However, Git also knows about the subtree directory. We can leverage this information to merge changes between the local version of our shared library and the remote library repository, without merging the separate project histories (which have nothing in common other than the use of the library project source code).
As we will see, there are a few things to keep in mind when doing so, but overall it is pretty simple.
Update the Library Sub-Project from the Shared Library Remote
If changes are made to the shared library at the source repository, we will likely want to bring them into our sub-project so that the subproject is up-to-date with the latest code. The process is a little different than the standard pull, fetch/merge, or rebase process we are accustomed to, because most of the time, we don’t want to bring the commit history from the shared library remote into our main project (as they are unrelated and share no common commits).
Before performing this update, I recommend committing any other local changes you may have introduced in MainProject so that, when you commit the imported changes retrieved for the shared library code, it will stand as its own unique commit.
Step 1: Checkout the Shared Library Remote Tracking Branch:
$ git checkout <library_branch_name>
e.g:
$ git checkout itext_branch
Step 2: Pull from Remote Library Repository:
When you set up the remote tracking branch originally, you also pointed it at the the shared library remote repo, so Git understands that git pull from this branch means to fetch/merge from the proper location:
$ git pull
Step 3: Checkout Target Branch
Before we perform any merging, we obviously need to make sure we are back on the branch we want to merge the changes into:
$ git checkout <target_branch>
e.g:
$ git checkout master
Step 4: Merge Changesets without Merging History
This is where things differ a little from a normal merge or rebase. We want to merge all of the changes, but we don’t want to bring any commit history with it. Also, because we are targeting our subtree directory, we need to explicitly specify the subtree merge strategy. In the below, the –s flag tells git we will be explicitly prescribing a merge strategy, followed by the strategy to use (obviously, subtree in this case):
$ git merge --squash -s subtree --no-commit <library_branch>
e.g:
$ git merge --squash -s subtree --no-commit itext_branch
Step 5: Commit:
Now the sub-project version of the shared library is up-to-date with the remote. Commit as normal. I recommend using a descriptive commit message indicating that this commit is specifically related to updating the subtree project.
Update the Shared Library Remote with Changes from the Library Sub-Project
It may happen that while working in your main project, you introduce modifications or additional features to the shared library code that you decide should be integrated into the Shared Library source repository. Doing so is essentially the reverse of the preceding. Simply merge the changes from your working branch out to the shared library remote tracking branch (again, without bringing the local project history along), then push to the shared library remote (and/or submit a pull request, depending upon who maintains the shared library repository).
Obviously, prior to performing the following steps, commit your changes in the working tree on branch master (or whatever your working branch is) in your main project.
Step 1: Checkout the Shared Library Remote Tracking Branch
$ git checkout <library_branch>
e.g:
$ git checkout itext_branch
Step 2: Merge Changes from the Library Sub-Project Without Merging History
As with the previous situation, in all but some very odd circumstance we don’t want to merge the two projects histories together, so we use the --squash option when we merge. Also as before, we explicitly specify the subtree merge strategy:
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge –squash –s subtree –no-commit master
Step 3: Commit the Merged Changes:
Now, assuming there were no merge conflicts, commit as normal, again using a descriptive commit message identifying the commit as a merge from a subtree in an outside project.
Step 4: Push Changes to the Shared Library Repo
When we go to push changes from our remote tracking branch to the shared library remote, the command below will actually push our changes on a new branch (named, not surprisingly, “itext_branch”). For me, this is the behavior I want. This way, I (or whoever is the maintainer of the remote library) can integrate the changes into the mainline of development from within the library remote. In other words, we are not merging directly back into master (rarely how we would want to do things under these circumstances). Obviously, if we are NOT the maintainer of the shared library, this is where a pull request would come in instead of a direct merge.
$ git push <library_remote> <library_branch>
e.g:
$ git push itext_remote itext_branch
Final Thoughts
I do not claim to have the authoritative post on this topic by any means. I have written this, and the longer-form post, as much as a learning exercise and reference for myself as anything else. I have sought to present the information in a way which would have helped me understand ti when I was first learning some of Git’s more advanced features and techniques.
I am aware that there are more condensed version of some of the commands used, and quite possibly better ways of doing this. If you have read this far, and have noted some incredible stupidity on my part herein, please do point it out in the comments.
Additional Resources
As I learned this, I referred frequently to the following, among other sources:
John on Google CodeProject
I recently posted about using PdfBox.net to manipulate Pdf documents in your C# application. This time, I take a quick look at iTextSharp, another library for working with Pdf documents from within the .NET framework.
Some Navigation Aids:
What is iTextSharp?
iTextSharp is a direct .NET port of the open source iText Java library for PDF generation and manipulation. As the project’s summary page on SourceForge states, iText “ . . . can be used to create PDF Documents from scratch, to convert XML to PDF . . . to fill out interactive PDF forms, to stamp new content on existing PDF documents, to split and merge existing PDF documents, and much more.”
iTextSharp presents a formidable set of tools for developers who need to create and/or manipulate Pdf files. This does come with a cost, however. The Pdf file format itself is complex; therefore, programming libraries which seek to provide a flexible interface for working with Pdf files become complex by default. iText is no exception.
I noted in my previous post on PdfBox that PdfBox was a little easier for me to get up and running with, at least for rather basic tasks such as splitting and merging existing Pdf files. I also noted that iText looked to be a little more complex, and I was correct. However, iTextSharp does not suffer some of the performance drawbacks inherent to PdfBox, at least on the .net platform.
As I observed in my previous post, PdfBox.net is NOT a direct port of the PdfBox Java library, but instead is a Java library running within .net using IKVM. While I found it very cool to be able to run Java code in a .NET context, there was a serious performance hit, most notably the first time the PdfBox library was called, and the massive IKVM library spun up what amounts to a .Net implementation of the Java Virtual Machine, within which the Java code of the PdfBox library is then executed.
Needless to say, iTextSharp does not suffer this limitation. the library itself it relatively lightweight, and fast.
One of the most common tasks we need to do is extract pages from one Pdf into a new file. We’ll take a look at some relatively basic sample code which does just that, and get a feel for using the iTextSharp programming model.
In the following code sample, the primary iTextSharp classes we will be using are the PdfReader, Document, PdfCopy, and PdfImportedPage classes.
My simplified understanding of how this works is as follows: The PdfReader instance contains the content of the source PDF file. The Document class, once initialized with the PdfReader instance and a new output FileStream, essentially becomes a container into which pages extracted from the source file represented in the PdfReader class will be copied. Note that the Document class represents the Pdf content as HTML, which will be used to construct a properly formatted Pdf file. The result is then output to the Filestream, and saved to disk at the location specified by the destination file name.
You can download the iTextSharp source code and binaries as a single package from Files page at the iTextSharp project site. Just click on the “Download itextsharp-all-5.4.0.zip” link. Extract the files from the .zip archive, and stash them somewhere convenient. Next, set a reference in your project to the itextsharp.dll. You will need to browse to the folder where you stashed the extracted contents of the iTextSharp download.
NOTE: The complete example code for this post is available at my Github Repo.
I went ahead and created a project named iTextTools, with a class file named PdfExtractorUtility. Add the following using statements at the top of the file:
using iTextSharp.text;
using iTextSharp.text.pdf;
using System;
// CLASS DEPENDS ON iTextSharp: http://sourceforge.net/projects/itextsharp/
namespace iTextTools
{
public class PdfExtractorUtility
{
}
}
First, I’ll add a simple method to extract a single page from an existing PDF file and save to a new file:
public void ExtractPage(string sourcePdfPath, string outputPdfPath,
int pageNumber, string password = "")
{
PdfReader reader = null;
Document document = null;
PdfCopy pdfCopyProvider = null;
PdfImportedPage importedPage = null;
try
{
// Intialize a new PdfReader instance with the contents of the source Pdf file:
reader = new PdfReader(sourcePdfPath);
// Capture the correct size and orientation for the page:
document = new Document(reader.GetPageSizeWithRotation(pageNumber));
// Initialize an instance of the PdfCopyClass with the source
// document and an output file stream:
pdfCopyProvider = new PdfCopy(document,
new System.IO.FileStream(outputPdfPath, System.IO.FileMode.Create));
document.Open();
// Extract the desired page number:
importedPage = pdfCopyProvider.GetImportedPage(reader, pageNumber);
pdfCopyProvider.AddPage(importedPage);
document.Close();
reader.Close();
}
catch (Exception ex)
{
throw ex;
}
}
As you can see, simply pass in the path to the source document, the page number to be extracted, and an output file path, and you’re done.
If we want to be able to a range of contiguous pages, we might add another method defining a start and end point:
public void ExtractPages(string sourcePdfPath, string outputPdfPath,
int startPage, int endPage)
{
PdfReader reader = null;
Document sourceDocument = null;
PdfCopy pdfCopyProvider = null;
PdfImportedPage importedPage = null;
try
{
// Intialize a new PdfReader instance with the contents of the source Pdf file:
reader = new PdfReader(sourcePdfPath);
// For simplicity, I am assuming all the pages share the same size
// and rotation as the first page:
sourceDocument = new Document(reader.GetPageSizeWithRotation(startPage));
// Initialize an instance of the PdfCopyClass with the source
// document and an output file stream:
pdfCopyProvider = new PdfCopy(sourceDocument,
new System.IO.FileStream(outputPdfPath, System.IO.FileMode.Create));
sourceDocument.Open();
// Walk the specified range and add the page copies to the output file:
for (int i = startPage; i <= endPage; i++)
{
importedPage = pdfCopyProvider.GetImportedPage(reader, i);
pdfCopyProvider.AddPage(importedPage);
}
sourceDocument.Close();
reader.Close();
}
catch (Exception ex)
{
throw ex;
}
}
What if we want non-contiguous pages from the source document? Well, we might override the above method with one which accepts an array of ints representing the desired pages:
public void ExtractPages(string sourcePdfPath,
string outputPdfPath, int[] extractThesePages)
{
PdfReader reader = null;
Document sourceDocument = null;
PdfCopy pdfCopyProvider = null;
PdfImportedPage importedPage = null;
try
{
// Intialize a new PdfReader instance with the
// contents of the source Pdf file:
reader = new PdfReader(sourcePdfPath);
// For simplicity, I am assuming all the pages share the same size
// and rotation as the first page:
sourceDocument = new Document(reader.GetPageSizeWithRotation(extractThesePages[0]));
// Initialize an instance of the PdfCopyClass with the source
// document and an output file stream:
pdfCopyProvider = new PdfCopy(sourceDocument,
new System.IO.FileStream(outputPdfPath, System.IO.FileMode.Create));
sourceDocument.Open();
// Walk the array and add the page copies to the output file:
foreach (int pageNumber in extractThesePages)
{
importedPage = pdfCopyProvider.GetImportedPage(reader, pageNumber);
pdfCopyProvider.AddPage(importedPage);
}
sourceDocument.Close();
reader.Close();
}
catch (Exception ex)
{
throw ex;
}
}
Scratching the Surface
Obviously, the example(s) above are a simplistic first exploration of what appears to be a powerful library. What I notice about iText in general is that, unlike some API’s, the path to achieving your desired result is often not intuitive. I believe this is as much to do with the nature of the PDF file format, and possibly the structure of lower-level libraries upon which iTextSharp is built.
That said, there is without a doubt much to be discerned by exploring the iTextSharp source code. Additionally, there are a number of resources to assist the erstwhile developer in using this library:
Lastly, there is a book authored by one of the primary contributors to the iText project, Bruno Lowagie:
CodeProject John on Google
I have found two primary libraries for programmatically manipulating PDF files; PdfBox and iText. These are both Java libraries, but I needed something I could use with C Sharp. Well, as it turns out there is an implementation of each of these libraries for .NET, each with its own strengths and weaknesses:
Some Navigation Links:
PdfBox - .Net version
The .NET implementation of PdfBox is not a direct port - rather, it uses IKVM to run the Java version inter-operably with .NET. IKVM features an actual .net implementation of a Java Virtual Machine, and a .net implementation of Java Class Libraries along with tools which enable Java and .Net interoperability.
PdfBox’s dependency on IKVK incurs a lot of baggage in performance terms. When the IKVM libraries load, and (I am assuming) the “’Virtual’ Java Virtual Machine” spins up, things slow way down until the load is complete. On the other hand, for some of the more common things one might want to do with a PDF programmatically, the API is (relatively) straightforward, and well documented.
When you run a project which uses PdfBox, you WILL notice a lag the first time PdfBox and IKVM are loaded. After that, things seem to perform sufficiently, at least for what I needed to do.
Side Note: iTextSharp
iTextSharp is a direct port of the Java library to .Net.
iTextSharp looks to be the more robust library in terms of fine-grained control, and is extensively documented in a book by one of the authors of the library, iText in Action (Second Edition). However, the learning curve was a little steeper for iText, and I needed to get a project out the door. I will examine iTextSharp in another post, because it looks really cool, and supposedly does not suffer the performance limitations of PdfBox.
Getting started with PdfBox
Before you can use PdfBox, you need to either build the project from source, or download the ready-to-use binaries. I just downloaded the binaries for version 1.2.1 from this helpful gentleman’s site, which, since they depend on IKVM, also includes the IKVM binaries. However, there are detailed instruction for building from source on the PdfBox site. Personally, I would start with the downloaded binaries to see if PdfBox is what you want to use first.
Important to note here: apparently, the PdfBox binaries are dependent upon the exact dependent DLL’s used to build them. See the notes on the PdfBox .Net Version page.
Once you have built or downloaded the binaries, you will need to set references to PdfBox and ALL the included IKVM binaries in your Visual Studio Project. Create a new Visual Studio project named “PdfBoxExamples” and add references to ALL the PdfBox and IKVM binaries. There are a LOT. Deal with it. Your project references folder will look like the picture to the right when you are done.
The PdfBox API is quite dense, but there is a handy reference at the Apache Pdfbox site. The PDF file format is complex, to say the least, so when you first take a gander at the available classes and methods presented by the PDF box API, it can be difficult to know where to begin. Also, there is the small issue that what you are looking at is a Java API, so some of the naming conventions are a little different. Also, the PdfBox API often returns what appear to be Java classes. This comes back to that .Net implementation of the Java Class libraries I mentioned earlier.
Things to Do with PdfBox
It seems like there are three common things I often want to do with PDF files: Extract text into a string or text file, split the document into one or more parts, or merge pages or documents together. To get started with using PdfBox we will look at extracting text first, since the set up for this is pretty straightforward, and there isn’t any real Java/.Net weirdness here.
To do this, we will call upon two PdfBox namespaces (“Packages” in Java, loosely), and two Classes:
The namespace org.apache.pdfbox.pdmodel gives us access to the PDDocument class and the namespace org.apache.pdfbox.util gives us the PDFTextStripper class.
In your new PdfBoxExamples project, add a new class, name it “PdfTextExtractor," and add the following code:
The PdfTextExtractor Class
using System;
using org.apache.pdfbox.pdmodel;
using org.apache.pdfbox.util;
namespace PdfBoxExamples
{
public class pdfTextExtractor
{
public static String PDFText(String PDFFilePath)
{
PDDocument doc = PDDocument.load(PDFFilePath);
PDFTextStripper stripper = new PDFTextStripper();
return stripper.getText(doc);
}
}
}
As you can see, we use the PDDocument class (from the org.apache.pdfbox.pdmodel namespace) and initialize is using the static .load method defined as a class member on PDDocument. As long as we pass it a valid file path, the .load method will return an instance of PDDocument, ready for us to work with.
Once we have the PDDocument instance, we need an instance of the PDFTextStripper class, from the namespace org.apache.pdfbox.util. We pass our instance of PDDocument in as a parameter, and get back a string representing the text contained in the original PDF file.
Be prepared. PDF documents can employ some strange layouts, especially when there are tables and/or form fields involved. The text you get back will tend not to retain the formatting from the document, and in some cases can be bizarre.
However, the ability to strip text in this manner can be very useful, For example, I recently needed to download an individual PDF file for each county in the state of Missouri, and strip some tabular data our of each one. I hacked together an iterator/downloader to pull down the files, and the, using a modified version of the text stripping tool illustrated above and some rather painful Regex, I was able to get what I needed.
At the simplest level, suppose you had a PDF file and you wanted to split it into individual pages. We can use the Splitter Class, again from the org.apache.pdf.util namespace. Add another class to you project, named PDFFileSplitter, and copy the following code into the editor:
The PdfFileSplitter Class
using org.apache.pdfbox.pdmodel;
using org.apache.pdfbox.util;
namespace PdfBoxExamples
{
public class PDFFileSplitter
{
public static java.util.List SplitPDFFile(string SourcePath,
int splitPageQty = 1)
{
var doc = PDDocument.load(SourcePath);
var splitter = new Splitter();
splitter.setSplitAtPage(splitPageQty);
return (java.util.List)splitter.split(doc);
}
}
}
Notice anything strange in the code above? That’s right. We have declared a static method with a return type of java.util.List. WHAT? This is where working with PdfBox and more importantly, IKVM becomes weird/cool. Cool, because I am using a direct Java class implementation in Visual Studio, in my C# code. Weird, because my method returns a bizarre type (from a C# perspective, anyway) that I was unsure what to do with.
I would probably add to the above class so that the splitter persisted the split documents to disk, or change the return type of my method to object[], and use the .ToArray() method, like so:
The PdfFileSplitter Class (improved?)
public static object[] SplitPDFFile(string SourcePath,
int splitPageQty = 1)
{
var doc = PDDocument.load(SourcePath);
var splitter = new Splitter();
splitter.setSplitAtPage(splitPageQty);
return (object[])splitter.split(doc).toArray();
}
In any case, the code in either example loads up the specified PDF file into a PDDocument instance, which is then passed to the org.apache.pdfbox.Splitter, along with an int parameter. The output in the example above is a Java ArrayList containing a single page from your original document in each element. Your original document is not altered by this process, by the way.
The int parameter is telling the Splitter how many pages should be in each split section. In other words, if you start with a six-page PDF file, the output will be three two-page files. If you started with a 5-page file, the output would be two two-page files and one single-page file. You get the idea.
Something slightly more useful might be a method which accepts an array of integers as a parameter, with each integer representing a page number within a group to be extracted into a new, composite document. For example, say I needed pages 1, 6, and 7 from a 44 page PDF pulled out and merged into a new document (in reality, I needed to do this for pages 1, 6, and 7 for each of about 200 individual documents). We might add a method to our PdfFileSplitter Class as follows:
The ExtractToSingleFile Method
public static void ExtractToSingleFile(int[] PageNumbers,
string sourceFilePath, string outputFilePath)
{
var originalDocument = PDDocument.load(sourceFilePath);
var originalCatalog = originalDocument.getDocumentCatalog();
java.util.List sourceDocumentPages = originalCatalog.getAllPages();
var newDocument = new PDDocument();
foreach (var pageNumber in PageNumbers)
{
// Page numbers are 1-based, but PDPages are contained in a zero-based array:
int pageIndex = pageNumber - 1;
newDocument.addPage((PDPage)sourceDocumentPages.get(pageIndex));
}
newDocument.save(outputFilePath);
}
Below is a simple example to illustrate how we might call this method from a client:
Calling the ExtractToSingleFile Method:
public void ExtractAndMergePages()
{
string sourcePath = @"C:\SomeDirectory\YourFile.pdf";
string outputPath = @"C:\SomeDirectory\YourNewFile.pdf";
int[] pageNumbers = { 1, 6, 7 };
PDFFileSplitter.ExtractToSingleFile(pageNumbers, sourcePath, outputPath);
}
Limit Class Dependency on PdfBox
It is always good to limit dependencies within a project. In this case, especially, I would want to keep those odd Java class references constrained to the highest degree possible. In other words, where possible, I would attempt to either return standard .net types from my classes which consume the PdfBox API, or otherwise complete execution so that client code calling upon this class doesn’t need to be aware of IKVM, or funky C#/Java hybrid types.
Or, I would build out my own “PdfUtilities” library project, within which objects are free to depend upon and intermix this Java hybrid. However, I would make sure public methods defined within the library itself accepted and returned only standard C# types.
In fact, that is precisely what I am doing, and I’ll look at that in a following post.
John on Google CodeProject
I would make clear before I proceed that I am attempting to create a rational examination of the authors points from my own perspective. Like Mr. Cain, I am far from an economist or a study of political science, and I may well lack critical information here. I also want to make clear that I ultimately respect Mr. Cain’s opinions. Where I disagree, I am completely open to discussion and enlightenment. This post should be read from the perspective of a rational and respectful debate, rather than an ideological monologue. I imagine Mr. Cain and I could have a very interesting discussion over coffee, and both emerge the better for it.
I recently read a fascinating blog post which undertakes to describe author Caleb Crain’s understanding of predominant metaphors and concepts used to describe the American economy of the past century. That post can be found here, and it is an interesting read:
Topics within this post, in order for easier navigation:
The ultimate point of the author’s work appears to be that, through increasing computerization and automation, society will (has?) create(d) an economic engine dependent primarily upon machines, with little meaningful production from human power beyond building, operating, and maintaining the machines which produce the bulk of our economic output. Oh, and that the owners of such machines will realize most of the revenue and profit to be had. The author uses this scenario to make the case for Democratic Socialism and wealth re-distribution as the only effective mechanism for society to function within these constraints.
Mr. Crain’s theories about the “Robot Utopia” (or dystopia, as the case may be) is not where I take exception with the article. Also, I am not here to argue for or against any one economic or political system. Where I take exception, and would offer some constructive criticism of the author’s work, is in his presentation of flawed metaphors and concept to set the stage for his proposal that wealth redistribution and Democratic Socialism is the only logical conclusion to the story.
Most people make a natural comparison between a nation's budget and a family's. If a family is sliding into debt, the only remedies are to earn more and spend less.
Note the use of the term “family budget” in the above quote. It is this which which is the important concept, as we shall see shortly. Crain takes up the economic metaphor of the family budget commonly employed to describe proposed solutions to our economic woes, and calls it . . . flawed. The author counters with this seemingly sensible argument:
But a nation's economy is not at all like a family's. For one thing, within most families, communism prevails: the rule governing money is, From each according to his abilities, to each according to his needs. For better or worse, this doesn't happen to be the rule governing money in America at large. Also, within most families, money is not exchanged for labor.
This is so true. Within the micro economy that is the typical American family, one might say that some form of “communism” prevails. However, Mr. Crain undermines his argument with the very next statement:
“. . . In a pedagogical, largely symbolic way, Jimmy may be given $2 a week in exchange for taking out the garbage. But the person who cooks and cleans does not clock his hours; the children do not buy their dinners. The exchange of labor and goods within a family is for the most part unmeasured and invisible, and it makes more sense to understand a family as a group of people functioning [as] a single economic agent.” [emphasis mine]
Mr. Crain’s attempt to nullify the metaphor of the family budget-style economic prescription is invalid. Even communist countries, and the families within them, must have a budget, and exist within the restrictions of that budget. The author attempts to compare the micro-economic activity within a family unit with the macro-economic activity in which the family, as whole, participates. Also, note for later that within said family, there is a familiar central planning authority - namely, the parents. Yes, communism has been shown to work on a micro scale, within families, and within larger agricultural collectives. It has also been shown to fail miserably as a macro-economic model at the national level, especially as societies move beyond agriculture into mechanized and militarized police states. That central planning authority doesn’t work with nations. But I digress.
The idea that a nation, any nation, must exist within budget constraints which include revenue, expenses, and debt is 100% valid. And when the combination of expenses and debt-service payments exceed that nation’s ability to pay, expenses must be reduced, and/or revenue must go up. Which leads us to the next point, on which the author and I agree in substance, but for which again, the author makes a logically flawed argument and thus weakens his larger position.
A favorite of the American Republican party, Trickle-Down economics, or Supply-Side economics, is a largely political (rather than economic) idea which proposes that tax benefits or other economic incentives provided by government to businesses and the wealthy will spur economic investment, job creation, and overall economic growth. While there are numerous arguments against this so-called theory, the author once again attempts to use the micro-economic situation within a family to illustrate why the idea fails within a national economy. He is correct in pointing out that there is no real point of comparison here. But that, in essence, is the problem. There is not a valid point of comparison, and the author expends a longish paragraph describing why.
Proponents of trickle-downism will argue that the little-taxed corporate executive will in fact share his wealth by spending it, and that his purchase of goods and services will drive economic growth more efficaciously than mere giveaways would. But it turns out that the executive doesn't spend more, or not enough more for his increased spending to be helpful to the economy—for the simple reason that he doesn't need to. In the hands of rich people, money moves slowly.
Ahhh, Now HERE the author approaches the real reason why trickle-down/”supply-side” economics fails. What economists would call the “Marginal Propensity to Save.” When all of your needs are met, as well as the bulk of your “wants",” it makes more sense to sock that money away. If you own a business, your argument might go like this:
Government: “We’ll give you a real big break on your taxes. You pay nothing in taxes this year. Are you going to hire more people?”
Business Owner: “Don’t think so, unless my business grows. My current crew is keeping up with demand at the moment.”
Government: “Will you increase pay?”
Business Owner: “I do my best to pay a fair wage, based on what the market will bear, and employee merit.”
While there is a case to be made that tax breaks might allow for expansion into new markets, or of existing product lines, I think we can count on the idea that if business sees a profit in either of these areas, the expansion will occur with or without tax breaks. You know, because of the profit. If tax breaks are necessary in order for the move to be profitable, then possibly we need to examine the tax structure as a whole, and not just for this business.
While it is certainly true that individuals and corporations do in fact invest (even putting money into a savings account is a form of investing), which creates desirable economic activity, the economic disparity between the wealthiest among us and the rest of us grows faster under these conditions. Investment in potential job-creating activity, while most definitely a part of the process, is an inherently more risky thing than investing in a blue-chip stock. As an example, examine the trend in relative compensation of corporate executives compared to middle-class workers over the past three decades. Trickle-down economics tends to target benefits toward a certain group, instead of spreading economic benefit and liability equally across all.
The economic iniquity we see in American society today is growing at an alarming rate. Most likely, this represents the growing influence of moneyed interests in our political system, and the continuing perforation of our tax code. I suspect we are approaching a breaking point, in which the 24 hour news cycle and the internet propaganda machine
As importantly, levying taxes as an equal fraction of everyone’s income above and beyond the poverty line, and doing so at a level which is adequate to provide for the costs of running the country and service the debt is inherently more fair. The discussion of what costs are incurred, and the level of debt which should be allowed, are a different discussion, beyond the scope of this post.
I realize I am over-simplifying the situation here, and I am by no means attempting to take up a debate about Supply-Side economic theory itself (I lack the education and knowledge to do so). However, my point is that, the author and I agree that trickle-down economic theory is inherently flawed.
Perhaps my biggest complaint with Caleb Crain’s arguments lies in his proposal that the economic prosperity in America following World War II arose from some magical re-distribution of wealth within the American economy as a result of the war itself. From Mr. Cain’s post:
What if it wasn't the deficit spending of World War II that stimulated the American economy, but the war's redistribution of wealth? The war obliged America to employ a literal army of people as soldiers and factory workers, and after the war, America felt obliged to continue to reward the working classes with expanded social services, including free higher education for veterans.
This much is true. The war itself caused massive employment, both domestically, and with an immense increase in armed forces enlistment. No doubt, America’s entry into WWII caused a gigantic stimulus to the previously flagging economy. But it is the author’s next statement which falls short of the true mark:
The period from World War II to the 1970s turned out to be the greatest era of prosperity America has ever known. Is it a coincidence that it followed a massive, government-run redistribution of wealth, which happened to take the form of a war?
The war itself was most definitely a contributing factor, but not so much in the way the author suggests. Think about it - the wealth re-distribution (if you really need to call it that) achieved by paying the soldiers, and the women and others who assumed the previously male-dominated factory jobs for the course of the war, would soon end with the conclusion of the war. Granted, the increase in industrial capacity, and the massive production effort needed to sustain the war effort helped kick the economy into gear, but these things merely set the stage for the real reason America emerged from WWII into what “turned out to be the greatest era of prosperity America has ever known.”
Mr. Crain correctly notes that America was exceptionally prosperous in the post war years, through most of the 1970’s. However, he attempts to attribute this to the stimulus effect of wartime spending, and some sort of socio-economic engineering on the part of the federal government. The reality is quite different.
During the period from WWII through the 1970’s, America was dominant in nearly every sector of manufacturing. Why? Because the manufacturing capabilities of most of America’s potential competition were in ruins. Japan. Germany. England, France. The Soviet Union. While some were allies, and some were vanquished enemies, most of these nation’s suffered enormous damage to industrial production capacity as a result of the war, whereas America was left with a surplus of factories previously devoted specifically to feeding the national war effort (and to no small degree, that of our allies, as well).
As a side benefit, one of the “spoils” of war was an infusion (in many cases, of dubious ethical standing) of refugee Nazi scientists and engineers fleeing Nazi Europe for fear of becoming guests of the Russians.
In the years following World War II, while the rest of the Western world was struggling to rebuild shattered infrastructure and production capacity, America was enjoying a near-monopoly on manufacturing, and technological innovation.
At the conclusion of World War II America was blessed with conditions perfect for her rise as a the pre-eminent economic force in the post-war world, and she benefitted from this for several decades. While the war itself may have had some stimulating economic effect, the prosperity of the decades following WWII was far more about the incapacitated economic state of the other participant nations at war’s end.
This is the question posed by Caleb Crain as he leads us to the central tenet of his article, and he makes no claim to know an answer to this question, though he mentions the US possibly “not paying soldiers enough” and the possibility that the military’s investment in technology and equipment has muted war’s impact as a redistributor of wealth. To this I say, war is NOT an effective “redistributor of wealth” within a nation, although possibly, wealth is redistributed from the vanquished nation to the victor. In any case, the only redistribution of wealth in modern brush wars seems to be from the taxpayer to the defense contracting industry.
I suspect, in light of my forgoing argument that America’s economic prosperity resulted from the devastation of most of the rest of the world’s modern production capacity, that you know where I am going with this.
The wars (or “police actions'” or “conflicts”) in which America participated after WWII were an entirely different kind of conflict, and bear no comparison to the second world war. Yes, the government did spend tax dollars and create additional debt to fund those wars. But for the most part, these conflicts began and ended as proxy conflicts in third-world countries. At war’s end, the US had expended great capital, and possibly created a handful of jobs in the defense contracting industry, but in the meantime, other countries around the world (such as Germany, Japan, and China), unencumbered by international conflicts, continued to pump out products, and build/rebuild manufacturing infrastructure.
Indeed, it would seem that US military engagements in the years since WWII appear to be the fulfillment of Dwight D. Eisenhower’s prophetic warning about the dangers of the “Military/Industrial Complex",” creating wealth building opportunity for large defense contractors at the expense of the American taxpayer, and American soldier’s lives. Not to mention the apparent influence of the defense industry upon our policy-makers.
In point of fact, one could argue that the “stimulus” affect of such government spending and debt creation is demonstrated, in the form of these lesser conflicts, to be ineffective as a long-term economic solution. As I write this little missive, we have been embroiled in various parts of the Middle East and Afghanistan for most of the previous decade. To what economic result?
In making his ultimate point, Mr. Crain presents a dual notion, the first part of which is intriguing (and definitely something for economists to ponder), and the second part of which he spends most of the article trying to support, and in my view fails.
Thanks in large part to computerized mechanization, manufacturing productivity in the past century has increased many times over. Standards of living are higher than they ever were, but we no longer need as many humans to work as we once did. Perhaps not coincidentally, human wages, in America at least, have stagnated since the 1970s. If humans made no more money in the past four decades, where did the wealth created by the higher productivity go? Toward robot wages, as it were. The owners of the robots took the money—that is, the capitalists.
Above is the first part of the author’s final premise, which is most definitely food for thought. What happens when a society is sufficiently automated such that fewer and fewer actual workers are needed? This is potentially rich soil for thought. However, I suspect that people have had this concern since the first tools were invented, and especially when the industrial age began. New jobs will be invented to support new technologies, and new technologies will continue to spawn new work which needs to be done.
After all, while buggy whip makers were sadly and tragically displaced by the invention of the automobile, the economy did not collapse. In fact, new and fantastic opportunities arose in the form of interstate transportation, automobile service, and automobile manufacturing. Likewise, the age of the personal computer has eliminated typing pools and decimated the slide rule manufacturing sector, but has spawned an enormous industry around software development. And so on.
At some point society has to choose. Either society accepts the robots' gift as a general one, and redistributes the wealth that the robots inadvertently concentrate, or society allows the robots to become the exclusive tools of an ever-shrinking elite, increasingly resented, in confused fashion, by the people whom the robots have displaced.
You might compare the political challenge they represent to what's known as the "resource curse"—the infamous difficulty that oil-rich nations have in preserving democracy while sharing the oil's proceeds. Do we want to be Norway or Saudi Arabia? The choice seems to be between democratic socialism and tyranny
And that is the final part of the author’s conclusion. I contend that such is not the choice, or at least, not the only choice. More importantly, the author appears to lead the reader to this binary option through the previously examined arguments about wealth redistribution, which for myself, anyway, I feel are deeply flawed, and lacking in factual support in the real world.
The Relative Ease of Successive Generations
Author Crain’s expression of the Robot Utopia has been expressed similarly in the past. Though the degree of paranoia may vary, each technological innovation brings with it some fear of ultimate catastrophic impact. Hell, I remember my mother, on seeing her first digital clock, saying “gosh, if everybody has those, no one will be able to tell time anymore".” No, really. She felt that way, and was concerned.
As humankind continues to automate areas of manufacturing and other repetitive or unpleasant segments of the economy, the economic landscape will change, and our socio-political systems along with it. America, in particular, has felt for numerous generations that we are somehow entitled to be the economic center of the universe. Now, as an American, I find this idea seductive, and I would that my country remain at the top of its game. However, the reality is that great economies rise and fall, expand and contract. The void left by one will be filled by another, guaranteed. Political and economic systems are nothing but models, created around sets of ideals. In general, the ideals which form the basis for those models regularly exceed our limited human capacity to fulfill, and the result is the model breaks down.
I truly enjoyed reading Caleb Crain’s article, and while I find fault with some of his arguments, he makes interesting and though-provoking points nonetheless. What do YOU think?
John on Google