Wednesday, May 20, 2009

Primitive Tooling

I don't know that I'm such a big fan of IDE's. They seem to run counter to the Unix Philosophy, which is to have a large number of small tools that do one specific thing, and do it very well. These small tools are all bound together through dead simple, absolutely standard and rock-solid message-passing interfaces (stdin, stdout, stderr; all parameters are Strings). A big IDE that attempts to do everything and the kitchen sink often just feels like the wrong level at which to be working. Certainly, for my current project, being strongly bound to Eclipse for development has so far been counterproductive. This is because the GWT-Eclipse tooling is not yet matured... which is to say that the GWT module concept is really very different from the basic Java package concept, and this is a gap that the current tooling does not yet bridge. The result, for me at least, has been a great deal of confusion. Then again, I'm attempting to use GWT for things for which it was never intended to be used ;)

In any case, it has felt very good to ditch the IDE for a while and return to my humble yet powerful development tools: GNU Screen, Vim, Bash and Ant. I won't describe here what these tools are or how to use them; this is covered very well in other places. But I will post a nice screenshot of my current desktop, which I feel is very close to complete User Interface zen:




One component currently missing from my toolbox is a good, preferably console-based file managers. I don't really have very strong preferences with regard to file managers, but I really should, because there are many times when I could really use some powerful features. I just haven't quite figured out what those are yet. Vi keybindings are a major plus, though. That rules out the famous midnight commander, unfortunately, as its keybindings are not configurable. Really, it only leaves two options: vifm, and a Vim plugin called VimExplorer. vifm was quite nice, a nice solid little file manager with vi-like keybindings. Unfortunately, it had some trouble installing on Ubuntu Hardy, and it was just different enough to what I was used to with a proper Vim instance to be annoying. That left me VimExplorer. I was a bit wary of this, because it hadn't been updated since 2007, and the last update was to fix a critical bug that could cause data loss. But I installed it, tried it out, and found it to be quite good. It gives you a small file manager instance that lives right inside of vim. All of your vim skills then can be applied to managing files, including: regular expression search, macros, manipulating vim windows and buffers, and using "!" to trigger shell scripts. There's something really great about being about to just say "yy" and then "p" to copy and paste a file. Same for move and delete.

Unfortunately, I still feel like this tool is lacking. 90% of what I do when developing is in done in just a few files and directories. What I want is to be able to just type the name of a file on my filesystem (e.g. "Accessibility.gwt.xml"), and immediately have an editor up for that file. The system should behave intelligently when there are naming collisions. I believe that GNOME-Do might actually be able to do this, and I should look into that. For the most part though, I think most of this could be accomplished by just tagging certain files or directories as favorites, and maintaining a simple hash to map short variable names to long paths. In any case, I'm continuing to iterate on this component of my set of simple tools.

Convergence 1

Warning: Extremely Boring, Project-Specific Post

I worked really hard today, and I have lots to report. I started at 6:30am and worked straight through on GSoC until about 7:30pm. I also spent some time investigating alternate tooling, and I'll talk a bit about this in another post.

So, to recap, we have this stack: Demos/GEF/Draw2d/SWT/JRE. The goal of this week is to get Draw2d to compile under GWT. After that, I'll have to try to get it to run, but for now, I'm just concerned with getting it to compile. This, of course, means figuring out what Draw2d's dependencies are in SWT and the JRE, and where they are not satisfied, writing out stub code and small TODO comments. Also, I will atempt, where possible, to reuse the existing, non-functional code from the previous project that used GWT to compile SWT to js.

To summarize my experiences of today: JRE == SUCCESS, SWT == FAIL.

First some background: GWT already provides some JRE emulation, but it is not sufficient to compile the SWT codebase. To get SWT to compile, one must at least implement certain stub interfaces that GWT does not provide. There are additional issues, such as reflection, that may be more serious, and I'll talk about those in a later post. The extension to the GWT emulation classes live in the project org.eclipse.swt.e4.jcl. Left over from the previous project were three GWT modules which I was never able to compile. Additionally, it was unclear to me how these modules were supposed to compile, as they used the GWT module "source" tag, rather than the "super-source" tag, which, according to all of the documentation I have read, is what one is supposed to use when implementing JRE classes.

Today, I successfully compiled the previous project's JCL classes. In order to accomplish this, I applied Technique 1 from my previous post, which is to say, I added a prefix ("e4") to the JCL packages, where before there was no prefix, and I used <super-source path="e4"/> in my GWT modules. I had to futz a bit to discover in which order I should import the modules, but after that, they compiled without complaint, w00t

//TODO: insert complete project structure here

Note that, since yesterday, I have been developing this prototype outside of the Eclipse environment. My toolset has consisted of GNU Screen, Vim, a Vim plugin called VimExplorer, and Ant. This has allowed me to ignore all of the IDE-specific confusion that I have so far encountered. My next post will be about my experiences using some of these tools.

I was feeling very optimistic after succeded in getting JCL to compile, so I set it as my goal to get the necessary SWT libraries to compile as well. Unfortunately, I failed in this task, because, I believe, I took the wrong approach. The approach I took was to attempt to reuse directly the SWT emulation classes left over from the previous project, just as I reused the previous project's JCL classes. Unfortunately, these SWT emulation classes had many missing dependencies. My strategy was to just keep hacking on it (adding in missing classes, stub methods, etc.), until the thing worked. I spent most of the afternoon working on this, probably about 4 hours. Ultimately, I ended up pulling in most of the code in "org.eclipse.swt/Eclipse SWT/common". I didn't get to the end of the dependency chain, but I think I got far enough along to recognize a trend that I wasn't happy with, namely, pulling in dependencies that in turn rely on other dependencies, etc. It should be a major priority to limit dependencies, because the more deps I cut out, the less complex my project will be. Given that this project is still at the prototyping stage, reducing complexity is of critical importance.

Note that I also tried compiling "org.eclipse.swt/Eclipse SWT/common", but this pulled in some native code which GWT was unhappy with. So, I really do need to be selective.

The alternative to today's approach is to start at the top of the stack (work top-down) and discover what the dependencies are for the Draw2d demos; then, find out what the dependencies are for Draw2d; then, very carefully, attempt to pull in those deps from org.eclipse.swt, or stub them out. I intend to use Doxygen to figure out the complete dependency chain. This will be tomorrow's work.

Tuesday, May 19, 2009

Summary of Some of the Pecularities Involving GWT Eclipse Tooling and Super-Source

I've been having some difficulties figuring out how to use the super-source tag for GWT's module files. See the following forum posts for more background on this:

http://groups.google.com/group/Google-Web-Toolkit/browse_thread/thread/933d511f524cc355/6c9216fd943e3f47?lnk=gst&q=otakuj462#6c9216fd943e3f47
http://groups.google.com/group/Google-Web-Toolkit/browse_thread/thread/776660639069c8e9/98bc3c99c7a2848f?lnk=gst&q=otakuj462#98bc3c99c7a2848f

In short, there is a kind of impedance mismatch between the functionality that GWT provides the functionality that GWT's Eclipse tooling provides, at least with respect to super-source. Here's my understanding of how all of this breaks down:

1. You can use a package structure in which the native packages you would like to emulate (e.g. java.*) are prefixed by some other package (e.g. hack.java.io). So you might have a project structure that looks like this:


testsuper/
super/
test/
TestSuper.gwt.xml
test.hack.java.io/
OutputStream.java


Your GWT module super-source tag would then use path="hack".
The package declaration in your emulated Java classes would then say "java.io" (no hack prefix!).

The Pros: This works great in both GWT hosted mode and compiled mode.
The Cons: The Eclipse IDE will think that something is wrong with your package declaration (it thinks it should be hack.java.io), and will give you errors.

2. You can use a package structure which does not use prefixed packages. So, you have something like:


testsuper/
super/
test/
TestSuper.gwt.xml
test.java.io/
OutputStream.java


Your GWT module super-source tag would then use path="".
The package declaration in your emulated Java classes would then say "java.io", as per normal.

You then have two further possibilities:

2.a. Put super/ on the Eclipse build path.

The Pros: GWT Compilation mode works, and the Eclipse IDE doesn't throw errors regarding packaging.
The Cons: GWT Hosted mode fails, ostensibly because it's using the incorrect Java classes. In fact, I'm not really sure why GWT Hosted mode fails with this setup, but it surely does.

2.b. Do not put super/ on the Eclipse build path.

The Pros: GWT Hosted mode works (although you must set the runtime classpath to point to whatever dependencies TestSuper has; you set this as an Eclipse run configuration), and the Eclipse IDE doesn't throw errors.
The Cons: GWT Compiled mode fails for the Eclipse GWT tooling, if you have projects that import this project, because the super/ folder is not on the Eclipse build path. If you were using Ant, this would not be a problem, because you can simply set the build-time classpath, but it does not seem to be possible to do this yet with the Eclipse tooling. Also, because super/ is not on the build path, you don't get all of the advantages of the Java tooling that you would normally get from Eclipse.
Workaround: Use ant within Eclipse to Compile, and use regular GWT for Hosted mode.


It's important to point out that all of these issues are related to the GWT Eclipse tooling, not GWT itself. Nevertheless, for my project, I am strongly dependent on Eclipse, so it is necessary for me to find the best way to resolve these issues. I'm not 100% certain about this, but at the moment, I feel like my best option is to go with option 2.b., so to stay in Eclipse, but move all of the build logic to Ant.

Saturday, May 9, 2009

Top-Down 2

Warning: Extremely Boring, Project-Specific Post

Onto the next puzzle: I'm trying to compile without importing any JCL code, in order to find out what JRE code needs to be emulated that is not covered by GWT. I'm getting this very peculiar output:


Buildfile: /home/jacob/workspace/gsoc-phase-1/org.eclipse.swt.e4.examples/dojo/build.xml

init:

build-dojo:
[java] WARNING: 'com.google.gwt.dev.GWTCompiler' is deprecated and will be removed in a future release.
[java] Use 'com.google.gwt.dev.Compiler' instead.
[java] (To disable this warning, pass -Dgwt.nowarn.legacy.tools as a JVM arg.)
[java] Compiling module controlexample
[java] Refreshing module from source
[java] Validating newly compiled units
[java] Removing units with errors
[java] [ERROR] Errors in 'file:/home/jacob/workspace/gsoc-phase-1/org.eclipse.swt/Eclipse%20SWT/common/org/eclipse/swt/internal/image/PNGFileFormat.java'
[java] [ERROR] Line 294: The method getProperty(String) is undefined for the type System
...


Why on earth is the GWT compiler trying to go inside of org.eclipse.swt/Eclipse SWT/common/org/eclipse/swt/internal?

Top-Down 1

Warning: Extremely Boring, Project-Specific Post

Moving forward. In the post before my last post, I mentioned the swtlibs.gwt.xml file which was confusing me. Here it is again:


<module>
<source path=""/>
<source path="events"/>
<source path="graphics"/>
<source path="layout"/>
<source path="widgets"/>
<source path="dnd"/>
<source path="printing"/>
<source path="animation"/>
<source path="accessibility"/>
<source path="custom"/>
<source path="graphics2"/>
<source path="internal"/>
<source path="net"/>
<script src="swt/loading.js"/>
</module>


The thing that was confusing me was the path entries. One would assume that they are relative paths... and I've now decided that, in fact, they are. First of all, the GWT documentation makes it pretty clear that this is the case:


<source path="path" /> : Each occurrence of the tag adds a package to the source path by combining the package in which the module XML is found with the specified path to a subpackage. Any Java source file appearing in this subpackage or any of its subpackages is assumed to be translatable. The <source> element supports pattern-based filtering to allow fine-grained control over which resources get copied into the output directory during a GWT compile.


But this was nonobvious in this case, because all of the paths referenced except for graphics and widget, are not accessible as relative paths. One could interpret them as being poorly-documented indicators of "future work", but only if the it was certain that the GWT compiler wasn't doing something clever like pulling in other, actual SWT folders on the classpath, that have the same name; for this to be true, the GWT compiler needed to be silently ignoring the missing referenced paths. To test this, I set up a sample GWT project with a source package that does exist. I also added a reference to a source package that did not exist:


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 1.6.4//EN" "http://google-web-toolkit.googlecode.com/svn/tags/1.6.4/distro-source/core/src/gwt-module.dtd">
<module rename-to='test'>
<!-- Inherit the core Web Toolkit stuff. -->
<inherits name='com.google.gwt.user.User'/>

<!-- Inherit the default GWT style sheet. You can change -->
<!-- the theme of your GWT application by uncommenting -->
<!-- any one of the following lines. -->
<inherits name='com.google.gwt.user.theme.standard.Standard'/>
<!-- <inherits name='com.google.gwt.user.theme.chrome.Chrome'/> -->
<!-- <inherits name='com.google.gwt.user.theme.dark.Dark'/> -->

<!-- Other module inherits -->

<!-- Specify the app entry point class. -->
<source path="exists"/>
<source path="doesnotexist"/>
<source path="client"/>
<entry-point class='foo.bar.test.client.Test'/>
</module>


End result: the Java classes in subpackage exists were found usable in both hosted mode and compiled mode. The compiler just ignored the reference to a package that did not exist.

So what does all of this finally mean for my project? It shows me the scope of the SWT emulation that they had at one point presumably been working. At the very least, it sets an upper bound on the work that was done in this regard: emulation of graphics and widgets package. That's all. But they were aiming to emulate the entire SWT library.

With that in mind, I dug a bit deeper into the widgets package. As expected, there were a bunch of methods using JSNI. Not the nicest JavaScript code... But I won't criticize it to much, as I understand that it never got past the prototype phase.

Bottom-Up 1

Warning: Extremely Boring, Project-Specific Post

I'm ignoring the work that has already been done on this, for the moment, and imagining how I would do it if I were starting from scratch.

There are two components that would need to be implemented: the jre emulation layer, and the SWT emulation layer. GWT already provides a JRE emulation layer, but it will likely be necessary to enhance it.

SWT is big. All we really care about is GraphicsContext and event handling.

The graphics part should slot nicely into the way SWT graphics is already organized, in that it has different drawing backends. We just need to use the API exposed by GWTCanvas. It should be possible to look to swt graphics for guidance on this, in that it already supports many different target implementations.

Draw2d may have other dependencies on SWT beyond GC and events. Here's a bit of script I used to found out what dependencies org.eclipse.draw2d has on SWT:


jacob@jacob-laptop:~/workspace/gsoc-phase-1/org.eclipse.draw2d$ grep -o -r "import org.eclipse.swt.*" . | sed 's/[^:]*:\(.*\)/\1/' | sort | uniq

import org.eclipse.swt.accessibility.ACC;
import org.eclipse.swt.accessibility.AccessibleAdapter;
import org.eclipse.swt.accessibility.AccessibleControlAdapter;
import org.eclipse.swt.accessibility.AccessibleControlEvent;
import org.eclipse.swt.accessibility.AccessibleControlListener;
import org.eclipse.swt.accessibility.AccessibleEvent;
import org.eclipse.swt.accessibility.AccessibleListener;
import org.eclipse.swt.events.ControlAdapter;
import org.eclipse.swt.events.ControlEvent;
import org.eclipse.swt.events.DisposeEvent;
import org.eclipse.swt.events.DisposeListener;
import org.eclipse.swt.events.FocusEvent;
import org.eclipse.swt.events.FocusListener;
import org.eclipse.swt.events.KeyAdapter;
import org.eclipse.swt.events.KeyEvent;
import org.eclipse.swt.events.KeyListener;
import org.eclipse.swt.events.MouseEvent;
import org.eclipse.swt.events.MouseListener;
import org.eclipse.swt.events.MouseMoveListener;
import org.eclipse.swt.events.MouseTrackAdapter;
import org.eclipse.swt.events.MouseTrackListener;
import org.eclipse.swt.events.SelectionAdapter;
import org.eclipse.swt.events.SelectionEvent;
import org.eclipse.swt.events.TraverseEvent;
import org.eclipse.swt.events.TraverseListener;
import org.eclipse.swt.graphics.Color;
import org.eclipse.swt.graphics.Cursor;
import org.eclipse.swt.graphics.Font;
import org.eclipse.swt.graphics.FontData;
import org.eclipse.swt.graphics.FontMetrics;
import org.eclipse.swt.graphics.GC;
import org.eclipse.swt.graphics.Image;
import org.eclipse.swt.graphics.ImageData;
import org.eclipse.swt.graphics.LineAttributes;
import org.eclipse.swt.graphics.PaletteData;
import org.eclipse.swt.graphics.Path;
import org.eclipse.swt.graphics.PathData;
import org.eclipse.swt.graphics.Pattern;
import org.eclipse.swt.graphics.Point;
import org.eclipse.swt.graphics.Rectangle;
import org.eclipse.swt.graphics.RGB;
import org.eclipse.swt.graphics.TextLayout;
import org.eclipse.swt.graphics.TextStyle;
import org.eclipse.swt.graphics.Transform;
import org.eclipse.swt.printing.Printer;
import org.eclipse.swt.SWT;
import org.eclipse.swt.SWTError;
import org.eclipse.swt.SWTException;
import org.eclipse.swt.widgets.Canvas;
import org.eclipse.swt.widgets.Caret;
import org.eclipse.swt.widgets.Composite;
import org.eclipse.swt.widgets.Control;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Event;
import org.eclipse.swt.widgets.Listener;
import org.eclipse.swt.widgets.Shell;


GWT modules will glue the project together. There will be a top-level example module which has an entry point, and imports the modules for SWT emulation and JRE emulation. That's basically it.

Friday, May 8, 2009

Strategy, and what I'm finding out

Warning: Extremely Boring, Project-Specific Post

So, my strategy for my GSoC project is going to simultaneously encompass two approaches: top-down and bottom-up. The top-down approach would have me re-use the pre-existing code as a starting point as much as possible. The bottom-up approach would have me start from scratch. I think the success of my project depends on me embracing both: I should try to understand the code as much as possible in order to maximally reuse it. At the same time, said code appears to be in a rather broken state, so I need to forge ahead and dolve problems in my own way. It would be really nice if the code was not in a broken state, though, as I can see where my contribution would slot right in. Oh well...

Here's what I've determined so far. The goal of this phase of the project is to use GWT to implement the SWT API, so that GWT may then be used to compile any Java code that targets SWT down to JavaScript. It appears that the work that has been done so far has been to implement this API for most of the SWT controls. A notable exception to this is the GraphicsContext (GC) API. This is where my contribution would slot in, if any of the previous work was functioning correctly.

The best place to zoom in on this is org.eclipse.swt.e4.examples/dojo/controlexample/controlexample.gwt.xml. This top-level GWT module illustrates what other projects I would likely need to depend on, and where the GWT modules have been inserted into that project structure.


<module>
<!-- //////// FIXED PATHS //////// -->
<!-- Inherit the SWT-Dojo libs and the SWT common libs -->
<inherits name="org.eclipse.swt.swtlibs"/>

<!-- Inherit the base Dojo libs -->
<inherits name="dojoLib"/>

<!-- Inherit the SWT javascript libs, css styles, etc. -->
<inherits name="resources"/>

<!-- Inherit java.io + java.lang emulated libs. -->
<inherits name="JCL"/>
<inherits name="extended_JCL"/>
<inherits name="extended_js_JCL"/>

<!-- Inherit the core Web Toolkit stuff. -->
<inherits name='com.google.gwt.core.Core'/>
<!-- //////// FIXED PATHS //////// -->

<!-- Specify the app entry point class. -->
<entry-point class="org.eclipse.swt.examples.controlexample.Main"/>

</module>



So, basically, we're pulling in the SWT lib, dojo, some resources, classes for custom JRE emulation, and GWT core. Pretty straightforward so far. There's an ant build.xml file that accompanies this which sets up the appropriate classpath and runs the GWT compiler against the module. If I was using the GWT plugin for Eclipse, I imagine that I could do away with the ant script and simply press the GWT compile button to begin compilation.

So, that part is fine.

It gets more complicated, though, when you drill down into the above modules. I spent the first week-and-ahalf of this project trying to make sense of the *JCL modules, before finally deciding that they were, in actual fact, in a disfunctional state. See, for example, the swtlibs.gwt.xml module:


<module>
<source path=""/>
<source path="events"/>
<source path="graphics"/>
<source path="layout"/>
<source path="widgets"/>
<source path="dnd"/>
<source path="printing"/>
<source path="animation"/>
<source path="accessibility"/>
<source path="custom"/>
<source path="graphics2"/>
<source path="internal"/>
<source path="net"/>
</module>


These might appear to be relative paths, but they are not, because the directory structure in which they find themselves is as follows:


org.eclipse.swt/EclipseSWT/
  common/
    org/eclipse/swt/
      events/
      graphics/
      internal/
      layout/
      widgets/
      package.html
      SWT.java
      SWTError.java
      SWTException.java
  dojo/
    org/eclipse/swt/
      graphics/
      widgets/
      swtlibs.gwt.xml


By process of elimination, it appears that they are referencing libs in the org.eclipse.swt project.


jacob@jacob-laptop:~/workspace/eclipse$ find -name layout
./org.eclipse.swt/Eclipse SWT/common/org/eclipse/swt/layout


That's the only folder named "layout" in the entire workspace.

There's a bunch more strangeness here involving importing the project, which I'll cover in a later blog post.

GNU/Linux Desktop Distress

Right now, I'm running Kubuntu 8.04, which uses the very excellent KDE 3.5.10 desktop environment. Unfortunately, Kubuntu 8.04, unlike its close cousin Ubuntu 8.04, is not an LTS (Long-Term Support) release. This means that, instead of support being discontinued in April 2011, it will be discontinued in October of this year. I suppose this makes sense, as KDE technology has moved on to the 4.* branch. The KDE developers apparently no longer even fix bugs in KDE3 apps. This makes me sad, because I'm totally happy with my current setup. I really don't want to change my system when everything is working so nicely, but I'm afraid I'm just going to have to bite the bullet and do it. The question is, what should I change to? It seems like my main options are to either upgrade to Kubuntu 9.10 (which will be out in October), which should ship with the KDE 4.3 desktop; or, I stay in 8.04, but step over to the Ubuntu desktop.

One reason I've been avoiding using GNOME/Ubuntu is that by using Kubuntu 8.04, I've been able to avoid PulseAudio. I have had terrible experiences with PulseAudio so far. It is really not ready for normal users. My roommate Chris has discreetly switched back to using Windows because we couldn't adequately resolve his audio problems. I don't blame him; PulseAudio is not ready to go. Neither is Phonon (the KDE4 media backend) for that matter.

BUT, I have used PulseAudio enough to apprehend the vision behind it, and I like it. I like the idea of having audio sources, sinks and streams; having the ability to redirect those streams arbitrarily, even across a network. It's a lovely, simple vision. I would love to write a beautiful, graph-like UI on top of it. I want to believe in PulseAudio. I want it to work. So, if I can get the minimum apps that I need to work, including the proprietary ones (Flash and Skype, although I'm hoping to finally begin to migrate rapidly away from Skype very soon), given my very minimal hardware requirements, and have it work reliably, then I will be a happy camper. Anyways, I've sudo apt-get installed ubuntu-desktop, and I'm going to try it and see what the state of things are.

The other reason why I've been hesitant to switch to GNOME/Ubuntu is because of GNOME's use of Mono. I'm in the camp that says that relying on Mono is bad. Microsoft is litigious and Mono is patent-encumbered. It should therefore be avoided; it's as simple as that. What that ultimately means in terms of how I use my desktop, though, is not clear. The only applications that come bundled with Ubuntu that use Mono appear to be FSpot and Tomboy. That's not so bad; these are two apps I can live without. But I was hoping to try out Gnome-Do, which is also based on Mono. It's tempting to just ignore the issues surrounding it, and just install the app, but it feels dirty and wrong, so I just won't do it.

There are certain KDE apps that I would miss if I used GNOME: Amarok is a big one. Konqueror is another. KDESVN rocks my socks. Of course, I can use all of these apps in GNOME as well, but then it pulls in all of the KDE libraries.

In any case, these are all KDE3 apps running on top of KDE3. From what I've seen, the KDE4 apps are often radically different. One reason I'm even engaging in this discussion is because my experience of KDE4 has so far been quite bad. On every platform I've used it, KDE4 has felt slow and buggy. I got a chance to really sink my teeth into it in Kubuntu 9.04 (the last *buntu I installed on my roommate Chris's computer before he finally gave up on it), and everything regarding usability was just awful. For just an example, look at what they did to the available apt front-ends. Kubuntu 8.04 shipped with aptitude_manager. It was fast, had a fast incremental search, a simple direct UI. I never once had a problem with it. Not a single instance of it not being able to install packages correctly or crashing. Even slightly weird packages like the java-sun* packages that require you to agree a special license installed easily and reliably. In KDE4, on the other hand, you have KPackageKit. The app feels slow, the UI is terrible, and it choked multiple times when trying to install software. In particular, it choked on java-sun, and actually asked me to try using a better apt front-end. Ridiculous. There is also a version of Aptitude for KDE4, although though it doesn't ship by default, but the UI is also very different from the KDE3 version, and, in my opinion, is not very usable.

My experience of KDE4 was like this constantly while I was setting Chris up with Kubuntu 9.04. So I feel I may not be very happy in KDE4. Too much has changed, most of it for the worse.

So where does that leave me as a full-time GNU/Linux user, looking toward the future of my OS of choice?

...

Probably GNOME, sans Mono. Although I keep waffling about this. Hopefully, in six months Kubuntu will have turned things around, and I can reassess my decision. But who knows, by that time I might be a happy GNOME user.

Thursday, May 7, 2009

Fun with Git

I met a bunch of smart people at the Montreal Ubuntu 9.04 release party, and I talked to one person a lot about the distributed SCM Git. I had heard of distributed SCM's but had never had any reason to try anything other than SVN. Still, I was intrigued, and so decided to look into it. Here are some useful links:

For an overview of what Git is, see Linus Torvald's tech talk at Google. For an overview of how to use Git, see Randall Schwartz's talk at Google. For an overview of how I have so far successfully used it in my GSoC project, read on.

My goal was to use Git to solve a particular problem, which was as follows:

There is an SVN repository A, and a CVS repository B. The code in B was probably forked at some point from the code in A. B was iterated on in a private sandbox before being put into CVS, so B has no history, just HEAD. A has also changed since the time it was forked. I need to take the code in B and compare it to each of the major release tags in A, so as to determine which tag is the most similar to B, and therefore which tag B was most likely to have been forked from. This basically means diffing two different filesystem trees. Because Git is supposed to be very good at merging different branches based on content, I assumed it would be able to give me some very intelligent diff output.

In this contex, A is the GWT SVN repo, and B is a project in Eclipse's CVS repo. The Eclipse project org.eclipse.swt.e4.jcl appears to be forked from GWT, so I needed to determine when it was forked, what changes were made after, and (hopefully) why.

The person I met at the Ubuntu party, Derek, wrote me very excellent instructions on how to do this, which I have gratuitously copied and pasted here.


Here are the general steps:

- Use git-svn-clone to clone the Subversion repository A.
- Checkout the CVS HEAD of repository B into a separate workspace.
- Create a new branch in the git-svn repository. You can branch
starting from any tag or commit, but the closer that the branch origin
is to the original fork tag, the fewer differences you will likely
encounter. So, try to correctly guess the fork tag and branch from
that guess.
- Overwrite the workspace associated with the new branch with all the
files in the CVS HEAD workspace. Stage and then commit the changes
into the Git repository.
- For every Subversion "tag" branch in Git (Git stores every
Subversion tag as a Git branch, not as a Git tag), display the
differences between the branch that contains CVS HEAD and the Git
Subversion "tag" branch. Note the one that gives you the fewest
differences.

Specific commands (untested):

# Clone the Subversion repository A. Option "--stdlayout" tells Git
that the repository has the standard branches, tags, trunk layout
git svn clone --stdlayout http://domain.com/repositoryA

# Create a branch from a tag
git branch subversiontag cvsbranch

# Switch to branch "cvsbranch"
git checkout cvsbranch

# Overwrite files in Git "cvsbranch" workspace with those from CVS
HEAD workspace. Make sure that you don't copy the CVS metadeta
directory in each workspace directory. Consider using rsync which can
ignore CVS and Subversion metadata directories.

# Stage all CVS changes
git add --all

# Commit CVS changes
git commit --message "Added changes from CVS repository B"

# List all Git Subversion tag branches
git branch | grep svn

# Compare each Git Subversion tag branch with tag "cvsbranch". Option
"--name-only" displays only the names of the files that changed:

for tag in `git branch -r | grep tags`; do git diff --name-only $tag
cvsbranch; done

# Note the tag that produced the fewest differing files.



While this mostly worked great, I ran into one gotch using Git from the Ubuntu 8.04 repo (git version 1.5.4.3), which was that git-svn did not capture all of the tags from svn. When I tried it with git from the 8.10 repo (git version 1.5.6.something), it worked fine. Really, I should be git-cloning the latest git, but I haven't quite figured out how to do that successfully. Nevertheless, beware.

I also tried using svn2git, which is supposed to basically be svn-git with some manipulations to make the imported svn repo more git-like. I spent an hour trying to get this to work, and met with marginal success. It turned out not to be neccessary.

In the end, I ended up using the following code to guage similarity.

The first one just counts the number of files that changed between each tag and the cvs branch.


olpc@OLPC:~/workspace-gsoc/svn/user/super/com/google/gwt/emul/java$ for tag in `git-branch -a | grep tag`; do echo $tag; git-diff --relative --name-only $tag cvsbranch | wc; done;

tags/1.3.1
151 151 3767
tags/1.3.3
151 151 3767
tags/1.3.3@288
151 151 3767
tags/1.4.10
150 150 3746
tags/1.4.59
150 150 3746
tags/1.4.60
150 150 3746
tags/1.4.60@1399
150 150 3746
tags/1.4.61
150 150 3746
tags/1.4.61@1504
150 150 3746
tags/1.4.62
150 150 3746
tags/1.4.62@2104
150 150 3746
tags/1.5.0
78 78 1874
tags/1.5.0@2941
78 78 1874
tags/1.5.1
70 70 1709
tags/1.5.1@3391
70 70 1709
tags/1.5.2
70 70 1709
tags/1.5.2@3587
70 70 1709
tags/1.5.3
73 73 1756
tags/1.5.3@3776
73 73 1756
tags/1.6.0
78 78 1863
tags/1.6.0@4621
78 78 1863
tags/1.6.1
78 78 1863
tags/1.6.1@4846
78 78 1863
tags/1.6.2
78 78 1863
tags/1.6.2@5035
78 78 1863
tags/1.6.3
78 78 1863
tags/1.6.3@5110
78 78 1863
tags/1.6.4
78 78 1863
tags/1.6.4@5112
78 78 1863
tags/1.6.4@5189
78 78 1863


This second analysis actually counts the number of lines in each diff
between each tag and the cvs branch.


olpc@OLPC:~/workspace-gsoc/svn/user/super/com/google/gwt/emul/java$ for tag in `git-branch -a | grep tag`; do echo $tag; git-diff --relative $tag cvsbranch | wc; done;

tags/1.3.1
18257 77427 515421
tags/1.3.3
18257 77427 515421
tags/1.3.3@288
18257 77427 515421
tags/1.4.10
18261 78017 524249
tags/1.4.59
18158 77486 520272
tags/1.4.60
18158 77486 520272
tags/1.4.60@1399
18158 77486 520272
tags/1.4.61
18158 77486 520272
tags/1.4.61@1504
18158 77486 520272
tags/1.4.62
18158 77486 520272
tags/1.4.62@2104
18158 77486 520272
tags/1.5.0
5667 22620 165036
tags/1.5.0@2941
5667 22620 165036
tags/1.5.1
4357 17759 127102
tags/1.5.1@3391
4357 17759 127102
tags/1.5.2
3920 15766 113373
tags/1.5.2@3587
3920 15766 113373
tags/1.5.3
4045 16322 117623
tags/1.5.3@3776
4045 16322 117623
tags/1.6.0
4974 19823 144457
tags/1.6.0@4621
4974 19823 144457
tags/1.6.1
4986 19881 144836
tags/1.6.1@4846
4986 19881 144836
tags/1.6.2
4986 19881 144836
tags/1.6.2@5035
4986 19881 144836
tags/1.6.3
4986 19881 144836
tags/1.6.3@5110
4986 19881 144836
tags/1.6.4
4986 19881 144836
tags/1.6.4@5112
4986 19881 144836
tags/1.6.4@5189
4974 19823 144457


So, it looks like 1.5.2 has significantly fewer changes than any other tag. The diff to 1.5.2 is attached to this email. It basically looked like 48 new files were added, 0 were removed and many were modified.

Once I had established which GWT SVN tag was most similar, it was trivial to diff the branches and get meaningful output:


olpc@OLPC:~/workspace-gsoc/svn/user/super/com/google/gwt/emul/java$ git-diff --relative cvsbranch tags/1.5.2 > 1-5-2.diff


I won't post the resulting diff here, but suffice it to say that it was exactly what I was looking for, and I am now moving onto the task of analyzing the diff to figure out exactly what was changed, and why.

My final impressions of git: I'm very happy with it. The ability to import an entire SVN repo so that it can be used offline; the ease with which one may branch and merge; and the ability to push changes back up into the svn repo, are all things that will be very useful to me. I think I'm actually going to get to use Git a lot on my GSoC project.