Tuesday, November 27, 2012

Minimal eclipse-like application

For some of my applications I need to create a minimal eclipse like application. Something that uses a workspace, has preferences, help, p2, logger and the look and feel of eclipse. Yet I do not want JDT or any other programming language support. The question is: how would a minimal setup for such an application look like?

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online.

Step 1: create a plug-in and a feature

As products generally should be feature based we first create an empty plug-in project named com.codeandme.eclipse.minimal and a feature com.codeandme.eclipse.minimal.feature that contains this project. Simple as that.

Step 2: create the product

Create a new Product Configuration named Minimal.product in your product plug-in.

On the Overview tab create a new Product Definition using the button on the righthand side. Select a nice Product Name and a unique Product ID. To use the same application as eclipse select org.eclipse.ui.ide.workbench.

Afterwards define your product to be feature based instead of the default plug-in based selection.

Switch to the Contents tab and add
  • org.eclipse.e4.rcp
  • org.eclipse.platform
  • com.codeandme.eclipse.minimal.feature
Hit Add Required to resolve additionally needed features. When you add features manually they usually are referenced by name and version number. You should get rid of the version information by selecting such features and removing the version string that can be found under Properties.... To depend on specific versions for your build use target definitions.

Save your product and test it by selecting Launch an Eclipse Application on the Overview tab. From there you can also use the export wizard to package your application.

If you do get an error 'No application id has been found' then you quite likely forgot to add your plug-in to your feature.

Sunday, November 25, 2012

Using your gerrit server in eclipse

So you followed the tutorial on how to setup a gerrit server and want to use it with eclipse?  Keep on reading...

We are using eclipse 4.2 here, not sure if all the required egit features are already present in previous versions.

Preparations

Gerrit is configured through its web frontend. As mentioned the first user to login will gain admin rights, so login with your administrator user. On the Projects tab you may Create a New Project.


I'm not going into details on project options here. The default settings should be sufficient for us to commit and review code.

Step 1: Configure a user

Now restart your browser, clear your passwords or do whatever necessary to get the authentication dialog back when accessing gerrit (in Firefox you might start a private session). Login with a user account, in the tutorial here I will use user1. You need to use the password you provided in the htaccess file on your server.

On the righthand side there is a Settings link. Fill in your Contact Information: provide your real name and a valid email address. You will receive a confirmation mail to verify your account.

Next add your SSH Public Key. If you don't have one yet you may create one in eclipse preferences.


Make sure you hit Save Private Key... otherwise your created key would not be saved permanently.

Back to gerrit: as a last step create a HTTP Password.

Every user of your system should follow these steps before he actually may use gerrit.

Step 2: Add gerrit repository in eclipse

The Git Repositories view allows to Clone a Git repository. The wizard will first ask for the repository type, obviously we want Gerrit. On the next page click on Add... and add your gerrit repository location.


The Server address is the same as you just entered in your browser before. Use your User ID and the htaccess file Password. Validate your settings before you hit Finish.

You should see your playground repository now. Gerrit allows to access this repository in multiple ways. We will use http (authorized) access in our example, but anonymous http and ssh will work, too.


Select the master branch and continue the wizard accepting the default settings.

Step 3: Committing & reviewing

Now you can start to share projects the same way as you would do it for git. On your first commit you might change a difference: There is a change ID provided for you:


On your first push you will be asked for credentials. The user name is still user1, but the password is the one you created on the gerrit webpage in your user settings. It is NOT the password from the htaccess file. Make sure you don't mix up those two.

You can see your change in gerrit by now:


Step 4: Mylyn integation

The gerrit web frontend is nice, but fun starts when you use mylyn integration. Unfortunately gerrit 2.5 support seems to be broken in Juno SR1 (see bug 393807). You still can use mylyn if you update your installation using the weeky builds update site: http://download.eclipse.org/mylyn/snapshots/weekly

I updated Mylyn Task List, Mylyn Reviews Connector: Gerrit and Mylyn Versions Connector: Git. Possibly updating the gerrit connector might be sufficient.

Open the Task List view to see pending review tasks. If you don't have defined queries yet you should find an "Unmatched" folder containing an entry for your first import. Now you can directly edit your reviews within eclipse.




Setup a dedicated gerrit server

Update:
This tutorial is somewhat outdated. Maybe relevant setup parts are still valid, however nowadays you should find plenty of ready-to-use docker images out there providing a working gerrit setup. I leave the tutorial online, just remember that it was written in 2012.


When you followed my previous post on git you might already have expected this one to come. Today we will setup a gerrit code review server.

A typical setup would use the integrated jetty server, an H2 database and openID for authentication. As this might not always be the first choice of components we will use a different approach here:

As a servlet container we will use tomcat running behind an apache webserver. We will use apache to deal with authentication so that we don't need openID for that. Finally we will replace H2 with mysql. To make this work we need to tweak a bit as especially mysql does not seem to be the first choice for gerrit programmers. For all you guys out there that deal with apache/tomcat/reverse proxy setups: please feel free to help to improve the setup as I (as a novice) struggled to make this work. To all the others: be warned that it might have some security issues.

That said, we are ready to start. Like the git server setup we will implement this on gentoo, my linux distribution of choice.

To actually work with gerrit you need access to a working SMTP server. I guess you'll figure out how to do this on your own.

Step 1: Install components

When building apache we need some extra modules named proxy, proxy_ajp and proxy_http. To enable these modules we need to add them to the APACHE2_MODULES configuration. So edit /etc/make.conf and add the line:
APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias proxy proxy_ajp proxy_http"
I left the other modules as defined by the default settings. If you are going to experiment with these you do this at your own risk.

Now install all needed components:
emerge apache tomcat mysql git commons-dbcp commons-pool
After starting apache and tomcat you should be able to see the welcome pages when pointing your browser to http://localhost and http://locahost:8080.
/etc/init.d/apache2 start
/etc/init.d/tomcat-7 start
Step 2: Configure mySQL

If you installed mysql for the first time, you need to configure it before starting it.
emerge --config =dev-db/mysql-5.1.62-r1
/etc/init.d/mysql start
mysql -u root -p
On the mysql shell enter following commands
CREATE USER 'gerrit'@'localhost' IDENTIFIED BY 'gentoo';
CREATE DATABASE gerritdb;
ALTER DATABASE gerritdb charset=latin1;
GRANT ALL ON gerritdb.* TO 'gerrit'@'localhost';
FLUSH PRIVILEGES;
exit;
We created a database gerritdb and a user gerrit using the password gentoo.

Step 3: Configure tomcat

We need a tomcat user to access the manager console. So edit /etc/tomcat-7/tomcat-users.xml
<tomcat-users>
    <role rolename="manager-gui"/>
    <user username="root" password="gentoo" roles="manager-gui"/>
</tomcat-users>
Additionally we need a database connector for the mysql database. Open /etc/tomcat-7/context.xml and add:
<Context>
    <!-- DB connector for gerrit -->
    <Resource name="jdbc/ReviewDb" auth="Container"
          type="javax.sql.DataSource" 
          maxActive="100" maxIdle="30" maxWait="10000"
          username="gerrit" password="gentoo"
          driverClassName="com.mysql.jdbc.Driver"
          factory="org.apache.commons.dbcp.BasicDataSourceFactory"
          url="jdbc:mysql://localhost:3306/gerritdb"/>
</Context>
Now we need to tell tomcat where to find commons-dbcp.jar and commons-pool.jar.
ln -s /usr/share/commons-dbcp/lib/commons-dbcp.jar /usr/share/tomcat-7/lib/
ln -s /usr/share/commons-pool/lib/commons-pool.jar /usr/share/tomcat-7/lib/
Restart tomcat to use the new settings:
/etc/init.d/tomcat-7 restart
Step 4: Configure apache

Apache will act as reverse proxy, which allows us to hide tomcat behind the apache webserver. Apache acts as a proxy between the client and the tomcat application. As we normally do not want to relay all traffic to tomcat we will create our own named vhost

First enable proxy support by changing APACHE2_OPTS in /etc/conf.d/apache2
APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PROXY"
Add a new file /etc/apache2/vhosts.d/01_gerrit_proxy.conf to host our virtual server:
<IfDefine DEFAULT_VHOST>
Listen 81
NameVirtualHost *:81

<VirtualHost *:81>
        ServerName localhost
        Include /etc/apache2/vhosts.d/default_vhost.include

        <IfModule mpm_peruser_module>
                ServerEnvironment apache apache
        </IfModule>

        ProxyRequests Off
        <proxy *="">
                Order deny,allow
                Allow from all
        </proxy>

        ProxyPass /gerrit/ http://localhost:8080/gerrit/
        ProxyPassReverse /gerrit/ http://localhost:8080/gerrit/

        <Location /gerrit/login/>
                AuthType Basic
                AuthName "Gerrit Code Review"
                AuthBasicProvider file
                AuthUserFile /var/www/localhost/passwords
                Require valid-user
        </Location>
</VirtualHost>
</IfDefine>
We defined to use a ht password file to access gerrit, so we need to create it first.
htpasswd -c /var/www/localhost/passwords administrator # create first entry
htpasswd/var/www/localhost/passwords user1 # create additional entry
Remember that every gerrit user needs its own entry here.

Finally restart apache
/etc/init.d/apache2 restart
Step 5: Setup gerrit

Next we need to download gerrit. At the time of writing gerrit-full-2.5.war was the latest release. Rename your downloaded file to gerrit.war. Open up the local tomcat manager application http://localhost:8080/manager and scroll down to WAR file to deploy. Select your downloaded gerrit.war and deploy it.

Gerrit cannot install when binlogs are enabled in mysql. So lets disable them during setup. Modify /etc/mysql/my.cnf and comment out log-bin:
#log-bin
afterwards restart mysql
/etc/init.d/mysql restart
Now we are ready to setup gerrit:
java -jar /var/lib/tomcat-7/webapps/gerrit.war init -d /opt/gerrit
The setup script will ask some questions, see my installation log below for answers:
*** Gerrit Code Review 2.5
*** 
Create '/opt/gerrit'           [Y/n]? 

*** Git Repositories
*** 
Location of Git repositories   [git]: /var/lib/git/repositories

*** SQL Database
*** 

Database server type           [H2/?]: mysql

Gerrit Code Review is not shipped with MySQL Connector/J 5.1.10
**  This library is required for your configuration. **
Download and install it now [Y/n]? 
Downloading http://repo2.maven.org/maven2/mysql/mysql-connector-java/5.1.10/mysql-connector-java-5.1.10.jar ... OK
Checksum mysql-connector-java-5.1.10.jar OK
Server hostname                [localhost]: 
Server port                    [(MYSQL default)]: 
Database name                  [reviewdb]: gerritdb
Database username              [root]: gerrit
gerrit's password              : gentoo
              confirm password : gentoo

*** User Authentication
*** 

Authentication method          [OPENID/?]: http
Get username from custom HTTP header [y/N]? 
SSO logout URL                 : 

*** Email Delivery
*** 

SMTP server hostname           [localhost]: <enter your smtp server>
SMTP server port               [(default)]: <...>
SMTP encryption                [NONE/?]: <...>
SMTP username                  : <...>

*** Container Process
*** 

Run as                         [root]: 
Java runtime                   [/opt/icedtea-bin-6.1.11.5/jre]: 
Copy gerrit.war to /opt/gerrit/bin/gerrit.war [Y/n]? 
Copying gerrit.war to /opt/gerrit/bin/gerrit.war

*** SSH Daemon
*** 

Listen on address              [*]: 
Listen on port                 [29418]: 

Gerrit Code Review is not shipped with Bouncy Castle Crypto v144
  If available, Gerrit can take advantage of features
  in the library, but will also function without it.
Download and install it now [Y/n]? 
Downloading http://www.bouncycastle.org/download/bcprov-jdk16-144.jar ... OK
log-bin
Checksum bcprov-jdk16-144.jar OK
Generating SSH host key ... rsa... dsa... done

*** HTTP Daemon
*** 

Behind reverse proxy           [y/N]? 
Use SSL (https://)             [y/N]? 
Listen on address              [*]: 
Listen on port                 [8080]: 8081

*** Plugins
*** 

Prompt to install core plugins [y/N]? 

Initialized /opt/gerrit
Executing /opt/gerrit/bin/gerrit.sh start
Starting Gerrit Code Review:  * WARNING: -c/--chuid is deprecated and will be removed in the future, please use -u/--user instead
log-bin
 * WARNING: -a/--startas is deprecated and will be removed in the future, please use -x/--exec or -n/--name instead
OK
Waiting for server to start ... OK
Opening browser ...
Please open a browser and go to http://vserver:8081/#/admin/projects/
Your settings are stored in /opt/gerrit/etc/gerrit.config. See the configuration docs for details. In case you want to change some of them simply run the setup again.

SMTP data is up to you to provide. You'll need it as gerrit depends on sending emails.

You may revert the binlogs settings of mysql now if you wish.

As the setup automatically starts gerrit (and ssh, jetty) we have to stop it:
/opt/gerrit/bin/gerrit.sh stop
Some final settings to make tomcat happy:
chown tomcat:tomcat -R /opt/gerrit 
chown tomcat:tomcat -R /var/lib/git/repositories 
cp /opt/gerrit/lib/bcprov-jdk16-144.jar /usr/share/tomcat-7/lib/
cp /opt/gerrit/lib/mysql-connector-java-5.1.10.jar /usr/share/tomcat-7/lib/
Step 6: Fine tuning of mysql

Gerrit is a little picky about text encodings in mysql. Typically all fields should be encoded latin1_bin, but for the web frontend to work some of them need to be changed to utf8_bin. This seems to be a long lasting struggle between gerrit and mysql. To fix those issues you need to convert some text fields to utf8. I tried to locate them all by playing around with the web interface. You might change them by running following sql statements in mysql:
USE gerritdb;
ALTER TABLE `accounts` CHANGE `full_name` `full_name` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL , CHANGE `preferred_email` `preferred_email` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL;
ALTER TABLE `account_external_ids` CHANGE `email_address` `email_address` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_bin NULL DEFAULT NULL; 

If you encounter application errors, please consult /var/logs/tomcat-7/localhost.<current_date>.log and look for lines like this one:
com.google.gwtorm.server.OrmException: fetch failure on account_external_ids
It tells you that table acount_external_ids needs some tweaking. Here the field email_address needs conversion to utf8-bin.

As sql is not a language that I 'speak' fluently I used phpmyadmin to change the database layout.

Please let me know of any additional changes so that I can update this post.

Step 7: Restart servers

Now all that's left to do is restart your servers and add them to your default runlevel.
/etc/init.d/mysql restart
/etc/init.d/apache2 restart
/etc/init.d/tomcat-7 restart

rc-update add mysql default
rc-update add apache2 default
rc-update add tomcat-7 default
Point your browser to http://localhost:81/gerrit/ to see your server in action.

Take care to first login with your designated administrator user as the very first user to login to gerrit will automatically gain admin rights.

Final thoughts

The setup as described is working, but I guess the apache/tomcat setup might need some tweaking. I'd be glad to get some comments from more experienced users on this. Eg. after entering login data the user is redirected to an error site. When entering the base address for a second time you end up in the gerrit application.

Setting gerrit.canonicalWebUrl in the gerrit config file is highly recommended as gerrit wants to provide some links to itself. With the reverse proxy such links will only work when a correct host is specified here (like http://hostname:81/gerrit).