Sunday, September 25, 2011

JRuby and RubyGems and JavaClassloader

jruby and classloading

rubygems from MRI point if view has nothing do with java classloader. but for jruby you need to deal with this from ruby side. jruby comes with a neat feature(s) that you can 'require "myjar.jar"' and use the java classes more or less in ruby manner. that jar just gets added to the jruby classloader during runtime.

now as a gem developer I might to use certain java libraries and a common way to do it, is to put the jar into the lib directory and just require it when needed. as java developer I first needed to think about: duplicated jar with different version is a source for classloader problems. and if any gem just can add any time a new jar to classloader, there is basically no control over what is going on, and when !

duplicated libraries in the LOAD_PATH was also a common source of annoying errors with rubygems in the time before bundler. bundler works great but for jar dependencies there is no(?) way to do things right now.

example

I set up a a gem and jar which is used inside a java-servlet with embedded jruby. so it is a java application using ruby and rubygems: http://github.com/mkristian/examples/tree/webapp_with_nokogiri_without_maven

the web archive has the following dependencies:
 com.example:web:war:1.0-SNAPSHOT
 +- com.example:app:jar:1.0-SNAPSHOT:compile
 |  +- xerces:xercesImpl:jar:2.10.0:compile
 |  |  \- xml-apis:xml-apis:jar:1.4.01:compile
 |  +- rubygems:gem:gem:0.0.0:compile
 |  |  \- rubygems:nokogiri:gem:1.5.0:compile
 |  \- org.jruby:jruby-complete:jar:1.6.4:compile
 +- javax.servlet:servlet-api:jar:2.5:provided
 +- junit:junit:jar:3.8.1:test
 \- rubygems:jruby-openssl:gem:0.7.4:compile
    \- rubygems:bouncy-castle-java:gem:1.5.0146.1:compile
this info got extracted by the package descriptors (gemspec) and there there are lots of hidden jars:
bouncycastle:bcmail-jdk15:jar:140
bouncycastle:bcprov-jdk15:jar:140
joda-time:joda-time:jar:1.6
org.yaml:snakeyaml:jar:1.8
msv:isorelax:jar:20050913
thaiopensource:jing:jar:20030619
nekohtml:nekodtd:jar:0.1.11
net.sourceforge.nekohtml:nekohtml:jar:1.9.15
xerces:xercesImpl:jar:2.9.1
the last line even shows a jar which is twice in the classloader with different versions xerces:xercesImpl:jar with 2.9.1 and 2.10.0. this is very similar to the pre-bunlder times.

'jar as a gem' approach

one approach was to wrap a jar into gem and use the gem as normal gem. if you install jruby-1.6.x you will get a "experimental" feature which allows you to install any maven artifact (mainly jar files) as gem. great idea BUT the moment you upgrade your rubygems installation on your system that rubygems maven support is gone. and it does not look like this will improve in the near future. so there is no way to include maven-gem dependencies in your gems and publish these gems on rubygems.org

there was/is the attempt to launch a gems repository with all the maven-gem from central maven repository. even if this goes online I do not see how to use it for your regular gems, unless any jruby installation makes sure you are using both rubygems.org and maven-gem repository as source for you gem installations. and it has some small short comings due to the difference rubygems and maven resolve dependencies.

even if one of those work it is only a partial solution of the problem. if you want to run your rails or sinatra application on java application server where you have a more complex classloader hierarchy you end up with classloader having "hidden" jars. the latest nokogiri release recommends to "comment out" some require statement in certain environments (if I remember the mail right).

bundler for jar files ?

one possible solution is to deal with the jar dependencies "outside" of ruby. a bit like have a tool setting up your environment variable CLASSPATH and you just execute jruby as usual but have all the jar you need ready to use. I will show a slightly different way of doing it: ruby-maven gem.

just define you jar dependencies in a Mavenfile like (taken from http://github.com/mkristian/jibernate/blob/master/Mavenfile):
jar 'org.hibernate:hibernate-core', '3.3.2.GA'
jar 'org.hibernate:hibernate-annotations', '3.4.0.GA'
jar 'org.hibernate:hibernate-tools', '3.2.4.GA'
jar 'javax.transaction:jta', '1.1'
jar 'javassist:javassist', '3.8.0.GA'
jar 'mysql:mysql-connector-java', '5.1.9'
jar 'postgresql:postgresql', '8.4-701.jdbc4'
jar 'com.h2database:h2', '1.2.138'
jar 'org.apache.derby:derby' ,'10.5.3.0_1'
jar 'org.hsqldb:hsqldb', '2.0.0'
this almost looks like a Gemfile for jars, but the version is compulsory.

now we need to get that info into the gemspec of gem, then ruby-maven can use both the usual part of gemspec to setup gems (via bundler) and the jar dependencies to setup the classpath. there is a little feature in gemspec which allows to add requirements. jar dependencies are a sort of requirements - just add in your gemspec:
s.requirements.put File.read('Mavenfile')
that basically adds human readable requirements to you gemspec and do it in a way that it can be user/parsed by ruby-maven.

back to example using ruby-maven

this example uses nokogiri-maven gem instead of nokogiri and jruby-openssl-maven instead of jruby-openssl and instead of jruby-complete it uses jruby-core and jruby-stdlib. the later pulls in all its  dependencies via maven and the "maven" gems do not come with the jar files and do not require them instead using the requirements declaration of the gemspec to specify the jar dependencies. with this you get following overview (rmvn dependency:tree)
 com.example:web:war:1.0-SNAPSHOT
 +- com.example:app:jar:1.0-SNAPSHOT:compile
 |  +- xerces:xercesImpl:jar:2.10.0:compile
 |  |  \- xml-apis:xml-apis:jar:1.4.01:compile
 |  +- rubygems:gem:gem:0.0.0:compile
 |  |  \- rubygems:nokogiri-maven:gem:1.5.0:compile
 |  |     +- msv:isorelax:jar:20050913:compile
 |  |     +- thaiopensource:jing:jar:20030619:compile
 |  |     +- nekohtml:nekodtd:jar:0.1.11:compile
 |  |     \- net.sourceforge.nekohtml:nekohtml:jar:1.9.15:compile
 |  +- org.jruby:jruby-core:jar:1.6.4:compile
 |  |  +- org.jruby.joni:joni:jar:1.1.6:compile
 |  |  +- org.jruby.extras:jnr-netdb:jar:1.0.3:compile
 |  |  +- org.jruby.ext.posix:jnr-posix:jar:1.1.8:compile
 |  |  +- org.jruby.extras:bytelist:jar:1.0.8:compile
 |  |  +- org.jruby.extras:constantine:jar:0.6:compile
 |  |  +- org.jruby.jcodings:jcodings:jar:1.0.5:compile
 |  |  +- org.jruby.extras:jffi:jar:1.0.8:compile
 |  |  +- org.jruby.extras:jaffl:jar:0.5.11:compile
 |  |  |  +- org.jruby.extras:jffi:jar:native:1.0.6:runtime
 |  |  |  +- asm:asm:jar:3.2:compile
 |  |  |  +- asm:asm-commons:jar:3.2:compile
 |  |  |  |  \- asm:asm-tree:jar:3.2:compile
 |  |  |  \- org.jruby.extras:jnr-x86asm:jar:1.0.0:compile
 |  |  +- org.yaml:snakeyaml:jar:1.8:compile
 |  |  +- jline:jline:jar:1.0:compile
 |  |  \- joda-time:joda-time:jar:1.6:compile
 |  \- org.jruby:jruby-stdlib:jar:1.6.4:compile
 +- javax.servlet:servlet-api:jar:2.5:provided
 +- junit:junit:jar:3.8.1:test
 \- rubygems:jruby-openssl-maven:gem:0.7.4:compile
    \- bouncycastle:bcmail-jdk15:jar:140:compile
       \- bouncycastle:bcprov-jdk15:jar:140:compile
now you have the control of your jars back. there is only one xercesImpl and you can "update" joda-time, snackyaml, bouncycastle, etc with backward compatible versions and use them instead inside your warfile.

since the whole thing was born within the jibernate project see the demo https://github.com/mkristian/dm-hibernate-adapter/tree/master/demo there (not sure if it is in a working state) and look the Gemfile which just adds dm-hibernate-adapter and you have all the jars in place if you use ruby-maven (rmvn rails server) to start the server:

 . . . .
 +- rubygems:dm-hibernate-adapter:gem:0.1pre:compile
 |  +- org.hibernate:hibernate-core:jar:3.3.2.GA:compile
 |  |  +- antlr:antlr:jar:2.7.6:compile
 |  |  +- commons-collections:commons-collections:jar:3.1:compile
 |  |  \- dom4j:dom4j:jar:1.6.1:compile
 |  |     \- xml-apis:xml-apis:jar:1.0.b2:compile
 |  +- org.hibernate:hibernate-annotations:jar:3.4.0.GA:compile
 |  |  +- org.hibernate:ejb3-persistence:jar:1.0.2.GA:compile
 |  |  \- org.hibernate:hibernate-commons-annotations:jar:3.1.0.GA:compile
 |  +- org.hibernate:hibernate-tools:jar:3.2.4.GA:compile
 |  |  +- org.beanshell:bsh:jar:2.0b4:compile
 |  |  +- freemarker:freemarker:jar:2.3.8:compile
 |  |  \- org.hibernate:jtidy:jar:r8-20060801:compile
 |  +- javax.transaction:jta:jar:1.1:compile
 |  +- javassist:javassist:jar:3.8.0.GA:compile
 |  +- mysql:mysql-connector-java:jar:5.1.9:compile
 |  +- postgresql:postgresql:jar:8.4-701.jdbc4:compile
 |  +- org.apache.derby:derby:jar:10.5.3.0_1:compile
 |  \- org.hsqldb:hsqldb:jar:2.0.0:compile
 . . . .

so basically you get "normal" workable gems with some extra info to setup a classloader/classpath. you can do that setup with ruby-maven or any other possible way. with this you regain control over your classloader, can exclude certain jars or replace them with a different version (maven allows all this), build war-files suitable for package them into an enterprise archive (ear-file), etc.

things to come

bundler offers the binstubs which is a convenient way of calling ruby commands in bundler context, the next release of ruby-maven will create binstubs which uses bundler for the gems and sets up a classpath for the jars and execute jruby in that context.

finally

I published the nokogiri-maven and jruby-openssl-maven gems to rubygems.org download and have a look. there are still more things to consider like sometimes the JRubyClassLoader is the ContextClassLoader some times not (standalone jruby vs. war-context), etc.

feedback is welcome.

1 comment:

  1. You have a broken link to https://github.com/mkristian/jibernate/tree/master/demo

    Not sure if it should be corrected to
    https://github.com/headius/jibernate
    or
    https://github.com/mkristian/dm-hibernate-adapter

    ReplyDelete