Monday, December 29, 2014

improvements of jruby java integration with jruby-9.0.0.0

detection of jruby's classloader


this simple example
ScriptingContainer jruby = new ScriptingContainer();
jruby.runScriptlet( "p 'hello'" );
it just works with jruby-9k, but  only works in jruby-1.7 when the jruby classes are loaded via the Thread.currentThread().getContextClassLoader(). a work around for jruby-1.7.x is to set the right classloader.
ScriptingContainer jruby = new ScriptingContainer();
// jruby-1.7 only
jruby.setClassLoader( jruby.getClass().getClassLoader() );
jruby.runScriptlet( "p 'hello'" );
for example any OSGi framework will load jruby in its bundle classloader which is not a context classloader.

default gems


jruby comes with a couple of gems bundled as default gems. when running jruby within a java application those default gems usually come from within a jar file, but the rubygems code need directory globs to "activate" those default gems as gems, otherwise those gems are just a library which is already on the LOAD_PATH and you can not upgrade or downgrade these gems.

in jruby-9k
ScriptingContainer jruby = new ScriptingContainer();
jruby.runScriptlet( "require 'openssl';p Gem.loaded_specs" );
this will always printout that jruby-openssl is loaded.

in jruby-1.7 the result depends on which path is set to jruby-home. jruby knows various uri-like protocols for paths. possible jruby-homes pathes:


  • /some/path/to/jruby/home
  • jruby-complete.jar!/MEATA-INF/jruby.home
  • file:jruby-complete.jar!/MEATA-INF/jruby.home
  • jar:file:jruby-complete.jar!/MEATA-INF/jruby.home
  • classpath:/MEATA-INF/jruby.home
  • uri:classloader:/MEATA-INF/jruby.home


  • with the first 4 the default gems are working as they should. with classpath:/ they will not work with uri:classloader:/ they will work with jruby-1.7.17+.

    classpath: is used when jruby cannot find the jar-file which contains jruby-stdlib (its bundled default gems). this is in case of exotic j2ee environments or OSGi frameworks. if you have a war-file and do not know on which servlet container it will run, you need to set jruby.home explicitly to something which works:
    ScriptingContainer jruby = new ScriptingContainer();
    // jruby-1.7.17+ only
    jruby.setJRubyHome( "uri:classloader://META-INF/jruby.home" );
    // jruby-1.7 only
    jruby.setClassLoader( jruby.getClass().getClassLoader() );
    jruby.runScriptlet( "require 'openssl';p Gem.loaded_specs" );
    
    

    embedded gems inside a jar-file


    the same problem with the default gems are facing embedded gems as well: they only work when jruby can perform directory globs "inside" this jar-file.

    just adding the /specifications and the /gems directory to the jar-file only works if the framework allows jruby to "find" the jar, then directory globs are possible. the uri:classloader:// protocol works differently, it looks for a file uri:classloader://specifications/.jrubydir or uri:classloader://gems/.jrubydir. these files contain the directory entries - one entry per line.

    before packing the somewhere/specifications and the somewhere/gems you need to run something like
    require 'jruby/commands'
    JRuby::Commands.generate_directory_info( 'somewhere/specifications' )
    JRuby::Commands.generate_directory_info( 'somewhere/gems )
    
    a jar-file with such embedded gems will work with jruby-9k but with jruby-1.7.17+ one more step is needed:
    ScriptingContainer jruby = new ScriptingContainer();
    // jruby-1.7.17+ only
    jruby.setJRubyHome( "uri:classloader://META-INF/jruby.home" );
    // jruby-1.7 only
    jruby.setClassLoader( jruby.getClass().getClassLoader() );
    // jruby-1.7.17+ only
    jruby.runScriptlet( "require 'rubygems/defaults/jruby'; Gem::Specification.add_dir 'uri:classloader://specifications'" );
    
    

    embedding ruby scripts


    in some cases embedding ruby scripts need to perform directory globs as well - like loading all initializers inside the /initializers directory.

    you need to use the same generate directory info command as with the embedded gems.
    require 'jruby/commands'
    JRuby::Commands.generate_directory_info( 'somewhere/initializers )
    
    and now you need to tell jruby how to find the embedded ruby scripts by adding uri:classloader:// to your $LOAD_PATH:
    ScriptingContainer jruby = new ScriptingContainer();
    // jruby-1.7.17+ only
    jruby.setJRubyHome( "uri:classloader://META-INF/jruby.home" );
    // jruby-1.7 only
    jruby.setClassLoader( jruby.getClass().getClassLoader() );
    // jruby-1.7.17+ only
    jruby.runScriptlet( $LOAD_PATH.unshift 'uri:classloader://'" );
    // jruby-1.7.17+ only
    jruby.runScriptlet( "require 'rubygems/defaults/jruby'; Gem::Specification.add_dir 'uri:classloader://specifications'" );

    default LOAD_PATH for java applications


    in jruby-1.7 there is some "magic" going on when creating a ScriptingContainer: it adds all the jars from the "classpath" of the running java application to the ruby $LOAD_PATH. which can be quite a list of jars which may or may not be related to your ruby application.

    I saw those "extra" $LOAD_PATH entries to slow down the startup of the ruby application from 8 seconds to 20 seconds. tidy up could make sense especially if you do not know where your library is going to be used.

    usually after creating the
    ScriptingContainer jruby = new ScriptingContainer();
    // jruby-1.7.17+ only
    jruby.setJRubyHome( "uri:classloader://META-INF/jruby.home" );
    jruby.setLoadPaths( Arrays.asList( "uri:classloader://" ) );
    // jruby-1.7 only
    jruby.setClassLoader( jruby.getClass().getClassLoader() );
    // jruby-1.7.17+ only
    jruby.runScriptlet( "require 'rubygems/defaults/jruby'; Gem::Specification.add_dir 'uri:classloader://specifications'" );
    this also finds the embedded ruby scripts on the $LOAD_PATH.

    do not use the user installed gems from ~/.gem/jruby/


    the gems installed with gem install --user-install my.gem were installed in $HOME/.gem/jruby and can interfere with your embedded gems. rubygems gives preference to those user-installed over the embeded gems. Actually it is jruby which adds those embedded gems to the Gem::Specification.dirs which is the set of directories where rubygems searches for installed gems.

    jruby-9k uses uri:classloader://specifications at the first entry of Gem::Specification.dirs which gives preference to the embedded gems but still might pick a gem from the user-installed gems. so best to tidy up the Gem::Specification.dirs

    ScriptingContainer jruby = new ScriptingContainer();
    // jruby-1.7.17+ only
    jruby.setJRubyHome( "uri:classloader://META-INF/jruby.home" );
    // jruby-1.7 + jruby-9k
    jruby.setLoadPaths( Arrays.asList( "uri:classloader://" ) );
    // jruby-1.7 only
    jruby.setClassLoader( jruby.getClass().getClassLoader() );
    // setup the isolated GEM_PATH, i.e. without $HOME/.gem/**
    jruby.runScriptlet("require 'rubygems/defaults/jruby';"
                     + "Gem::Specification.reset;"
                     + "Gem::Specification.add_dir 'uri:classloader://META-INF/jruby.home/lib/ruby/gems/shared';"
                     + "Gem::Specification.add_dir 'uri:classloader:';");
    

    the downside is that it also ignores any GEM_PATH setting which is there !

    IsolatedScriptingContainer


    the IsolatedScriptingContainer does all the setup already which allows to create jar library using jruby, embedded gems and embedded ruby scripts which just works on all possible java environments:
    ScriptingContainer jruby = new IsolatedScriptingContainer();
    nothing else to do.

    OSGi support


    the IsolatedScriptingContainer to add a classloader uri to the $LOAD_PATH and Gem::Specification.dirs. since inside OSGi you are more familiar how to get the osgi-bundle which contains your ruby library you can add the bundle to the $LOAD_PATH and Gem::Specification.dirs as well.
    IsolatedScriptingContainer jruby = new IsolatedScriptingContainer();
    jruby.addBundleToLoadPath( FrameworkUtil.getBundle( SomeClassFromTheBundle.class ) );
    jruby.addBundleToGemPath( FrameworkUtil.getBundle( SomeClassFromTheBundle.class ) );
    
    or the classloader variant (which is not bound to OSGi):
    IsolatedScriptingContainer jruby = new IsolatedScriptingContainer();
    jruby.addLoadPath( SomeClassFromTheBundle.class.getClassLoader() );
    jruby.addGemPath( SomeClassFromTheBundle.class.getClassLoader() );
    important here is the directory info files to get them working for OSGi.

    pitfall


    there is still a pitfall left where jruby treats Thread.currentThread().getContextClassLoader() inconsistently.
    class M; end
    m = M.new
    jm = JRuby.become_java! m
    p jm.java_class == Thread.current_thread.context_class_loader.load_class( jm.name )
    
    this **ONLY** works with jruby command line execution !!!!

    so any ruby code which takes advantage of this special treatment of the context classloader in the command line case will FAIL when using it with embedded jruby !!!

    next


    enjoy and give feedback !

    Wednesday, January 15, 2014

    rubygems with java extensions

    this blog tries to show ways how to create gems which have a jar file as part of their gem. sometimes that is called gem with java-extension. but those java extentions have are vendored jar files which is very different from rubygems native extensions.

    the tool I use here is the ruby-maven gem which is mainly a proper maven-3.1.x  as a gem and which comes with some ruby bin stub to start it and a small set of rake tasks. it also adds a ruby pom DSL to maven which is part of github.com/tesla/tesla-polyglot. the rake tasks along with some ruby DSL is the topic of this blog post.

    for all examples from below you can find a working version in github.com/mkristian/ruby-maven-demo.

    simple gem

    a minimal gemspec looks like this
    Gem::Specification.new do |s|
      s.name = 'simple'
      s.version = "0"
      s.author = 'sample person'
      s.email = [ 'mail@example.com' ]
      s.summary = 'simple gem'
      s.description = 'simple gem with empty jar'
    end
    assume our ruby files are in ./lib and the java files in ./src/main/java all you need to compile the java sources and add a jar file in lib is Rakefile
    require 'maven/ruby/tasks'
    now with
    $ rake build
    we compile the java sources and create lib/simple.jar and pack the gem.

    gem with jar dependencies

    usually you need some jar dependencies for your java code to work. Gem::Specification does not allow you to declare jar dependencies but it has a requirements list which is a free text to describe requirements for that gem. ruby-maven does 'use' this to declare jar dependencies. the notation used for this is the same used by github.com/mkristian/jbundler. so all we need to change is the gemspec which now looks as such simple.gemspec:
    Gem::Specification.new do |s|
      s.name = 'simple'
      s.version = "0"
      s.author = 'sample person'
      s.email = [ 'mail@example.com' ]
      s.summary = 'gem with jar'
      s.description = 'gem with empty jar and jar dependencies'
      s.requirements << "jar org.bouncycastle:bcpkix-jdk15on, 1.49"
      s.requirements << "jar org.bouncycastle:bcprov-jdk15on, 1.49"
    end
    nothing else is needed and
    $ rake build
    the will include them into the classpath for compilation. unfortunately rubygems has no way (yet) to provide those jar during runtime. jbundler does include them !

    gem with vendored jar dependencies

    now we need a way to tell the ruby-maven to include those jar into the gem. for this we need another configuration file Mavenfile:
    gemspec :include_jars => true
    that looks similiar to a Gemfile. now
    $ rake build
    will put the dependent jar files into the ./lib directory of the gem. with that the gem is self-contained and can be used is the same manner as all other gems from rubygems.org.

    gem depending on jars from other gems

    having those jar dependencies declared in the gemspec allows to use those jar dependencies to compile your java code. let's have a look at a gem which depends on the simple gem from above inherited.gemspec:
    Gem::Specification.new do |s|
      s.name = 'simple'
      s.version = "0"
      s.author = 'sample person'
      s.email = [ 'mail@example.com' ]
      s.summary = 'gem depending on deps of other gems'
      s.description = 'gem depending on deps of other gems'
      s.add_runtime_dependency 'simple', '=0'
    end
    now your java code can use the bouncy-castle jars from the simple.gem.

    customizations

    all customizations go into the Mavenfile like those few example below. the Mavenfile allows you to use the full pom DSL from maven, i.e. you can use any plugin you need for your jar file, define a parent pom, running java unit tests, etc. you also can tell ruby-maven to dump a pom.xml which again can be used in multi module maven setup.

    the rake tasks also have a few more tasks
    $ rake -T
    rake build    # Build gem into the pkg directory
    rake clean    # Clean up the build directory
    rake compile  # Compile any java source configured - default java files are in src/main/java
    rake jar      # Package jar-file with the compiled classes - default jar-file lib/{name}.jar
    rake junit    # Run the java unit tests from src/test/java directory
    rake maven    # Setup Maven instance
    rake push     # Push gem to rubygems.org
    there is plenty of space to customize the rake side of things as well.

    more

    the current situation with vendored jars is probably still doable but there is no control over it and conflicts with those jars will arise sooner or later.

    there is no way to find out which gem adds which jar to the jruby classloader and when and which version of the jar. using current jruby (1.7.x) and the scripting container you can add create all kind of classloader issues.

    once the gem do declare their vendored jar as dependencies within the gemspec then at least it is possible to have a look the dependency graph/tree for both gems as well for jars.

    enjoy !!!

    Thursday, September 26, 2013

    use war files made by warbler with maven

    setup

    you have (j)ruby project and you are using warbler to create a war file. now you want to use that war file as maven artifact and/or deploy it on a maven repository (nexus, etc). this post is about how to link these two worlds.

    people how use warbler are more at home on the ruby side of things so I am choosing ruby tools for maven to show how this is done. the respective pom.xml are linked at the respective places. tesla is polyglot maven which supports a ruby DSL. the ruby-maven gem is basically the ruby DSL from tesla packed into gem.

    simple approach

    pom.xml for this.
    this will create a war artifact and can be installed or deployed as with any other maven project.

    note

    you will need to make sure that all gems are installed and that warbler is part of the search path of the OS when executing maven. this makes it a more manual approach since all these steps need to be done step by step.

    maven approach

    now you want to add the project be part of multi module maven project. or as a standalone maven project which is 'selfcontained', i.e. git clone myproject followed by cd myproject and finally maven deploy it to a repository.

    so the solution has one big requirements
    • all gems must come from rubygems.org, i.e. no gems with path or git declaration inside the Gemfile are possible
    pom.xml for this.
    this approach has one major disadvantage: you need to keep the list of gems from Gemfile in sync with pom.xml !! but if you use tesla or ruby-maven then the ruby DSL just imports the gem list from the Gemfile into the pom.

    remarks on tesla and ruby-maven

    tesla is a drop in replacement of maven when you deal with pom.xml only. the other DSLs of tesla do work already OK but there are and will be certain plugins which work only with the XML-DSL (pom.xml).

    ruby-maven can run with MRI (needs java installed on the OS) or JRuby. both still have their own individual problems. the jruby version has classloading problems when using the jruby-maven-plugins and the MRI version does not work with multi module projects and parents projects using the ruby DSL.

    altogether the DSL might have a few limitation though is fairly complete.

    my personal approach right now is to use tesla/ruby-maven to 'translate' the Mavenfile into a pom.xml and include it into version control system. with this anyone with only maven installed can use the project as is. when I work on the Mavenfile or Gemfile I use tesla/ruby-maven to test it which produces the generated pom.xml (you can configure that with property inside the Mavenfile).

    enjoy !

    Monday, February 18, 2013

    getting up and running with OAuth using signet gem (ruby)

    since the readme of github.com/google/signet has a straight forward example of how to use OAuth version 1, it misses the OAuth version 2 completely.being new to OAuth I did havesome problems to get the example from the readme working at first. so here is the slightly modified version of the example

    now I was able to use the same for twitter after creating an application on dev.twtter.com

    now I wanted to use OAuth2 with my google account so first I created a clientId on code.google.com/apis/console. with that it was possible to get data from my google acount

    maybe these scripts might be of any help . . .

    Thursday, March 8, 2012

    simple portknocking on debian/ubuntu

    since I needed some time to get it all nicely working here how I did to hide the ssh port.

    knockd setup

    install knockd package
    apt-get install knockd
    added following config /etc/knockd.conf
    [options]
      UseSyslog
    [openCloseSSH]
      sequence      = 7000,9000,8000
      seq_timeout   = 15
      tcpflags      = syn
      start_command = /sbin/iptables -I INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
      cmd_timeout   = 15
      stop_command  = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
    to get the daemon running on reboot, edit /etc/default/knockd and change to
    START_KNOCKD=1
    now you can start the daemon
    service knockd start
    basically I followed ubuntu help PortKnocking but did use -I INPUT to insert the rule to open the ssh port at the beginning of the chain.

    iptables setup

    so the firewall shall keep existing ports open and reject any connection to the ssh port. the knoch daemon will open the port for the knocking IP and close it after 15s (see config above).
    iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
    iptables -A INPUT -p tcp --dport ssh -j REJECT
    now to make we want this after reboot as well (see IptablesHowTo):
    iptables-save > /etc/iptables.rules
    create file /etc/network/if-pre-up.d/iptablesload
    #!/bin/sh
    iptables-restore < /etc/iptables.rules
    exit 0
    change the permissions
    chmod +x /etc/network/if-pre-up.d/iptablesload
    now with nmap you will see no trace of ssh:
    $ nmap 192.168.56.101
    Nmap scan report for example.net (192.168.56.101)
    Host is up (0.0088s latency).
    Not shown: 998 closed ports
    PORT    STATE SERVICE
    80/tcp  open  http
    
    Nmap done: 1 IP address (1 host up) scanned in 0.18 seconds
    hope that help !

    simple portknocking on debian/ubuntu

    since I needed some time to get it all nicely working here how I did to hide the ssh port.

    knockd setup

    install knockd package
    sudo apt-get install knockd
    added following config /etc/knockd.conf
    [options]
      UseSyslog
    [openCloseSSH]
      sequence      = 7000,9000,8000
      seq_timeout   = 15
      tcpflags      = syn
      start_command = /sbin/iptables -I INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
      cmd_timeout   = 15
      stop_command  = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
    to get the daemon running on reboot, edit /etc/default/knockd and change to
    START_KNOCKD=1
    basically I followed ubuntu help PortKnocking but did use -I INPUT to insert the rule to open the ssh port at the beginning of the chain.

    iptables setup

    so the firewall shall keep existing ports open and reject any connection to the ssh port. the knoch daemon will open the port for the knocking IP and close it after 15s (see config above).
    iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
    iptables -A INPUT -p tcp --dport ssh -j REJECT
    now to make we want this after reboot as well (see IptablesHowTo):
    iptables-save > /etc/iptables.rules
    create file /etc/network/if-pre-up.d/iptablesload
    #!/bin/sh
    iptables-restore < /etc/iptables.rules
    exit 0
    change the permissions
    chmod +x /etc/network/if-pre-up.d/iptablesload
    now with nmap you will see no trace of ssh:
    $ nmap 192.168.56.101
    Nmap scan report for example.net (192.168.56.101)
    Host is up (0.0088s latency).
    Not shown: 998 closed ports
    PORT    STATE SERVICE
    80/tcp  open  http
    
    Nmap done: 1 IP address (1 host up) scanned in 0.18 seconds
    hope that help !

    Saturday, February 18, 2012

    simple guard for rails authorization

    due to some recent dicussions about authorization I feel obliged to show how to guard your controller actions with a simple Hash per controller and a single before fitler. the whole will be orthogonal to the application, i.e. no coding required beside setting up this Hash.

    the only assumption is that the permissions are organized with user/group model or user/role model, similar what you find on linux/unix systems or with PosixAccounts/PosixGroups on LDAP systems.

    in case you need a full fledged authroization framework with DSL and protecting the models instead of the controllers probably github.com/brayn/cancan

    setup the permissions per controller

    let's have a scaffolded account resource (rails g scaffold accounts name:string) and for each action for the acounts controller you configure a list of groups which are allowed to perform that action.

    PERMISSIONS = {
     :accounts => {
       :index => [:root, :supervisor, :accountant],
       :show =>[:root, :supervisor, :accountant],
       :new =>[:root, :accountant],
       :create =>[:root, :accountant],
       :update =>[:root, :accountant],
       :edit =>[:root, :accountant],
       :destroy =>[:root]
     }
    }

    to check the permissions you need some method like this

    def allowed?(action = param[:action], controller = params[:controller])
      perm = PERMISSIONS[controller.to_sym] || {}
      allowed = perm[action.to_sym]
      if allowed
        found = allowed & (current_user_groups || [])
        found.size > 0
      end
    end
    

    now the guard is almost in done. just add a before filter and a method which returns the an array of groups (here these are group-names) of the current-user.

    before_fitler :guard
    def guard
      if allowed?
        raise "permission denied"
      else
        raise "no permission for #{params[:action]}"
      end
    end
    def current_user_groups
      [:root] if current_user # change this to something real !!
    end
    

    the allowed? method from above can be put into the application_helper.rb to have it available in the view templates.

    def allowed?(action, controller = params[:controller])
      controller.allowed?(action, controller)
    end
    

    the controller defaults to the current controller but the action is compulsory for the view.

    add some default permissions

    with this allowed? method

    def allowed?(action = param[:action], controller = params[:controller])
      perm = PERMISSIONS[controller.to_sym] || {}
      allowed = perm[action.to_sym] || perm[:defaults]
      if allowed
        allowed & (current_user_groups || [])
      end
    end
    

    the config is reduced to

    PERMISSIONS = {
      :accounts => {
        :defaults =>[:root, :accountant],
        :index => [:root, :supervisor, :accountant],
        :show =>[:root, :supervisor, :accountant],
        :destroy =>[:root]
      }
    }
    

    add a superuser

    the superuser group is :root so we can change the allowed? method to

    def allowed?(action = param[:action], controller = params[:controller])
      perm = PERMISSIONS[controller.to_sym] || {}
      allowed = perm[action.to_sym] || perm[:defaults]
      if allowed
        found = (allowed | [:root]) & (current_user_groups || [])
        found.size > 0
      end
    end
    

    this reduces the PERMISSIONS setup to this

    PERMISSIONS = {
      :accounts_with_defaults_and_root => {
        :defaults =>[:accountant],
        :index => [:supervisor, :accountant],
        :show =>[:supervisor, :accountant]
      }
    }
    

    this is not perfect but for most cases sufficiently compact and readable.

    groups

    looking at the code then the groups do pop up at two places
    • the PERMISSIONS
    • the current_user_groups method
    with such a setup you can make the users and groups totally orthogonal to the application. for example they can be stored as PosixAccounts/PosixGroups in an LDAP system. the allowed? method uses only the controller and action name as parameter. the current_user_groups method can be generic depending on your user/group model and the PERMISSIONS could be a database table as well, allowing to configure permissions on the fly without changing the code !

    one advanced example

    now we assume a group can be associated with an organization (which is part of domain model).

    first we change the allowed? method to check the found allowed groups for agiven controller-action-pair if any if it is allowed to access the current organization. this check is done by a given block.

    def allowed?(action = param[:action], controller = params[:controller], &block)
      perm = PERMISSIONS[controller.to_sym] || {}
      allowed = perm[action.to_sym] || perm[:defaults]
      if allowed
        found = (allowed | [:root]) & (current_user_groups || [])
        if block
          permitted = found.detect do |group|
            block.call(group)
          end
          permitted.size > 0
        else
          found.size > 0
        end
      end
    end
    

    now we add some extra methods to the application_controller.rb

    def guard_organization
      guard do
        GroupsOrganziationsUser.first(:conditions => ["group_name = ? or org_id = ? and user_id = ?", group, current_organization.id, current_user.id]).size > 0
      end
    end
    
    def current_organization
      @org ||= Organization.find(params[:org_id])
    end
    

    to change the account_controller.rb to use the restricted group the before filters need to reflect this

    skip_before_filter :guard
    before_filter :guard_organization
    

    more

    there is also a gem available which wraps this approach and adds some yaml configuration for the permissions: ixtlan-guard