MC Scanning lib

Introduction to new MC Scanning lib.

 

From requirements  gathered here:  http://community.jboss.org/wiki/PapakiAnnotationScanningRequirements
I've  introduced a new MC Scanning sub-project:   http://anonsvn.jboss.org/repos/jbossas/projects/scanning/trunk/

 

The  main goal or idea behind this lib is very simple: unify all of JBossAS  component's scanning into a single pass scan.
So, instead of doing  the resources scanning for every component, we simply do it once, and  properly delegate the work to these component's handles.

 

Another  goal was to also enable usage of pre-indexed information, so the there  would actually be no need for the scanning itself.
e.g. one could  pre-index all of jar's annotations; e.g. its locations / class owners

 

Project  structure

 

* scanning-spi

 

It contains simple scanner  and metadata spi and initial helpers to help you extend / use simple  versoin of this lib.

 

* scanning-impl

 

It provides component  agnostic scanning api.
It also includes generic metadata  implementation and its usage.

 

* plugins

 

This module holds  custom component scanning implementations.
Current implementations  are:
- annotations
- Hibernate
- hierarchy
- jsf
- web
-  Weld

 

* deployers

 

Integration with VDF; new custom  deployers.

 

* indexer

 

This module contains utils for  creating pre-indexed handles and merging them into existing jars.
It  includes Ant task and Maven plugin.

 

* testsuite

 

Tests for  all other modules.

 

Basic building blocks

 

The  Scanner class is trivial, it only has scan() method, so it's not really  useful. :-)

 

The main thing is ScanningPlugin:

 

package  org.jboss.scanning.spi;

import java.io.IOException;
import  java.io.InputStream;
import java.io.OutputStream;

import  org.jboss.classloading.spi.visitor.ResourceFilter;
import  org.jboss.classloading.spi.visitor.ResourceVisitor;

/**
 *  Scanning plugin.
 * Defines what to do with a resource.
 *
 *  @param <T> exact handle type
 * @param <U> exact handle  interface
 * @author <a  href="mailto:ales.justin@jboss.org">Ales Justin</a>
 */
public  interface ScanningPlugin<T extends ScanningHandle, U> extends  ResourceFilter, ResourceVisitor
{
   /**
    * Create plugins  handle/utility.
    * e.g. AnnotationRepository for annotations  scanning
    *
    * @return new handle instance
    */
   T  createHandle();

   /**
    * Read serialized handle.
    *
     * @param is the serialized handle's input stream.
    * @return  de-serialized handle
    * @throws Exception for any error
    */
    ScanningHandle readHandle(InputStream is) throws Exception;

    /**
    * Write / serialize handle.
    *
    * @param os the  output stream to serialize handle.
    * @param handle the handle
     * @throws IOException for any IO error
    */
   void  writeHandle(OutputStream os, T handle) throws IOException;

   /**
     * Cleanup handle.
    *
    * @param handle the handle to cleanup
     */
   void cleanupHandle(U handle);

   /**
    * Get  handle interface.
    *
    * @return the handle interface
     */
   Class<U> getHandleInterface();

   /**
    * Get  handle's key.
    * Used to attach handle to map/attachments.
     *
    * @return the handle's key
    */
   String  getAttachmentKey();

   /**
    * Get handle's file name.
     * Used to attach handle to jar and/or get pre-indexed.
    *
    *  @return the handle's file name
    */
   String getFileName();

    /***
    * Get recurse filter.
    *
    * @return the recurse  filter
    */
   ResourceFilter getRecurseFilter();
}

 

Most  of the stuff is already implemented in its abstract form  (AbstractScanningPlugin) so you only need to provide the custom logic.

 

As  you can already see from the plugin's signature, the plugin introduces a  handle.
A handle is what will hold the scanning information for  particular component; e.g. an annotation repository
We can see  handle's implementation defined as parameter T, where handle's interface  is parameter U.

 

/**
 * Scanning handle.
 * 
 *  Represents a simple interface resource scanning results must implement
 *  in order to be able to merge pre-existing results.
 *
 * @param  <T> exact handle type
 * @author <a  href="mailto:ales.justin@jboss.org">Ales Justin</a>
 */
public  interface ScanningHandle<T extends ScanningHandle>
{
   /**
     * Merge existing handle with sub-handle / pre-existing handle.
    *
     * @param subHandle the sub handle
    */
   void merge(T  subHandle);
}

 

The main purpose of handle introduction is to  allow for pre-existing handle's merging in type safe manner.

 

How  to make usage of plugins as easy as possible in MC?
As we can see  Scanner (or its actual implementations) take a set of plugins to handle  artifacts.
But since plugins are mostly mutable, we need some sort of  factory to help use create these plugins.

 

For VDF based usage  this is how our factory looks like

 

import  org.jboss.deployers.structure.spi.DeploymentUnit;
import  org.jboss.scanning.spi.ScanningHandle;
import  org.jboss.scanning.spi.ScanningPlugin;

/**
 * Deployment based  scanning plugin factory.
 * Used for incallback automatching.
 *
 *  @param <T> exact handle type
 * @author <a  href="mailto:ales.justin@jboss.org">Ales Justin</a>
 */
public  interface DeploymentScanningPluginFactory<T extends ScanningHandle,  U>
{
   /**
    * Is this plugin relevant to unit.
    *
     * @param unit the unit to check against
    * @return true if it's  relevant, false otherwise
    */
   boolean  isRelevant(DeploymentUnit unit);

   /**
    * Create scanning  plugin from deployment unit.
    *
    * @param unit the  deployment unit
    * @return new scanning plugin
    */
    ScanningPlugin<T, U> create(DeploymentUnit unit);
}

 

Also,  as the javadoc says, this interface is nicely used for MC's incallback  usage.

 

Usage example

 

Lets see what we need to  implement in order to get annotation scanning into repository.

 

public  class AnnotationsScanningPluginFactory implements  DeploymentScanningPluginFactory<DefaultAnnotationRepository,  AnnotationIndex>
{
   public boolean isRelevant(DeploymentUnit  unit)
   {
      // any better check? -- metadata complete is  already done elsewhere
      // see JBossMetaDataDeploymentUnitFilter  in JBossAS
      return true;
   }

   public  ScanningPlugin<DefaultAnnotationRepository, AnnotationIndex>  create(DeploymentUnit unit)
   {
      ReflectProvider provider =  DeploymentUtilsFactory.getProvider(unit);
      ResourceOwnerFinder  finder = DeploymentUtilsFactory.getFinder(unit);
      return new  AnnotationsScanningPlugin(provider, finder, unit.getClassLoader());
    }
}


public class AnnotationsScanningPlugin extends  AbstractClassLoadingScanningPlugin<DefaultAnnotationRepository,  AnnotationIndex>
{
   /** The repository */
   private final  DefaultAnnotationRepository repository;
   /** The visitor */
    private final ResourceVisitor visitor;

   public  AnnotationsScanningPlugin(ClassLoader cl)
   {
       this(IntrospectionReflectProvider.INSTANCE,  ClassResourceOwnerFinder.INSTANCE, cl);
   }

   public  AnnotationsScanningPlugin(ReflectProvider provider, ResourceOwnerFinder  finder, ClassLoader cl)
   {
      repository = new  DefaultAnnotationRepository(cl);
      visitor = new  GenericAnnotationVisitor(provider, finder, repository);
   }

    protected DefaultAnnotationRepository doCreateHandle()
   {
       return repository;
   }

   protected ClassLoader  getClassLoader()
   {
      return repository.getClassLoader();
    }

   @Override
   public void cleanupHandle(AnnotationIndex  handle)
   {
      if (handle instanceof  DefaultAnnotationRepository)
          DefaultAnnotationRepository.class.cast(handle).cleanup();
   }

    public Class<AnnotationIndex> getHandleInterface()
   {
       return AnnotationIndex.class;
   }

   public ResourceFilter  getFilter()
   {
      return visitor.getFilter();
   }

    public void visit(ResourceContext resource)
   {
       visitor.visit(resource);
   }
}

public class  GenericAnnotationVisitor extends ClassHierarchyResourceVisitor
{
    /** The mutable repository */
   private MutableAnnotationRepository  repository;

   public GenericAnnotationVisitor(ReflectProvider  provider, ResourceOwnerFinder finder, MutableAnnotationRepository  repository)
   {
      super(provider, finder);
      if  (repository == null)
         throw new  IllegalArgumentException("Null repository");
      this.repository =  repository;
   }

   protected boolean isRelevant(ClassInfo  classInfo)
   {
      return  repository.isAlreadyChecked(classInfo.getName()) == false;
   }

    public ResourceFilter getFilter()
   {
      return  ClassFilter.INSTANCE;
   }

   @Override
   protected void  handleAnnotations(ElementType type, Signature signature, Annotation[]  annotations, String className, URL ownerURL)
   {
      if  (annotations != null && annotations.length > 0)
      {
          for (Annotation annotation : annotations)
         {
             repository.putAnnotation(annotation, type, className, signature,  ownerURL);
         }
      }
   }
}

 

While the  repository is just a bit smarter Map.

 

Integration with VDF

 

   <bean name="ScanningMDDeployer"  class="org.jboss.scanning.deployers.metadata.ScanningMetaDataDeployer"/>

   <bean name="ScannerDeployer"  class="org.jboss.scanning.deployers.ScanningDeployer">
     <property name="filter">
      <bean  class="org.jboss.scanning.deployers.filter.ScanningDeploymentUnitFilter"/>
     </property>
    <incallback method="addFactory" />
     <uncallback method="removeFactory" />
  </bean>

   <bean name="AnnScanningPlugin"  class="org.jboss.scanning.annotations.plugins.AnnotationsScanningPluginFactory"/>  

 

Using the Indexer

 

public class Main
{
    private static final Logger log =  Logger.getLogger(Main.class.getName());

   /**
    * Usage
     */
   private static void usage()
   {
       System.out.println("Usage: Indexer <input-jar>  <scanning-plugins-comma-delimited> <classpath*>");
   }

    /**
    * Main.
    * The output is file named  <input-jar>.jar.mcs.
    *
    * @param args the program  arguments
    */
   public static void main(String[] args)
   {
       try
      {
         int offset = 2;
         if (args.length  < offset)
         {
            File input = new  File(args[0]);
            String[] providers = args[1].split(",");
             URL[] urls = new URL[args.length - offset];
            // add the  rest of classpath
            for (int i = 0; i < urls.length;  i++)
               urls[i] = new File(args[i +  offset]).toURI().toURL();

            ScanUtils.scan(input,  Constants.applyAliases(providers), urls);
         }
         else
          {
            usage();
         }
      }
      catch  (Throwable t)
      {
         log.log(Level.SEVERE,  t.getMessage(), t);
      }
   }
}

 

Pre-existing or  pre-indexed information

 

For each scanning plugin we look for  artifact's META-INF/<plugin::getFileName> entry.

 

               String fileName = plugin.getFileName();
               for (URL root  : roots)
               {
                  InputStream is =  getInputStream(root, Scanner.META_INF + fileName);
                   if (is != null)
                  {
                      ScanningHandle pre = plugin.readHandle(is);
                      handle.merge(pre);

 

It's plugin's responsibility to know how to  read pre-existing handle.
By default we use plain Java serialization  together with gzip.

 

   public ScanningHandle  readHandle(InputStream is) throws Exception
   {
      try
       {
         GZIPInputStream gis = new GZIPInputStream(is);
          ObjectInputStream ois = createObjectInputStream(gis);
          return (ScanningHandle) ois.readObject();
      }
      finally
       {
         is.close();
      }
   }

   public void  writeHandle(OutputStream os, T handle) throws IOException
   {
       GZIPOutputStream gos = new GZIPOutputStream(os);
       ObjectOutputStream oos = new ObjectOutputStream(gos);
      try
       {
         oos.writeObject(handle);
         oos.flush();
       }
      finally
      {
         oos.close();
      }
    }

 

How to limit scanning?

 

There already was a  jboss-scanning.xml, I've just enhanced it a bit.

 

<scanning  xmlns="urn:jboss:scanning:1.0">
  <path name="myejbs.jar">
     <include name="com.acme.foo"/>
    <exclude  name="com.acme.foo.bar"/>
  </path>
  <path  name="my.war/WEB-INF/classes">
    <include  name="com.acme.foo"/>
  </path>
  <path  name="esb.sar/lib/ui.jar">
    <include name="com.esb.bar"  recurse="true"/>
  </path>
</scanning>

 

The  recurse filter is now a bit smarter than it used to be in previous  version. :-)

 

package org.jboss.scanning.plugins.filter;

import  java.net.URL;
import java.util.HashMap;
import java.util.List;
import  java.util.Map;
import java.util.Set;

import  org.jboss.classloading.spi.visitor.ResourceContext;
import  org.jboss.classloading.spi.visitor.ResourceFilter;
import  org.jboss.scanning.spi.metadata.PathEntryMetaData;
import  org.jboss.scanning.spi.metadata.PathMetaData;
import  org.jboss.scanning.spi.metadata.ScanningMetaData;
import  org.jboss.vfs.util.PathTokenizer;

/**
 * Simple recurse  filter.
 *
 * It searches for path substring in url string,
 *  and tries to match the tree structure as far as it goes.
 */
public  class ScanningMetaDataRecurseFilter implements ResourceFilter
{
    /** Path tree roots */
   private Map<String, RootNode> roots;

    public ScanningMetaDataRecurseFilter(ScanningMetaData smd)
   {
       if (smd == null)
         throw new IllegalArgumentException("Null  metadata");

      List<PathMetaData> paths =  smd.getPaths();
      if (paths != null && paths.isEmpty() ==  false)
      {
         roots = new HashMap<String,  RootNode>();
         for (PathMetaData pmd : paths)
         {
             RootNode pathNode = new RootNode();
             roots.put(pmd.getPathName(), pathNode);
             Set<PathEntryMetaData> includes = pmd.getIncludes();
             if (includes != null && includes.isEmpty() == false)
             {
               pathNode.explicitInclude = true;
                for (PathEntryMetaData pemd : includes)
               {
                   String name = pemd.getName();
                  String[] tokens =  name.split("\\.");
                  Node current = pathNode;
                   for (String token : tokens)
                     current =  current.addChild(token);
                  if (pemd.isRecurse())
                      current.recurse = true; // mark last one as recurse
               }
             }
         }
      }
   }

   public boolean  accepts(ResourceContext resource)
   {
      if (roots == null)
          return false;

      URL url = resource.getUrl();
      String  urlString = url.toExternalForm();
      for (Map.Entry<String,  RootNode> root : roots.entrySet())
      {
         if  (urlString.contains(root.getKey()))
         {
             RootNode rootNode = root.getValue();
            if  (rootNode.explicitInclude) // we have explicit includes in path, try  tree path
            {
               String resourceName =  resource.getResourceName();
               List<String> tokens =  PathTokenizer.getTokens(resourceName);
               Node current =  rootNode;
               // let's try to walk some tree path
                for (String token : tokens)
               {
                  //  if we're here, the rest is recursively matched
                  if  (current.recurse)
                     break;

                   current = current.getChild(token);
                  // no fwd path
                   if (current == null)
                     return false;
                }
            }
            return true;
         }
      }
       return false;
   }

   private static class Node
   {
       private Map<String, Node> children;
      private boolean  recurse;

      public Node addChild(String value)
      {
          if (children == null)
            children = new HashMap<String,  Node>();

         Node child = children.get(value);
          if (child == null)
         {
            child = new Node();
             children.put(value, child);
         }
         return child;
       }

      public Node getChild(String child)
      {
          return children != null ? children.get(child) : null;
      }
   }

    private static class RootNode extends Node
   {
      private  boolean explicitInclude;
   }
}

 

Javassist based JBoss  Reflect

 

In order not to load the actual resource's underlying  class, we use Javassist under covers - via JBoss Refect project.

 

       DeploymentUnit unit = assertDeploy(jar);
      try
      {
          TIFScanningPlugin plugin = unit.getAttachment(TIFScanningPlugin.class);
          assertNotNull(plugin);

         Kernel kernel =  assertBean("Kernel", Kernel.class);
         KernelConfigurator  configurator = kernel.getConfigurator();

         ClassLoader cl =  unit.getClassLoader();

         String name =  JarMarkOnClass.class.getName();
         TypeInfo ti =  configurator.getTypeInfo(name, cl);
         TypeInfo visited =  plugin.getResources().get(name);
         assertSame(ti, visited); //  let's check if the cache is working

         Method  findLoadedClass = ClassLoader.class.getDeclaredMethod("findLoadedClass",  String.class);
         findLoadedClass.setAccessible(true);
          Object clazz = findLoadedClass.invoke(cl, name);
          assertNull(clazz); // should not be loaded
      }
      finally
       {
         undeploy(unit);   
      }

 

But the overall  usage of helper utils is plugable:

 

/**
 * Find the util for  deployment.
 * Newly created utils are grouped per module.
 *
 *  @author <a href="mailto:ales.justin@jboss.org">Ales  Justin</a>
 */
public class DeploymentUtilsFactory
{
    /** The default impls */
   private static Map<Class<?>,  UtilFactory<?>> defaults = new WeakHashMap<Class<?>,  UtilFactory<?>>();

   static
   {
       addImplementation(ReflectProvider.class, new  ReflectProviderUtilFactory());
       addImplementation(ResourceOwnerFinder.class, new  ResourceOwnerFinderUtilFactory());
   }

   /**
    * Add  the util impl.
    *
    * @param iface the interface
    *  @param factory the util factory
    */
   public static <T>  void addImplementation(Class<T> iface, UtilFactory<T>  factory)
   {
      defaults.put(iface, factory);
   }

    /**
    * Remove the util impl.
    *
    * @param iface the  interface
    */
   public static void  removeImplementation(Class<?> iface)
   {
       defaults.remove(iface);
   }

   /**
    * Get util.
    *
     * @param unit the deployment unit
    * @param utilType the util  type
    * @return util instance
    */
   public static  <T> T getUtil(DeploymentUnit unit, Class<T> utilType)
   {
       if (utilType == null)
         throw new  IllegalArgumentException("Null util type");

      DeploymentUnit  moduleUnit = getModuleUnit(unit);

      T util =  moduleUnit.getAttachment(utilType);
      if (util == null)
      {
          UtilFactory factory = defaults.get(utilType);
         if (factory  == null)
            throw new IllegalArgumentException("No util  factory defined for " + utilType);

         Object instance =  factory.create(moduleUnit);
         util = utilType.cast(instance);

          moduleUnit.addAttachment(utilType, util);
      }
      return  util;
   }

   /**
    * Get module unit.
    *
    *  @param unit the current unit
    * @return unit containing Module, or  exception if no such unit exists
    */
   public static  DeploymentUnit getModuleUnit(DeploymentUnit unit)
   {
      if  (unit == null)
         throw new IllegalArgumentException("Null  unit");

      // group util per module
      DeploymentUnit  moduleUnit = unit;
      while(moduleUnit != null &&  moduleUnit.isAttachmentPresent(Module.class) == false)
          moduleUnit = moduleUnit.getParent();

      if (moduleUnit ==  null)
         throw new IllegalArgumentException("No module in unit:  " + unit);

      return moduleUnit;
   }

   /**
     * Wrap util lookup in lazy lookup.
    *
    * @param unit the  deployment unit
    * @param utilType the util type
    * @return  lazy util proxy
    */
   public static <T> T  getLazyUtilProxy(DeploymentUnit unit, Class<T> utilType)
   {
       // null check is in handler
      LazyUtilsProxyHandler<T>  handler = new LazyUtilsProxyHandler<T>(unit, utilType);
       Object proxy = Proxy.newProxyInstance(unit.getClassLoader(), new  Class[]{utilType}, handler);
      return utilType.cast(proxy);
    }

   /**
    * Get reflect provider.
    *
    * @param  unit the depoyment unit
    * @return the provider
    */
    public static ReflectProvider getProvider(DeploymentUnit unit)
   {
       return getUtil(unit, ReflectProvider.class);
   }

   /**
     * Get finder.
    *
    * @param unit the depoyment unit
    *  @return the finder
    */
   public static ResourceOwnerFinder  getFinder(DeploymentUnit unit)
   {
      return getUtil(unit,  ResourceOwnerFinder.class);
   }

   /**
    * Cleanup the  util.
    *
    * @param util the util to cleanup
    */
    public static void cleanup(Object util)
   {
      if (util  instanceof CachingResourceOwnerFinder)
          CachingResourceOwnerFinder.class.cast(util).cleanup();
   }
}

 

Meaning  it's easy to swap utils behavior for particular deployment unit.
e.g.  different ResourceOwnerFinder or ReflectProvider

 

Reporting  issues

As usual, use the forums:
* MC user forum:  http://community.jboss.org/en/jbossmc
* MC dev forum:  http://community.jboss.org/en/jbossmc/dev

 

Or custom JIRA:
https://jira.jboss.org/jira/browse/JBMCSCAN