Wednesday, December 16, 2015

A new interpreter for EASE (4): Provide modules support

EASE is shipped with modules: libraries written in Java, made available for script interpreters. The intention of such modules is to write them once and use them in all available script languages.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 


Introduction

Typically modules are plain java classes. We could easily create an instance in BeanShell and then call instance.method(). While this is perfectly ok for the average programmer, some of our user might expect a more simple, functional interface. What module loading does is:
  • create an instance of the module
  • inject the instance into the interpreter
  • create dynamic script code to wrap method calls
  • execute the created code
For BeanShell we would create code like
method(param1, param2) { 

result = __MOD_module_instance.method(param1, param2);

return result;

}
After loading this fragment we could access method() without the knowledge of modules, instances and object orientation at all.

Step 1: The code factory

To dynamically create code EASE needs an ICodeFactory implementation. To start you may look at existing implementations, basically we just need to build strings containing script code.
package org.eclipse.ease.lang.beanshell;

import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.Arrays;
import java.util.List;

import org.eclipse.ease.Logger;
import org.eclipse.ease.modules.AbstractCodeFactory;
import org.eclipse.ease.modules.IEnvironment;
import org.eclipse.ease.modules.IScriptFunctionModifier;
import org.eclipse.ease.modules.ModuleHelper;

public class BeanShellCodeFactory extends AbstractCodeFactory {

 public static final List<String> RESERVED_KEYWORDS = Arrays.asList("abstract", "continue", "for", "new", "switch", "assert", "default", "goto", "package",
   "synchronized", "boolean", "do", "if", "private", "this", "break", "double", "implements", "protected", "throw", "byte", "else", "import", "public",
   "throws", "case", "enum", "instanceof", "return", "transient", "catch", "extends", "int", "short", "try", "char", "final", "interface", "static",
   "void", "class", "finally", "long", "strictfp", "volatile", "const", "float", "native", "super", "while");

 private static boolean isValidMethodName(final String methodName) {
  return BeanShellHelper.isSaveName(methodName) && !RESERVED_KEYWORDS.contains(methodName);
 }

 @Override
 public String getSaveVariableName(final String variableName) {
  return BeanShellHelper.getSaveName(variableName);
 }

 @Override
 public String createFunctionWrapper(final IEnvironment environment, final String moduleVariable, final Method method) {

  final StringBuilder scriptCode = new StringBuilder();

  // parse parameters
  final List<Parameter> parameters = ModuleHelper.getParameters(method);

  // build parameter string
  final StringBuilder parameterList = new StringBuilder();
  for (final Parameter parameter : parameters)
   parameterList.append(", ").append(parameter.getName());

  if (parameterList.length() > 2)
   parameterList.delete(0, 2);

  final StringBuilder body = new StringBuilder();

  // insert hooked pre execution code
  body.append(getPreExecutionCode(environment, method));

  // insert deprecation warnings
  if (ModuleHelper.isDeprecated(method))
   body.append("\tprintError('" + method.getName() + "() is deprecated. Consider updating your code.');\n");

  // insert method call
  body.append("\t ");
  if (!method.getReturnType().equals(Void.TYPE))
   body.append(IScriptFunctionModifier.RESULT_NAME).append(" = ");

  body.append(moduleVariable).append('.').append(method.getName()).append('(');
  body.append(parameterList);
  body.append(");\n");

  // insert hooked post execution code
  body.append(getPostExecutionCode(environment, method));

  // insert return statement
  if (!method.getReturnType().equals(Void.TYPE))
   body.append("\treturn ").append(IScriptFunctionModifier.RESULT_NAME).append(";\n");

  // build function declarations
  for (final String name : getMethodNames(method)) {
   if (!isValidMethodName(name)) {
    Logger.error(IPluginConstants.PLUGIN_ID,
      "The method name \"" + name + "\" from the module \"" + moduleVariable + "\" can not be wrapped because it's name is reserved");

   } else if (!name.isEmpty()) {
    scriptCode.append(name).append("(").append(parameterList).append(") {\n");
    scriptCode.append(body);
    scriptCode.append("}\n");
   }
  }

  return scriptCode.toString();
 }

 @Override
 public String classInstantiation(final Class<?> clazz, final String[] parameters) {
  final StringBuilder code = new StringBuilder();
  code.append("new ").append(clazz.getName()).append("(");

  if (parameters != null) {
   for (final String parameter : parameters)
    code.append(parameter).append(", ");

   if (parameters.length > 0)
    code.delete(code.length() - 2, code.length());
  }

  code.append(")");

  return code.toString();
 }

 @Override
 public String createFinalFieldWrapper(final IEnvironment environment, final String moduleVariable, final Field field) {
  return getSaveVariableName(field.getName()) + " = " + moduleVariable + "." + field.getName() + ";\n";
 }

 @Override
 protected String getNullString() {
  return "null";
 }
}
createFunctionWrapper() is called for each exposed method, createFinalFieldWrapper() is called for each exposed final field. The function wrapper injects preExecutionCode and postExecutionCode. This is interesting for modules implementing IScriptFunctionModifier. If such a module is loaded, it may influence code generation for all other modules. Use cases are creation of log files or unit testing.

Step 2: Register code factory

Now we need to register our code factory, so EASE is able to find it. Open the Extensions tab of org.eclipse.ease.lang.beanshell/plugin.xml. Navigate to the scriptType extension and add the code factory class to the BeanShell type.
Step 3: Bootstrapping

There exists a dedicated Environment module that provides basic commands like loadModule() to load further modules. Lets try to load it in a BeanShell:

run EASE and open a bean shell in the Script Shell View. We will create an instance of EnvironmentModule and advise it to load itself:
new org.eclipse.ease.modules.EnvironmentModule().loadModule("/System/Environment");
If your code factory works as expected, commands like print() and loadModule() should be available.

To automate this task we will add a launch extension to our BeanShell language definition.
Switch to the plugin.xml and add a launchExtension to the org.eclipse.ease.language node.We bind it to the engineID of BeanShell and need to provide an implementation:
public class BootStrapper implements IScriptEngineLaunchExtension {

 @Override
 public void createEngine(final IScriptEngine engine) {
  ICodeFactory codeFactory = ScriptService.getCodeFactory(engine);
  if (codeFactory != null) {
   StringBuilder stringBuilder = new StringBuilder();
   stringBuilder.append(codeFactory.classInstantiation(EnvironmentModule.class, new String[0]));
   stringBuilder.append(".loadModule(\"");
   stringBuilder.append(EnvironmentModule.MODULE_NAME);
   stringBuilder.append("\");\n");

   engine.executeAsync(new Script("Bootloader", stringBuilder.toString()));
  }
 }
}
Of course we could directly use the string we tried manually before, but using the code factory is somewhat nicer. The implementation given above is already available from the BootStrapper class from EASE. If you do want to add more stuff to the bootloader you need to provide your own implementation.

With that in place you should be able to load all available modules into BeanShell and use its functions.

Optional: Loading external jars

We added support to register URLs to the classloader in our previous tutorial. Now as we have modules working we may try this out by calling
loadJar("http://central.maven.org/maven2/com/github/lalyos/jfiglet/0.0.7/jfiglet-0.0.7.jar")
 true
com.github.lalyos.jfiglet.FigletFont.convertOneLine("I      love      scripting");
  ___        _                                             _         _    _               
 |_ _|      | |  ___  __   __  ___        ___   ___  _ __ (_) _ __  | |_ (_) _ __    __ _ 
  | |       | | / _ \ \ \ / / / _ \      / __| / __|| '__|| || '_ \ | __|| || '_ \  / _` |
  | |       | || (_) | \ V / |  __/      \__ \| (__ | |   | || |_) || |_ | || | | || (_| |
 |___|      |_| \___/   \_/   \___|      |___/ \___||_|   |_|| .__/  \__||_||_| |_| \__, |
                                                             |_|                    |___/ 


Tuesday, December 15, 2015

A new interpreter for EASE (3): Loading native classes

In this tutorial we will teach our interpreter to load all kinds of classes from Eclipse and dedicated URLs. So we will have a closer look on the classloader, but don't worry - it will not get too complicated.

Our BeanShell interpreter is already capable of calling java code, so we only need to make sure that its classloader will have access to all desired resources.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 

Step 1: Access java classes from eclipse

Our BeanShell interpreter already accepts calls to JRE classes like
java.lang.System.out.println("Hello world");
However if we try to access eclipse classes like
org.eclipse.jface.resource.JFaceColors
    Sourced file: inline evaluation of: ``org.eclipse.jface.resource.JFaceColors;'' :
    Class or variable not found: org.eclipse.jface.resource.JFaceColors
We get an exception indicating that the class cannot be found. This is a classloader issue as the interpreter uses the bundle classloader from our org.eclipse.ease.lang.beanshell bundle. We could add dependencies to jface to solve the problem, but how would we introduce dependencies to all plug-ins that might be used by a customer?

Fortunately eclipse has a solution for this. We may allow plug-ins to access classes from all exported packages out there without  adding a dedicated dependency by defining
Eclipse-BuddyPolicy: global
in the MANIFEST.MF file. This comes with a penalty on class loading performance but allows to access all available classes.

Add this setting to the manifest of org.eclipse.ease.lang.beanshell. Afterwards switch to the BeanShellEngine and add following line to setupEngine():
fInterpreter.setClassLoader(getClass().getClassLoader());
This line changes the classloader of the BeanShell with the one from the org.eclipse.ease.lang.beanshell bundle.

Now go and try to load some eclipse classes:
org.eclipse.jface.resource.JFaceColors
    Class Identifier: org.eclipse.jface.resource.JFaceColors

Step 2: Load classes from external URLs

EASE allows to register external jar files and make them available to scripting. This mechanism  is provided by a module - a concept we will introduce in the next tutorial. For now we will make all necessary preparations. EASE provides a generic classloader with all necessary functionality. Go to your BeanShellEngine and modify the code as follows:
public class BeanShellEngine extends AbstractScriptEngine {

 private DelegatingJarClassLoader fClassLoader;

 @Override
 protected boolean setupEngine() {
  fInterpreter = new Interpreter();

  fClassLoader = new DelegatingJarClassLoader(getClass().getClassLoader());
  fInterpreter.setClassLoader(fClassLoader);

  [...]
 }

 @Override
 public void registerJar(final URL url) {
  if (fClassLoader != null)
   fClassLoader.registerURL(url);
 }
}
The DelegatingClassLoader will take care of registered URLs. For everything else its parent classloader will be used. We cannot test this currently, so you need to trust me on this until the next tutorial.

Friday, December 11, 2015

A new interpreter for EASE (2): Shell and script integration

Today we will focus on integrating BeanShell into the script shell and to allow to execute .bsh files.

To follow this tutorial you will either need to install EASE into your development IDE or even better create a target platform containing EASE core plugins. I expect that you are familiar with that process.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 

Step 1: A very basic EASE interpreter

Create a new Plug-in Project called org.eclipse.ease.lang.beanshell. Add dependencies to org.eclipse.ease and to our org.beanshell plug-in.

Switch to the Extensions tab and add a new extension point for org.eclipse.ease.language. Create an engine, provide an ID, a nice name and a link to the implementing class.

Instead of implementing the IScriptEngine interface directly you may inherit from AbstractScriptEngine:
package org.eclipse.ease.lang.beanshell;

import org.eclipse.ease.AbstractScriptEngine;
import org.eclipse.ease.Script;

import bsh.Interpreter;

public class BeanShellEngine extends AbstractScriptEngine {

 private Interpreter fInterpreter = null;

 public BeanShellEngine() {
  super("BeanShell");
 }

 @Override
 protected boolean setupEngine() {
  fInterpreter = new Interpreter();

  fInterpreter.setOut(getOutputStream());
  fInterpreter.setErr(getErrorStream());

  return true;
 }

 @Override
 protected boolean teardownEngine() {
  fInterpreter = null;

  return true;
 }

 @Override
 protected Object execute(final Script script, final Object reference, final String fileName, final boolean uiThread) throws Throwable {
  return fInterpreter.eval(script.getCode());
 }
}
Use the QuickFix to Add unimplemented methods.

Launch a new RCP application adding org.eclipse.ease.* and org.beanshell plugins to your run configuration.

Now open the Script Shell view and use the engine selection to switch to BeanShell. Your console name should change to BeanShellScript Shell and you should be able to enter and execute some beanshell commands.

Step 2: Adding launch target support

To launch files with an engine we need to bind our engine to dedicated content types. That means that we first need a content type for BeanShell files. Open the Plug-in Manifest Editor, switch to the Extensions tab and add a new extension for org.eclipse.contenttype.contentTypes. Add a new content-type based on org.eclipse.core.runtime.text and set file-extensions to bsh.

Now create a new extension for org.eclipse.ease.scriptType. Set name to BeanShell and the defaultExtension to bsh.  Afterwards add a binding to our previously created content type.

Finally switch to your engine definition and create a binding to the scriptType.

When launching you may now run .bsh files from your workspace using the EASE launch target (eg by using Run As/EASE Script from the context menu).

Step 3: Add variables support

Getting and setting variables improves the Script Shell experience but is also needed for modules support, which we investigate in a following tutorial.

Open your BeanShellEngine and replace your default methods for variables with these:
 @Override
 protected Object internalGetVariable(final String name) {
  try {
   return fInterpreter.get(name);
  } catch (EvalError e) {
   Logger.error("org.eclipse.ease.lang.beanshell", "Cannot retrieve variable \"" + name + "\"", e);
  }

  return null;
 }

 @Override
 protected Map<String, Object> internalGetVariables() {
  Map<String, Object> variables = new HashMap<String, Object>();

  for (Variable variable : fInterpreter.getNameSpace().getDeclaredVariables())
   variables.put(variable.getName(), internalGetVariable(variable.getName()));

  return variables;
 }

 @Override
 protected boolean internalHasVariable(final String name) {
  for (Variable variable : fInterpreter.getNameSpace().getDeclaredVariables()) {
   if (variable.getName().equals(name))
    return true;
  }

  return false;
 }

 @Override
 protected void internalSetVariable(final String name, final Object content) {
  try {
   fInterpreter.set(name, content);
  } catch (EvalError e) {
   Logger.error("org.eclipse.ease.lang.beanshell", "Cannot set variable \"" + name + "\"", e);
  }
 }

 @Override
 protected Object internalRemoveVariable(final String name) {
  Object content = internalGetVariable(name);
  fInterpreter.getNameSpace().unsetVariable(name);

  return content;
 }

 @Override
 public String getSaveVariableName(final String name) {
  return BeanShellHelper.getSaveName(name);
 }
We purely dig into the beanshell interpreter and investigate its namespace. The BeanShellHelper class is copied from the JavaScriptHelper and provides basic checks on variable names.

Once launched, the Script Shell will show a list of all available variables in your interpreter attached in the container right to the shell.

Monday, December 7, 2015

A new interpreter for EASE (1): Playing with beanshell

This new series will provide a guide how to integrate your own interpreter in EASE. As a showcase we will implement an interpreter for BeanShell.

Read all tutorials from this series.

Source code for this tutorial is available on github as a single zip archive, as a Team Project Set or you can browse the files online. 

Step 1: Evaluate interpreter candidates

The bare minimum requirement for an interpreter to do something useful with EASE is to support an execute() method that accepts and executes arbitrary strings of script code. This will allow to use the interpreter in the script shell and to create script files and execute them using the EASE launch target.

Now EASE typically integrates interpreters that run natively on the JRE. This allows scripts to seamlessly interact with the current application. Such interpreters have some additional requirements:
  • access native java objects (eg. java.lang.System)
  • set/get variables in the interpreter
  • allow to define functions/methods
  • redirect input/output/error streams
Interpreters like Rhino or Jython support these extended requirements and therefore allow to write scripts that fully integrate into EASE.

However much simpler interpreters may make sense in some cases. I have played around with such interpreters and built a basic git console and a bash console. Both are in a proof of concept state, but might indicate what can be done with EASE.

Step 2: Integrating BeanShell

We will concentrate on BeanShell, so download the latest stable build.

Now create a new Plug-in from Existing JAR Archives. On the next page hit Add External and add the downloaded bsh-<version>.jar. Set the project name to org.beanshell and select Unzip the JAR archives into the project.

Open the Plug-in Manifest Editor and make sure that the bsh package is correctly exported on the Runtime tab.

Step 3: Play with the interpreter

To learn how the interpreter works we create a Java Project org.beanshell.playground. Add our org.beanshell project to the Java Build Path and create a test class with a main method:
package org.beanshell.playground;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Date;

import bsh.EvalError;
import bsh.Interpreter;

public class RunEmbedded {

 private static final String[] CODE_FRAGMENTS = new String[] { "2+3", "foo*10", "new java.io.File(\"/\").exists()",
   "sum(a,b){\n return a+b;\n }\nsum(40,2)" };

 public static void main(String[] args) throws EvalError, FileNotFoundException, IOException {
  Interpreter interpreter = new Interpreter(); // Construct an interpreter

  // set variables
  interpreter.set("foo", 5);
  interpreter.set("date", new Date());

  // get variables
  Object date = interpreter.get("date");

  // evaluate statements
  for (String code : CODE_FRAGMENTS) {
   System.out.println(interpreter.eval(code));
  }
 }
}

A simple embedding example was quickly found using google, now our tests show how to get/set variables and how to execute various pieces of code. From what we can see so far BeanShell is a perfect candidate for scripting.

The following tutorials will show how to integrate the interpreter in EASE, stay tuned!.