At the weekend I did a talk on OpenJDK lambda at FOSDEM.
You can find the slides here, unfortunately the slides don't convey everything that a video would, since I was switching between slides, the IDE, and command line.
At the end I demonstrated how to code up the "Game of Life" using the lambda Streams API, which was inspired by a Clojure version created by Christophe Grand. You can find the gist here.
Wednesday, February 6, 2013
Tuesday, July 17, 2012
Java 8, Lambda and method references
It could be said that Java 8 with Lambda has a simple form of structural typing.
A method on an object can be transformed into an instance of a matching functional interface. A functional interface is an interface with one method. This is achieved by using method references. For more details see Brian Goetz's State of the Lambda document, section 8.
For example, one can do this:
public class Main {
public static class NotAutoCloseable {
public void close() throws Exception {
System.out.println("CLOSE");
}
}
public static void main(String... args) throws Exception {
NotAutoCloseable nac = new NotAutoCloseable();
try (AutoCloseable ac = nac::close) {
}
}
}
The class NotAutoCloseable does not implement AutoCloseable but it has a method, close, that matches the functional interface AutoCloseable. So NotAutoCloseable#close can be transformed into an instance of AutoCloseable using the method reference nac::close.
This is equivalent to the following with a lambda expression:
try (AutoCloseable ac = () -> { nac.close(); }) {
}
The variable nac is effectively final.
Compiling the above Main class currently generates an inner class that implements AutoCloseable and encapsulates the invocation of nac.close(). Thats gonna change, expect to see invokedynamic instructions in the byte code! Such an instruction will describe a recipe to construct the lambda expression at runtime without specifying the implementation details.
In addition to method references there are also constructor references. So constructors can be transformed into factories.
I think method and constructor references are expressive little gems, but they are one of many new features in the Java language, Java virtual machine and Java libraries that combined will bring significant improvements in Java 8.
A method on an object can be transformed into an instance of a matching functional interface. A functional interface is an interface with one method. This is achieved by using method references. For more details see Brian Goetz's State of the Lambda document, section 8.
For example, one can do this:
public class Main {
public static class NotAutoCloseable {
public void close() throws Exception {
System.out.println("CLOSE");
}
}
public static void main(String... args) throws Exception {
NotAutoCloseable nac = new NotAutoCloseable();
try (AutoCloseable ac = nac::close) {
}
}
}
The class NotAutoCloseable does not implement AutoCloseable but it has a method, close, that matches the functional interface AutoCloseable. So NotAutoCloseable#close can be transformed into an instance of AutoCloseable using the method reference nac::close.
This is equivalent to the following with a lambda expression:
try (AutoCloseable ac = () -> { nac.close(); }) {
}
The variable nac is effectively final.
Compiling the above Main class currently generates an inner class that implements AutoCloseable and encapsulates the invocation of nac.close(). Thats gonna change, expect to see invokedynamic instructions in the byte code! Such an instruction will describe a recipe to construct the lambda expression at runtime without specifying the implementation details.
In addition to method references there are also constructor references. So constructors can be transformed into factories.
I think method and constructor references are expressive little gems, but they are one of many new features in the Java language, Java virtual machine and Java libraries that combined will bring significant improvements in Java 8.
Friday, July 6, 2012
Modular services with OpenJDK Jigsaw and Guice
This blog entry describes an experiment exploring the connection between Java modules and services in OpenJDK Jigsaw and the dependency injection framework Guice.
Modular services in Jigsaw define a very simple way to bind a (service) interface to implementations (service provider classes) and, using java.util.ServiceLoader, a way to iterate through all (service) instances of implementations bound to an interface.
Guice supports a rich binding model and scoping model. Guice can support a JDK services style approach using multibindings where service instances are obtained by referencing an injection type Set<T> where T is the service interface. (Note that when instance of the Set<T> is injected all members are instantiated according to the scoping rules; the set is not lazy.)
Can Jigsaw and Guice be combined for richer modular services? Perhaps :-)
Before we can start playing around with Jigsaw and Guice we first need to modularize Guice, so this is also an experiment in taking a popular framework and modularizing it to work in the modular OpenJDK.
The complete set of code can be found here on GitHub.
A modular jar file is a jar file that contains a compiled module description, module-info.class, at the top-level. The module-info.class is the result of compiling the module declaration source, module-info.java. The module declaration declares important information such as the a module name, version, dependent modules, and what types are exported.
The approach to work out that set of jar files was to manually analyze the runtime dependencies of Guice modules on maven central (com.google.inject:guice:3.0 and com.google.inject.extensions:guice-multibindings:3.0) to obtain the complete set of jar files required for for Guice to run.
For each jar a corresponding source module was created that contained just a module-info.java. The Java compiler, javac, was used to compile all the source modules to obtain corresponding module-info.class files.
Finally each jar file was downloaded from maven central and updated to include the corresponding module-info.class, thus transforming each jar into a modular jar.
The ant script which automates all this can be found here.
I have no doubt this can be further automated. Alan Bateman showed of a demo at JavaOne last year that processed maven pom files, slurped stuff from maven central, and automatically created modular jars from analyzing the dependencies.
This black box approach does not work in my case as i need tweak the module declarations (as explained later in this section). Some sort of automated white-box approach is required where an initial module-info.java can be created from which it can then be modified.
There is also some particularly thorny issues lurking here. Maven and Jigsaw have different dependency resolution algorithms. Maven, using dependency management, allows a pom to override versions of it's transitive dependencies. Jigsaw currently does not support this (it's an open question as to whether it will). Thus anything produced from such automation may not be reusable in other contexts since the dependency graphs might be different.
A total of four jars were identified:
module aopalliance@1.0 {
exports org.aopalliance.aop;
exports org.aopalliance.intercept;
}
The module declaration for the javax.inject module is:
module javax.inject@1 {
exports javax.inject;
}
Both of the above modules do not have any dependencies and just export types contained in the declared packages.
The module declaration for the guice module is:
module guice@3.0 {
requires public javax.inject@1;
requires public aopalliance@1.0;
requires jdk.logging;
exports com.google.inject;
exports com.google.inject.binder;
exports com.google.inject.matcher;
exports com.google.inject.name;
exports com.google.inject.spi;
exports com.google.inject.util;
view guice.internal {
permits guice.multibindings;
exports com.google.inject.internal;
exports com.google.inject.internal.util;
}
}
Now it gets a little bit more interesting. First the guice module requires (or depends on) the javax.inject and aopalliance modules. Those dependencies are qualified as public so that any module requiring the guice module will automatically have access to the types exported by the other two modules; the types in those modules are re-exported.
Note that i am not sure if types in the aopalliance module are referenced in the Guice API, if not there is no need for the types in that module to be re-exported. Certainly types in the javax.inject module are referenced in the Guice API.
Guice requires the jdk.logging module but even though some types from this module are bound, by default, by Guice no such types are (AFAICT) exposed publicly hence the types of this module are not re-exported.
Guice takes particular care of placing all implementation classes underneath the package com.google.inject.internal. The expectation is that developers should not directly reference any such classes that are publicly accessible. Where possible such implementation classes are package private. This made it easy produce a list of exports clauses for the public API. More importantly with Java modularity the developers need not know anything about the internal classes and the module developer can be assured that at compilation and runtime such non-exported types are not visible and accessible.
It might be reasonable to suggest why not allow wildcards in exports clauses?, for example "exports com.google.inject.*;". In this case wildcards would be not be a good idea because types in the internal packages would be exported. In general the use of wildcards makes it all too easy to expose more than necessary and unintentionally, for example if packages are renamed.
However, there is a little problem. The exported types are not sufficient for other Guice modules to function, such as the guice.multibindings module. The Guice multibinding code imports types from the packages com.google.inject.internal and com.google.inject.internal.util:
The module declaration for the guice.multibindings module is:
module guice.multibindings@3.0 {
requires public guice@3.0;
requires guice.internal@3.0;
exports com.google.inject.multibindings;
}
It has a dependency on the guice.internal view of module guice. For ease of use this module re-exports all the types in the guice module, otherwise this would be redundant since the guice.internal view inherits from the guice module (or default view).
I had to go through a couple of iterations of editing the module declarations, producing the modular jars, installing those into a library, and running a test application. The perils of modularizing from jars files rather than compiling the source code!
Without the view and permits clauses a dreaded NoClassDefFoundError was thrown:
Exception in thread "main" java.lang.NoClassDefFoundError: Lcom/google/inject/internal/util/$ImmutableList;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2318)
at java.lang.Class.getDeclaredFields(Class.java:1771)
at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:649)
...
at com.google.inject.multibindings.Multibinder$RealMultibinder.configure(Multibinder.java:269)
...
at com.google.inject.Guice.createInjector(Guice.java:95)
...
at mapp.App.main(App.java:9)
Caused by: java.lang.ClassNotFoundException: com.google.inject.internal.util.$ImmutableList : requested by +guice.multibindings
at org.openjdk.jigsaw.Loader.loadClass(Loader.java:116)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
So Guice is doing some reflection accessing fields of some type in the guice.multibindings module that has a field whose type is in the internal package of the guice module, see here. Note that the Guice build process re-names certain class files e.g. ...util.ImmutableList to ...util.$ImmutableList. There is enough information here to guess what is going on but the indirection can throw one slightly off course.
Tooling analyzing class files and presenting class dependencies would be most helpful.
In the example on github the modular jars are installed into a guice-specific library. The application library (containing the installed application modules using Guice) is hooked up to the guice-specific library. This is primarily to avoid doing too much work when building the application library but it does show that libraries can be reusable (although not necessarily portable) components.
There are five modules that are compiled and installed into the application library:
Essentially Jigsaw modular services is used to bootstrap Guice services.
Each module that participates in Guice services provides an implementation of GuiceInjectorService service interface:
import com.google.inject.Injector;
public interface GuiceInjectorService {
Injector getInjector();
}
which supports the getting of a Guice Injector instance. java.util.ServiceLoader can be used to obtain all Injector instances, and then the bindings of those instances can be introspected.
An alternative solution might be to obtain Guice Module instances then create one Injector for all those instances. However, since the Java modules are decoupled from each other it cannot be guaranteed that no binding conflicts will arise.
The GuiceInjectorService implementation and the Guice module for the rotter module are as follows:
public class RotterInjectorProvider implements GuiceInjectorService {
@Override
public Injector getInjector() {
return Guice.createInjector(new RotterModule());
}
}
public class RotterModule extends AbstractModule {
@Override
protected void configure() {
Multibinder<StringTransformer> uriBinder = Multibinder.
newSetBinder(binder(), StringTransformer.class);
uriBinder.addBinding().to(RotterStringTransformer.class).
in(Scopes.SINGLETON);
}
}
The Guice module is using the multibinding support to bind RotterStringTransformer to a singleton instance. Multibinding makes it easier to bind two or more implementations to the same interface (internally it will create unique keys, containing unique annotations so that each individual binding is unique).
The GuiceInjectorService implementation and the Guice module for the hasher module are as follows:
This Guice module registers two bindings of the StringTransform service interface. The first uses the programmatic binding. The latter uses the annotation-based binding to adapt the former and convert the String to lower case. To differentiate the later from the former so it can be bound the later is named.
The main class in the mapp module is as follows:
public class App {
public static void main(String[] args) throws Exception {
for (StringTransformer s : GuiceServiceLoader.
load(StringTransformer.class)) {
System.out.println(s + " " + s.description());
}
for (StringTransformer s : GuiceServiceLoader.
load(StringTransformer.class)) {
System.out.println(s + " " + s.description());
}
}
}
It just iterates through the StringTransformer service instances using the GuiceServiceLoader much like one can do with ServiceLoader.
So how does GuiceServiceLoader work? The Injector instances are lazily cached as follows:
public class GuiceServiceLoader<T> implements Iterable<T> {
static class Initializer {
private static Set<Injector> injectors = load();
private static Set<Injector> load() {
final Set<Injector> giss = new LinkedHashSet<>();
for (GuiceInjectorService g : ServiceLoader.
load(GuiceInjectorService.class)) {
giss.add(g.getInjector());
}
return giss;
}
}
static Set<Injector> getInjectors() {
return new LinkedHashSet<>(Initializer.injectors);
}
...
It was an iterative process to produce the right module declarations. Better tooling could help in this regard to generate initial module declaration source from say maven with further analysis from class file dependencies. Guice is a logically and physically well-structured project which helped make this process easier.
Guice has a much richer binding model than Jigsaw modular services, as demonstrated by the hasher and rotter modules. However, the resolver knows nothing about Guice-based service interfaces and bindings. Ideally only the Java modules that bind required service interfaces should be resolved and included in the configuration of the application. To achieve that requires either the expression of a richer binding model in the module declaration or a more general way to express requirements and capabilities.
Finally the Guice modularization effort uncovered a bug in the jdk.logging module. The mapp module needs to require the jdk.logging module otherwise an annoying exception is logged:
Can't load log handler "java.util.logging.ConsoleHandler"
java.lang.ClassNotFoundException: java.util.logging.ConsoleHandler : requested by +mapp
java.lang.ClassNotFoundException: java.util.logging.ConsoleHandler : requested by +mapp
at org.openjdk.jigsaw.Loader.loadClass(Loader.java:116)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.util.logging.LogManager$3.run(LogManager.java:418)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:405)
...
at com.google.inject.internal.util.$FinalizableReferenceQueue.<init>(FinalizableReferenceQueue.java:131)
...
at com.google.inject.Guice.createInjector(Guice.java:62)
at hasher.HasherInjectorProvider.getInjector(HasherInjectorProvider.java:11)
...
at guice.service.GuiceServiceLoader.<init>(GuiceServiceLoader.java:72)
at guice.service.GuiceServiceLoader.load(GuiceServiceLoader.java:118)
at mapp.App.main(App.java:9)
The culprit is at line 418 of java.util.logging.LogManager:
Class<?> clz = ClassLoader.getSystemClassLoader()
.loadClass(word);
The system class loader, which is the class loader of the mapp module, is being used to load a class, java.util.logging.ConsoleHandler, in the jdk.logging module, but that class is not visible to the mapp module, hence the ClassNotFoundException is thrown and logged. This area is a strong candidate to be modified to support to modular services.
Modular services in Jigsaw define a very simple way to bind a (service) interface to implementations (service provider classes) and, using java.util.ServiceLoader, a way to iterate through all (service) instances of implementations bound to an interface.
Guice supports a rich binding model and scoping model. Guice can support a JDK services style approach using multibindings where service instances are obtained by referencing an injection type Set<T> where T is the service interface. (Note that when instance of the Set<T> is injected all members are instantiated according to the scoping rules; the set is not lazy.)
Can Jigsaw and Guice be combined for richer modular services? Perhaps :-)
Before we can start playing around with Jigsaw and Guice we first need to modularize Guice, so this is also an experiment in taking a popular framework and modularizing it to work in the modular OpenJDK.
The complete set of code can be found here on GitHub.
A modular Guice
The set of jar files that comprise of the Guice jar files and it's runtime dependent jar files need to be converted into modular jar files so that those modular jar files can be installed into a library.A modular jar file is a jar file that contains a compiled module description, module-info.class, at the top-level. The module-info.class is the result of compiling the module declaration source, module-info.java. The module declaration declares important information such as the a module name, version, dependent modules, and what types are exported.
The approach to work out that set of jar files was to manually analyze the runtime dependencies of Guice modules on maven central (com.google.inject:guice:3.0 and com.google.inject.extensions:guice-multibindings:3.0) to obtain the complete set of jar files required for for Guice to run.
For each jar a corresponding source module was created that contained just a module-info.java. The Java compiler, javac, was used to compile all the source modules to obtain corresponding module-info.class files.
Finally each jar file was downloaded from maven central and updated to include the corresponding module-info.class, thus transforming each jar into a modular jar.
The ant script which automates all this can be found here.
I have no doubt this can be further automated. Alan Bateman showed of a demo at JavaOne last year that processed maven pom files, slurped stuff from maven central, and automatically created modular jars from analyzing the dependencies.
This black box approach does not work in my case as i need tweak the module declarations (as explained later in this section). Some sort of automated white-box approach is required where an initial module-info.java can be created from which it can then be modified.
There is also some particularly thorny issues lurking here. Maven and Jigsaw have different dependency resolution algorithms. Maven, using dependency management, allows a pom to override versions of it's transitive dependencies. Jigsaw currently does not support this (it's an open question as to whether it will). Thus anything produced from such automation may not be reusable in other contexts since the dependency graphs might be different.
A total of four jars were identified:
- aopalliance-1.0.jar
- javax.inject-1.jar
- guice-3.0.jar
- guice-multibindings-3.0.jar
$ find msrcThe module declaration for the aopalliance module is:
msrc
msrc/aopalliance
msrc/aopalliance/module-info.java
msrc/guice
msrc/guice/module-info.java
msrc/guice.multibindings
msrc/guice.multibindings/module-info.java
msrc/javax.inject
msrc/javax.inject/module-info.java
module aopalliance@1.0 {
exports org.aopalliance.aop;
exports org.aopalliance.intercept;
}
The module declaration for the javax.inject module is:
module javax.inject@1 {
exports javax.inject;
}
The module declaration for the guice module is:
module guice@3.0 {
requires public javax.inject@1;
requires public aopalliance@1.0;
requires jdk.logging;
exports com.google.inject;
exports com.google.inject.binder;
exports com.google.inject.matcher;
exports com.google.inject.name;
exports com.google.inject.spi;
exports com.google.inject.util;
view guice.internal {
permits guice.multibindings;
exports com.google.inject.internal;
exports com.google.inject.internal.util;
}
}
Now it gets a little bit more interesting. First the guice module requires (or depends on) the javax.inject and aopalliance modules. Those dependencies are qualified as public so that any module requiring the guice module will automatically have access to the types exported by the other two modules; the types in those modules are re-exported.
Note that i am not sure if types in the aopalliance module are referenced in the Guice API, if not there is no need for the types in that module to be re-exported. Certainly types in the javax.inject module are referenced in the Guice API.
Guice requires the jdk.logging module but even though some types from this module are bound, by default, by Guice no such types are (AFAICT) exposed publicly hence the types of this module are not re-exported.
Guice takes particular care of placing all implementation classes underneath the package com.google.inject.internal. The expectation is that developers should not directly reference any such classes that are publicly accessible. Where possible such implementation classes are package private. This made it easy produce a list of exports clauses for the public API. More importantly with Java modularity the developers need not know anything about the internal classes and the module developer can be assured that at compilation and runtime such non-exported types are not visible and accessible.
It might be reasonable to suggest why not allow wildcards in exports clauses?, for example "exports com.google.inject.*;". In this case wildcards would be not be a good idea because types in the internal packages would be exported. In general the use of wildcards makes it all too easy to expose more than necessary and unintentionally, for example if packages are renamed.
However, there is a little problem. The exported types are not sufficient for other Guice modules to function, such as the guice.multibindings module. The Guice multibinding code imports types from the packages com.google.inject.internal and com.google.inject.internal.util:
$ grep import *.java | grep internalSo there is some tighter coupling between the Guice modules. It is necessary to create a non-default module view, guice.internal (the default view is the module guice), that exports types from the internal packages but only permits the guice.multibindings module to require this module view. Non-default views are never versioned and inherit the version, export and requires clauses from the default view.
MapBinder.java:import com.google.inject.internal.util.ImmutableList;
MapBinder.java:import com.google.inject.internal.util.ImmutableMap;
MapBinder.java:import com.google.inject.internal.util.ImmutableSet;
MapBinder.java:import com.google.inject.internal.util.Lists;
Multibinder.java:import com.google.inject.internal.Annotations;
Multibinder.java:import com.google.inject.internal.Errors;
Multibinder.java:import com.google.inject.internal.util.ImmutableList;
Multibinder.java:import com.google.inject.internal.util.ImmutableSet;
Multibinder.java:import com.google.inject.internal.util.Lists;
The module declaration for the guice.multibindings module is:
module guice.multibindings@3.0 {
requires public guice@3.0;
requires guice.internal@3.0;
exports com.google.inject.multibindings;
}
I had to go through a couple of iterations of editing the module declarations, producing the modular jars, installing those into a library, and running a test application. The perils of modularizing from jars files rather than compiling the source code!
Without the view and permits clauses a dreaded NoClassDefFoundError was thrown:
Exception in thread "main" java.lang.NoClassDefFoundError: Lcom/google/inject/internal/util/$ImmutableList;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2318)
at java.lang.Class.getDeclaredFields(Class.java:1771)
at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:649)
...
at com.google.inject.multibindings.Multibinder$RealMultibinder.configure(Multibinder.java:269)
...
at com.google.inject.Guice.createInjector(Guice.java:95)
...
at mapp.App.main(App.java:9)
Caused by: java.lang.ClassNotFoundException: com.google.inject.internal.util.$ImmutableList : requested by +guice.multibindings
at org.openjdk.jigsaw.Loader.loadClass(Loader.java:116)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
So Guice is doing some reflection accessing fields of some type in the guice.multibindings module that has a field whose type is in the internal package of the guice module, see here. Note that the Guice build process re-names certain class files e.g. ...util.ImmutableList to ...util.$ImmutableList. There is enough information here to guess what is going on but the indirection can throw one slightly off course.
Tooling analyzing class files and presenting class dependencies would be most helpful.
In the example on github the modular jars are installed into a guice-specific library. The application library (containing the installed application modules using Guice) is hooked up to the guice-specific library. This is primarily to avoid doing too much work when building the application library but it does show that libraries can be reusable (although not necessarily portable) components.
Services and Guice
The project that experiments with services and Guice can be loaded into NetBeans just like the modular services example.There are five modules that are compiled and installed into the application library:
- The guice.service module that exports the GuiceInjectorService service interface and the GuiceServiceLoader for obtaining service instances bound using Guice.
- The stringer module that exports the StringTransformer service interface.
- The hasher and rotter modules that both provide a service provider class for the GuiceInjectorService service interface and implement a Guice module that binds the StringTransformer service interface to one or more implementations.
- The mapp module that provides the main entry point that looks up StringTransformer instances using GuiceServiceLoader.
Essentially Jigsaw modular services is used to bootstrap Guice services.
Each module that participates in Guice services provides an implementation of GuiceInjectorService service interface:
import com.google.inject.Injector;
public interface GuiceInjectorService {
Injector getInjector();
}
which supports the getting of a Guice Injector instance. java.util.ServiceLoader can be used to obtain all Injector instances, and then the bindings of those instances can be introspected.
An alternative solution might be to obtain Guice Module instances then create one Injector for all those instances. However, since the Java modules are decoupled from each other it cannot be guaranteed that no binding conflicts will arise.
The GuiceInjectorService implementation and the Guice module for the rotter module are as follows:
public class RotterInjectorProvider implements GuiceInjectorService {
@Override
public Injector getInjector() {
return Guice.createInjector(new RotterModule());
}
}
public class RotterModule extends AbstractModule {
@Override
protected void configure() {
Multibinder<StringTransformer> uriBinder = Multibinder.
newSetBinder(binder(), StringTransformer.class);
uriBinder.addBinding().to(RotterStringTransformer.class).
in(Scopes.SINGLETON);
}
}
The GuiceInjectorService implementation and the Guice module for the hasher module are as follows:
public class HasherInjectorProvider implements GuiceInjectorService {
@Override
public Injector getInjector() {
return Guice.createInjector(new HasherModule());
}
}
return Guice.createInjector(new HasherModule());
}
}
public class HasherModule extends AbstractModule {
@Override
protected void configure() {
bind(StringTransformer.class).
to(HasherStringTransformer.class).
to(HasherStringTransformer.class).
in(Scopes.NO_SCOPE);
}
@Provides
@Named("ToLowerCaseHasher")
@Singleton
protected StringTransformer toLowerCase(
final StringTransformer st) {
final StringTransformer st) {
return new StringTransformer() {
@Override
public String description() {
return "LowerCase: " + st.description();
}
@Override
public String transform(String s) {
return st.transform(s).toLowerCase();
}
};
}
}
This Guice module registers two bindings of the StringTransform service interface. The first uses the programmatic binding. The latter uses the annotation-based binding to adapt the former and convert the String to lower case. To differentiate the later from the former so it can be bound the later is named.
The main class in the mapp module is as follows:
public class App {
public static void main(String[] args) throws Exception {
for (StringTransformer s : GuiceServiceLoader.
load(StringTransformer.class)) {
System.out.println(s + " " + s.description());
}
for (StringTransformer s : GuiceServiceLoader.
load(StringTransformer.class)) {
System.out.println(s + " " + s.description());
}
}
}
It just iterates through the StringTransformer service instances using the GuiceServiceLoader much like one can do with ServiceLoader.
So how does GuiceServiceLoader work? The Injector instances are lazily cached as follows:
public class GuiceServiceLoader<T> implements Iterable<T> {
static class Initializer {
private static Set<Injector> injectors = load();
private static Set<Injector> load() {
final Set<Injector> giss = new LinkedHashSet<>();
for (GuiceInjectorService g : ServiceLoader.
load(GuiceInjectorService.class)) {
giss.add(g.getInjector());
}
return giss;
}
}
static Set<Injector> getInjectors() {
return new LinkedHashSet<>(Initializer.injectors);
}
...
}
When an instance of GuiceServiceLoader is created for a given service interface the bindings of each Injector is introspected to find the list of all Guice Key instances whose type literal is equal to the service interface.
When the GuiceServiceLoader instance is iterated over the Keys are used to get the instances from the corresponding Injector. It's reasonably straightforward although i complicated the implementation with some additional caching.
When the GuiceServiceLoader instance is iterated over the Keys are used to get the instances from the corresponding Injector. It's reasonably straightforward although i complicated the implementation with some additional caching.
Observations
I have used javac to compile modular source for module declarations that export types in packages that don't exist in that source. This may seem a little odd but package names are sort of an artificial concept when it comes to compilation and runtime. So to me that seems reasonable.It was an iterative process to produce the right module declarations. Better tooling could help in this regard to generate initial module declaration source from say maven with further analysis from class file dependencies. Guice is a logically and physically well-structured project which helped make this process easier.
Guice has a much richer binding model than Jigsaw modular services, as demonstrated by the hasher and rotter modules. However, the resolver knows nothing about Guice-based service interfaces and bindings. Ideally only the Java modules that bind required service interfaces should be resolved and included in the configuration of the application. To achieve that requires either the expression of a richer binding model in the module declaration or a more general way to express requirements and capabilities.
Finally the Guice modularization effort uncovered a bug in the jdk.logging module. The mapp module needs to require the jdk.logging module otherwise an annoying exception is logged:
Can't load log handler "java.util.logging.ConsoleHandler"
java.lang.ClassNotFoundException: java.util.logging.ConsoleHandler : requested by +mapp
java.lang.ClassNotFoundException: java.util.logging.ConsoleHandler : requested by +mapp
at org.openjdk.jigsaw.Loader.loadClass(Loader.java:116)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.util.logging.LogManager$3.run(LogManager.java:418)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:405)
...
at com.google.inject.internal.util.$FinalizableReferenceQueue.<init>(FinalizableReferenceQueue.java:131)
...
at com.google.inject.Guice.createInjector(Guice.java:62)
at hasher.HasherInjectorProvider.getInjector(HasherInjectorProvider.java:11)
...
at guice.service.GuiceServiceLoader.<init>(GuiceServiceLoader.java:72)
at guice.service.GuiceServiceLoader.load(GuiceServiceLoader.java:118)
at mapp.App.main(App.java:9)
The culprit is at line 418 of java.util.logging.LogManager:
Class<?> clz = ClassLoader.getSystemClassLoader()
.loadClass(word);
The system class loader, which is the class loader of the mapp module, is being used to load a class, java.util.logging.ConsoleHandler, in the jdk.logging module, but that class is not visible to the mapp module, hence the ClassNotFoundException is thrown and logged. This area is a strong candidate to be modified to support to modular services.
Friday, June 15, 2012
Modular services with OpenJDK Jigsaw
Services are a simple but effective way to decouple interface and implementation.
Services in the classpath universe
The class java.util.ServiceLoader was introduced in Java SE 6 and formalized a pattern that many developers were already implementing prior to SE 6 (especially for JSR implementations).ServiceLoader provides a simple way to bind a Java interface, a service interface, to an instance of a Java class, a service provider class, that implements the interface. Such classes are declared in files located in the META-INF/services directory, where the file name is the fully qualified name of the service interface. The contents of a META-INF/services file contains one or more lines, each line of which declares the fully qualified class name of a service provider class implementing the service interface.
Using ServiceLoader one can lazily iterate over all service instances. All META-INF/services files visible to the class loader, that is used to get those the files (see ClassLoader.getResources), are parsed, service provider classes declared in those files are loaded, and then those classes are instantiated.
It's a neat little class that helps decouple interface and implementation.
That is not to say there are no problems with it.
The Java compiler knows nothing about META-INF/services files, so, unless an IDE groks those files and helps the developer, runtime errors can sneak in, for example if the service interface and/or a service provider class name does not exist, perhaps there was a spelling mistake, perhaps a class was renamed or moved to another package, or a service provider class does not implement the service interface.
A service provider class may have dependencies on third party libraries. The developer has to ensure the classpath is set up correctly, otherwise be prepared for errors such as NoClassDefFoundError.
Bundling multiple jar files into one uber jar may result in missing service provider classes if two or more jars contain the same META-INF/services files.
For the latter two issues maven is your friend. It can ensure the classpath is set up correctly. The maven shade plugin is smart enough to combine META-INF/services files.
Services in the modular universe
The Java multiverse is expanding with OpenJDK Jigsaw to embrace the modular universe.Developers can choose to stay in the classpath universe or take a quantum leap into the modular universe, where the classpath no longer exists, and modularity become a first class citizen of the Java language, compilation and runtime.
It is important to stress that the JDK has always maintained backwards compatibility between releases, nothing is taken away from the developer, thus the classpath universe will not collapse into a singularity, no impending entropy death is to be anticipated.
In the modular universe less can be more, since the JDK itself is being modularized using the module system. Smaller JDK installations are possible (no need for CORBA?, fine with me!).
Services too become a first class citizen of the Java language, compilation and runtime. No more META-INF/services files!
Recently i have been delving into the design and implementation of services in Jigsaw. While there is an implementation in place it is recognized as a temporary solution to get something mostly working with the JDK usage of ServiceLoader.
The Jigsaw team have come up with an alternative cleaner approach that works well in modular universe. For more information see this email and corresponding presentation. However, in this blog i don't want to get into the details of that email and instead want to describe the basics of how to use services in the modular universe (the presentation is helpful in that respect and i will be reusing terms defined in that presentation).
I have have pushed a simple example to GitHub. If you have available a recent build of Jigsaw then it is possible to compile and execute this example. (See here for a recent developer preview or if you are on the Mac you can also select more recent builds created and uploaded by Henri Gomez.)
This example consists of four modules all within the same src directory:
- A service interface module, mstringer, exporting a service interface for transforming strings
- Two service provider modules, mhasher and mrotter, that declare service provider classes that produce an MD5 checksum and a ROT13 transformation of a string respectively; and
- A service consume module, mapp, that creates service instances (using ServiceLoader) and transforms strings.
The module declaration for the service interface module mstringer is:
module mstringer@1.0 {
exports stringer;
}
This module declaration declares that all publicly accessible Java classes in the stringer package of module mstringer are visible and accessible to any code in a dependent module. There is only one Java class which is the service interface:
module mstringer@1.0 {
exports stringer;
}
This module declaration declares that all publicly accessible Java classes in the stringer package of module mstringer are visible and accessible to any code in a dependent module. There is only one Java class which is the service interface:
package stringer;
public interface StringTransformer {
String description();
String transform(String s);
}
Note that there is nothing specific to this module that says it has anything to do with services. Any non-final Java class can become a service interface.
The module declaration for the service provider module mrotter is:
module mrotter@1.0 {
requires mstringer;
provides service stringer.StringTransformer
with rotter.RotterStringTransformer;
}
This module declaration declares a dependency on the mstringer module since it's service provider classe will implement the service interface, stringer.StringTransformer.
module mrotter@1.0 {
requires mstringer;
provides service stringer.StringTransformer
with rotter.RotterStringTransformer;
}
This module declaration declares a dependency on the mstringer module since it's service provider classe will implement the service interface, stringer.StringTransformer.
Now we get into some service specifics. This module provides a service provider class rotter.RotterStringTransformer that implements the service interface stringer.StringTransformer. In effect the "provides <S> with <I>" is a very simple binding language: bind this interface to that implementation.
Note that the package rotter is never exported. This means that code in any other module cannot see and access the class RotterStringTransformer. In the modular universe service provider classes can remain private to the service provider module.
Also note that, although not present in this example, the module could require other modules that are needed for the implementation of the service interface.
Also note that, although not present in this example, the module could require other modules that are needed for the implementation of the service interface.
The module declaration for the service provider module mhasher is very similar.
Finally the module declaration for the service consumer module mapp is:
module mapp@1.0 {
requires mstringer;
requires service stringer.StringTransformer;
class app.Main;
}
This module declaration declares a dependency on the mstringer module since it will use service instances that implement stringer.StringTransformer.
module mapp@1.0 {
requires mstringer;
requires service stringer.StringTransformer;
class app.Main;
}
This module declaration declares a dependency on the mstringer module since it will use service instances that implement stringer.StringTransformer.
The module declares it requires the service interface with "requires <S>". This informs the module system to include any service provider modules that provide for the service interface, and most importantly any dependencies of those modules, in the dependency resolution and linking phases, which occur both at compile time and when modules are installed into a library.
The module also declares an entry point, a class with a static main(String[] args) method that is invoked when the module is executed.
Notice that service consumer module mapp knows nothing about the service provider modules mrotter and mhasher. They are decoupled. New services can be installed and old ones removed without necessarily affecting mapp.
See the execution code here. The class ServiceLoader is being used even though there are no longer any META-INF/services files. The approach that is being proposed in the email linked to previously ensures that the use of ServiceLoader in the modular universe is much easier to grok (hint: it does not matter what class loader is used as long as it is a module class loader).
Since javac indirectly knows about services (by way of the module system) errors can be produced at compile time if, for example, a service provider class does not implement the corresponding service interface.
Hopefully this blog entry and example will help developers get started playing around with modular services and Jigsaw. I have deliberately avoided going into the details of compilation, installation and execution. Take a closer look at the ant build script to understand how the Java command line tools are being used.
What if a module could declare richer bindings to be consumed by dependent modules? Could Guice like modules and bindings be supported such that the module system could create appropriate injectors for consuming modules? If a module defines an entry point perhaps that could be instantiated by the dependency injection system thereby allowing for annotation-based injection out-of-the-box?
Lots of questions!
See the execution code here. The class ServiceLoader is being used even though there are no longer any META-INF/services files. The approach that is being proposed in the email linked to previously ensures that the use of ServiceLoader in the modular universe is much easier to grok (hint: it does not matter what class loader is used as long as it is a module class loader).
Since javac indirectly knows about services (by way of the module system) errors can be produced at compile time if, for example, a service provider class does not implement the corresponding service interface.
Hopefully this blog entry and example will help developers get started playing around with modular services and Jigsaw. I have deliberately avoided going into the details of compilation, installation and execution. Take a closer look at the ant build script to understand how the Java command line tools are being used.
Dependency injection in the modular universe
The modular service clauses and the use of ServiceLoader is really a very simple form binding and explicit dependency injection.What if a module could declare richer bindings to be consumed by dependent modules? Could Guice like modules and bindings be supported such that the module system could create appropriate injectors for consuming modules? If a module defines an entry point perhaps that could be instantiated by the dependency injection system thereby allowing for annotation-based injection out-of-the-box?
Lots of questions!
Wednesday, April 11, 2012
Building Jigsaw on Mac OS X natively
No sooner had I got Jigsaw builds working on the Mac, using Virtual Box, Michael McMahon updates the code base to build natively and then before you can blink Henri Gomez pushes out some jigsaw DMGs to install. Great! now it is even easier to play with Jigsaw on the Mac.
However, if like me you want to hack and build the source, here are some details.
The build dependencies are:
If there is a previous version of Xcode installed I recommend removing it:
The installation of Xcode does not install the command line tools, such as gcc, g++ and make. Such tools can be installed by running Xcode, selecting the Preferences dialog, selecting the Downloads tab, and clicking to install the Command Line Tools.
Finally, if OpenJDK 7u4 was installed before Xcode 4.2.3 was installed it necessary to reinstall the former (as stated here).
If you have the Oracle OpenJDK 7 distribution installed then set the environment variable ALT_BOOTDIR to:
While on the subject of building there is work going on to improve the build system, plus the build structure will at some point be modularized as part of the JDK modularization effort, thus making building separate parts easier. All in all this means building is gonna get easier and faster.
However, if like me you want to hack and build the source, here are some details.
The build dependencies are:
- OS X Lion 10.7.3
- Xcode 4.2.3
- OpenJDK 7u4 (or using Henri's distribution)
If there is a previous version of Xcode installed I recommend removing it:
sudo /Developer/Library/uninstall-devtools --mode=all
The latest version Xcode is a little saner and installs in one location. Then, switch Xcode to the latest installation:
sudo xcode-select -switch /Applications/Xcode.app/Contents/DeveloperSome of the JDK build scripts will use "xcode-select -print-path" to determine the path to the command line tools.
The installation of Xcode does not install the command line tools, such as gcc, g++ and make. Such tools can be installed by running Xcode, selecting the Preferences dialog, selecting the Downloads tab, and clicking to install the Command Line Tools.
Finally, if OpenJDK 7u4 was installed before Xcode 4.2.3 was installed it necessary to reinstall the former (as stated here).
If you have the Oracle OpenJDK 7 distribution installed then set the environment variable ALT_BOOTDIR to:
export ALT_BOOTDIR=/Library/Java/JavaVirtualMachines/1.7.0.jdk/Contents/HomeBuilding natively took about 22 minutes, approximately one third faster than on Ubuntu within VirtualBox. The gain was mostly for building hotspot (implying VirtualBox is not so efficient at managing multiple CPUs).
While on the subject of building there is work going on to improve the build system, plus the build structure will at some point be modularized as part of the JDK modularization effort, thus making building separate parts easier. All in all this means building is gonna get easier and faster.
Thursday, April 5, 2012
Building Jigsaw on Mac OS X using VirtualBox
Jigsaw is the OpenJDK project for modularity in the JDK.
Currently the easiest way I have found to get started with Jigsaw on a Mac is to check out and build the source on a virtual machine running Linux.
I boot-strapped from reading Julien's very useful blog entry on building Jigsaw. My experience was a little smoother.
I am using Mac OS X 10.7.3 with VirtualBox 4.1.10.
Create a new virtual machine with at least 1GB of memory and 16GB of disk space (space gets tight if the default 8GB is selected).
Download the Ubuntu 11.10 32 bit ISO and hook up the ISO to the CD drive of the virtual machine (see the Storage section of the settings dialog).
Start up the virtual machine, install Ubuntu, and update the OS to the latest packages.
The above steps should take about 40 to 50 minutes to complete, given a reasonable network connection.
Install OpenJDK 7 and it's build dependencies:
Install mercurial:
I can get this down to about 30 minutes by reconfiguring the virtual machine to have 4 CPUs (the same as the host machine) and setting the following environment variables:
Currently the easiest way I have found to get started with Jigsaw on a Mac is to check out and build the source on a virtual machine running Linux.
I boot-strapped from reading Julien's very useful blog entry on building Jigsaw. My experience was a little smoother.
I am using Mac OS X 10.7.3 with VirtualBox 4.1.10.
Create a virtual machine
First, the virtual machine needs configured and installed with an operating system.Create a new virtual machine with at least 1GB of memory and 16GB of disk space (space gets tight if the default 8GB is selected).
Download the Ubuntu 11.10 32 bit ISO and hook up the ISO to the CD drive of the virtual machine (see the Storage section of the settings dialog).
Start up the virtual machine, install Ubuntu, and update the OS to the latest packages.
The above steps should take about 40 to 50 minutes to complete, given a reasonable network connection.
Prepare the virtual machine
Next, the virtual machine needs to be prepared to checkout the Jigsaw source and build it.Install OpenJDK 7 and it's build dependencies:
sudo apt-get build-dep openjdk-7
sudo apt-get install openjdk-7-jdkJigsaw will be built using OpenJDK 7, commonly referenced in this context as the bootstrap JDK.
Install mercurial:
sudo apt-get mercurialThe Jigsaw repository (like that for other OpenJDK projects) uses the Mercurial distributed version control system.
Check out and build
Now the source can be checked out and built. Check out the source forest:hg clone http://hg.openjdk.java.net/jigsaw/jigsawand then execute:
cd jigsaw
bash get_sources.sh
to get all source trees in the forest i.e. this is a multi-project repository (i don't yet know if the forest extension can be utilized).
Set the following environment variables:
Set the following environment variables:
export LANG=C
export ALT_BOOTDIR=/usr/lib/jvm/java-7-openjdk-i386
export ALLOW_DOWNLOADS=true
The ALT_BOOTDIR variable declares the location of the bootstrap JDK that will be used to build Jigsaw. The ALLOW_DOWNLOADS variable ensures that source not within the repository, namely that for JAXP and JAX-WS, will be downloaded.
Then make Jigsaw:
make sanity
make all
On my machine it took about 50 minutes to build. The installation of Jigsaw is located in the directory build/linux-i586. Execution can be verified:
./build/linux-i586/bin/java -version
and the output should be something like:
openjdk version "1.8.0-internal"
OpenJDK Runtime Environment (build 1.8.0-internal-sandoz_2012_04_04_13_06-b00)
OpenJDK Client VM (build 23.0-b11, mixed mode)
export HOTSPOT_BUILD_JOBS=4
export NO_DOCS=trueThe next blog entry will explain how to compile, install and execute modules.
Wednesday, April 4, 2012
Java Boomerang
Friday 30th of March was my last day at CloudBees.
Monday 2nd of April was my first day back at Oracle. I have joined the Java Platform team (more on that later in another blog entry).
CloudBees is a great company full of great people and is on track to be a success disrupting the middleware market with the RUN@cloud, DEV@cloud, the integrated set of services, and Jenkins Enterprise. It's fantastic to see Jenkins and the community go from strength to strength. I wish the team the very best of luck. CloudBees is two years old today, Happy Birthday! Keep up the internal meme generation service!
However, personally, the time is not right for me to work at home in a startup.
So, it is with mixed feelings that I say I am sad about leaving CloudBees but also very excited about joining Oracle and the Java Platform team.
Monday 2nd of April was my first day back at Oracle. I have joined the Java Platform team (more on that later in another blog entry).
CloudBees is a great company full of great people and is on track to be a success disrupting the middleware market with the RUN@cloud, DEV@cloud, the integrated set of services, and Jenkins Enterprise. It's fantastic to see Jenkins and the community go from strength to strength. I wish the team the very best of luck. CloudBees is two years old today, Happy Birthday! Keep up the internal meme generation service!
However, personally, the time is not right for me to work at home in a startup.
So, it is with mixed feelings that I say I am sad about leaving CloudBees but also very excited about joining Oracle and the Java Platform team.
Subscribe to:
Posts (Atom)