AspectJ 1.8.12 - Faster Spring AOP

by Andy Clement (noreply@blogger.com) at October 21, 2017 05:02 AM

AspectJ 1.8.12 is now out. It is actually hot on the heels of 1.8.11. That previous release was really just about doing some Java 9 toleration in the 1.8 line. 1.8.12 includes some real changes that have been brewing for a while but were a bit too radical to rush into 1.8.11.

The aim here is to speed up Spring AOP - or really any system that consumes AspectJ like Spring does. Instead of just using the weaver as-is, Spring uses the pointcut parser and matcher independently of the weaver (unless LTW). Typically when used as a whole, the matching and weaving is all underpinned by type information parsed from class files. There are alternative ways to get that type information and Spring, when consuming just the parser and matcher, actually uses a Java reflection driven system.

What drove interest in doing something here is when Dave Syer from the Spring team recently created a series of benchmarks for checking the speed of Spring-AspectJ: https://github.com/dsyer/spring-boot-aspectj

Here is a run of those on 1.8.11 (on my macbook pro). For this set I actually added one extra case beyond the standard benchmark (the a20_100) case:


Benchmark (scale) Mode Cnt Score Error Units
StartupBenchmark.ltw N/A avgt 10 2.656 ~ 0.166 s/op
StartupBenchmark.ltw_100 N/A avgt 10 2.618 ~ 0.063 s/op
StartupBenchmark.spring v0_10 avgt 10 2.071 ~ 0.044 s/op
StartupBenchmark.spring v1_10 avgt 10 2.210 ~ 0.058 s/op
StartupBenchmark.spring v1_100 avgt 10 2.260 ~ 0.068 s/op
StartupBenchmark.spring v10_50 avgt 10 2.933 ~ 0.039 s/op
StartupBenchmark.spring v20_50 avgt 10 3.832 ~ 0.094 s/op
StartupBenchmark.spring v20_100 avgt 10 3.959 ~ 0.047 s/op
StartupBenchmark.spring a0_10 avgt 10 2.073 ~ 0.028 s/op
StartupBenchmark.spring a1_10 avgt 10 2.729 ~ 0.061 s/op
StartupBenchmark.spring a1_100 avgt 10 2.750 ~ 0.029 s/op
StartupBenchmark.spring a10_50 avgt 10 7.153 ~ 0.075 s/op
StartupBenchmark.spring a10_100 avgt 10 7.152 ~ 0.059 s/op
StartupBenchmark.spring a20_50 avgt 10 11.430 ~ 0.105 s/op
StartupBenchmark.spring a20_100 avgt 10 11.497 ~ 0.162 s/op

This is the average *startup time* of an app affected by aspects applying to the beans involved. So this is not about the impact of AOP when the system is up and running, just about the impact of parsing pointcuts and matching join points on the startup of the application.

Where numbers are referenced above the first is the number of aspects/pointcuts (there is one pointcut per aspect) and the second is the number of beans. The 'a' indicates an annotation based pointcut vs a non-annotation based pointcut ('v'). It is pretty obvious that the performance of annotation based pointcuts is not great - it scales really badly. At 20 pointcuts and 50 beans the app is 9 seconds slower to startup!

The key change in 1.8.12 (and 1.9.0) is to recognize that it seems much more common to use annotations with runtime retention than those with just class retention, at least in most situations that I've seen (and the benchmarks here). Retrieving annotations with class retention is costly because we must open the bytes for the class file and dig around in there, as opposed to those with runtime retention which are easily accessible simply using Java reflection.

Now, if you need all the annotations, we must do that digging but there are situations in matching where we know exactly the annotation we are looking for - and if we know that, we know whether it has class or runtime retention. Until now we haven't been making this distinction when looking for them and always fetching them all. In 1.8.12 we are smarter and avoid having to locate and parse the class file unless necessary. There is also some deeper work in 1.8.12 related to activating caches that were not previously being used correctly but the primary change is factoring in the annotation retention.

What difference does that make? AspectJ 1.8.12:


Benchmark (scale) Mode Cnt Score Error Units
StartupBenchmark.ltw N/A avgt 10 2.620 ~ 0.130 s/op
StartupBenchmark.ltw_100 N/A avgt 10 2.567 ~ 0.038 s/op
StartupBenchmark.spring v0_10 avgt 10 2.044 ~ 0.027 s/op
StartupBenchmark.spring v1_10 avgt 10 2.195 ~ 0.026 s/op
StartupBenchmark.spring v1_100 avgt 10 2.237 ~ 0.039 s/op
StartupBenchmark.spring v10_50 avgt 10 2.774 ~ 0.038 s/op
StartupBenchmark.spring v20_50 avgt 10 3.488 ~ 0.116 s/op
StartupBenchmark.spring v20_100 avgt 10 3.642 ~ 0.080 s/op
StartupBenchmark.spring a0_10 avgt 10 2.067 ~ 0.034 s/op
StartupBenchmark.spring a1_10 avgt 10 2.159 ~ 0.030 s/op
StartupBenchmark.spring a1_100 avgt 10 2.207 ~ 0.020 s/op
StartupBenchmark.spring a10_50 avgt 10 2.471 ~ 0.031 s/op
StartupBenchmark.spring a10_100 avgt 10 2.517 ~ 0.045 s/op
StartupBenchmark.spring a20_50 avgt 10 2.842 ~ 0.049 s/op
StartupBenchmark.spring a20_100 avgt 10 2.916 ~ 0.145 s/op

Look at the a20_100 case - instead of impacting start time by 9 seconds, it impacts it by 1 second.

In fact those numbers look too good to be true... which worries me a bit! So if you are reading this and using Spring AOP, please kick the tires on this 1.8.12 release and raise any issues on our Bugzilla.

Barring any unexpected regressions this will be the final release of 1.8.12 and 1.9.0 is next.


by Andy Clement (noreply@blogger.com) at October 21, 2017 05:02 AM

A rose by any other name…

by Donald Raab at October 21, 2017 04:03 AM

Similar iteration patterns, different naming patterns.

A Rosetta Stone for Smalltalk, Eclipse Collections and Java 8 Streams

I used this slide at JavaOne 2017 in a talk titled API Deep Dive: Designing Eclipse Collections [CON6133] which I presented with Nikhil Nanivadekar. This is a translation guide for iteration pattern names between Smalltalk, Eclipse Collections and Java 8 Streams.

As I pointed out in the talk, I chose Smalltalk as the source of inspiration for the names of the basic iteration patterns in Eclipse Collections.

If you know Smalltalk, learning Eclipse Collections just requires you to learn the syntax and types of the Java language. You will already be familiar with the iteration patterns. If you know Java 8 Streams, learning Eclipse Collections will require you to add nine synonyms to your vocabulary for common iteration patterns. Three of these patterns start with the same prefix (any/all/none), and differ only in the suffix (Satisfy vs. Match). That’s an easy place to get started. Now you only have to learn six new names to become as fluent in the basic iteration patterns of Eclipse Collections as you are in Java 8 Streams. Eclipse Collections types allow you to use the API directly on collections without having to call the “bun” operations (e.g. .stream().something().collect()). You can also use the Streams API if you want to with Eclipse Collections types. You get to choose whichever option you like best.

Let’s learn the iteration pattern synonyms by example. The following examples will be based on the Eclipse Collections Pet Kata domain.

Select / Filter

If you want to include items of a collection that satisfy a given condition (Predicate) you use select (EC) or filter (Streams). Eclipse Collections also has “With” versions of the iteration patterns that allow for more usages of method references.

// Eclipse Collections (Select)
MutableList<Person> peopleWithCats =
this.people.select(person -> person.hasPet(PetType.CAT));
// Eclipse Collections (SelectWith) 
MutableList<Person> peopleWithCats =
this.people.selectWith(Person::hasPet, PetType.CAT);
// Streams (Filter)
List<Person> peopleWithCats =
this.people.stream()
.filter(person -> person.hasPet(PetType.CAT))
.collect(Collectors.toList());

Reject / Filter (!)

If you want to exclude items of a collection that satisfy a given condition (Predicate) you use reject (EC) or filter (Streams) with a negation of the predicate.

// Eclipse Collections (Reject)
MutableList<Person> peopleWithoutCats =
this.people.reject(person -> person.hasPet(PetType.CAT));
// Eclipse Collections (RejectWith) 
MutableList<Person> peopleWithoutCats =
this.people.rejectWith(Person::hasPet, PetType.CAT);
// Streams (Filter !)
List<Person> peopleWithoutCats =
this.people.stream()
.filter(person -> !person.hasPet(PetType.CAT))
.collect(Collectors.toList());

Collect / Map

If you want to transform a collection of one type to a collection of another type, you use collect (EC) or map (Streams) with a Function. There is a method named collect on Streams as well, but it means something different. It is referred to as a mutable reduction. What is reduces to is up to you, so the return type of the Stream version of collect is defined by the Collector you give it. I wish the name of the method was different (aka. mutableReduce). C’est la vie.

// Eclipse Collections
MutableList<String> firstNames =
this.people.collect(Person::getFirstName);
// Streams
MutableList<String> firstNames =
this.people.stream()
.map(Person::getFirstName)
.collect(Collectors2.toList());

I used Collectors2 in the above example to show you that you can also return Eclipse Collections types using Streams if you want to as well.

FlatCollect / FlatMap

If you want to transform a collection of collections to a single collection of another type, you use flatCollect (EC) or flatMap (Streams) with a Function that either returns an Iterable (EC) or a Stream (Streams).

// Eclipse Collections (FlatCollect - Eager)
MutableSet<PetType> petTypes =
this.people.flatCollect(Person::getPetTypes)
.toSet();
// Eclipse Collections (FlatCollect - Lazy)
MutableSet<PetType> petTypes =
this.people.asLazy()
.flatCollect(Person::getPetTypes)
.toSet();
// Streams
MutableSet<PetType> petTypes =
this.people.stream()
.flatMap(person -> person.getPetTypes().stream())
.collect(Collectors2.toSet());

In this example, I also show that you can choose either Eclipse Collections eager API (EC is eager by default), or you can save creating a temporary collection by using the lazy API. The asLazy() call will return a LazyIterable which has the same protocols available. The return type for flatCollect will change from MutableList in the first example to LazyIterable in the second. A LazyIterable is reusable (it is an Iterable), unlike a Stream which can only be used once (a Stream is like an Iterator).

Detect / Filter + FindFirst + Get

If you want to return the first element of a collection which matches a given condition (Predicate), you use detect (EC) or filter + findFirst + get (Streams). If you want to return an optional value representing either the first item that matches the condition or null, you can use detectOptional (EC) or filter + findFirst (Streams).

// Eclipse Collections (DetectWith)
Person firstPersonWithCat =
this.people.detectWith(Person::hasPet, PetType.CAT);
// Eclipse Collection (DetectWithOptional)
Optional<Person> optionalFirstPerson =
this.people.detectWithOptional(Person::hasPet, PetType.CAT);
// Streams (Filter + FindFirst + Get)
Person firstPersonWithCat =
this.people.stream()
.filter(person -> person.hasPet(PetType.CAT))
.findFirst();
.get();
// Streams (Filter + FindFirst)
Optional<Person> personOptional =
this.people.stream()
.filter(person -> person.hasPet(PetType.CAT))
.findFirst();

InjectInto / Reduce

If you want to inject an initial value into a function that then injects the result of each application of the function into the next iteration, you can use injectInto (EC) or reduce (Streams). This is probably the hardest method to understand, but has the easiest examples. You can use this method to calculate sum, min, max, product, etc.

// Eclipse Collections (InjectInto)
int numberOfPets =
this.people.collectInt(Person::getNumberOfPets)
.injectInto(0, Integer::sum);
// Streams (Reduce)
int numberOfPets =
this.people.stream()
.mapToInt(Person::getNumberOfPets)
.reduce(0, Integer::sum);

I hope this translation guide of basic iteration patterns will be helpful for developers who want to learn Eclipse Collections and are either familiar with the names used with Java 8 Streams or the names used with Smalltalk.


by Donald Raab at October 21, 2017 04:03 AM

Eclipse Xtext 2.13 is released!

by Karsten Thoms (thoms@itemis.de) at October 20, 2017 04:04 PM

Right on time for EclipseCon Europe the Xtext project managed to release its new version 2.13. Xtext 2.13 is the first version for the Eclipse Photon simultaneous release and was designed to fit well with the recently released Eclipse Oxygen.1a version. For users of older Eclipse versions, Xtext remains compatible back to Eclipse Luna.

whats-new.jpg

Xtext's New Project Wizard: Option to build Language Server Support

We have enhanced Xtext’s New Project Wizard, which now adds an option to build Language Server packages for languages created with Xtext. The wizard will enhance the build scripts to produce the packages, which can be integrated with any tool that supports language integration through the Language Server Protocol (LSP), e.g. VS Code, Eclipse Che and others.

Xtext-Release-new-project-language-server-protocol.png

Users can choose between a Fat Jar packaging, which packages the DSL and all required dependencies into a single executable jar, or a “Regular” format, which will have each jar on its own and includes launch scripts for the language server. A basic language server package is around 13 MB size, more depends on a DSLs individual dependencies.

The full support of the LSP is still on top of Xtext’s agenda. With this release will finally get support for rename refactoring.

Xtext 2.13: Quality, stability, robustness

The processed bugs of a release are often just a link to a list in the release notes. But it should be worth to mention that this is the most important and least thankful job to keep a project alive and kicking.

federico-tomassetti-quote-twitter.png

 

It is especially Christian Dietrich’s invaluable work that is addressing the problems that real users have out there, on the support forum and by fixing bugs. Also I am continuously working down the “big pile of shit”. That might not be the fun work, but it is crucial for the project’s health.



We have focussed also on issues that are reported to Eclipse’s
Automatic Error Reporting System (AERI), where actual problems are reported by users automatically. You might have noticed in Eclipse this feature when a pop up message appears in Eclipse’s lower right corner and asks you to send to report when something goes wrong. These reports are very important to us, and we are analyzing them continuously.

With this release we finally managed to reach a break even point where we don’t get more reports inside than we are able to process. And for some weeks we are even not getting new requests, and fight against the remaining ones.

As of today, Xtext has only about 10% of the reports open, and I believe this is a top quota. I believe that Xtext is amongst the most active projects using AERI at Eclipse.

Xtext-problem-status-report.png

New Xbase Library Functions

A new flatMap() function has been added to Xtext’s library bundle org.eclipse.xtext.xbase.lib. It combines map with flatten, a likely combination in functional programming.

Further mutable collection literals finally get no-args factory methods in class CollectionLiterals. Usage of these methods yield better performance, and Xtext’s code has been refactored to use these methods.

New Model Manipulation API

Synchronization of text with the EMF resources is one of the magic things Xtext is doing for you without usually noticing it. For advanced users who need to influence the process of changes to the EMF model, there is now a new API. Its core concept is the IChangeSerializer interface.

This API is still in incubation status, so we expect that it might change later. For the rare case that you might need this API, please give feedback so that we can improve it.

The road to Eclipse Photon

Xtext 2.13 is our first milestone towards the Eclipse Photon simultaneous release. As of today we plan to deliver Xtext 2.15 for this release in June 2018, so there is likely another release in between.

Our focus at itemis will be on language server, Java 9, usability and performance. We have found first issues with Java 9 and are still getting more insights with it, but could not solve all of them for 2.13 already. We warmly encourage you to provide feedback or even contributions. To ease your contribution, we polished our Oomph Setup, so that setting up a fully working dev workspace is a matter of coffee drinking.

As Xtext is a non-trivial framework, so it is important to us to lower the hurdle to use it as much as possible. We expect to add more features for build, Xtext’s UI (e.g. wizards), and documentation. Therefore we listen to requirements users have or what we need in concrete projects. Also here, your input is valuable to us.

Xtext @ EclipseCon Europe 2017

The complete itemis Xtext team will be at EclipseCon Europe in Ludwigsburg next week. All of us have multiple talks there, and some are Xtext related. If you can make it to ECE 2017, make sure to come to our talks. We will often hang around the itemis booth in front of the Theater, and would happily talk with you about your experience and ideas on Xtext. It is YOU that can shape the future of Xtext! We will help you wherever we can.

TL;DR

Xtext 2.13 was released on October 20th 2017. The release’s main new features are enhanced Language Server Support, additional library functions and an API for EMF model manipulation. With this release Xtext has improved much on stability again, and is one of the projects with highest AERI problem report fixing rate at Eclipse. Future focus for Eclipse Photon are language servers, usability and scalability.

Xtext 2.13 is available now on the release update site. Download now and enjoy! Read the full release notes here.


by Karsten Thoms (thoms@itemis.de) at October 20, 2017 04:04 PM

Xtext 2.13.0 Released: Semantic Editing Made Easy

by Moritz Eysholdt at October 20, 2017 03:58 PM

Today we are excited to release Xtext 2.13.0 and I would like to give you an opinionated tour through some of the new features. There is one area where particularly much happened: Refactoring support and QuickFix support. I wan to thank Jan and Dennis for implementing significant parts of it. Refactoring and QuickFix are similar: They take the DSL documents and turn them into some improved version of themselves. While refactoring focusses on restructuring, QuickFixes focus on resolving problems. The modifications applied to the DSL documents can range from very simple cases like updating an attribute’s value to complex cases such as renaming or moving multiple elements at once. To implement an operation like this, in the past, it was either necessary to obtain tokens from the node model and directly modify the text or obtain the AST and serialize the modified subtree. Unfortunately, both approaches have drawbacks:

  • Changing text directly, even with the help of the node model, is only practical for small changes and insertions. Your language’s formatter is not used and syntactic correctness of the change is your very own responsibility.
  • Neither approach can detect and update cross references that point to the modified piece of the text or AST.
  • The serialization approach is too fragile to work with broken models since it needs to re-serialize the full sub-tree of the modified element. Unfortunately, since we’re working with text that’s being edited by a human, broken models are more common than valid ones.
  • And more.

ChangeSerializer: Engine for Semantic Editing

So we spend lots of thought on how Xtext could provide hooks where all you need to do is to update your AST (the EMF model) and Xtext will take care of updating the textual DSL documents. The result is an amazingly simple service:

public interface IChangeSerializer {
  
  interface IModification<T extends Notifier> {
      void modify(T context);
  }
  
  <T extends Notifier> void addModification(T context, IModification<T> modification);
  void applyModifications(IAcceptor<IEmfResourceChange> acceptor);
  void setUpdateCrossReferences(boolean value);
  void setUpdateRelatedFiles(boolean value);
  
  // (...)
}

To use this service, first, you call addModification(context, modification) one or multiple times. The parameter context is the EMF Resource or EObject that (children included) you want to modify. The parameter modification is an object or a lambda that executes the modification.

When all modification have been added, it is time to call applyModifications(acceptor). With acceptor, you provide a call-back that will be called once for every EMF Resource that needs to be changed. For XtextResources, you’ll get instances of

public interface ITextDocumentChange extends IEmfResourceChange {
  URI getOldURI();
  URI getNewURI();
  XtextResource getResource();
  List<ITextReplacement> getReplacements();
}

As promised, this gives the all ITextReplacements that are needed to apply the model change back to the textual DSL documents. A lot is being considered during this process:

  • The changes are kept as minimal as possible. For example, when only the value of a single EAttribute is changed, only the EAttributes text region will be updated. This keeps serialization robust in times of broken models. To add new EObjects or to handle very complicated changes, the ChangeSerializer will delegate to ISerializer.
  • Modifications can occur in one or many files at the same time. This feature is important because often files are interconnected via cross references and therefore it can be impossible to treat the individually.
  • XtextResources and other EMF resources can be mixed. For example, XML or XMI resources can be involved.
  • Cross references are updated automatically when setUpdateCrossReferences(true) has been set. We build in this feature because we realized that more advanced modifications often involve many rename-refactorings.
  • Related files are automatically detected and updated, when setUpdateRelatedFiles(true) has been called. Related files are the ones for which there is no explicit modification, but which contain cross references that need to be updated. Related files are determined using Xtext’s index.
  • There is a hook to update the model of related files, for example to update a section of import statements.
  • Code comments are handled properly by being moved/deleted according to how the associated model elements are moved/deleted.
  • Transactional model modification: Since all changes are computed before they are applied, a modification can be aborted at any time withour having caused side-effects.
  • Your IFormatter2 is called for all model modifications. Thus, the resulting text changes are always nicely formatted according to how you implemented your formatter.

The IChangeSerializer is part of xtext.ide and therefore available on all platforms supported by Xtext: Language Server Protocol, Eclipse IDE, etc.

File/Folder Copy/Move/Rename Refactoring

In short: Resource Relocation Refactoring.

The motivation behind this refactoring is the fact that many languages support syntactical elements that directly relate to the structure of the file system:

  • Java-style package names: By definition symmetric to the file’s path within the source folder.
  • C-style import statements: They’re like cross references pointing to file names.

If the file structure is changed by renaming files or folders or by moving them around, the contents of one or more files needs to be updated. A perfect case for the change serializer! Also, when a file it copied it must have a different path/name compared to the original, thus making file contents changes necessary.

Implementing such a hook is spectacularly simple:

class MyStrategy implements IResourceRelocationStrategy {

  @Inject IResourceServiceProvider language

  override applyChange(ResourceRelocationContext context) {
    context.changes.filter[language.canHandle(fromURI)].forEach [ change |
      context.addModification(change) [ resource |
        val rootElement = resource.contents.head
        if (rootElement instanceof PackageDeclaration) {
          val newPackage = change.toURI.trimSegments(1).segmentsList.drop(2).join('.')
          rootElement.name = newPackage
        }
      ]
    ]
  }
}

Even though the hook’s implementation is language-specific, it receives the URIs of all changed files, independently of which language they belong to. This enables you to implement piggyback renaming, e.g. rename a diagram file when its model file has been renamed. Therefore, the first thing this example does is filter out all URIs of other languages via canHandle. Then it creates a modifcation lambda in which the AST is accessible and it simply updates the EAttibute name with the new package name. The ChangeSerializer will automatically detect which other files are referencing this files and update those accorrdingly.

Multi-Quickfixes

We also made available a new QuickFix API based on the ChangeSerializer. It has interesing capabilities:

  • Multi-QuickFixes are now supported by Xtext: As a user you can select multiple markers from the Eclipse Problems View and fix them all at once with a single action. It’s hard to over-state how awesome this is!
  • Cross-File QuickFixes: A single QuickFix can now update several files at once.
  • File rename/move simply by changing the URI in the EMF resource.
  • Automatic updating of affected cross references: Changing a model element’s name can now be a rename refactoring!

@Fix(MyValidator.ISSUE_CODE_1)
public void fix1(final Issue issue, IssueResolutionAcceptor acceptor) {
  acceptor.acceptMulti(issue, "Add Doc1", "Adds Documentation", null, 
    (MyElement myEObject) -> {
      myEObject.setDoc("Better documentation");
    }
  );
}

@Fix(MyValidator.ISSUE_CODE_2)
public void fix2(final Issue issue, IssueResolutionAcceptor acceptor) {
  acceptor.acceptMulti(issue, "Add Doc2", "Adds Documentation", null, 
    (MyElement myEObject, ICompositeModificationContext<MyElement> context) -> {
      ctx.addModification(main, (obj) -> {
        obj.setDoc("Better documentation");
      }
    );
  });
}

The API works just like the old one, except that that now you’ll need to call acceptor.acceptMulti() instead of acceptor.accept(). There are two flavors of the API:

  • The one with one parameter: myEObject: In the lambda, you may modify any EObject from the same resource. Cross references and related files will be updated.
  • The one with two parameter: myEObject and context. The context object gives you fine-grained control over the ChangeSerializer which is operating behind the scenes. AST-modifications are only allowed via ctx.addModification(). Since you tell ctx.addModification about the root element about the modification, elements from other resources can be modified and/or you can keep the recorded model-subtree as small as possible. Also, via the context object, you can enable/disable updating of cross references and related files.

The second QuickFix enables a pattern that’s important for Multi-Fixes: First resolve everything, then modify. For example when you’re storing EObject URIs in an issue’s user data, you’ll need to resolve them before the first modification is applied. Otherwise, URIs may become unresolvable because the model has changed. These two stages are automatically handled properly when you do the resolution outside of ctx.addModification() and the actual modification inside ctx.addModification‘s lambda.

Rename Element Refactoring

Rename refacoring for model elements has long been avilable in Eclipse Xtext. New in Xtext 2.13.0 is a flag to enable the ChangeSerializer to be the driving engine:

Workflow {
  component = XtextGenerator {
    ...
    language = StandardLanguage {
      ...
      renameRefactoring = {
        useChangeSerializer = true
      ...

And more!

Among the other changes there is:

  • Tons of bug-fixes, discussion, reviews. Thank you to all committers and Christian in particular.
  • The project creation wizard can now create projects that contain build scripts that assemble regular or fat JARs for Language Servers. Thx Karsten!
  • Improvements for indentation-based languages. Thank you Sebasitan!
  • new collection literals for empty collection and Iterables.flatMap() for xbase.lib (Thx Karsten!)

by Moritz Eysholdt at October 20, 2017 03:58 PM

The Future of Java EE, A Q&A with David Blevins: The Eclipse Foundation, EE4J and MicroProfile

by Daniel Bryant at October 20, 2017 12:00 PM

This year’s JavaOne saw many interesting announcements, and within the enterprise domain the most important news was that the Eclipse Foundation will become the new custodians of Java EE. InfoQ recently sat down with David Blevins, a long standing member of the Java EE community, and discussed what this move will mean, and how developers can get involved with “EE4J”, the future of Java EE.

By Daniel Bryant

by Daniel Bryant at October 20, 2017 12:00 PM

Vert.x 3.5.0 released !

by vietj at October 20, 2017 12:00 AM

The Vert.x team is pleased to announce the release of Vert.x 3.5.0.

As usual it delivers an impressive number of high quality features.

Let’s go RxJava2

First and foremost this release delivers the RxJava2 API with support of its full range of types.

In addition of Single, Rxified APIs also expose theCompletable and Maybe types:

// expose Handler<AsyncResult<Void>>
Completable completable = server.rxClose();

completable.subscribe(() -> System.out.println("closed"));

// expose Handler<AsyncResult<String>> where the result can be null
Maybe<String> ipAddress = dnsClient.rxLookup("www.google.com");
ipAddress.subscribe(
  value -> System.out.println("resolved to " + value),
  err -> err.printStackTrace(),
  () -> System.out.println("does not resolve"));

RxJava augments Vert.x streams with a toObservable() method, likewise RxJava2 adds the toFlowable() method:

// Flowable maps to a ReadStream<Buffer>
// back-pressured stream
Flowable flowable = asyncFile.toFlowable();

// but we still can get an Observable<Buffer>
// non back-pressured stream
Observable flowable = asyncFile.toObservable();

What’s so different between Flowable and Observable? the former handles back-pressure, i.e the subscriber can control the flow of items and the later can not!!!

You can read the documentation in the section of the docs or go straight to the examples

Kotlin coroutines

Support for Kotlin Coroutines is one of my favourite 3.5 features (by the way I’ll present a talk about Vert.x and coroutines at KotlinConf).

Coroutines allows you to reason about asynchronous flow the same way you do with traditional sequential flow with the extra bonus to use try/catch/finally super combo:

val movie = ctx.pathParam("id")
val rating = Integer.parseInt(ctx.queryParam("getRating")[0])
val connection = awaitResult<SQLConnection> { client.getConnection(it) }
try {
  val result = awaitResult<ResultSet> { connection.queryWithParams("SELECT TITLE FROM MOVIE WHERE ID=?", json { array(movie) }, it) }
  if (result.rows.size == 1) {
    awaitResult<UpdateResult> { connection.updateWithParams("INSERT INTO RATING (VALUE, MOVIE_ID) VALUES ?, ?", json { array(rating, movie) }, it) }
    ctx.response().setStatusCode(200).end()
  } else {
    ctx.response().setStatusCode(404).end()
  }
} finally {
  connection.close()
}

This example is borrowed from our examples.

NOTE: I’ve used try/finally purposely instead of Kotlin’s use extension method

MQTT Client

In Vert.x 3.4 we added the MQTT server, 3.5 completes the MQTT story with the MQTT client:

MqttClient mqttClient = MqttClient.create(vertx,
   new MqttClientOptions()
     .setPort(BROKER_PORT)
     .setHost(BROKER_HOST)).connect(ar ->
  if (ar.succeeded()) {
    System.out.println("Connected to a server");

    mqttClient.publish(
      MQTT_TOPIC,
      Buffer.buffer(MQTT_MESSAGE),
      MqttQoS.AT_MOST_ONCE,
      false,
      false,
      s -> mqttClient.disconnect(d -> System.out.println("Disconnected from server")));
  } else {
    System.out.println("Failed to connect to a server");
    ar.cause().printStackTrace();
  }
});

You can find MQTT client and server examples here

Web API contracts

With the new OpenAPI router factory we can focus on the API implementation and not on the validation of the input. The usage is quite simple:

OpenAPI3RouterFactory.createRouterFactoryFromFile(vertx, "petstore.yaml", ar -> {
  if (ar.succeeded()) {
    // Spec loaded with success
    OpenAPI3RouterFactory routerFactory = ar.result();

    // add your API and security handlers to the factory

    // add it to a server
    vertx.createHttpServer()
      .requestHandler(routerFactory.getRouter()::accept)
      .listen();
  } else {
    // Something went wrong during router factory initialization
  }
});

Now as a developer you only need to care about the API and not on the validation. The OpenAPI router will ensure that a request to an API will first to the contract before your handler is invoked.

Java 9 support

Java 9 was released a few days ago and the Vert.x stack has been carefully tested on Java 9 and most of our components run on Java 9 (Groovy does not run well on Java 9, please see the support matrix)

As a bonus you can now use HTTP/2 out of the box with JDK SSL!

You can also use Vert.x jars as anonymous modules.

Event driven JSON Parsing

We provide now an event driven JSON Parser emitting parse events that is very handy when you need to handle very large JSON structures and you don’t want to buffer it which introduce extra latency and increase the memory consumption.

The parser allows you to switch between fine grained JSON parse events or full structures, for instance you can parse an array of object very efficiently:

JsonParser parser = JsonParser.newParser();

// The parser will handle JSON objects as values
parser.objectValueMode();

parser.handler(event -> {
  switch (event.type()) {
    case START_ARRAY:
      // Start the array
      break;
    case END_ARRAY:
      // End the array
      break;
    case VALUE:
      // Handle each object
      break;
  }
});

Single SQL operations

Single SQL operations (aka one-shot) have been drastically simplified: most of the SQLOperations operations can now be performed directly on the SQLClient:

client.queryWithParams("SELECT AVG(VALUE) AS VALUE FROM RATING WHERE MOVIE_ID=?", new JsonArray().add(id), ar2 -> {
  if (ar.succeeded()) {
    int value = ar.result().get(0).getInteger("VALUE");
    // Continue
  }
});

Under the hood, the client takes care of the pool acquire/release interaction for you.

Native transport and domain sockets

We now support native transports on Linux (Epoll) and MacOS (KQueue), as well as UNIX domain sockets for NetServer/NetClient (HttpServer/HttpClient should support UNIX domain sockets soon).

Auth handler chaining

There are times when you want to support multiple authN/authZ mechanisms in a single application.

Vert.x Web supports auth handlers chaining

Vert.x config improvements

Vert.x Config allows configuring your application by assembling config chunks from different locations such as file, http, zookeeper…

In this version, we have added the support for Consul and Vault.

With the Consul config store, you can retrieve your configuration from a Consul server - so in other words, distribute the configuration from your orchestration infrastructure.

The Vault config store lets you retrieve secrets avoiding hard coding secrets or distributing credentials using an insecure way. Vault enforces the security of your secrets and only allowed applications can retrieve them. In other words, now you can keep your secrets secret.

ACKs

I want on behalf of the team to thank all the contributors for this release including the Google Summer of Code students (Pavel Drankov, Francesco Guardiani and Yunyu Lin) that delivered an impressive work.

Finally

The release notes

Docker images are also available on the Docker Hub. The Vert.x distribution is also available from SDKMan and HomeBrew.

The event bus client using the SockJS bridge are available from NPM, Bower and as a WebJar:

The artifacts have been deployed to Maven Central and you can get the distribution on Bintray.


by vietj at October 20, 2017 12:00 AM

Eclipse Vert.x 3.5.0 released !

by vietj at October 20, 2017 12:00 AM

The Vert.x team is pleased to announce the release of Eclipse Vert.x 3.5.0.

As usual it delivers an impressive number of high quality features.

Let’s go RxJava2

First and foremost this release delivers the RxJava2 API with support of its full range of types.

In addition of Single, Rxified APIs also expose theCompletable and Maybe types:

// expose Handler<AsyncResult<Void>>
Completable completable = server.rxClose();

completable.subscribe(() -> System.out.println("closed"));

// expose Handler<AsyncResult<String>> where the result can be null
Maybe<String> ipAddress = dnsClient.rxLookup("www.google.com");
ipAddress.subscribe(
  value -> System.out.println("resolved to " + value),
  err -> err.printStackTrace(),
  () -> System.out.println("does not resolve"));

RxJava augments Vert.x streams with a toObservable() method, likewise RxJava2 adds the toFlowable() method:

// Flowable maps to a ReadStream<Buffer>
// back-pressured stream
Flowable flowable = asyncFile.toFlowable();

// but we still can get an Observable<Buffer>
// non back-pressured stream
Observable flowable = asyncFile.toObservable();

What’s so different between Flowable and Observable? the former handles back-pressure, i.e the subscriber can control the flow of items and the later can not!!!

You can read the documentation in the section of the docs or go straight to the examples

Kotlin coroutines

Support for Kotlin Coroutines is one of my favourite 3.5 features (by the way I’ll present a talk about Vert.x and coroutines at KotlinConf).

Coroutines allows you to reason about asynchronous flow the same way you do with traditional sequential flow with the extra bonus to use try/catch/finally super combo:

val movie = ctx.pathParam("id")
val rating = Integer.parseInt(ctx.queryParam("getRating")[0])
val connection = awaitResult<SQLConnection> { client.getConnection(it) }
try {
  val result = awaitResult<ResultSet> { connection.queryWithParams("SELECT TITLE FROM MOVIE WHERE ID=?", json { array(movie) }, it) }
  if (result.rows.size == 1) {
    awaitResult<UpdateResult> { connection.updateWithParams("INSERT INTO RATING (VALUE, MOVIE_ID) VALUES ?, ?", json { array(rating, movie) }, it) }
    ctx.response().setStatusCode(200).end()
  } else {
    ctx.response().setStatusCode(404).end()
  }
} finally {
  connection.close()
}

This example is borrowed from our examples.

NOTE: I’ve used try/finally purposely instead of Kotlin’s use extension method

MQTT Client

In Vert.x 3.4 we added the MQTT server, 3.5 completes the MQTT story with the MQTT client:

MqttClient mqttClient = MqttClient.create(vertx,
   new MqttClientOptions()
     .setPort(BROKER_PORT)
     .setHost(BROKER_HOST)).connect(ar ->
  if (ar.succeeded()) {
    System.out.println("Connected to a server");

    mqttClient.publish(
      MQTT_TOPIC,
      Buffer.buffer(MQTT_MESSAGE),
      MqttQoS.AT_MOST_ONCE,
      false,
      false,
      s -> mqttClient.disconnect(d -> System.out.println("Disconnected from server")));
  } else {
    System.out.println("Failed to connect to a server");
    ar.cause().printStackTrace();
  }
});

You can find MQTT client and server examples here

Web API contracts

With the new OpenAPI router factory we can focus on the API implementation and not on the validation of the input. The usage is quite simple:

OpenAPI3RouterFactory.createRouterFactoryFromFile(vertx, "petstore.yaml", ar -> {
  if (ar.succeeded()) {
    // Spec loaded with success
    OpenAPI3RouterFactory routerFactory = ar.result();

    // add your API and security handlers to the factory

    // add it to a server
    vertx.createHttpServer()
      .requestHandler(routerFactory.getRouter()::accept)
      .listen();
  } else {
    // Something went wrong during router factory initialization
  }
});

Now as a developer you only need to care about the API and not on the validation. The OpenAPI router will ensure that a request to an API will first to the contract before your handler is invoked.

Java 9 support

Java 9 was released a few days ago and the Vert.x stack has been carefully tested on Java 9 and most of our components run on Java 9 (Groovy does not run well on Java 9, please see the support matrix)

As a bonus you can now use HTTP/2 out of the box with JDK SSL!

You can also use Vert.x jars as anonymous modules.

Event driven JSON Parsing

We provide now an event driven JSON Parser emitting parse events that is very handy when you need to handle very large JSON structures and you don’t want to buffer it which introduce extra latency and increase the memory consumption.

The parser allows you to switch between fine grained JSON parse events or full structures, for instance you can parse an array of object very efficiently:

JsonParser parser = JsonParser.newParser();

// The parser will handle JSON objects as values
parser.objectValueMode();

parser.handler(event -> {
  switch (event.type()) {
    case START_ARRAY:
      // Start the array
      break;
    case END_ARRAY:
      // End the array
      break;
    case VALUE:
      // Handle each object
      break;
  }
});

Single SQL operations

Single SQL operations (aka one-shot) have been drastically simplified: most of the SQLOperations operations can now be performed directly on the SQLClient:

client.queryWithParams("SELECT AVG(VALUE) AS VALUE FROM RATING WHERE MOVIE_ID=?", new JsonArray().add(id), ar2 -> {
  if (ar.succeeded()) {
    int value = ar.result().get(0).getInteger("VALUE");
    // Continue
  }
});

Under the hood, the client takes care of the pool acquire/release interaction for you.

Native transport and domain sockets

We now support native transports on Linux (Epoll) and MacOS (KQueue), as well as UNIX domain sockets for NetServer/NetClient (HttpServer/HttpClient should support UNIX domain sockets soon).

Auth handler chaining

There are times when you want to support multiple authN/authZ mechanisms in a single application.

Vert.x Web supports auth handlers chaining

Vert.x config improvements

Vert.x Config allows configuring your application by assembling config chunks from different locations such as file, http, zookeeper…

In this version, we have added the support for Consul and Vault.

With the Consul config store, you can retrieve your configuration from a Consul server - so in other words, distribute the configuration from your orchestration infrastructure.

The Vault config store lets you retrieve secrets avoiding hard coding secrets or distributing credentials using an insecure way. Vault enforces the security of your secrets and only allowed applications can retrieve them. In other words, now you can keep your secrets secret.

ACKs

I want on behalf of the team to thank all the contributors for this release including the Google Summer of Code students (Pavel Drankov, Francesco Guardiani and Yunyu Lin) that delivered an impressive work.

Finally

The release notes

Docker images are also available on the Docker Hub. The Vert.x distribution is also available from SDKMan and HomeBrew.

The event bus client using the SockJS bridge are available from NPM, Bower and as a WebJar:

The artifacts have been deployed to Maven Central and you can get the distribution on Bintray.


by vietj at October 20, 2017 12:00 AM

New Industry 4.0 Open Testbed Addresses Performance Monitoring and Management in Manufacturing

October 19, 2017 04:00 PM

Bosch Software Innovations, CONTACT Software, Eurotech, fortiss GmbH, and InfluxData collaborate to demonstrate Product Performance Management.

October 19, 2017 04:00 PM

Industry 4.0 Meets Open Source

by Ian Skerrett at October 19, 2017 03:03 PM

A key industry that is driving the Internet of Things is Manufacturing. Companies are always looking to make their manufacturing process more efficient, flexible and improve quality, while lowering costs. Therefore, modernizing factory floors and integrating factory equipment with the enterprise IT systems is of interest to many manufacturing companies. In Germany, they call this trend Industrie 4.0. They see IoT as being the fourth industrial revolution to fundamentally change the manufacturing industry. For history buffs, the other industrial revolutions are considered to be: mechanization with water and steam, mass production, and robotic automation.

The Eclipse IoT community has been developing open source technology that can be used to implement Industry 4.0 solutions. A key strength of our community is that we have companies like Bosch, Red Hat, Eurotech, Sierra Wireless, IBM and others,  who are experts in operational technology (OT) and experts in enterprise IT technology (IT). It is really the integration of OT and IT that leads to successful Industry 4.0 deployments.

To help educate organizations on how open source software can be used to implement Industry 4.0 solutions, the Eclipse IoT Working Group has announced two new initiative:

  1. A new white paper titled ‘Open Source for Industry 4.0‘ has been published. This white paper describes the software features required for Industry 4.0 and what open source projects are available.
  2. A new Eclipse Open IoT Testbed has been launched to showcase how Eclipse IoT technology and commercial solutions can be used to implement Production Performance Management (PPM) for factory floors. The PPM Testbed is a collaboration of Bosch SI, Contact Software, InfluxData, Eurotech and fortis GmbH. It is a great example of how open source software can be used to solve important challenges in the manufacturing industry.

Both these initiatives are great examples of collaborations made possible by the Eclipse IoT Working Group. It also shows how open source is really ready to solve Industry 4.0 challenges.

Next week we will be talking a lot about the PPM testbed, Industry 4.0 and All Things IoT.  I hope you will join us.

 

 

 



by Ian Skerrett at October 19, 2017 03:03 PM

Theia Update – Extensions, Git Support & More

by Sven Efftinge at October 19, 2017 11:30 AM

Back in May, we announced a new open-source IDE framework for build Desktop & Cloud IDEs implemented in TypeScript. Since then, Theia has become more mature and feature rich, approaching its 1.0 BETA version on December 6.

During the last couple of months, the Theia developers (we count 19 contributors so far with two companies backing up the development) have been busy laying out important groundwork as well as implementing features. In this post, I want to give you an overview of the current state.

Extension System

Besides being able to run both a desktop and a cloud IDE, from the same source code, an extendable platform has been one of the primary motivations for Theia. We had mainly two requirements:

  • Extensions should be first class citizens
    An extension should be able to contribute and change any aspect of a Theia-based application. The goal really is to allow developers to compose custom products from extensions. Therefore, in Theia, everything is an extension. So even in the core, we eat our own dog food. This is different to e.g. VS Code which runs extensions in an isolated sandbox with only a limited API exposed.
  • Discover and Install Extensions
    The other requirement was to allow users to change the set of installed extensions at runtime, by searching a registry/marketplace and installing/updating/removing extensions. This is very common, but e.g. Eclipse Che does not support such a feature yet.

The extension system has been one of the most important tasks, as it lays out how future development can scale and an eco-system can grow. Therefore another important goal was to make it as simple as possible and rely on existing knowledge and widely used concepts.

Theia extensions are regular Node.js packages which you publish to npmjs.or or another npm registry. To turn your package into a Theia extension, you only need to add minimal information to the package.json. At the core, we use dependency injection (InversifyJS) to wire the contributions from all extensions together. Theia comes with a CLI tool that makes it easy to create an application from a list of extensions.

The dynamic part that allows to install and uninstall extensions at runtime, really just changes the list of extensions and runs the CLI again. When an extension has been successfully installed it informs the user that the application has been updated and restarted. After a quick reload the new extensions is ready to be used.

Furthermore, for extension developers we’ve created a small Yeoman generator that gets you started with the initial project setup. You can find more detailed documentation here.

Git Support

Another new extension is the Git extension, that supports the user to author commits and execute basic git commands. It is very similar to the git support in VS Code, so you can

  • see the list of local and staged changes,
  • open diff editors to see more details and
  • execute standard git operations. 

Under the hood, the extension uses GitHub’s dugite library, which we extended a bit.

Here’s ashort screencast showing it in action:

Language Tooling

Intelligent support for languages is one of the main features an IDE absolutely needs to have. With Theia, we decided to primarily support the Language Server Protocol. By integrating the awesome Monaco editor, that also drives VS Code we already have an excellent foundation. Today, Theia has full support for the LSP and we even have additional extensions for some languages. For instance, in Java one can navigate to files contained in jars.

All 46 available language servers will work fine in Theia, we just haven’t wrapped all of them in a Theia extension for easy consumption, yet. As of today, the following languages are supported:

  • Java
  • JavaScript / TypeScript
  • Python
  • C/C++ (Clangd in development)
  • Go
  • YANG

The list of features Theia supports includes:

  • content assist
  • find references
  • go to definition
  • diagnostics (incl. a problems-view listing all of them)
  • outline
  • quick outline
  • hover
  • signature help
  • formatting
  • rename refactoring
  • syntax coloring
  • folding
  • bracket matching
  • toggle comment
  • code lenses

There are many more important editing features, provided by Monaco, such as moving lines, multi cursor support and so on. Also, Theia has gained lots of other smaller nice goodies, like the quick command palette, preference support or the open file command.

Quick Open in Theia

Furthermore, the colleagues from Ericsson are actively working on managing processes, debugging support and other useful extensions.

Summary

Theia has advanced to a solid foundation for future browser and desktop IDEs. If you were looking for an easy-to-use, open-source framework to build extendable IDEs, you can consider Theia now.

Have a look at the backlog to get an idea of what we want to accomplish in the next weeks.

Besides using Theia for your own projects/products we are welcoming any kind of contribution. If you want to get involved with Theia, please ping us on GitHub or Gitter (or StackOverflow, Twitter, etc.). There are many fun things to work on and some are explicitly marked for starting contributors.

Let us know what you think!


by Sven Efftinge at October 19, 2017 11:30 AM

Next stop: EclipseCon Europe 2017

by kthoms at October 18, 2017 03:38 PM

As every year EclipseCon Europe is fixed in my schedule, and I am excited to go there. It is a melting pot for the Eclipse Community, a big family come together. I am in the last preparations before departure to Ludwigsburg, and can’t wait to finally go there.

FullSizeRender.jpg

itemis is of course sponsoring EclipseCon Europe, and we will have a booth in front of the theater. With 12 talks, 8 speakers and about 20 attendees I am expecting that the it(emis)-Crowd is again the biggest party at EclipseCon Europe. The itemis booth will be the place where you will have the highest chance to meet me, but of course there is plenty of time to meet each other.

Meeting the Scout and Modeling community

As in the past years, I will start my journey already on sunday. I will again join the Eclipse Scout community’s pre-conference dinner at the Rossknecht. The past years I was joining on monday the Scout User Group meeting, but this year I committed to join the Guided Tour on Eclipse Modeling at the Unconference. At 14:40 I will give there my first talk, an insight on the Xtend language.

Xtext and Platform development

This Friday, Oct 20th, Xtext 2.13 will be released finally and I have invested quite some time in the past months to find and resolve bugs. I have worked intensively with the Eclipse EARI system, and investigated together with Christian Dietrich the problem reports we are getting in. Until we now managed to get less problem reports in than we are processing. Christian and I are those who care most about user problems and we have fixed together the majority of bugs for this and the past releases. Christian does an incredible job! We will happily share insights on the current state on Xtext at the Unconference and all conference days.

The past months I starting getting involved into the Eclipse Platform itself. While before I completely focussed on Xtext development and just used the platform, I thought it was time to give something back. I am using Eclipse every day and still love to work with it. I am recognizing that others prefer other IDEs, or even new ones are built, and there are reasons for that. But still for complex development tasks I believe that an extensible desktop IDE like Eclipse is the best tool. Eclipse has some flaws, and I could help there. Now I am frequently contributing to the Eclipse Platform (with focus on performance, usability and stability) and found into the development process, which took me some time. Because of this engagement I am expecting to have some interersting talks on platform development with some driving persons like Lars Vogel, Dani Megert, Alexander Kurtakov, Mickael Istria, Mikaël Barbero or Andrey Loskutov. I have to thank them for guiding me in the process and reviewing my changes carefully. Guys, I owe you a beer at the Nestor bar!

A conference day (almost) never ends

Nestor bar, the place to be after the long conference day! You will find me there each evening from monday on till late. Like every year it will be hard to celebrate long and get up early. But be sure, I’ll manage that. It isn’t the first time, and won’t be the last. The party ain’t over until it’s over.

I’m not staying at Nestor; like last year I reserved early a room at the nearby Villa Forêt. It is just a 5 minute walk (and some walking does not harm) and fine for me. It was in this hotel where I met Philip Wenig some years ago at breakfast. Ever since then I had nice talks with him and I always enjoy that. This year I already met him twice: At the Eclipse DemoCamp in Zurich, and at Eclipse Hackathon Hamburg.

Talks at the main conference

This year, besides my Xtend talk at the Unconference, I will give 2 talks at EclipseCon:

screenshot 21.png

“Introduction to Expression Languages with Xtext” (Tuesday 14:30, Silchersaal) will give you some patterns in Xtext grammers when you need to embed expressions in your language. Xtext ships with Xbase, which is a full expression langauge that you can easily integrate, but sometimes Xbase is not the right choice for you. Xbase is tightly bound to the Java typesystem and JDT, and for your language this could be undesired. Then you have to build your own expression language, which is a bit advanced. But you can learn a lot from Xbase, and in this talk I will show some grammar patterns that you could take from Xbase. I already gave this talk at EclipseCon France this year, so most of the slides are fortunately prepared.

screenshot 20.png

Different for my second talk: “Advanced Oomph Setup Authoring” (Wednesday 12:00, Silchersaal). This is completely new, and I am right now working on the slide deck. I was responsible for developing Xtext’s Oomph setup, which is compared to other setups at Eclipse more complex. But I have learned much from the other setups. Again, there are some patterns that can be recognized among the different setups. I will show and explain screenshots from different setups and discuss some advantages or disadvantages from then. Oomph is a mighty framework, and creating good setups is a time consuming and error prone work. The information given in this talk should give some help to author more robust setups, and build them faster. Advanced Oomph users might recognize that they do already much right, but even they might get the one or other idea to enhance their setups. Users rather new with Oomph will get the most out of this talk. They should at least have a basic idea about Oomph project setups.

Lightning Talks at the itemis booth

This year we will give some 5-Minute Lightning Talks at our booth in pauses. We have a lot of interesting small talks this time, from Xtext to Java 9, and even where plastic plants play a role. Just come around to the exhibitor’s area in front of the theater and get some inspiration.

Also here I have 2 slots:

  • Tuesday 15:50: “What’s new in Eclipse Photon?”: Let me show you a sneak preview on some features coming in Photon. You will see some of my contributions and some other.
  • Wednesday 15:20: “A committer’s view on AERI”: As said before, AERI helped me a lot to improve Xtext and Eclipse Platform. I’ll show you what committers see from problems reported to it and how it can help to find bugs. Also, a big Thank You to the guys from Codetrails for the support!

Follow @itemis on Twitter to get notice on further talks from us!

The most important thing at EclipseCon is…

the people! I love to meet all the people again, from which most of them I only see once a year. This year I have already attended Eclipse Converge/, DevoXX US and EclipseCon France, so some of you folks I have already met again. But EclipseCon Europe is by far larger and more intensive. To all the people I already know, from year to year they become more.

Eclipse on the roll

It is a pleasure to see which companies and projects joined the Eclipse Foundation recently. Since I have a background in Java Enterprise development from early beginnings (yes, I had to implement bean managed persistence with EJB 1.0 in the ancient days and it was NOT funny!), I was delighted to see EE4J at Eclipse. Then IBM’s J9, Deeplearning4J, and the story is not over yet. If this continues, the Eclipse Foundation has a bright future and I am glad that itemis is a driving part of the story.

For this year’s EclipseCon Europe some of this hot new stuff might come a bit too late, but I expect more talks related to these exciting technologies next year. Yes, there are already some talks, but I think the focus will shift from now on.

And finally: Time to rest

After EclipseCon I’m taking a week off. I need this already, and will desperately need this after this exciting and exhausting week. Back to my beloved family, who is awaiting me after a long week.

 



by kthoms at October 18, 2017 03:38 PM

About the Power of Open Source Development at Eclipse

by Karsten Thoms (thoms@itemis.de) at October 18, 2017 02:40 PM

As a project deeply integrated with Eclipse JDT, Xtext has sometimes also to use internal API. This is, for example, necessary to leverage best performance of accessing Java elements. We try to avoid this whenever possible sometimes even by duplicating the code, and we never faced a problem for several years in this regard.

This October Eclipse released Oxygen.1a, which added Java 9 and JUnit 5 support to the recently released Oxygen.1 release. Java 9 support was a major effort for the JDT team and the fast availability to Eclipse users right after release of Java 9 last month proves the vibrant nature of Eclipse projects and the enthusiasm developers put into their projects. And naturally this feature required some major changes to JDT’s internals.

Small change in JDT – great impact for Xtext users

One of these refactorings lead to a signature change in one of JDT’s internal methods that Xtext makes use of. This is a small change, but crucial for Xtext users. All current productive versions of Xtext including Xtext 2.12, which participates in the Eclipse Oxygen release train, are compiled against this method and Xtext plugins would fail when running on JDT when this method would not be available in the specific signature.

At itemis we care about quality of the Eclipse projects we are involved in. A very valuable source information for us is the Eclipse Automatic Error Reporting System (AERI). We actively scan reports there, since these are real issues real users have out there. Without AERI, the hurdle to report problems was rather high: You need an Eclipse Bugzilla account, find the right category and describe manually your problem. Many issues were not reported then. Now it is just one click to automatically submit a report. And the more users report the same issue, the more prominent this is shown to us developers.

On Monday before the final release candidate there was suddenly a report on a linking problem, which showed the following excerpt in its stack trace:

java.lang.NoSuchMethodError: 
org.eclipse.jdt.internal.core.JavaProject.computePackageFragmentRoots
(Lorg/eclipse/jdt/core/IClasspathEntry;Lorg/eclipse/jdt/internal/compiler/util/ObjectVector;Ljava/
util/HashSet;Lorg/eclipse/jdt/core/IClasspathEntry;ZLjava/util/Map;)V

at org.eclipse.xtext.common.types.access.jdt.JdtTypeProvider.collectSourcePackageFragmentRoots
(JdtTypeProvider.java:539)


From time to time there are similar reports, which usually indicate faulty configurations like a version mixture of Xtext bundles. AERI reports contain some information on the available bundles and their versions in the execution environment, which help identifying the context of the problem. Now this one was strange: It showed the version 3.1.50 of bundle org.eclipse.jdt.core installed. This version was unknown to us – Eclipse Oxygen.1 was delivered with version 3.1.0.

As we detected this problem it was my itemis colleague Christian Dietrich who immediately got into contact with the JDT team to discuss this issue and raised Bug#525462. Stephan Herrmann from the JDT team, also an active Xtext user, and Dani Megert worked with us to address this issue. Finally, it was decided together that the missing method would be added again. The release of Oxygen.1a was already in progress, and developers were informed through the cross-project mailing list about this important issue.

Fortunately the Eclipse Platform team agreed to respin the whole Oxygen.1a simultaneous release. There were two other issues with the JDT and Maven tooling, and these problems are solved with the Oxygen.1a release before all users are affected.

We from the Xtext team will now carefully review and reduce internal API usage and work together with the JDT team to find proper replacements or evolve suitable public API where it is missing. Thanks to Christian the issue was immediately resolved in Xtext’s source, so that upcoming Xtext 2.13 already does not use this specific internal API anymore.

Since Xtext is used in many different release versions out there it will take time until the internal API is not used by users out there, although the code base might be clean soon already. Further we will discuss internally and cross-team at EclipseCon how we will detect such situations even earlier. In the past we had builds running against several Eclipse versions, but you can imagine that keeping this alive is eating plenty resources both of developers and build servers. To assure our high support requirements it might be necessary to build this up again.

open-source-team-work.jpg


Eclipse: Open source development at its best

At the time when the problem was detected there were 15 reports of this problem in AERI. Sounds much, but actually this is really just a few. Just consider the case that Oxygen.1a went live without detecting this, all Xtext users would have been affected. It is people like Christian, Stephan and Dani to thank that a disaster was avoided. This again shows the great co-working at Eclipse across projects and across companies.

Speaking of Xtext 2.13, the Xtext team itself is in its final release phase and we’ll release on October 20th, just in time for EclipseCon Europe in Ludwigsburg! If you are attending EclipseCon and have questions on the new release, the future of Xtext or just want to drink a beer with us, come to the itemis booth (booth 13) or later to the bar at Nestor hotel. We can be bribed with a beer to share valuable internal insights and tips for you :)

After this EclipseCon and Xtext release we are not going to rest a moment and will continue to make Xtext even more rock solid and improve its usability, stability and performance. Note that our Xtext team at itemis offers Professional Support Services for Xtext support plans. For your project consider to get us on board, together we can make your project even more successful.

Get Professional Support


by Karsten Thoms (thoms@itemis.de) at October 18, 2017 02:40 PM

Eclipse Science Advances Open Source Technology for Scientific Research

October 18, 2017 02:00 PM

The Eclipse Science Working Group, a working group of the Eclipse Foundation, today announced the new releases of five open source projects used by the scientific research community.

October 18, 2017 02:00 PM

Eclipse Newsletter | Utility Belt: Projects and Tools

October 18, 2017 10:02 AM

Learn more about JUnit 5, e4 on JavaFX, scripting with EASE, embedded tools and Eclipse CDT, the Ferret search tool, and software ethics.

October 18, 2017 10:02 AM

Download the EclipseCon 2017 App

by Anonymous at October 17, 2017 02:22 PM

Create your EclipseCon schedule with the app from EclipseSource. The app provides you with the program organized by track or by date and time with the complete abstract and speaker information. You can locate your destination with the built in maps of the conference venue and its surroundings. Login to eclipsecon.org during the conference to submit feedback on the talks. Download from these links:

We asked the EclipseSource team to tell us more about the app:


by Anonymous at October 17, 2017 02:22 PM

Formatting for the Masses

by Holger Schill (schill@itemis.com) at October 17, 2017 01:28 PM

Reading source code happens way more often than writing it. And reading code is a lot easier when it is well formatted. Linebreaks and indentations are guiding the reader and it is more obvious in which context things are happening. Doing formatting completely manually is simply not an option, especially for larger changes. For that reason general purpose languages, like Java, do have many different implementations for automatic code formatters or “pretty printers”. 

All implementations come with a default behaviour that works, but this is not enough. Formatting has always been a matter of taste and therefore users want to customize all the different settings to their liking. Therefore professional formatting tools like Eclipse JDT offer a gazillion number of options.

Formatting-code-Eclipse-JDT.png

These options can be stored as preferences and shared together with the code. Sounds cool, but it is still not enough for “everybody”. People want to adjust those settings or when, they come to the point where the tool does not offer the 'right' options, they will use tag-comments in JDT. This will make the formatter shut up for a specific area.

formatting-code.png


The formatter can be explicitly switched on and off to manually format the code. This is a workaround and not a solution. The problem is that “everybody” wants to format their code differently. Some language designers try to circumvent this problem by mandating the correct formatting on the syntax level as Python does it, but this is oftentimes limited to the indentation level and not to the overall formatting rules.

For most general purpose languages, there are a gazillion tools to format the code in different ways with different options and the user has the choice – the choice of the tool, the choice of options – everything can be tailored to specific needs.

Formatting code for DSLs with Xtext

For domain specific languages (DSL) there is nothing like that coming out of the box. Looking at frameworks like Eclipse Xtext, it is really a breeze to create languages, but there is no formatter coming for free. Of course, the framework is offering a powerful API to define formatters. Decisions can be made on grammar level for all instances of the DSL and in addition you have access to a specific part of the abstract syntax tree so can make decisions based on the structure and values of your model, too. Having the option to format code based on syntax and structure sounds really cool and actually it is, but it’s a lot of code that needs to be written and it’s not “that” easy.

Formatting.code.png


After you are satisfied with your implementation and your formatter works as you wanted, your users might see that differently. As already said, formatting is a matter of taste and they might come up with the need for options. An additional newline here, no space there... these options do add a lot of complexity to the formatter code. And that's certainly not for free. Time to define the canonical formatting for your DSL? Or can we do better? How about formatters that take examples and learn how to format similar documents? Intelligent formatters that nobody needs to write manually and that are driven by your coding style? Can this be done?

We started to ask ourselves the same questions and looked at existing libraries out there. As we use AntLR in Xtext heavily, we looked in that direction, too. And we found a very promising project named Codebuff, that is driven by Terence Parr, the mastermind behind AntLR. It aims to do formatting by example and the only thing you need is a grammar and a bunch of examples.

formatting-for-the-masses.jpg


We have played around with it and it really looks promising. In our
talk at the EclipseCon Europe at Wednesday, October 25, 2017, 02:45 pm to 03:20 pm Sebastian and I will give you an overview of what we have figured out, how it works and how it might integrate with Xtext.

See you there!


by Holger Schill (schill@itemis.com) at October 17, 2017 01:28 PM

SpotBugs 3.1.0 RC7 with Java 9 support

by Andrey Loskutov (noreply@blogger.com) at October 17, 2017 07:37 AM

FindBugs is dead, long live SpotBugs!

I would like to introduce here SpotBugs, a spiritual successor of FindBugs, crafted by the community.

We've just released 3.1.0 RC7 version and hope to release the final 3.1.0 this weekend.
One of the main drivers to release SpotBugs now is to support Java 9, but there were also few other exciting changes in the previous SpotBugs milestones, please check them for more details: https://github.com/spotbugs/spotbugs/releases.

We also have a new SpotBugs Eclipse plugin, here are the links to the update sites: http://spotbugs.readthedocs.io/en/latest/eclipse.html and Eclipse marketplace entry.

We need your help!
Please check if you see regressions and don't hesitate to report issues to the bug tracker!

SpotBugs links

Home: https://spotbugs.github.io
Mailing list: https://github.com/spotbugs/discuss/issues?q=
Bug tracker: https://github.com/spotbugs/spotbugs/issues
Git repository: https://github.com/spotbugs/spotbugs
Manual: http://spotbugs.readthedocs.io/en/latest
Migration guide: http://spotbugs.readthedocs.io/en/latest/migration.html
Eclipse plugin nightly build: https://spotbugs.github.io/eclipse-latest

P.S.
If you are curious about the history of SpotBugs and why FindBugs isn't there anymore, please read through this mails:

by Andrey Loskutov (noreply@blogger.com) at October 17, 2017 07:37 AM

Zip Symmetry

by Donald Raab at October 13, 2017 05:31 PM

Another tale of Symmetric Sympathy in the land of object and primitive collections.

Island of Balos, Crete

I gave a talk this year at JavaOne titled API Deep Dive: Designing Eclipse Collections [CON6133] with Nikhil Nanivadekar. I really enjoyed this talk, as I got to recall twelve plus years of development in forty-five minutes. The slides are now available online here.

Symmetry continues to drive the design and implementation of new features in Eclipse Collections. On slide 36 of the presentation (see below), I indicated where we are currently missing some symmetry between object and primitive collections.

Where is primitive eager zip?

After returning home from JavaOne, I decided to spend Columbus Day implementing zip for primitive eager lists in Eclipse Collections.

What is zip?

According to Wikipedia:

In computer science, specifically formal languages, convolution (sometimes referred to as zip) is a function which maps a tuple of sequences into a sequence of tuples.

To put this slightly differently, if you have two lists of objects, and you zip them together, you will wind up with a single list of pairs of objects.

Here’s an example of how you can use zip today in Eclipse Collections (EC) for object collections.

MutableList<String> one = Lists.mutable.with("1", "2", "3");
ImmutableList<String> two = Lists.immutable.with("one", "two", "three");
MutableList<Pair<String, String>> zipOneAndTwo = one.zip(two);
ImmutableList<Pair<String, String>> zipTwoAndOne = two.zip(one);

If we output zipOneAndTwo and zipTwoAndOne they look as follows:

[1:one, 2:two, 3:three]
[one:1, two:2, three:3]

We can change the first list to Integer objects like so.

MutableList<Integer> one = Lists.mutable.with(1, 2, 3);
ImmutableList<String> two = Lists.immutable.with("one", "two", "three");
MutableList<Pair<Integer, String>> zipOneAndTwo = one.zip(two);
ImmutableList<Pair<String, Integer>> zipTwoAndOne = two.zip(one);

We still get the same output, because of how Pair defines toString().

[1:one, 2:two, 3:three]
[one:1, two:2, three:3]

I hate boxing ints as Integers. I’d like to be able to use an IntList instead of a List of Integer objects. Previously in EC, we could not change the MutableList of Integer to a MutableIntList. Now we can, but this new ability highlights a slight problem.

MutableIntList one = IntLists.mutable.with(1, 2, 3);
ImmutableList<String> two = Lists.immutable.with("one", "two", "three");
MutableList<IntObjectPair<String>> zipOneAndTwo = one.zip(two);
// The following code does not compile
ImmutableList<ObjectIntPair<String>> zipTwoAndOne = two.zip(one);

I have added a method zip to primitive lists with the following signature (IntList used as an example below).

/**
* Returns a {
@code MutableList} formed from this {@code MutableIntList} and a {@code ListIterable} by
* combining corresponding elements in pairs. If one of the two Lists is longer than the other, its
* remaining elements are ignored.
*
*
@since 9.1.
*/
default <T> MutableList<IntObjectPair<T>> zip(ListIterable<T> list)

We cannot (yet) zip an object list with a primitive list. By adding a feature in one direction (primitive+object), I have discovered (or created) a lack of symmetry in the other direction (object+primitive).

One small step for a code generator…

We can zip two object lists together regardless of the object type, so I wanted to at least be able to zip two int lists, two double lists, two short lists, etc. This is in addition to now being able to zip a primitive list with an object list. So the following signature is also included and code generated today (example below with zipInt on IntList). A method zipDouble is also generated for DoubleList, zipFloat for FloatList, etc. I used a naming pattern which will easily allow additional forms to be added later if there are use cases for them that surface.

/**
* Returns a {
@code MutableList} formed from this {@code MutableIntList} and another {@code IntList} by
* combining corresponding elements in pairs. If one of the two {
@code IntList}s is longer than the other, its
* remaining elements are ignored.
*
*
@since 9.1.
*/
default MutableList<IntIntPair> zipInt(IntList list)

Here’s an example zipping two IntLists together using the method zipInt.

MutableIntList one = IntLists.mutable.with(1, 2, 3);
ImmutableIntList two = IntLists.immutable.with(10, 20, 30);
MutableList<IntIntPair> zipOneAndTwo = one.zipInt(two);
ImmutableList<IntIntPair> zipTwoAndOne = two.zipInt(one);

If I output both zipped lists the result is as follows:

[1:10, 2:20, 3:30]
[10:1, 20:2, 30:3]

We can now zip eight forms of primitive lists to object lists today. We can also zip “like” forms of primitive lists as well (IntList->zipInt, DoubleList->zipDouble, etc.). I am hopeful that this functionality will satisfy most needs for the foreseeable future.

The North Remembers

Symmetry does not forget, and when it is missing and needed, developers will not always easily forgive. Achieving perfect symmetry for the sake of symmetry itself is a fool’s errand. If you find yourself in the position of needing more symmetry in this space, remember Eclipse Collections is open source. You can put in a request and perhaps someone may implement it on your behalf, but the quickest way to completion may be getting actively involved in the Eclipse Collections community and making a contribution. We will make sure you get all the help you need if you are interested in contributing.

The sequel?

We have zip today in Eclipse Collections but we do not yet have its natural opposite: unzip. Unzip would take a list of pairs, return a pair of lists. Unzip only makes sense on the object collection side, because you can only unzip a list of pairs. Sounds easy right? Unzip winds up more complex to define with primitives now supported in zip. If you have a List of Pair, you will get back a Pair of Lists. If you have a List of ObjectIntPair, you would want to get back a Pair of List and IntList… and so on. The return type for unzip will be tricky to define, at least without complicating the API, which should require no parameters. I think we should wait a while before trying to solve this symmetry problem. However, there is a pattern which would be very nice to have, and may make using zip less attractive once we do have it. This pattern is collectWithIndex. We have forEachWithIndex and injectIntoWithIndex (primitive only) today. Oh, no... Stay tuned.

I hope this latest chapter in my tale of Symmetric Sympathy was interesting and informative.


by Donald Raab at October 13, 2017 05:31 PM

An invitation from the Science community

by Anonymous at October 13, 2017 09:32 AM

by Tracy Miranda

You're invited to join the Science activities at EclipseCon Europe. Get involved by participating in:

And the program committee has selected 5 talks from the Eclipse science community:


by Anonymous at October 13, 2017 09:32 AM

Eclipse Hono : Virtual IoT meetup

by ppatierno at October 12, 2017 05:54 AM

Virtual IoT - Hono

Yesterday, thanks to the Eclipse Foundation I had the chance to talk about Eclipse Hono as speaker for this virtual IoT meetup as part of a meetup series where the focus is on the Eclipse IoT projects. I was with Kai Hudalla (Chief Software Architect at BoschSI) who is co-lead and main contributor on Hono.

It was my first virtual meetup and a really exciting experience for me with almost 90 “virtual” attendees and a lot of interesting questions showing the interest that developers had about this “new” project.

If you didn’t have a chance to watch the session or you want to re-watch it, you can find the recording on YouTube; the slides deck is available here as well.



by ppatierno at October 12, 2017 05:54 AM