IDYUTS

This document and revision Copyleft Saturday, July 12, 2003

By James Northrup

Email is grrrrr at users.sourceforge.net

Executive Summary:

IDYUTS “I dare you to use this shell”

This is a building block and an early evolution of smarter control paradigms in robust system development.

• Idyuts is a pipeline paradigm code generation solution developed from scratch based on the need for more flexible late binding java constructs. These concepts have eluded sharing of discussion until the term “IoC” became a buzzword in the blogs.
• This loose discipline of simple scripts and simple mechanisms provides publish and subscribe template generation at its cleanest fundamental representation without religion.
• This politely makes up for the lack of templates in Java (ala C++), the lack of true first class object abstractions in Java(ala Jython), and the lack of functor classes in Java (ala Jython and C++ templates).

Background and philosophy:

IDYUTS has seen implementation in half a dozen projects, some embedded. It is frugal to design real-time systems with fixed pipeline activities and easy branch abstractions to provide the paradigm of assembly-line data transformations. Idyuts offers only interfaces, and code fragment templates, from some industrial strength Reflection tinkering.

This release is a quick packaging of idyuts, after 2 years of tinkering within several projects. Most of this was refined while using TogetherJ on much bigger projects; using UML tools is not an ideal IoC diagram notation, but it sure beats nothing.

For my needs, the best Class-diagram visual representation of the java pipeline classes comes from hiding NOTHING in a framework. This makes for tool-agnostic polymorphism to the nth degree but less overhead than an MQ or a Java Event bean facility for high performance computing. The default model simply calls recv() directly, which is efficient with or without a JIT.

Newer IoC and pub/sub standards exist, and heavier ones. Idyuts begs to be the internals of efficient computing systems without really getting in the way of progress. Pipelines can get in the way of project understanding at the cultural level.

Command Design Pattern: Idyuts was designed to operate without java coding as a beanshell pipeline scripting toolkit. This massively dispersed collection of Source and Filters was designed to funnel all of the diverse object input of a production real-time system through a constant sql event generator. IDYUTS cleaves the gap from persistence to view in a typical MVC design, without a controller glue layer. The binding language spec was never designed. It was that good without the condiments.

Methodology:

For first time adaptation of IDYUTS, proceed coding normally along the given course.

Search your design for areas where data is transformed from one state to another.

Create a new Class from this routine, destined for your Controller toolbox. This will alter program flow radically.

For example, lets start out with simple example requirements:

Begin:

Convert a string to an object called a Sentence. Count all verbs and remove all “foul” words from a dynamic list. Our class holds some state, creates a report, and updates this state whenever a new String is passed into the method.

 
public class MyBLOBObject{
int verbcount;
Sentence sentence;
 
int getVerbCount(){…};
Sentence getSentence(){…};
setInputString(){…};
OutputStream createReport(){…};
}

Next:

This class just seems a little odd. It stores data, it uses side effects, it sends streams of data, but its just an ugly maintenance nightmare waiting to happen. We should refactor a few things.

First, lets identify the start of an event by the input String. Lets set it up to receive a String generically, and produce generic Events.

This is easily done at the inheritance level. The actual code is less important than the class structure, as it should be.

Lets add interfaces to represent the input and output of this classes existing signatures:

…
public class MyBLOBObject implements StringFilter{
…

as you can see, we’ve told the compiler that we use a StringFilter base interface. StringFilterNode.frag provides all the methods we’ll need to implement our new interface:

Refactor your solution so that <String>Filter.recv(String) is the transforming method in your new class.

Decouple these transformations by refactoring this mechanism as a Source and Filter abstraction.

 
 …
public class MyBLOBObject implements StringFilter{
 String theString;
…
 
 public void recv( String data)
 { 
 theString=data; 
 }
…
};

continue:

generate templates for Sentence pipelines, we will provide Sentence objects to a pipeline (SentenceFilter Objects attached at runtime) client instead of method invocation likewise.

Administrator@mars /opt/idyuts-src/java/net/sf/idyuts/IOLayer

$ sh genClass.sh Sentence Sentence

public class MyBLOBObject implements StringFilter,SentenceSource{
 
 public void recv( String data)
 { 
 //SET OUR DATA
 theString=data; 
 // perform our work
 this.xmit();
 }
… 
 // SentenceSourceNode **BEGIN** **UNIQUE SIGNATURES**
 
 private java.util.List _Sentence_clients=new java.util.LinkedList( );
 public void attach(SentenceFilter filter){
 _Sentence_clients.add(filter);
 };
 
 public void detach(SentenceFilter filter){
 _Sentence_clients.remove(filter);
 };
 
 private static final Class[] parm_cls_Sentence=new Class[]{ Sentence.class};
 
 // SentenceSourceNode **DUPE**
 public void xmit(){
 try{
 Object data=theString;//YOUR OBJECT
 Iterator iter=_Sentence_clients.iterator();
 while(iter.hasNext()){
 SentenceFilter filter=(SentenceFilter)iter.next();
  filter.getClass().getMethod("recv", parm_cls_Sentence ).invoke(
filter,new Object[]{data});
 //YOUR CODE GOES HERE
 throw new Error("unfinished code");
 }
 }catch (Exception e){
 throw new Error("more debugging needed here");
 };
 }; 
 
 // SentenceSourceNode **END**
 
…
}

Then:

Almost refactored… we also have an integer byproduct and an OutputStream sideffect. Lets go ahead and pile on the inheritance. Repeat template cut and paste.

Note **DUPE** lines in your source code. The Source.xmit is one-per-class but our BLOB will have 3 subscriber filter lists.

 
public class MyBLOBObject implements StringFilter,SentenceSource,intSource,OutputStreamSource{
…
 
 // intSourceNode **BEGIN** **UNIQUE SIGNATURES**
 
 private java.util.List _int_clients=new java.util.LinkedList( );
 public void attach(intFilter filter){
 _int_clients.add(filter);
 };
 
 public void detach(intFilter filter){
 _int_clients.remove(filter);
 };
 
 private static final Class[] parm_cls_int=new Class[]{ int.class};
 




 // SentenceSourceNode **DUPE**
 public void xmit(){
 try{
 Sentence data= getSentence( );//YOUR OBJECT
 Iterator iter=_Sentence_clients.iterator();
 while(iter.hasNext()){
 SentenceFilter filter=(SentenceFilter)iter.next();
 filter.recv(data);
  …
//do the int xmit operations.
   iter=_int_clients.iterator();
 while(iter.hasNext()){
 intFilter filter=(intFilter)iter.next();
 filter.recv(theInt)
 …
//DO our Stream xmit.
 
 theClient.recv(new OutputStream…)
 }
 }catch (Exception e){
 throw new Error("more debugging needed here");
 };
 };
 
 // intSourceNode **END**
 
 // OutputStreamSourceNode **BEGIN** **UNIQUE SIGNATURES**
 
 private java.util.List _OutputStream_clients=new java.util.LinkedList( );
 public void attach(OutputStreamFilter filter){
 theClient=filter;
 };
 
 public void detach(OutputStreamFilter filter){
  theClient=null;
 };
 
 private static final Class[] parm_cls_OutputStream=new Class[]{ OutputStream
.class};
 // OutputStreamSourceNode **END**
 
};
…
}// so much interface inheritance there’s no room left for sloppy code!
 

The bad news of this approach:

You’ve Inverted the Control of MyBLOBObjects and now it wants to talk to 4 other pipeline classes with mated interfaces! Your refactoring effort is only just beginning. Before it was just a placholder for some methods, now its aligned with publish subscribe operations and can operate with any other compatible idyuts templates.

The BLOB design AntiPattern typically is the right place to assess a pipeline solution. Each BLOB class typically performs multiple data transformations. A replacement for a BLOB is to reduce each transformation method to its tightly coupled routines, and move those routines into two inherited end classes of a pipeline template. Take a step back, refold some adapter patterns, and repeat, as above.

Rete inference engines constantly assess based on facts added or adjusted to a system’s rules. Pipeline systems from inference rules can be codified with 1:1 push model event reactions. Decision trees can be laid out by interconnected and adjoining pipeline tees and Y’s. Inference Engines are typically BLOBS in a design. See BLOB commentary above.

Toolkit UML diagram:

Below are the supported classes. No magic or complexity here, just simple interfaces.

How to install:

The relevant makefile is java/net/sf/idyuts/IOLayer/Makefile

1. cd to java/net/sf/idyuts/IOLayer/ and type make

Administrator@mars /opt/idyuts-src/java/net/sf/idyuts/IOLayer 
$ cd java/net/sf/idyuts/IOLayer/
 

2. now you have generated plentiful default templates you can design and model on case tools with exact type-specific interfaces

 
Administrator@mars /opt/idyuts-src/java/net/sf/idyuts/IOLayer
$ make
sh genClass.sh Bool boolean
sh genClass.sh Integer Integer
sh genClass.sh String String
sh genClass.sh Stream java.io.InputStream
sh genClass.sh Reader java.io.Reader
sh genClass.sh Writer java.io.Writer
sh genClass.sh Object 'Object'
sh genClass.sh Array 'Object[]'
sh genClass.sh Matrix 'Object[][]'
sh genClass.sh Collection java.util.Collection
sh genClass.sh Map java.util.Map
sh genClass.sh MapArray "java.util.Map[]"
sh genClass.sh floatArray float[]
sh genClass.sh float float
sh genClass.sh intArray int[]
sh genClass.sh int int
sh genClass.sh Number Number
sh genClass.sh Graphics java.awt.Graphics
sh genClass.sh StreamTokenizer java.io.StreamTokenizer
sh genClass.sh DatagramPacket java.net.DatagramPacket
 

3. generate some Event-Object Pipeline templates as needed

Administrator@mars /opt/idyuts-src/java/net/sf/idyuts/IOLayer
$ sh genClass.sh Rune Rune
 
Administrator@mars /opt/idyuts-src/java/net/sf/idyuts/IOLayer
$ ls Rune*
RuneFilter.java RuneFilterNode.frag RuneSource.java RuneSourceNode.frag
 

4. idyuts-src/java to your project’s classpath. Prebuilt jarfiles of idyuts primitives are of little value

5. IDYUTS toolkit use will rarely warrant importing the whole toolkit. Likewise, prune to taste.

How to use:

how to build and customize a pub/sub event template

• Run genClass.sh within java/net/sf/IOLayer directory. – if you have a working shell environment (bash) you should produce several primitive pipeline files;

1. A Source Interface – this defines how to add and remove event subscribers to the notifier class.
2. Filter interface – this defines the type-specific pipepline event handler interfaces. Override recv(<event>)
3. *Node.frag files with ready-made method bodies –

1. These files contain impl methods for the interfaces and allow quick OO prototyping. When the events are fired, you will have to replace the deliberate exception with working code.
2. Rename these to .java if you’re lazy and trusting, or borrow the methods with cut and paste.

Using pub/sub java artifacts

Now that you’ve generated a template Pipeline, and chosen to cut and paste methods from .frag files or write efficient simple event handlers, the pub/sub mechanism is exactly the overhead of what you wrote in place. Next, move the pipeline artifacts to a package. IntelliJ works nicely for this task.

Publishing an event

is called by Source.xmit(void) and the Source object should send immutable references to each of the Filter objects it is attached to by calling the type-specific recv(). Or send Null’s, no one will stop you.
I provide 1 utility class for reflection-specific auto attach/detach which aides in chain-of-command assembly. this is found in net.sf.idyuts.IOUtil package.

Examples

net.sf.idyuts.IOUtil.Auto

Auto.attach(StringSource,StringFilter);
 
Auto.attach(Source,Filter)

the first viable type-specific attach/detach will be called appropriately. If you like multiple inheritances, be cautious with this technique. A simple pipeline that needs scripting will benefit from this utility (someday J).

coded by hand

The second way is not flashy but very safe for intricate pipelines:

StringSource.attach(StringFilter) ;

customize the templates for your use. modifying *.pre files allows you to change what is generated to .frag files.

Short circuit

Occasionally during prototyping in my performance tuning it just made sense to couple a source and filter without using a container of clients. Replacing in your Source class

Set clients

With

<EventFilter> client 
 
 

TODO:

Agent code? Nahh, been there done that.

JMX Flavors? We’ll see.

Finish the planned Command Pattern? For an embedded platform that needs a CLI, IDYUTS is the ideal small toolkit around which a binding language makes sense. I’ll gladly accept commissions to further this end. For any larger project, Idyuts is a matter of taste and Command Patterns are a dime a dozen.

Disclaimer

I am an effective implementer by borrowing from my shared and previous open-source works, no warranty is expressed or implied.