Implementing JMS communication at WMB

MQ

Message queues provide an asynchronous communications protocol, meaning that the sender and receiver of the message do not need to interact with the message queue at the same time.

JMS

JMS is a API which helps java applications to interact with other applications through JMS Api.JMS API helps java programs to connect to Enterprise systems like MQ world.

JMS is similar to JDBC.Like the JDBC API used to connect to different relational databases,the JMS API provides vendor independent access to Enterprise messaging system like JBOSS Messaging,Websphere MQ,BEA Weblogic. Same JMS code can be used to connect to Enterprise messaging system like JBOSS Messaging,Websphere MQ,BEA Weblogic.

JMS is a standard.

Still not getting clear idea on why JMS?Read further…

Now-a-days,the messaging World has expanded a lot.We not only have IBM MQ.There are other MQ products such as Sonic MQ,Apache MQ,MSMQ,etc.

You have your application with you handling only the application level logic and Integration layer is completely in Integration layer and out of Application layer.In such case,Application should not bother about which type of Integration MQ you are going to connect.This should not  be tightly coupled.Your application should be able to connect to any type of Integration .Any type of MQ.

If your application is java based ,then it will support JMS.And if the MQ product also supports JMS.Then done.All set.

Your application can act as JMS consumer/provider and the Messaging MQ can also act as JMS producer/consumer so that they both can start communicating with each other.So,this seems to be like a thin layer in between Application and the Messaging MQ product.

Hope now you can understand why JMS..

Now,my Messaging product is IBM MQ.Lets check whether IBM MQ can support JMS.Check whether JMS related files are present in the below path.If there,then your MQ product can support JMS.

C:\Program Files (x86)\IBM\WebSphere MQ\java\bin

JMS client is the Application.And JMS Server is the Message Product which supports JMS.

While communication is in place,client can act as both Consumer/Producer and similar way Server can also act as Producer/Consumer.

Two Models of JMS communication are available:

1)Point to Point 2)Publish/Subscribe Topics

Why we may need Message broker or other Integration Transformer/Mediator in the JMS communication paradigm?

Suppose your application has to communicate with multiple JMS providers of different Messaging MQ products,then the target applications which are connecting to those JMS provider would require messages in their compatible format.

But  you have one application in which you would have to write many messaging transformation logic to support each target application.

Here comes the product like WMB.

Any JMS client use JNDI to lookup JMS connection factories and destinations.All JMS nodes act as JMS clients and hence all the JMS nodes have to configured with Connection Factories and Destinations.

So,now  the Workflow will be as below,

à Your Application acts as  JMS client

  àLooks for JNDI objects to connect to JMS Provider and put message in Destination Queue

     à Integration product like  WMB JMS Input node which is  JMS Consumer ,acts as JMS client to connect to that Destination Queue and receives the message to transform

          à WMB flow (To do the transformation of message)

              à JMSOutput node acts as JMS Provider  to put message in Destination Output queue.Again this                   acts as JMS client to connect to JMS Provider

            à Target Application connect to this Destination Output queue as JMS client to receive the message

What are the components we need to establish connection in JMS?

To create Connection factory:

·         For the clients communicate with JMS Provider(Messaging system),they lookup for Connection factories in JNDI.To configure the Connection Factories in JNDI,follow the below steps

Create Initial Context Factory which specifies whether it is LDAP or FileSystem for JNDI and also the Provider URL

I have used MQQueueConnectionFactory to use purely Point-To-Point communication.And used Client mode.

So,below are the properties configured for Connection Factory

Creating Destination Queue-Finally its an MQ product after crossing JMS.So,as final destination JMS queues should be present for the clients to communicate.These should be local queues present in the queue manager.

I am creating here two local queues for the Destination Queues

2)Now Application is built in WMB as below,

JMSInput node:

JMSOutput node:

3)JMS client can be using Java code or using JMETER as given  below

Ensure that you have below jars in the JMETER lib folder to test JMS,

Testing

As the clients connect through SYSTEM.DEF.SVRCONN,the client user may not have permission to connect to QueueManager.By default in MQ7.5 has Security enabled for client connections.Hence the below error was thrown when trying to connect from JMS client.

So,Disabling for Client Authentication to ease the explaination of JMS flow here

Now,by placing message from JMS client,it works….

Below picture shows the JMS headers getting populated at the JMSInput node

Output Message at the JMS Destination Target Queue

Q&A

Question?
"This provider url (EG: F:/JMS) seems to be the path of .bindings file present in JMS Provider system." 

So,how does the client system resolves/recognises the F drive present in the Server system.Does it automatically resolves after it establishes connection to JMS provider via Server Connection Channel? 


An Application would like to communicate via JMS so that it can connect to any Messaging providers such as IBM MQ,TIBCO,SAP Netweaver PI etc...Having said that,why do we need ServerConnectionChannel which is more specific for an MQ Client to be used at Application side?Does it not fall back to the category of MQClient instead of acting as JMSClient?

Answer:

The provider URL points you with the JNDI to the .bindings file which contains the implementation information for the connection. 
The client system does not resolve anything at this point. 
The client system's OS knows the location of the .bindings file because the Provider URL told it where it was. 
Inside the .bindings file you have bound a JMS ConnectionFactory. There is an Implementation Class name behind it. So when you retrieve the JMS ConnectionFactory from the JNDI it already knows the relevant implementation parameters "decorating" it. 
I.E. all your JMS program needs to do is create a connection off of it. 

If you did create the JMS ConnectionFactory, instead of retrieving it from JNDI, you will have to decorate it with all the information it needs in order to be able to build the connection....

Read up on the basic JMS model. The JMS client knows nothing about the MQ Channel. The JMS client knows about the JMS ConnectionFactory. The connection factory in turn knows about the MQ channel and hostname / port etc.. and any other information needed to build the connection. 

So yes your JMS code is agnostic to the implementation. Your JNDI however is not. It is the JNDI that builds the glue between the JMS generic set up (program) and its practical implementation (MQ). 

So if you want to access MQ, Tibco, SAP PI or any other JMS implementation, all you need to do is correctly bind a "decorated" JMS ConnectionFactory for each of those providers into your JNDI. 
Retrieve from the JNDI the ConnectionFactory corresponding to the implementation you desire. 

Needless to say the same thing goes for the JMS Destinations bound in your JNDI. And no your code is not allowed to mix and match destinations with connection factories, or sessions.

The preferred thing is to define your destinations in the URI form and instantiate them from the session created off the connection (created off the connection factory). This gives you more flexibility than retrieving the queue from JNDI as you can have the URI retrieved from a resource bundle or properties file... 

The URI form of the destination should be as much implementation agnostic as possible. The following example ("queue://qmgr/qname?property=value&property=value...") is the MQ URI form for a queue. 
You may notice the qmgr/queue pair which could be considered as implementation specific... 

Your generic form is ("destinationtype://implementation naming?property=value&property=value..."). The property value pairs are mostly generic but can also be implementation specific. 

REF:

Very good website for JMS concepts

http://www.hartmannsoftware.com/JMS_Tutorial.pdf

http://www.youtube.com/watch?v=WH0ZaGwfmz4

http://www.ibm.com/developerworks/websphere/techjournal/0610_woolf/0610_woolf.html-For knowing more about standing Java application communicating with JMS Provider Messaging system

 http://jmeter.apache.org/usermanual/component_reference.html#JMS_Point-to-Point–jmeter configurations on JMS

Insight into how File nodes pick up files

Filenode pulling all the file in one shot for “n” number of files is only illusion. – Since the flow’s  processing time is in milli/nano seconds ,we will not be able to see the files getting placed intermediately in mqsitransitin folder .

Polling happens this way:

1)      It starts polling from the time of Deploying the flow

2)       It processes each file, one at a time from the folder.

3)      First it picks the first file.Until the file is successfully processed in the flow,it is in mqsitransitin folder

4)      While the  file is there in mqsitransitin folder under process,the FileInput node is still active.It does not go to sleep

5)      During this time of processing first file,if it sees next  files in the Input folder,it will pick the next files once the first file process is completed

6)      Now,after the files are successfully processed(files come out of mqsitransitin and moved to mqsiarchive as configured),if it  does not see any file,the FileInput will go to SLEEP mode for the duration of Polling interval

7)      When it is in SLEEP mode,if any file placed in the folder,it will not be picked up during this time.The file will be picked up only at next Polling

So the behavior is as below:

   If any file is in progress,it means that FileInput instance is still active.

   Hence if  next file is  placed during this time, will be picked up after previous  file is completely processed.

   If next  file is placed after the previous  file is completely processed,then those files will be picked up only in the next Polling.

I have tried a scenario where in esql I waited for 20 seconds( boolean b=SLEEP(20000) ) and did multiple test with Polling frequency as 100 seconds.Filename pattern as “*.txt”. The files are getting picked up as specified in the process steps given above.

Output as below:

Using LDAP Authentication in WMB-Part1

LDAP is hierarchical structure.Read is fast in LDAP.Hence it is suitable for storing data related to organisation and relationship.

Database is Flatfile.Write is fast.

What is the Protocol used for Authorisation? –LDAP

What is the Protocol used for Authentication? -Kerberos

 whats the relation between LDAP vs Kerberos ?
 Two _completely_ different protocols/systems. 
In short and generalized: LDAP is for AUTHORIZATION (access rules) 
 and Kerberos is for AUTHENTICATION (verify password). 

Lets say we need to have many users under  User group.

·         First download Apache Directory Studio

·         Start the Apache Server in Apache DS Eclipse

·         Right click on the Server and choose Create a connection

·         Right click on ou=system and  create Organisational Unit as ou=ibm

·         Right click on ou=ibm and  create Organisational Unit as ou=users

·         Now rightclick ou=users and create user using inetOrgPerson by cn=wmbuser1

What's an inetOrgPerson object?

The inetOrgPerson object class is a general purpose object class that
holds attributes about people. The attributes it holds were chosen
to accommodate information requirements found in typical Internet and
Intranet directory service deployments.

An inetOrgPerson can contain a user id (uid) and password (userPassword) which will be useful later for authenticating users from using LDAP.

·         Create two new attributes such as uid and userPassword for the user newly created

·         Now,to create Authorisation groups,right click on ou=users(you can even create a new organisational unit such as ou=group).

·         Choose groupofnamesand give cn=authorised.

·         Choose the member as wmbuser1 for the group cn=authorised

Creating Security Profiles in Message broker explorer

·         Right click on Broker name.And choose Properties.

·         In the Properties window,choose Security and Policy

·         Then choose,Security Policy

·         Fill up the below highlighted areas

·         For Authentication config :- (Right click on Users in Apache LDAP and get the baseDN from the properties window)

LDAP baseDN= “ou=users,ou=ibm,ou=system”

·         For Authorisation group baseDN config :- (Right click on authorisation group name in Apache LDAP and get the baseDN from the properties window)

LDAP group baseDN=”cn=authorized,ou=users,ou=ibm,ou=system”

Once the security profile is configured,use the below command to check the Security profile configured details.

mqsireportproperties <BrokerName> -c SecurityProfiles -o <SecurityProfileName> –r

mqsireportproperties RadBRK1 -c SecurityProfiles -o LDAPSP -r

One last point to be noted is,if you have setup your LDAP server to accept anonymous connection,then the below command can be executed.

mqsisetdbparms RadBRK1 -n ldap::localhost  -u anonymous

Enter password for user ID

Retype password for user ID.

BIP8071I: Successful command completion.

If not,then use the binding userid and password

mqsisetdbparms RadBRK1 -n ldap::localhost -u "uid=admin,ou=system" -p secret

Makesure that you use correct password for this binding userid .Otherwise WMB cannot connect to the LDAP server and binding related errors will be thrown.

How can you ensure that you use correct username and password for binding?

Verify password once by right clicking on the LDAP Connection,

Sometimes,if you feel that the previous security settings not cleared properly,then use the below command to clear the cache.

C:\Program Files\IBM\MQSI\9.0.0.1>mqsireloadsecurity  RadBRK1

BIP1026I: Modifying the security cache of broker 'RadBRK1' execution group 'defa

ult'...

BIP8071I: Successful command completion.

Using Security in HTTP nodes (Simple flow):

Identity Propagation

There are many ways to propagate Identity from client flow.

1.     Using tools like SOAPUI

At Client side - Use SOAPUI tool and use HTTPAuthentication headers as below

There is a tab named “Authentication and Security-related Settings”.Choose “Preemptive” and provide the user id and password to pass the Authentication headers in the message.

Identity Extraction

(Identity Extraction is only for Input nodes)

At Server side – HTTPInput node should be selected with “Transport Default” for Identity Token type.

·         If Identity token type is chosen as “Transport Default”,then the user name and password are extracted from the HTTP header.

2.     Using Static Security Profile from Broker Registry and setting this Security Profile  in HTTPRequest node Security Profile Property

3.     Using the Username and Password credentials directly in Compute node esql by base64encoding

4.     By setting  OutputRoot.Properties IdentitySourceType, IdentitySourceToken, IdentitySourcePassword.

I have explained how to setup Security in details for the above said methods in my earlier post.

Using Security Profiles:

If “Default Propagation” is chosen in Security Profile of Input nodes:

Enables or disables identity propagation on input,output and request nodes. On the security enabled input nodes, you can choose to select only identity propagation, without specifying any other security operations(Authorisations), to make the extracted incoming identity or security token available for use in the other nodes in the message flow, such as output or request nodes. 

In this example,let me show you how to use Security related operations in the Input side by having the Security Profile at the Input node of a simple flow as below :-

In the bar file,mention the SecurityProfile name we have created earlier,

As I am passing wrong user id and password “mqbrkrs” and “mqbrkrs” in the SOAPUI Request,it is failing as below

Now when I am using correct userid “wmbuser1” and its password “pass” which is configured in LDAP ,then I am getting proper results.

REF:

http://blog.xebia.com/2012/09/23/setting-up-your-own-ldap-server-with-apache-ds/

How to create connection in LDAP?

http://krams915.blogspot.in/2011/01/ldap-apache-directory-studio-basic.html

 https://access.redhat.com/documentation/en-US/Fuse_MQ_Enterprise/7.1/html/Security_Guide/files/LDAP-AddUserEntries.html

 http://madhukaudantha.blogspot.in/2012/11/apache-directory-studio-tutorial-for.html

https://confluence.atlassian.com/display/CROWD/Creating+a+Connection+to+your+LDAP+Directory

LDAP or RDBMS? http://support.novell.com/techcenter/articles/ana20011101.html

Using Security profiles in Message Broker:

http://www.ibm.com/developerworks/websphere/library/techarticles/1002_fan/1002_fan.html

http://www.ibm.com/developerworks/websphere/library/techarticles/1008_fan/1008_fan.html

Various ways of implementing Security in Broker

To use the security profile from Broker registry:
Check these steps

1. Register the u/p
mqsisetdbparms brokerName -n securityIdName -u username -p password
eg:mqsisetdbparms RadBRK  -n TestId  -u  MyName -p MyPass

2. Create a Security Profile configurable service by using the mqsicreateconfigurableservice command:
mqsicreateconfigurableservice brokerName -c SecurityProfiles -o securityProfileName -n "propagation,idToPropagateToTransport,transportPropagationConfig" -v "TRUE,STATIC ID,securityIdName"
eg:mqsicreateconfigurableservice RadBRK  -c SecurityProfiles -o MySecurityProfile  -n "propagation,idToPropagateToTransport,transportPropagationConfig" -v "TRUE,STATIC ID,TestId"
3. Register securityProfileName with the request node, either in the bar editor or by using mqsiapplybaroverride.
========================================================
If to use direct hardcoding of Security credentials,
CREATE COMPUTE MODULE SomeCompute
   CREATE FUNCTION Main() RETURNS BOOLEAN
   BEGIN
      CALL CopyEntireMessage();
      SET OutputRoot.HTTPRequestHeader."Authorization" = 'Basic '||base64Encode(CAST('admin:admin123' as BLOB CCSID InputRoot.Properties.CodedCharSetId));
      RETURN TRUE;
   END;

   CREATE PROCEDURE CopyEntireMessage() BEGIN
      SET OutputRoot = InputRoot;
   END;

   CREATE PROCEDURE base64Encode(IN source BLOB)
   RETURNS CHARACTER
   LANGUAGE JAVA
   EXTERNAL NAME "com.ibm.broker.javacompute.Base64.encode";
END MODULE;


Another way is using Identity Security Token:
Before sending request to HTTPRequest node,code as below in a compute node,

SET OutputRoot.Properties.IdentitySourceType='usernameAndPassword';
SET OutputRoot.Properties.IdentitySourceToken = 'myUser';
SET OutputRoot.Properties.IdentitySourcePassword = 'myPassw0rd';

And In the HTTPRequest node ,to pass the Headers,you need to set as below in the bar file

Output is shown as below,

Global Cache in WMB and IIB

Global Cache in Message broker

An in-memory cache where some details can be stored and deleted later.By this way information can be shared between processes(Brokers,EGs).

Egs:

Correlation information

Avoiding the frequent backend interactions by storing some frequently queried data

Scenarios

http://pic.dhe.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fbn23731_.htm

http://pic.dhe.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fbn23732_.htm

This is built based upon Websphere Xtreme scale technology.

WebSphere® eXtreme Scale is an elastic, scalable, in-memory data grid. The data grid dynamically caches, partitions, replicates, and manages application data and business logic across multiple servers. WebSphere eXtreme Scale performs massive volumes of transaction processing with high efficiency and linear scalability. With WebSphere eXtreme Scale, you can also get qualities of service such as transactional integrity, high availability, and predictable response times.

The WebSphere Message Broker global cache is implemented using embedded WebSphere eXtreme Scale (WebSphere XS) technology. By hosting WebSphere XS components, the JVMs embedded within WebSphere Message Broker execution groups can collaborate to provide a cache.

There are three components involved in Global cache:-

1)Catalog server-To control

2)Container server-To hold cache data

3)Map-To map key value pair    

Default Global cache is only one Broker scoped.First EG which startsup will act as both Catalogue Server and Container Server.The next 3 Egs will act as Container Servers.And further Egs will connect as clients to the cache hosted in Egs 1-4.

Note : If we set policy as none,then we can individually tune the Execution groups cache components instead of letting them to start dynamically.

How Catalog server and Container server are connected?

The execution group started first will act as both catalog server and container server.And 3  execution groups can act as container server.All of these 4 EGs will have a port.

The remaining EGs will connect to these container servers as clients.They will not have any ports.

When you restart the broker, the execution groups may start in a different order, in which case different execution groups might perform Roles1, 2, 3, and 4.

When you use the default topology, execution group properties are read only; an error is issued if you try to change them. You can switch off the default topology by selecting a broker cache policy of none, and set properties explicitly for each execution group. For example, you might want to specify particular execution groups to host the catalog and container servers so that you can tune broker performance. 

mqsicacheadmin

Since resource statistics and activity traces can provide only an isolated view from each execution group, there is also the mqsicacheadmin command, which provides information about the entire global cache.

What happens if I shut down an execution group that hosts a catalog server?

If it is the only catalog server, then your cache is gone. However, if you have configured your cache to have more than one catalog server, and they have successfully handshaked on startup, then the remaining catalog server (or servers) will continue to run the cache without interruption. When you restart the execution group, the catalog server will rejoin the topology.

Interacting with the global cache or external grid

You can interact with the global cache or external grid by using a JavaCompute node. The node can put data into a map, retrieve data, and create a map if one does not exist.

What is cache policy file?

This cache policy file helps us to shape the cache topology across one or more than one brokers.

Sample Policy files can obtained in :

C:\Program Files\IBM\MQSI\9.0.0.1\sample\globalcache

As soon as we restarted the Broker(Named as Integration Node in IIB),the first EG(Named as Integration Server in IIB) came up with container server and Catalog server,

Second EG came as Container server:

More about Policy files:

In the below picture,the 4 types of policy files are shown.

If Default –then Broker can chooses default topology. Default Global cache is only one Broker scoped.First EG which startsup will act as both Catalogue Server and Container Server.The next 3 Egs will act as Container Servers.And further Egs will connect as clients to the cache hosted in Egs 1-4.

If None – We need to explicitly define the Egs for Catalog and Container servers

If Disabled – Then Global cache is disabled;This is the Default broker configuration

In last option –we can specify policy xml file to structure the cache topology for more than one brokers

*How to interact with Global cache?

This can be done only through Java.Either you can use JavaCompute node as depicted in the below example or you can use Java API in ESQL.This is well explained in the link https://www.ibm.com/developerworks/community/blogs/c7e1448b-9651-456c-9924-f78bec90d2c2/entry/how_to_configure_the_global_cache_for_esql?lang=en

If you set Globalcache to none,then you need to set the EGs Globalcache configurations explicitly,

1) stop the broker and execute the below command :

mqsichangebroker MB8BROKER -b none

2) start the broker

** mqsichangeproperties MB8BROKER -b cachemanager -o CacheManager -n policy,portRange,listenerHost -v none,2800-2819,localhost

**mqsichangeproperties MB8BROKER -e MB8EG -o ComIbmCacheManager -n enableCatalogService -v true

**mqsichangeproperties MB8BROKER -e MB8EG -o ComIbmCacheManager -n enableContainerService -v true

MQ Explorer Global Cache monitoring

In MQExplorer resourcemanager-->Global Cahce Activity log

You will be able to see the container server and catalog servers are up

and Global cache connected to cache name WMB.

1)Execute the below code in a flow where you are creating the map and updating the map with key-value pair,

importcom.ibm.broker.javacompute.MbJavaComputeNode;

import com.ibm.broker.plugin.MbException;

import com.ibm.broker.plugin.MbGlobalMap;

import com.ibm.broker.plugin.MbMessage;

import com.ibm.broker.plugin.MbMessageAssembly;

import com.ibm.broker.plugin.MbOutputTerminal;

import com.ibm.broker.plugin.MbUserException;

public class GC_Sample_JavaCompute extends MbJavaComputeNode {

      public void evaluate(MbMessageAssembly inAssembly) throws MbException {

            MbOutputTerminal out = getOutputTerminal("out");

      //MbOutputTerminal alt = getOutputTerminal("alternate");

            String Varkey="MyKey";

            String Varval="Radha";

            MbGlobalMap MyglobalMap = MbGlobalMap.getGlobalMap("MyMap");

            MyglobalMap.put(Varkey, Varval);

            MbMessage inMessage = inAssembly.getMessage();

            MbMessageAssembly outAssembly = null;

            try {

                  // create new message as a copy of the input

                  MbMessage outMessage = new MbMessage(inMessage);

                  outAssembly = newMbMessageAssembly(inAssembly, outMessage);

                  // ----------------------------------------------------------

                  // Add user code below

                  // End of user code

                  // ----------------------------------------------------------

            } catch (MbException e) {

                  // Re-throw to allow Broker handling of MbException

                  throw e;

            } catch (RuntimeException e) {

                  // Re-throw to allow Broker handling of RuntimeException

                  throw e;

            } catch (Exception e) {

                  // Consider replacing Exception with type(s) thrown by user code

                  // Example handling ensures all exceptions are re-thrown to be handled in the flow

                  throw new MbUserException(this, "evaluate()", "", "", e.toString(),

                              null);

            }

            // The following should only be changed

            // if not propagating message to the 'out' terminal

            out.propagate(outAssembly);

      }

}

When you enable the Resource statistics for this Execution Group,you can see that 1 map is used:-

Types of Global cache:

In 8.0.0.1 broker,Global cache was introduced and in this version WMB can interact with only Embedded Global cache.

In IIB,broker can also interact with External Websphere Extreme Scale grid.For this,we need to create Configurable service and provide the details of External Websphere Extreme Scale grid server.

In WMB 8,Multiinstance brokers cannot host Catalog and Container servers.

In IIB, Multiinstance brokers can host Container server.But not Catalog server still.