Loved this SOA Presentation...very Catchy

Thought of sharing this video presentation which I liked a lot ..Hope you all enjoy... http://www.youtube.com/watch?v=A3_QlYJRVvk

HTTP in Message Broker

HTTP nodes in Message Broker

The HTTP-Input node accepts a request and theHTTP-reply node  replies to the destination

Various HTTP nodes in Message Broker

HTTP Input

HTTP Request

HTTP Reply

Through HTTPInput node ,we can listen to HTTP requests from Clients.So,if we use HTTPInput node ,then our  flow becomes the HTTP Server which is exposing its service as Webser vice.

The path url is property will get the HTTP requests intended to this webservice.Sample could be /CustomerTest

Hence the HTTP client should send the requests to the HTTP server as http://<server_ip>:7080 /CustomerTest

7080 – is the port of broker http listener.It is biphttplistener

This is the only HTTP port available for Broker.So,if we want to call another HTTP Service,we need to differentiate it based upon the path url. Eg: /EmployeeTest

If we want,we can also enable http port of execution groups.

To view Brokerwide listener

mqsireportproperties MyBroker -b httplistener -o AllReportableEntityNames -r

(To disable Brokerwide Listener set startListener='false' in mqsichange properties)

HTTPListener

  uuid='HTTPListener'

  enableSSLConnector='false'

  threadPoolSize=''

  traceOverrideLevel=''

  traceOverrideSize=''

  traceLevel='none'

  traceSize='102400 KB'

  javaDebugPort=''

  startListener='true'

HTTPConnector

  uuid='HTTPConnector'

  address=''

  port='7080'

  allowTrace=''

  maxPostSize=''

  acceptCount=''

  bufferSize=''

  compressableMimeTypes=''

  compression=''

  connectionLinger=''

  connectionTimeout=''

  maxHttpHeaderSize=''

  maxKeepAliveRequests=''

  maxSpareThreads=''

  maxThreads=''

  minSpareThreads=''

  noCompressionUserAgents=''

  restrictedUserAgents=''

  socketBuffer=''

  tcpNoDelay=''

  enableLookups='false'

HTTPSConnector

  uuid='HTTPSConnector'

  algorithm='Platform Default'

  clientAuth='Platform Default'

  keystoreFile='Platform Default'

  keystorePass='********'

  keystoreType='Platform Default'

  truststoreFile='Platform Default'

  truststorePass='********'

  truststoreType='Platform Default'

  sslProtocol='Platform Default'

  ciphers='Platform Default'

  address=''

  port=''

  allowTrace=''

  maxPostSize=''

  acceptCount=''

  bufferSize=''

  compressableMimeTypes=''

  compression=''

  connectionLinger=''

  connectionTimeout=''

  maxHttpHeaderSize=''

  maxKeepAliveRequests=''

  maxSpareThreads=''

  maxThreads=''

  minSpareThreads=''

  noCompressionUserAgents=''

  restrictedUserAgents=''

  socketBuffer=''

  tcpNoDelay=''

  enableLookups='false'

=====================================================================

To view Executiongroup HTTP listener

mqsireportproperties MyBroker  -e default -o HTTPConnector -r

HTTPConnector

  uuid='HTTPConnector'

  userTraceLevel='none'

  traceLevel='none'

  userTraceFilter='none'

  traceFilter='none'

  port='0'

  address=''

  allowTrace=''

  maxPostSize=''

  acceptCount=''

  bufferSize=''

  compressableMimeTypes=''

  compression=''

  connectionLinger=''

  connectionTimeout=''

  maxHttpHeaderSize=''

  maxKeepAliveRequests=''

  maxSpareThreads=''

  maxThreads=''

  minSpareThreads=''

  noCompressionUserAgents=''

  restrictedUserAgents=''

  socketBuffer=''

  tcpNoDelay=''

  explicitlySetPortNumber=''

  enableLookups=''

  enableMQListener=''

  shutdownDelay=''

  allowCrossConnectorPolling=''

  autoRespondHTTPHEADRequests=''

=====================================================================

To switch to the embedded execution group-wide listener:

mqsichangeproperties MyBroker -b httplistener -o HTTPListener -n startListener -v false

mqsichangeproperties MyBroker -e default -o ExecutionGroup -n httpNodesUseEmbeddedListener -v true

--------------------------------------------------------------------------------------

To switch to the Broker-wide listener:

mqsichangeproperties MyBroker -b httplistener -o HTTPListener -n startListener -v true

mqsichangeproperties MyBroker -e default -o ExecutionGroup -n httpNodesUseEmbeddedListener -v false

--------------------------------------------------------------------------------------

How to check HTTP port assigned for an execution group?

C:\Program Files\IBM\MQSI\8.0.0.1>mqsireportproperties RadBroker -e default -o H

TTPConnector -a

HTTPConnector

  uuid='HTTPConnector'

  userTraceLevel='none'

  traceLevel='none'

  userTraceFilter='none'

  traceFilter='none'

  port='0'

  address=''

  allowTrace=''

  maxPostSize=''

  acceptCount=''

  bufferSize=''

  compressableMimeTypes=''

  compression=''

  connectionLinger=''

  connectionTimeout=''

  maxHttpHeaderSize=''

  maxKeepAliveRequests=''

  maxSpareThreads=''

  maxThreads=''

  minSpareThreads=''

  noCompressionUserAgents=''

  restrictedUserAgents=''

  socketBuffer=''

  tcpNoDelay=''

  explicitlySetPortNumber=''

  enableLookups=''

  enableMQListener=''

  shutdownDelay=''

  allowCrossConnectorPolling=''

  autoRespondHTTPHEADRequests=''

--------------------------------------------------------------------------------------

HTTP Asynchronous nodes

You can make HTTP requests and receive a response asynchronously using the HTTPAsyncRequest and HTTPAsyncResponse nodes.

HTTP is synchronous: a client issues a request and waits for the corresponding response before anything else is done.
But we can make the client to act for Asynchronous HTTP client as well.In this way,there wont be any performance issues.

Basically, The HTTP client is only aware of a synchronous call.

Various scenarios of client connections:

1)Same client opening many number of connections to HTTP Server (HTTP URL: http://<server_ip>:7080 /CustomerTest)

2)Multiple clients opening multiple number of connections to HTTP Server :- same url (HTTP URL: http://<server_ip>:7080 /CustomerTest)

Workload balancing can be achieved via Load Balancer

If  I have multiple HTTP clients want to communicate with 2 brokers in load balanced way,but there is only one url, HTTP URL: http://<server_ip>:7080 /CustomerTest ,both the brokers are listening at same port.

When there is no load balancer,the client will have an url set at their request.So,only either one of the broker can be set at the url path.

In the case,if we use load balancer,then we can url the load balancer ip at the HTTP request url of HTTP client.By this way we can route the requests to both the brokers as the load balancer will act as network dispatcher.

Moving HTTP connection processing away from broker by using a proxy

The proxy is a servlet engine running inside Websphere Application Server just like Message broker internal process.

Message broker biphttpistener can support maximum upto 1200 concurrent sessions.

If we want more than 1200 sessions,then we can go for proxy engine.

Proxy engine is capable of handling more number of concurrent sessions than the default broker http listener.

HTTP Persistent and Non-Persistent connections

When HTTP request is made from a client,a new TCP connection is opened from client to Server.Once the response is sent by HTTP server through this TCP connection,the connection is terminates.For the next request from the same client ,a new TCP connection is opened.This is called non-persistent connections.This is also called as Short-lived connections.If the clients want to make individual connections then this type of connections can be choosen.But considering the memory and resource utilisations due to session creations ech and everytime would be a overhead.This could be preferred when many number of clients want to have individual connections with HTTP server.

For non-persistant connections,there may be ‘n’ number of clients connecting to the HTTP server.We can restrict the maximum number of concurrent connections by limiting ‘maxthreads’  property.Default is 250.

Command to change is: mqsichangeproperties MyBroker -b httplistener -o HTTPConnector -n maxThreads -v 2000

But machine resources, such as amount of addressable memory, determine the absolute maximum number of concurrent HTTP client connections that you can make to Message Broker, and these resources may be exhausted before the maxThreads limit is reached.

Increase the heap size of the JVM running within the biphttplistener process to let the JVM allocate more threads for the incoming connections. The default setting for the listener JVM is 192 MB, but you should increase it to 512 MB when dealing with large numbers of concurrent connections. To change the value, modify the Message Broker registry.

(http://www.ibm.com/developerworks/websphere/library/techarticles/0608_braithwaite/0608_braithwaite.html)

Hence always better to prefer HTTP persistent connections.This is also called as Long-Lived connections.This can be preferred when small number of clients want to have high volume of throughput from HTTP server.(When small number of clients are there,there will be small number of connections opened.But they are opened for a long time to serve all the requests instantly through the TCP session path).

There is  a property called maxKeepAliveRequests that specifies how many number of requests can be made in a single connection at the maximum.Default is 100. The next request the client sends creates a new socket and again can send up to 100 requests.        

Command to change this property according to our requirement

mqsichangeproperties MyBroker -b httplistener -o HTTPConnector -n maxKeepAliveRequests -v 0.

Promoted Properties vs UDP

Broker properties are divided into four categories:
Properties that relate to a specific node
Properties that relate to nodes in general
Properties that relate to a message flow
Properties that relate to the execution group
Experiment1
* Promote Property for main flow - Input Mq node Queue property and Output node Queue property
The properties are seen when seeing properties of message flow
Properties are promoted to message flow level
After creating bar file ,just the overriding properties are changing with new values
After overriding with new values,the bar file is just saved WITHOUT RECOMPILATION.
If we recompile,again it get backs to the same old value;the overridden values are not getting effected.
Just deploy now;The overriding properties works fine :)
Experiment 2
* Subflow is used in the middle of MQInput and MQOutput node in the main flow
*The subflow has a MQOutput node whose Queuename is promoted
*As it is promoted,the queue name has become the subflow's configurable property
*Now,when subflow is referred in the main flow,the subflow's property is shown.But it says,the subflow's promoted property cannot be overridden ."Property override is not supported on the subflow node"

WMB and Error Handling

(WMB) Error handling

When we design any message flow, we often do not give more emphasis on error handling. Well as per my experience I found this error handling techniques & design principles more crucial than designing the happy path.
So here I am with few details on how to handle unhappy path in WMBV6.0 with information about the message flow error behavior.


Design Consideration

·                     Connect the Failure terminal of any node to a sequence of nodes that processes the node's internal exception (the Failure flow).

·                     Connect the Catch terminal of the input node or a TryCatch node to a sequence of nodes that processes exceptions that are generated beyond it (the Catch flow).

·                     Insert one or more TryCatch nodes at specific points in the message flow to catch and process exceptions that are generated by the flow that is connected to the Try terminal.

·                     Ensure that all messages that are received by an MQInput node are processed within a transaction or are not processed within a transaction.

Understanding the Flow Sequence

·                     When an exception is detected within a node, the message and the exception information are propagated to the node's Failure terminal ( Diagnostic information is available in the ExceptionList). 

·                     If the node does not have a Failure terminal or if it is not connected, the broker throws an exception and returns control to the closest previous node that can process the exception. This node can be a TryCatch node  (Root & LocalEnvironment are reset to the values they had before) or the MQInput node .

·  If the catch terminal of the MQInput node is connected, the message is propagated there (ExceptionList entries are available; Root & LocalEnvironment are reset to the values they had before).
      1) If message is transactional and catch terminal is connected ,then message goes to catch terminal and thrown to Input node due to Throw node in Catch.And then backout count is checked.And based upon backout count,message is retried.Finally message goes to Failure terminal if it is connected.If Failure terminal is not connected,then Backout Q or System DLQ is checked.
      2)  If message is not transactional and catch terminal is connected,then message goes to catch terminal and thrown to Input node due to Throw node in Catch. But backout count is not checked and also does not go to Failure terminal even if it is connected and it is discarded.


What if in Failure path,there is some error?
Error path should be properly handled by Failure terminal's subsequent nodes.Otherwise,successive errors will be discarded.

Otherwise if the catch terminal is not connected, the transactionality of the message is considered.

·                     If the message is not transactional, the message is discarded. Otherwise , if it is transactional, the message is returned to the input queue, and it is read again, whereupon the backout count is checked.

·                     If the backout count has not exceeded its threshold, the message is propagated to the output terminal of the MQInput node for reprocessing. Otherwise if it is exceeded & if the failure terminal of the MQInput node is connected then the message is propagated to that path. (Root is available but ExceptionList is empty)

·                     If the failure terminal of the MQInput node is not connected, the message is put on an available queue, in order of preference; message is put in backout queue, if one is defined; otherwise in dead-letter queue, if one is defined. If the message cannot be put on either of these queues, it remains on the input queue in a retry loop until the target queue clears.(It also records the error situation by writing errors to the local error log)

The below are the info that can be captured from the ExceptionList:

·         MessageId of the incoming message

·         Flow name(Name of the message flow in which exception has occurred)

·         Node name in which the exception has occurred

·         Error time (Time when the exception has occurred)

·         Error Number (Error Number for the exception which is generated by Message broker)

·         Error Severity(Severity of the exception)

·         ErrorDetails(Complete description of the error)

My testing scenarios:

With Javacompute node:

Input queue -àHas Backout queueàBackout threshold =3

1)      Scenario:

Javacompute node has exception catching code as below.

Message flow  nodes’ Failure and Catch  are not connected

 Transactionality is “YES”,

catch (SQLException sqx ){

           MbUserException mbue = new MbUserException(this, "evaluate()", "","", sqx.toString(), null);

             throw mbue;

         }

      Observation:

·         Error thrown to MQInput node

·         Message is retrying with Backout count in MQMD as 1,2,3 consecutively

2)Scenario:

   Javacompute node has exception catching code

    MQInput node’s catch is connected

      Transactionality is “YES”

     Exception handling subflow ends with MQOUTPUT node (ERRORQ)  

      Observation:

·         Error thrown to MQInput node

·         Exception once thrown to MQInput node,is sent to catch terminal

·         The message  is written to MQOUTPUT node (ERRORQ).This message written has only original message

·         Retry is not actually performed because the message is not thrown to the input queue.

         3)Scenario

                        Javacompute node does not have exception catching code .Just like below,

catch (SQLException sqx ){

           

            }

                   MQInput node’s catch is connected

      Transactionality is “YES”

     Exception handling subflow ends with MQOUTPUT node (ERRORQ)  

                 The below Exceptionlist populating message is written in subflow,  

                    DECLARE start REFERENCE TO InputExceptionList.*[1];

             WHILE  start.Number IS NOT NULL DO

                 

                  SET OutputRoot.XML.Exception.messageNumber = start.Number;

                  SET  OutputRoot.XML.Exception.messageText = start.Text;

           

                  MOVE start LASTCHILD;

             END WHILE;

Observation:

Behaved same as Scenario 2 but the message at output is,

hi

<Exception>

 <messageNumber>6233</messageNumber>

 <messageText>Problem encountered obtaining JDBC connection</messageText>

</Exception>

4)Scenario

  Only if it is connectivity failure,the code has to retry.

 Otherwise the code should not retry

 

               

 

  

Database & WMB

Database node vs Compute node

The actual difference between the Database node and the Compute node is NOT the accessing of databases as both nodes can do this, which does mean their names can be slightly confusing. The actual (and only) difference is that a ComputeNode always creates a new message (even if you never use it), whereas a Database node never creates a new message. You can obviously do more in a Compute node than in a DB node (like create a changed OutputRoot message which you can not do in a DB node). But if all you want to do in the node is access and/or change the Envrionment or LocalEnvrionment trees then you can do this in the Database node which is why the patterns used the DB node where they do as they are simply setting a flag in the LE.

JDBC vs ODBC

i think JDBC may be faster than ODBC, our supplier's ODBC is not multi-threaded yet whereas their JDBC is, there are SQL limitations in the ODBC which are over-come by using JDBC, and ODBC may be phased out by Microsoft. i would propbaly offer JDBC as an alternative rather that a straight replacement to our customers.

As discussed below are the differences between ODBC and JDBC connections 

• OBDC is Microsoft drivers and uses dlls, JDBC is Java drivers and uses jar files. 
• ODBC is procedural oriented and JDBC is object oriented. 
• ODBC can't be directly used with Java because it uses a C interface. 
• ODBC makes use of pointers which have been removed totally from Java. 
• ODBC mixes simple and advanced features together and has complex options for simple queries. But JDBC is designed to keep things simple while allowing advanced capabilities when required. 
• ODBC requires manual installation of the ODBC driver manager and driver on all client machines. JDBC drivers are written in java and JDBC code is automatically installable, secure, and portable on all platforms. 
• JDBC API is a natural Java Interface and is built on ODBC. JDBC retains some of the basic feature of ODBC. 
• JDBC is Java Database Connectivity. It enables java programs to execute SQL statements. JDBC makes it possible to write a single database application that can run on different platforms and interact with different DBMS. JDBC is language dependent i.e it is used only to make connectivity with java. 
• ODBC is open database connectivity. The goal of ODBC is to make it possible to access any data from any application, regardless of which database management system (DBMS) is handling the data so it is language independent. 
• ODBC is language independent and JDBC is language dependent. 

·         If you are creating ODBC connections to your user databases, ensure that you correctly create 32-bit, 64-bit, or both, connections for each DSN. If you are creating JDBC connections, their use is independent of 32-bit or 64-bit mode.


There are no performance issues for JDBC connections as compared to ODBC connections, we need to configure it on the server to use it in InfoView , in the jar files, and the end users cannot modify it like ODBC connections. 

Yes JDBC libraries are faster than ODBC because it uses JAVA jars and not windows dlls, so there can be a slight better performance die to platform independency if using any non windows Server, but again as I told earlier it completely depends on the requirement and scenario . 

There are no major issues it’s just the configuration which is a bit complex as compared to ODBC , but it is more reliable and once done it works perfectly. 

Connecting to Database from message broker:

·         For one database ,one dsn ,one message flow,we need one thread to access.

·         When connection gets broker?

The broker makes the connections when it needs to use them, and they remain open until one of the following events occurs:

• The message flow has been idle for one minute
• The message flow is stopped
• The broker is stopped

ODBC.ini

ODBC .ini files that are supplied, and the information contained in these configuration topics, include all the connection parameters that are supported for connections to your databases.

ODBC connections –Creating DSNs

In Windows

Fro all databases like DB2,Mysql,… we just need to choose ODBC driver in User or System DSN and create a DSN.

But for ORACLE,we need to go to Windows registery REGEDIT ,then HKEY_LOCAL_MACHINE\SOFTWARE\ODBC\ODBC.INI. Expand that location, and right-click your DSN entry.Then give value in string(refer http://publib.boulder.ibm.com/infocenter/wmbhelp/v8r0m0/index.jsp?topic=%2Fcom.ibm.etools.mft.doc%2Fah14442_.htm)

In Linux/Unix (Using WMB ODBC Database extender)

Copy  the ODBC.ini and ODBCINST.ini  files from installation path(install_dir/ODBC/unixodbc/) and place in WMB /var/mqsi path.

Then point the environment variables ODBCINI  and ODBCSYSINI to this in

This copied path.

Then edit the ODBC.ini file with your required Database paramaters.

For DB2

The DB2 alias name configured during DB2 remote cataloguing for DB2 remote database in Client/server modal,should be used for DB2 alias name.

Enabling JDBC in WMB

First install the driver of the database if the driver is not in your broker machine.

Then mqsicreateconfigurable service to enable JDBC connection with that database.Include security identity.

Then we can use the JDBC only with DatabaseRetrieve node and DatabaseRoute nodes.

ODBC and JDBC drivers in Broker

Broker contains ODBC drivers for the Databases supported by Broker.

But the Broker will not contain the JDBC drivers for even the Databases supported by Broker.If any JDBC driver is required,it will has to be download and kept in BROKER lib folder and referred in jars url property of JDBCProvider.