Overview of MessageSight

As I told in earlier post,I am going to give some details about configuration of MessageSight as MQTT server and using a sensor as MQTT client.
Further posts explain about them in detail...

IBM MessageSight is an appliance-based messaging server that makes it

extremely easy to make existing applications much more dynamic, much

more “real time.” The Internet is transitioning to the Internet of Things (IoT), where things or machines, connect to machines (M2M), and interact with reduced, if any, human intervention. With the numbers of devices in use increasing, organizations require a scalable, reliable, and cost-effective solution for connecting these devices to their system of records.

The IBM MessageSight messaging appliance helps to deliver the performance, value, and simplicity that organizations need for accommodating this multitude of devices and processing large volumes of events in real time. It supports the lightweight MQTT protocol natively or over web sockets.

IBM MessageSight can handle a large amount of concurrently-connected devices, while at the same time ensuring that communications are secure.

By enabling the use of messaging protocols, such as Message Queuing Telemetry Transport(MQTT), MessageSight is a highly scalable middleware messaging product that provides the full-duplex web communication that is required for these mobile, intranet, and Internet applications.

Further chapters explain in detail about the the features of MQTT,MessageSight, Configurations of Arduino and Sensor,Executing Publish-Subscribe pattern.

MQTT Overview

In a traditional IoT architecture there are one or more sensors, actuators and controllers. These typically talk to an edge gateway box and the edge gateway box talks across the wide area network (typically the internet) to the cloud or enterprise or server. There are a number of variations, for instance where a device is formed from a combination of sensors and a gateway.

The communication between the sensor and gateway occurs over a sensor area network. If this is an IP based network then MQTT can be used.

MQTT is lightweight message queuing and Transport protocol. MQTT, as its name implies, is suited for the transport of telemetry data (sensor and actor data).

MQTT is very lightweight and thus suited for M2M (Mobile to Mobile), WSN (Wireless Sensor Networks) and ultimately IoT (Internet of Things) scenarios where sensor and actor nodes communicate with applications through the MQTT message broker.

MQTT is designed to provide very efficient communication across the wide area network portion of the network, if IP based.

HTTP can be used for this purpose but it has some limitations.

Comparison of MQTT and HTTP

Table 1. Comparison of MQTT and HTTP

	MQTT	HTTP
Pattern	Publish/Subscribe	Request and Response
Complexity	Simple	More complex
Messae size	Small	Larger
Header	Compact binary header that is just two bytes in size	Status detail and other Header details are Text based
Service levels	Three QoS settings	All messages get the same level of service
Data distribution	Supports 1 to zero,1 to 1 and 1 to n	1 to 1 only

Realtime scenario on HTTP Message pattern and MQTT benefits over HTTP

Most IoT systems deal in events where an event can occur at any point e.g. when someone presses a doorbell - most of the time we do not know when this will occur. The normal model is the door bell sends a doorbell event that a person listens for and responds to. If this is handled with HTTP, the HTTP application would have to poll to ask whether the door bell been pressed and most of the time the answer is no. In fact it has to continuously poll to find out if the door bell has been pressed. This is an exceedingly inefficient use of the network, CPU and battery.

MQTT like HTTP uses an IP based network but enables applications to listen for events and go to sleep (or do something else) until the event occurs at which point the event will be pushed to the listening application. As MQTT is a publish-subscribe protocol many applications can listen for the same event and the MQTT server will deliver the event to all who have expressed an interest in it. The application here can be a server side or device(client) side application.

Why LDAP?Why not Database?

I liked the below explanation which I read on why to choose LDAP ..Read and enjoy the concept..

The use model is similar like how people use library cards or phonebooks. When you have a task that requires “write/update once, read/query many times”, you might consider using LDAP. LDAP is designed to provide extremely fast read/query performance for a large scale of dataset. Typically you want to store only a small piece of information for each entry. The add/delete/update performance is relatively slower compared with read/query because the assumption is that you don’t do “update” that often.

Imagine you have a website that has a million registered users with thousands of page requests per second. Without LDAP, every time users click a page, even for static page viewing, you will probably need to interact with your database to validate the user ID and its digital signature for this login session. Obviously, the query to your database for user-validation will become your bottleneck. By using LDAP, you can easily offload the user validation and gain significant performance improvement. Essentially, in this example, LDAP is another optimization layer outside your database to enhance performance, not replacing any database functions.

LDAP is not just for user validation, any task that has the following properties might be a good use case for LDAP:

1) You need to locate ONE piece of data many times and you want it fast

2) You don’t care about the logic and relations between different data

3) You don’t update, add, or delete the data very often

4) The size of each data entry is small

5) You don’t mind having all these small pieces of data at a centralized place
========================================================================
Some Q&A

First Question: Why do we have to use LDAP instead of Database?
LDAP is more of hierarchical and database is relational.
Faster Reads in LDAP and Faster writes in Database.Database can be used to implement the same functionality, only not with the same performance (i.e. not read-optimized).

Second Question: What factors determine to use LDAP over DB and DB over LDAP?
If your application does mostly reads on the information being stored into LDAP/DB and performance is a priority, then LDAP could be considered for its better performance.
On the other hand, if you're out of budget, you may not be able to get the "extra" LDAP server but are forced to stick with the database.

Comparison of WMB 5,6,1,7,8,and IIB v9 -quick notes

I have gathered information about the new features introduced in various versions of WMB v5,6,7,8 and IIB9 from various links.Hope this would be useful as quick notes to compare the features between versions of WMB and IIBv9.

Differences between WMB 5 and 6.1 –Particularly talks about whats new  in WMBv6.1

1)                  Reducing the time to get started with Message Broker

2)                  Debugger is attached with the Toolkit. Removal of RAC prerequisite in WMB5.Native debug is introduced.

3)                  Supporting WS-Security and WS-Addressing in Webservices

4)                  Integration with WSRR-Nodes introduced for WSRR are RegistryLookup and EndpointLookup

5)                  Built-in nodes for EIS access: SAP, Siebel and PeopleSoft.JCA based Websphere Adapters introduced.

6)                  Native support for very large file processing, including FTP.In WMB v5,for FTP,plugins have to be installed.In 6.1, it is native.FileInput node ,FileOutput node introduced.

7)                  MB Explorer Eclipse administration have been introduced. Performance Monitor – Easily view CPU, IO and other metrics in Eclipse

8)                  WSDL Drag Drop – Quickly create Web Services solutions

9)                  Discovery wizards for SAP, SEBL, PeopleSoft

10)               High Performance XML Parsing using IBM Research Technology

11)               EmailOutput node have been introduced

12)               TCPIP nodes introduced

13)               Collector node which collects messages from multiple,disjoint sources under multiple input conditions is introduced

14)               Configurable Services -Modify operational parameters without redeploy. Email and FTP node server addresses, LDAP configuration parameters

Differences between WMB  6.1 and 7 –Particularly talks about whats new  in WMBv7

1)              Runtime now consists of a single component (the broker) .In any system, fewer interacting components means fewer opportunities for failure. Tools now connect directly to the broker, and do not use a configuration manager. 

2)             The broker's persistent configuration is now stored exclusively on the file system, which means that you no longer need to create and maintain a system database, and no database product is required unless you want to access user databases.

3)             Message Broker V7 needs only a broker and a queue manager for a working configuration

4)             All pub/sub administrative concepts now form part of the queue manager's configuration and are no longer exposed or managed using Message Broker tools. As a result, Message Broker no longer has, or exposes the concepts of, a User Name Server, topology, collectives, topics hierarchy, or subscriptions. The broker's Publication node now uses the MQ pub/sub engine.

5)             You can deploy message flows directly onto execution groups without having to build BAR files or change perspectives, and deployment results are displayed synchronously in a new Deployment log, which lets you quickly ensure that message flows are deployed and working as expected.

6)             Historically, Message Broker development has been a "bottom-up" process, with low-level components such as nodes joined together to gradually create a complete application connectivity solution. Pattern-based development is a "top-down" methodology, in which high-level application parameters such as queue names or WSDL documents are described to the broker as pattern parameters to form the solution. The Message Broker V7 Toolkit contains a Patterns Explorer, which lets you browse available patterns for ones that might be applicable to the current problem. By applying a set of pattern parameters to a pattern instance, you can quickly generate a set of Message Broker artifacts, with common algorithms for aspects such as logging and error handling. You can then customize and deploy these artifacts in the usual way.

7)              SCA nodes have been introduced for integrating with Websphere Process Server

8)             PHPCompute node -  apply general-purpose message transformation logic using the PHP language, and complements the JavaCompute, Compute, XSLTransform, and Mapping nodes

9)             Sequence and Resequence node - These new nodes in Message Broker V7 help ensure the correct processing of messages in scenarios where ordering is critical. The Sequence node causes the broker to apply a sequence number to messages. The Resequence node lets messages arrive in any order, but will only propagate messages in the correct order (which can be determined by any source, including the Sequence node). The node includes comprehensive detection and processing in the event of missing or duplicate messages.

10)          Audit and monitoring can be done with more enhanced features such as,

·                     Extraction of business-relevant information from messages, and the collation and reporting of this information in a product such as WebSphere Business Monitor

·                     Extraction of fields from messages for input into products designed to detect patterns in messages (for example, for fraud detection), using a product such as WebSphere Business Events

·                     Recording of each message's payload for auditing purposes, with the ability to modify, resend, or replay sequences of messages

11)           Resource statistics has been introduced which can capture resources such as  JVM,Network Sockets  etc,.

12)          WebSphere MQ V7.0.1 introduced the concept of multi-instance queue managers.Concept is storing QM configuration in a shared network storage, and if the active queue manager fails, a second instance automatically becomes active by assuming the stopped queue manager's configuration. Message Broker V7 builds on this feature to add the concept of multi-instance brokers. It works in exactly the same way -- hosting the broker's configuration on network storage -- and makes your brokers highly available in the same way as your queue managers are. If you only wish to make your brokers or queue managers highly available, multi-instances can save the administration overhead of a third-party high-availability solution such as HACMP. However, the multi-instance brokers feature does not make any connected resources (such as databases) highly available, so in these scenarios another product that provides high availability is still required.

13)          Existing message flows and related files can be imported into and used by a Message Broker V7 Toolkit workspace. Files saved in the V7 tools can only be deployed to V7 brokers, while files saved in the V6 tools can be deployed only to brokers of that version. However, unchanged V6 BAR files can be deployed into a V7 broker.

Differences between WMB  7 and 8 –Particularly talks about whats new  in WMBv8

1)                  Apps & Libs for streamlined development, packaging, deployment & management.Applications and Libraries are introduced for Development .Group of messageflows and messagesets can be deployed together,started/stopped together which is called as Application.Message flow/Message set which is in subflow can be reused wherever needed and is dynamically linked which is called as library.

·                     Encourages designing for reuse

·                     Simplifies deployment & management

2)                  Simple & high performing data modelling with DFDL

·                     New standard for binary, text & industry data formats

3)                  Healthcare Connectivity Pack

 • healthcare patterns -Range of integration patterns to quickly address common scenarios : HL7 to HL7, HL7 to Reports

·                     MB now has the ability to read data from a range of medical devices

- e.g. heart rate monitors, infusion pumps – Supports a wide set of device manufacturers

- GE, Philips, Cardinal Health  and more

4)                  Web 2.0 browser console with REST management API for ubiquitous access

·                     Focus on non-admins to understand broker resources

·                     Designed as a complement to MBExplorer; MB Administrators can continue to use MB Explorer

·                     Web UI features based on role • e.g. Power users can start message flows • e.g. End users can view only

5)                  Record & Replay to capture, view, edit & replay in-flight messages

6)                  Message flow activity trace for rapid flow analysis by end-users

·                     ActivityLog is a great tool for providing a general overview of the health of each flow, it is possible for example to see at a glance if a flow is processing message and if these are reaching the expected output nodes

7)                  Mobile Services support with IBM Worklight and MB Toolkit integration

8)                  Built-in WebSphere Extreme Scale global cache for highly available and scalable data sharing

·                     New Built-in facility to share data between multiple brokers –

·                     Global cache shared between flows, execution groups & brokers

·                      Seamless access to global cache using from all message broker flows and nodes –Typical scenarios include multi-broker request-reply and multi-broker aggregation

·                     Full resource manager statistics shows cache interactions and other relevant statistics – Activity log shows cache agent operations for write and read

9)                  Support for MQ 7.1 and 7.5

10)               Mqsipackagebar

WebSphere Message Broker (WMB) version 8 has seen several new features arrive in its first fixpack, 8.0.0.1. One of these is the ability to create deployable BAR files in your runtime environment.

Previously, a WMB Toolkit installation was needed to create BAR files through either the GUI or the 'mqsicreatebar' command. This command would compile all of your runtime resources (message flows, message sets, Java™ projects, etc) and place them in a single deployable file, that would then be interpreted by either the Configuration Manager (in older versions), or the Broker itself.

WMB 8.0.0.1 introduces the ability to deploy message flows without needing to compile them into *.cmf files. As a result, the 'mqsipackagebar' command was created. This command is shipped with the WMB runtime, not the Toolkit. This means you do not need the Toolkit installed to create a bar file. With the proper configuration, you can even run this command on machines that do not have a WMB runtime installed.

However, 'mqsipackagebar' isn't as powerful as its older brother: you will still need to compile your message sets and Java projects prior to using this command. The 'mqsipackagebar' command can be used to add these pre-compiled resources to a BAR file, but it cannot actually compile them. For the same reason, when you add a message flow or ESQL to a BAR file with 'mqsipackagebar', it will not be compiled into a .cmf, but will appear as a *.msgflow or *.esql file.

Differences between WMBv8 and IIBv9 –Particularly talks about whats new  in IIBv9

1)                  MQ service and DatabaseService discovery  to facilitate sharing of service definitions

Service definitions allow you to make best use of available resources – Facilitates sharing of service information between users and systems – Allows users to understand interfaces (e.g. CustomerAddress.Update operation) – Provides a connector with which to exchange technical configuration (e.g. hostname)

2)                  WESB Conversion: Import and conversion of mediation flows

Along with the IBM Integration Bus release, IBM consolidated ESB product offerings and that meant discontinuing WebSphere ESB. WESB will be fully supported for at least another 5 years, but there will not be any new major releases. To ease with migration, IBM will provide 'transfer' licenses to IBM Integration Bus. IBM will also provide WESB Conversion Tools on a phased approach.

3)                  Migration from WMB V6.1, V7 and V8

4)                  New Decision Service node

- The embedded rules engine is provided for high performance and it replaces the IAM9 Support Pac.

-There is a new Decision Service node which can invoke a new built-in rules engine or a third party rules engine.

-Identifies inputs to business rules from in-flight data

• e.g. the customers order from whole request • e.g. the item price from key fields…

 – Invokes the built-in rule engine to perform business logic

- Open interfaces for 3rd party and user engines(Optional governance of rules through remote ODM Decision Center)

      5)     Create rules directly inside Integration Bus toolkit

      6)    Improved performance in DFDL

      7) Integration with IBM MessageSight(Appliance based messaging)

      8) Single installation package contains ALL required software • MQ 7.5, Integration Bus (Runtime, Toolkit, Explorer) • Available on Windows and Linux platforms

  9) Sub-second timeout on Aggregation nodes – More granular timeout values (ms) can now be specified on the aggregation nodes – Allows for quicker timeouts when aggregating data from usually fast responding systems

 10) Developer Edition-There is now a free edition of Integration Bus for use for development and testing. It is fully functional with the only limitation being a throughput rate of 1 transaction per second per integration flow. The installation package includes all of the required software: runtime, toolkit and MQ 7.5. This is a great and exciting offering to make it easier for customers and developers to learn and try out Integration Bus.

OAUTH – explained with a simple analogy

OAuth is an authentication protocol that allows you to approve one application interacting with another on your behalf without giving away your password.

 Quiet confused isn’t it? Lets discuss this with a simple example..

 You got to know that there is Internet problem at your house.So,you call the Internet Service provider Customer care.

They send you a person.Now,you give your  house keys to that person and leave to office for some urgent work.

You are now taking a risk..

You need to trust the person for,

   1)He should not take away the things at your home

   2)He should not make copies of the keys and distribute to his friends

   3)He now knows you by face and can use the keys to enter into your house

 This is a simple analogy to the problem where  user shares his credentials of his gmail/facebook/twitter  or any  accounts with some third party website, so that  the third party website can access his gmail/facebook etc  to place  some updates.

 This is where OAuth comes to sort out the issue…

 Oauth gives the access to the third party to only the stuff the user wants them  to use in his gmail/facebook/twitter etc.

 In Oauth,Password is not shared.

 So,With Oauth ,its now like , 

You got to know that there is Internet problem at your house.So,you call the Internet Service provider Customer care.They send you a person.Now,you give a special key which can open only one room and which will restrict access to rectify only the Internet connection problem.You can anytime get back the special keys from the person.

 Now you handover the special keys to the person and  leave to office for some urgent work.

 You now can have a great relief,because..

   1)He can not take away the things at your home

   2)Even if he makes copies of the keys and distribute to his friends,its of no use.And anytime you can get back the special keys.

 This is the analogy of how Oauth works..

 1)You want a  thirdparty to access your facebook by posting Weather updates in your Timeline

2)You are not going to share  your facebook username and password to the thirdparty

3)First,the Thirdparty redirects you to your facebook after you enter your username and password(Note that you are not entering in thirdparty website.You are actually entering credentials in your facebook.)

4)Now facebook asks you,whether you are fine that the thridparty accessing your facebook for posting in your timeline on weather updates

5)You give Grant access.This is the stage where access token is generated

6)The Access token is shared with the Thirdparty

7)From then on,the Thirdparty can use the AccessToken to enter into your facebook and post the weather updates in your timelines

 What if the thirdparty tries to post some sports news?

Access denied error will be thrown for the thirdparty as it is restricted only for Weather updates.This is defined in Access token.

 What if the AccessToken is hacked by some other website from the thirdparty?

Since it is not your passwords,the thirdparty cannot misuse your facebook account.However,it will be still able to post Weather updates in your timelines (whatever the thirdparty was allowed to do).

Hence,you can deny access to that Thirdparty anytime by getting into your settings page.

 Now… go back to the first two lines of this post and read again..

Introduction to MQTT and MessageSight

Message Queuing Telemetry Transport (MQTT) can be used to connect various types of smart devices and applications that are measuring,monitoring, and, in certain cases, controlling the world today.

IBM MessageSight MessageSight is an appliance-based messaging server that is designed

to handle large numbers of connected clients and devices and process high volumes of

messages with consistent latency.

MQTT features:

·         Publish/subscribe

·         Topics and subscriptions

·         Quality of service (QoS) levels-MQTT defines three quality of service (QoS) levels for message delivery, with each level

designating a higher level of effort by the server to ensure that the message gets delivered.

Higher QoS levels ensure more reliable message delivery but might consume more network

bandwidth or subject the message to delays due to issues, such as latency.

·         Retained messages-With MQTT, the server keeps the message even after sending it to all current subscribers. If a new subscription is submitted for the same topic, any retained messages are then sent to the new subscribing client.

Comparison of MQTT and HTTP

Although comparison is often made between MQTT and other common protocols, the most

useful comparison is with the hypertext transfer protocol (HTTP) for the following reasons:

·         HTTP is the most widely used and available protocol. Almost all computing devices with a

TCP/IP stack have it. In addition, because HTTP and MQTT are both based on TCP/IP,

developers need to choose between them.

·         The HTTP protocol uses a request-and-response model, which is currently the most

common message exchange protocol. MQTT uses a publish/subscribe model. Developers

need to understand the relative advantages of each type of model.

                                      

Quick comparison of MQTT and HTTP

Good explanation on MessageSight

These days, mobile apps tend to follow a common well-established pattern when it comes to communications with enterprise systems. They tend to use standard paradigms, often leveraging HTTP and existing web server infrastructures with Enterprise JavaBeans(EJBs), request response and so forth. These models have proven to work well and to be scalable for traditional applications.

But once one tries to add more dynamic content to the applications, it becomes more difficult to scale. What if I would like to display continuously changing information in real time within my application? For instance, a live feed of news, play-by-play updates from a football game, continuous pressure readings from valves in a factory, incident reports in a city, the ability to chat with other users of the application and so on. It suddenly becomes more difficult to receive the data in real time when using conventional means. On the client side, using techniques like long polling are bad for the battery life of the device. On the server side, supporting a large number of concurrent users requires considerable amounts of resources.

That is when a new paradigm is required. Ideally, we need a lightweight bidirectional means of communication, one that is event driven with a very simple semantic: connect, subscribe, publish, disconnect.

If you want to display real time updates in your application, it should be as simple as connecting to your server and subscribing to a set of topics you are interested in. When a new message arrives, a callback function should automatically be called and then used to display the new content in the app.

That’s where IBM MessageSight can help you.

IBM MessageSight is an appliance-based messaging server that makes it extremely easy to make existing applications much more dynamic, much more “real time.” It supports the lightweight MQTT protocol natively or over web sockets. Using native clients for iOS and Android, or using a cross platform JavaScript client, developers can quickly start adding exciting dynamic real-time content to their existing applications.

Designed to scale since inception, IBM MessageSight can handle a large amount of concurrently-connected devices, while at the same time ensuring that communications are secure.

So, rethink what you can do with your mobile applications. It is time to think of enriching your application with dynamic, real-time content and differentiate yourself from the competition.

REF: http://www.slideshare.net/JohnSamuel17/ibm-messagesight-for-mobile-and-the-internet-of-things-29075268 -Good Introduction about MQTT and Messagesight