Sunday, February 9, 2014

The Need for Visualizing Data in the Enterprise and Satisfying it

We live in an interconnected world today. There are many systems which an individual would use to proceed with his or her daily tasks. Such systems generate huge amounts of data. Being able to visualize all this data on one personalized dashboard is an important requirement in today's business world.

Enterprises which provide such systems should provide tools with which they can access this data and then present it in a meaningful manner. Also, such tools need to provide the users of such systems the ability to personalize an individual's data dashboard according to his/her functional as well as aesthetic requirements. 

In turn, providers of tools need to cater data visualization toolboxes where enterprise users can create visualizations linking up different data views. This underlying tool should have the capability to link up to a multitude of such data sources, convert this data into a visually meaningful state and also provision for personalizing such representations. 

WSO2's User Engagement Server is one such tool which comes with the WSO2 middleware product suite. This post will look at how WSO2 UES caters to the data visualization needs of today's enterprise users.

UES in a Nutshell

The above picture gives the functionality of WSO2 UES in a nutshell. Within an organisation there are different sources of data. UES provides the ability to represent this data in suitable representations. Each such representation is called a gadget and a collection of such gadgets are put together in an appealing layout to create a dashboard. Gadgets and dashboards can be created and published to an enterprise store through which users can reuse these artifacts. These artifacts are shared across user categories based on roles. Once a dashboard or microsite is created and published these can be viewed through any device. 

Business Requirements and how WSO2 UES supports them

1. Rapid user interface development 
- In an organisation the main requirement would be the ability to create proper representations of data, in a rapid manner, with ease, and in a way which caters to changing needs. The ability of being able to create visually correct data views from data sources is a key requirement in a data visualizing tool. An enterprise should be able to cater to the changing needs of the environment in order to be agile - this requirement leads to being able to view the same data in different dimensions and in different views over different time periods. These requirements are satisfied via the provision of gadgets of various types such as line charts, bar charts, pie charts, trends etc to satisfy visual representation needs of different types of data. These gadgets can be arranged in a user's dashboard as he/she prefers based on the individuals viewing priorities. WSO2 UES provides the ability of a single dashboard shared among many users of a single role to be personalized for each individual. The intuitiveness that is displayed in WSO2 UES enables business users themselves to create the required views in a simple and rapid manner.

2. Advanced application development
- Each of the dashboards that are created via WSO2 UES is in fact a jaggery application. With jaggery these dashboards are powerful in their strength as a complete web application which can easily integrate with HTTP focused web services and server side logic support. WSO2 UES also comes with an inbuilt web IDE for jaggery which can be used for ease of use development of the dashboards created. Through the management console of WSO2 UES it is possible to govern the artifacts used and hence application governance of the visualization artifacts is also supported.

3. Providing role based views 
- In an enterprise there would be several categories of users which use enterprise systems. In order to better manage such users the concept of user roles is widely used where the individuals of the organisation are divided into groups which are assigned roles based on their functional areas and responsibilities. These principles also apply when providing data visualizations. Certain data views would be more beneficial to certain categories of users and also some of the data would be access controlled based on the confidentiality needs of the organisation. These requirements are satisfied via WSO2 UES by being able to define access to gadgets and dashboards to specific roles. The gadgets and dashboards published can be shared based on roles available in the organisation's user store which is linked to UES.

4. Discovery of views 
- Several individuals within an organisation would need to look at the same representation of data. It would be inefficient if each of these individuals had to create these representations over and over again to use in one's dashboard. Similarly there would be dashboards carrying information which are of interest to several users - again creating these for each and every user is pointless. WSO2 UES provides the ability to create and publish such data representational gadgets and  dashboards so that they can be shared. These are published into the embedded store within UES and can be either included or accessed by users who wish to do so. It is also possible to do activities such as searching for already created data representations, rating created data representations and providing social commenting on artifacts in the store.

5. Easy integration to an enterprise
- Having a visualization tool which is powerful in being able to amalgamate data from various sources and display this data in the way users want it is only one part of such a tool being successful. A data visualization tool should also be flexible and versatile enough to be able to integrate it into and existing enterprise environment. Each of the data gadgets and the dashboards created in WSO2 UES are governed by user roles as mentioned above. In order to do this WSO2 UES has provision to either allow the enterprise use it's inbuilt user store or link up to an external store which can for example be the organisational LDAP or Active Directory. WSO2 UES also provides out of the box single sign on capabilities so that embedding dashboards created within UES in an organisation's enterprise applications can be done seamlessly.

6. Providing an engaging user experience 
- For any user based tool to be successful it needs to provide a user experience which engages the user. This is dependent on the first impression such a tool creates, the interactiveness it supports, the intuitiveness of the user interface, and the ability to personalize. With WSO2 UES a user is easily guided through the creation of dashboards via a simple and easy interface to use. Due to the intuitiveness of the tool a business user can create a dashboard according to his/her preference in a very short time. The display of data via the gadgets in the dashboards provide the user with an easy to read and interact interface.

Looking at the above business requirements and how the various features of WSO2 User Engagement Server fits in to these, it can be easily seen that this is a product that empowers an organisational user to carry on his/her daily duties in a highly efficient and productive manner.

Tuesday, January 21, 2014

The place of WSO2 Storage Server in the land of aPaaS

What is an aPaaS

Gartner defines Application Infrastructure as enabling technology for delivering business applications. Such application infrastructure when considered on an old fashioned enterprise system is usually synonymous to middleware. With the advent of cloud computing such application infrastructure is being offered as a service which is in general called as middleware in the cloud or Platform as a service. The term PaaS is very broad including whole plethora of such application infrastructure such as application servers, application and data integration tools, DBMSs, ALMs etc.
Thinking more narrowly, an aPaaS is such a cloud platform which exclusively enables development and deployment of business applications. This includes tools for application development, application life cycle management, application deployment, application runtime provisioning and also eco system tools that are required in an application development environment such as source control management, issue trackers etc.


Storage requirements in an aPaaS

When talking about business applications the requirement for an application to be integrated to a persistent layer is a must. Various applications would have differing data persistence needs and all of these need to be provided and managed through the aPaaS that is used for building these.

This leads to the requirement for a tool within the aPaaS to provide such services to applications built with it. Some of the salient features of such a tool would be;

  • Cloud awareness from within the tool - in an aPaaS the persistence layer should be cloud aware with such features such as supporting multi-tenancy, being scalable, supporting metering and billing,
  • Provision to support all types of data storage options from relational, NoSQL and Distributed File Systems such as Hadoop,
  • Ability to provide a consistent interface to provision and manage the data storage layer irrespective of the type of storage,
  • Have a management console through which all such provisioned data storages can be viewed and managed when required,
  • Support the need to provision and not create data storage since the actual storage would reside either in the IaaS layer beneath or on bare hardware used for the aPaaS,
  • Provide monitoring information on storage provisioning and usage - this is especially useful for the aPaaS administrators,
  • Provide services such as backup, archive and recovery of the storage used, &
  • Visually show important statistics via dashboards which can be extended.

Most of these features are available in WSO2 Storage Server which is a cloud aware solution for storage provisioning and management.

Currently WSO2 AppFactory which powers the cloud and on-premise application cloud (aPaaS) uses the services provided via WSO2 Storage Server, thereby demonstrating that the Storage Server is invaluable within an aPaaS.

1. Platform as a Service: Definition, Taxonomy and Vendor Landscape, 2013 - Gartner

Wednesday, January 15, 2014

Why use WSO2 Storage Server

Typical Development Scenario with Data

Lets take a typical scenario of setting up an application development environment which needs to access a relational database.

With respect to handling the data side of things, the basic steps that need to be done are,
1. Create a database
2. Create permission templates
3. Create users with permission templates
4. Provide access to the database for different users with different permissions

The above can be done manually using the particular RDBMS tools. However, it is rare that one would simply setup a development environment for just one application with one type of data server. As various applications come through the data requirements would differ. Also it would be rare that all applications created can be satisfied with a relational database - there may be instances where a unstructured data storage mechanism is needed or a distributed file system is needed.

Whatever the data requirement is, the basic steps that need to be done would remain the same but the tool set used may differ depending on the data storage mechanism used. Thus arises the requirement for an abstraction which would provide such services providing the user a consistent interface to use while also doing the underlying actions with the required data storage system. WSO2's Storage Server provides the answer for this.

What is WSO2 Storage Server

WSO2 storage server is a platform which provides easy provisioning and managing of data storage systems for development purposes. This includes both structured as well as unstructured data storage systems. The storage server provides rapid provisioning and secure access to such data storage systems.
The storage server's management console provides a consistent management process for provisioning and managing relational, NoSQL Columnar and Hadoop distributed file system repositories alike. 

The highlight of WSO2 Storage server is that it is cloud aware and thus provides these services while taking into consideration cloud characteristics such as elasticity, scalability, tenant isolation, on demand self provisioning, metering and billing based on analytics.

What are the internal mechanics of WSO2 Storage Server

Looking at an example where an application uses a relational data store the following explains how WSO2's Storage Server works internally.
  • The storage server needs to be setup with the database server instance which acts as the container for the provisioned databases. A cluster of such containers is provided to the Storage Server configuration documents. 
  • Based on a round robin algorithm databases are created in each database server instance cluster.
  • The Storage  Server has it's own central database (RSS management database) which will contain all details pertaining to databases created, users created, users allocated and permissions created and granted. It is though this central database that all management activities are carried out.
  • If an application needs to have separate environments e.g. development and production then the storage server will enable provisioning environments which are logical boundaries separating database server instance cluster sets.
  • Once the databases have been created, privileges created, users created and allocated to the databases with privileges using the management console tables can be created and the created databases can be viewed.
  • Thereafter applications can access these databases in their applications.

Why you should use WSO2's Storage Server

When selecting a certain product for a need there are two main criteria that should be looked at.
  1. What the product does which are the basic features supported
  2. How these functional features are supported which points to the non-functional features of the product 

1. Functional features:
  • Provide a consistent management interface which handles provisioning and management of structured and unstructured data stores. 
  • Ability to be used on-premise as well as to be used in a private cloud.
  • Ability to monitor clusters created and obtain statistics of provisioning.
  • Support the concept of environment which enables an application to use different data stores based on the life cycle stage it is in through a single point interface.
  • Relational data stores
    • Ability to create databases, add users and privileges.
    • Provide a database console through which SQL queries can be run to create tables and also do data manipulation.
    • Ability to create data sources, edit privileges and detach users from databases.
  • Distributed File System based data stores - Hadoop
    • Provide an explorer to explore and navigate the file structure.
    • Ability to do operations such as folder management, file management, and file upload/download.
  • Columnar data stores - Cassandra 
    • Provide authentication and authorization.
    • Ability to create key spaces and create column families.
    • Ability to explore the cluster.
    • Provision to run CQL commands.
2. Non-functional features:
  • Usability
    • Consistency of operations across types of data storages.
    • Provides the basic functionality required.
    • Linked with WSO2 Developer Studio to provide an IDE based interface as well.
  • Availability and Reliability
    • The data stores are provisioned in a cluster which enables high availability and also reliability of data operations.
  • Scalability
    • Horizontally scalable through adding a node to the cluster.

So how do these features motivate one to use WSO2 Storage Server ?
  • It has a comprehensive set of features for provisioning and managing data storage requirements 
  • It's cloud nativeness opens up usability within a cloud solution. 
  • It does not lock you down to technology - users can always move if needed. 
  • It has high usability in terms of providing a consistent interface for all types of data storage.
  • It's linked up with a monitoring tool as well which provides critical information for decision making.
  • It understands the need for supporting scalability and providing a highly available service and provides the capability to execute these.

Looking at the above functional and non-functional features and the motivation points, it can be seen that WSO2's Storage Server provides all basic data storage provisioning and management functionality, supports the desired quality attributes, and, is cloud native as well which, enables it to be used in a cloudy environment thus qualifying as a comprehensive storage solution which can be used in any enterprise.