Hasith Yaggahavita's Blog

Propagating Identity Information with Thread Local Variables

Hasith Yaggahavita — Sat, 15 Mar 2008 15:31:00 +0000

Often in our web applications we need to pass the logged in user identity down to business and data layers. This is mostly required when the business processes need to behave differently for different users/roles.

Straight forward mechanism would be to pass user identity as a parameter for every business process method, but can quickly become verbose and painful.

Another option is to use 'session based singleton' to store the user identity. Once the user is authenticated we use this session based singleton to store the user identity. Business processes can access this session based singleton to obtain the same. Even though this works, the bi-directional dependency introduced (as explained below) limits the usefulness of this approach.

Session based singleton uses the http-session for storing/retrieving information. i.e it has a dependency upon the web ui libraries. The business processes depends on the session based singleton to obtain the user identity. This makes the business processes to be dependent on web ui libraries. As naturally web uis are dependent on the business layer this causes a bi-directional dependency between web ui and the business layer.

Use of thread local variables presents a viable alternative design to achieve the requirement. The thread locals stores information local to a particular thread. Variables placed by one thread is can be accessed only by the same thread (Each thread can have a separate value for the variable).

Remaining section of the post looks in to the details of a thread local variable implementation for achieving the user identity propagation in Java. Mainly there are two parts of the implementation, a servlet filter and custom 'ThreadContext' class.

Servlet filter is responsible of protecting the web application from unauthenticated access. Any request without a user identity (IUser instance) in the session are directed to "login.jsp" which places a user identity in the session after successful authentication. What is important for our discussion is the fact that this filter is also responsible of attaching the user identity to the request thread (see below).

public class AuthFilter implements Filter {

public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) {
    HttpServletRequest httpServletRequest = (HttpServletRequest) request;
    HttpServletResponse httpServletResponse = (HttpServletResponse) response;

    // get the user from session
    IUser user=(IUser)httpServletRequest.getSession().getAttribute(IUser.class.getName());
    if (user == null ) {
        //user is not in the session, not authenticated
        httpServletResponse.sendRedirect("login.jsp");
    } else {
            //Authenticated, lets store the user in the thread context
        ThreadContext.setUser(user);
        try {
                // call rest of the filters
            chain.doFilter(request, response);
        }finally {
                // remove the thread local variable
            ThreadContext.setUser(null);
        }
    }
}

public void init(FilterConfig filterConfig) throws ServletException {}
public void destroy() {}
}

In the above code if the user is authenticated then we call "ThreadContext.setUser(user)" to save the user information to the current thread local store. Once the processing is over (i.e after chain.doFilter()) we remove the user from the current thread bu calling "ThreadContext.setUser(null)".

Now lets look in to the other part of the implementation, i.e ThreadContext class. Not much of coding is required here.

public class ThreadContext {
  private static ThreadLocal threadLocalUser = new ThreadLocal();

  public static IUser getUser() {
      return threadLocalUser.get();
  }
  public static void setUser(IUser user) {
      if (user == null ) {
          threadLocalUser.remove();
      }
      threadLocalUser.set(user);
  }
}

You can see the static variable "threadLocalUser" is responsible of storing and managing the user identity for all the threads. Depending on the thread which calls the methods of the above class, appropriate IUser instances are selected by the "ThreadLocal" class.

Even though the thread local variables are really powerful construct you need to be very careful in using them. If the variables are not cleared at the end of the process (finally block) that memory can be leaked. Also this should not be used as an alternative to parameter passing but should only be used for contextual information propagation.

One other cool use of this would be to propagate transactions in declarative transaction management frameworks. I'm planning to release an new version of "Easy Data Access Framework" which makes use of thread local variables that completely hide the transaction instances from the developers.

Microsoft .NET framework has built-in support for passing the user identity through the thread. "System.Threading.Thread.CurrentPrincipal" can be used to get or set the user identity to the current thread.

Eurocenter BI Framework White Paper

Hasith Yaggahavita — Mon, 07 Jan 2008 17:18:00 +0000

We at Eurocenter were working hard to build few middleware frameworks throughout the past year. One of the most popular is 'Eurocenter Business Intelligence - ECBI' framework. We already have few customers using ECBI framework and are expecting to get in to the market strongly this year.

I was writing a product white paper on our ECBI framework during last week (Thanks Uchitha and Samudra for reviews). I thought of sharing the first draft through my blog. You can access the full draft here. Following is the introductory chapter of the white paper.

Challenge for Today’s Business Intelligence
Historically business intelligence was considered as a largely manual, back office work performed by high tech professionals. Data from different sources was periodically integrated in to a set of defined reports and forwarded to business users for their decision making.

Due to the ever increasing competition, the business world has demanded the technology to deliver much dynamic information for faster/better decision making. Over last 5 years business intelligence feature set had rapidly grown and the definition of business intelligence shifted to mean much more front office related activities by the information end users. During the period, numerous products appeared in the industry delivering lot of advanced and sophisticated features.
Although these products empowered users to make better decisions, most of the business intelligence products have unfortunately ignored an important fact. That is, ‘80 percent of the information users in a typical organization are often non-technical decision makers’. The complications introduced by the advanced feature set make these products to be overly difficult for end decision makers. According to our observations, it is only less than 20 percent of the available features are effectively used by business users in day-today decision making.

Evidently, the well-known 80-20 rule has a negative struck on the business intelligence product domain too. Overcoming this problem is becoming one of the most important challenges for BI products today. We believe that the business intelligence industry is now at its next maturity transition which can take the products to focus on ‘effective features’ by getting out of ‘features chaos’.

Proper channeling of right features to right users for right decision making is what is expected from the next generation business intelligence products. As a result, organizations are on the lookout for simpler and more focused BI products from the marketplace today.

Note: Credit for building this fantastic framework should go to Eurocenter BI Team lead by Samudra (Samudra, Ravith, Hiran, Eranga, Prashanthan, Vindya, Prasad, Janith, Lalinda, Rajive, Sandrina).

Conserns for large scale integration architectures

Hasith Yaggahavita — Sun, 30 Dec 2007 13:23:00 +0000

During last couple of days I was writing a report on some existing academic research papers as a part of my postgraduate studies. I selected to write on one of my favorite topics, 'Integration Architectures'. Following is an abstract (Introduction Chapter) of my report. You can access the full report here.

ABSTRACT:

Most of the organizations today live by the slogan “Buy the best, build the rest”. Cost and risk of building from ground-up has made most organizations to consider buying commercial-off-the-shelf (COTS) products. Effective integration of these purchased IT systems has become the main responsibility of IT managers today.

Unfortunately, it is not very common for an organization to find all the required systems to be homogeneous. Often they are from different vendors having diverse architectures and operating on different platforms. The schema of the information model can widely differ from one system to another. At all these complexities, a seamless integration and smooth business process flow is what expected from the IT infrastructure.

According to Pollock (2001), robust integration architecture should support both ‘Application Integration’ as well as ‘Information Integration’ against heterogeneity. ‘Application Integration’ is the process of linking different software systems to become a part of a larger system. This is the technical solution that decides the level of integration (data level, application level, transaction level, process level, or human level) and technology of communication. Therefore ‘Application Integration’ mainly deals with the transportation of data/objects/messages between heterogeneous systems.

On the other hand, ‘Information Integration’ deals with the meaning and semantics of the communication. The meta-data, business rules and domain schema of one party should be understood by the other party for the integration to be successful. Maximum exchange of meanings by transformation of one entire domain representation schema to another partially compatible domain representation schema is the challenge of ‘Information Integration’.

‘Application Integration’ aspects are primary requirements that need to be satisfied by any integration architecture. But that is still only half of the total picture. Most integration architectures fall in to the trap of focusing on much of these technical aspects but forget the quality aspects of ‘Information Integration’. Simple integration requirements may be full filled by architectures biased to one arena, but complex integrations definitely require lot of attention and balance of both these aspects.

Despite the number of integration technologies and patterns exists, the extent to which the above goal is realized is debatable. This report looks in to and evaluates two such integration architectures published in order to solve the integration puzzle. The selected two architectures present two dissimilar approaches towards enterprise integration. Main focus of the evaluation is to study the two architectural patterns to assess their support for ‘Application Integration’ and ‘Information Integration’ aspects as set by Pollock (2001) in his white paper.

Structuring .NET Windows Installations

Hasith Yaggahavita — Wed, 01 Aug 2007 04:02:00 +0000

How to structure the application installation directories and how the assemblies should be placed becomes much important when you have a suit of applications with several related apps. For this type of situations, it is very natural to have few core assemblies common for more than one app.

If you come from COM background, one immediate solution that come to mind may be to store in the GAC (Global Assembly Cache). But if you are coming from Java background (like me) then you may probably like to have your application to have minimum windows dependencies. The you would prefer a application which works simply when you copy the files to a destination folder. (still GAC has it's own share of advantages like assembly versioning)

If you want a non GAC solution, design your solution installation folder to support holding the common executables. You may have your installation structure as following:

Here all the executables are placed in the root installation directory and dependencies are placed in sub directories. Then in .config files you should specify the dependency probe paths:
<configuration>

</assemblyBinding>

</runtime>

</configuration>

But what if you want a much more modular structure with every application having a separate directory but common dlls are located in a separate dir, then you may have the following structure

In this case, "<probing privatePath="Common_DLLs"/>" will not work as now the dependencies are not stored in a sub directory of your executables. You have to use "<codeBase/>" directive in your configuration files. One important thing to note is that you have to make your dependency assemblies strong named in this case. Have the following in your .config files.

<configuration>
<runtime>

</dependentAssembly>

</assemblyBinding>

</runtime> </configuration>

If I had to select between, I would prefer second approach to the first one as this allows me to better structure my solution installation. Also strong naming can help avoiding much of the bad 'dll hell' experiences.

There are more interesting stuff to discuss here such as how you manage common configurations (in .config files) across these related applications. For an example if all the applications are connecting to a common database, then how do we keep connection string in a single location is a problem to solve (If you do not want to store in the Windows Registry either). I'm hoping to discuss this on a separate post.

Selling or Consulting?

Hasith Yaggahavita — Sat, 30 Jun 2007 15:43:00 +0000

Today is the last day of my 7 day visit to Norway for meeting 4 of our customers. We have managed to successfully conclude every meeting resulting a new project to us :). We were mainly focusing on presenting our new reporting and business-intelligence(BI) frameworks to the prospective customers.

In one of the meetings I learnt one important lessons for my career.

We met a customer who has already purchased one of the world's leading BI tools. Even though our BI framework was better in terms of simplicity, portal features and return on investment; the product our customer has purchased had lot more features and power. Anyway we were able to convince the value addition of our tool making it difficult for him to take a decision on whether to buy our tool or to continue with what he purchased.

But for me I was feeling uncomfortable as I actually believed that it makes his systems complicated if he buys our tool, as he has already purchased a tool for his organization. I step out of my sales mode and informed him that I honestly think that he should not buy our tool but should stick to what ever he already purchased. I also presented some new cool products that are recently released by that BI product company.

Even my colleague was bit surprised to see me doing that. But that resulted a very open discussion as trust level has well improved. He wanted us to start working on a data warehousing solution for his company which will be a much bigger project compared to BI implementation. We have opened up a new business direction for our company as data warehousing seems to be promising with possible "Data Explosions" in the industry. I ended up as a happy techie since I think that was the correct thing to do to advice him not to buy out tool.

This incident opened my mind to think between "Selling" and "Consultancy".

Success in your Organization

Hasith Yaggahavita — Sat, 21 Apr 2007 18:30:00 +0000

Being successful in their organization is every employee’s dream. There are few advices I normally provide to the juniors coming in to leadership roles in our organization. For me these qualities have helped a lot during my carrier.

Believe the organization: If you are hoping to be in the core team of your organization, then you must have a good trust towards the organization. If your heart does not trust the organization, then you are not in the correct place to act a leadership role. Either you give up being a leader or update your CV. When I mean trusting that doesn’t mean you should agree to every action your organization takes. But you should be certain that the organization’s acts are genuine and done with a good will.
Plan to grow with the organization: There is common mistake I see from most of the youngsters. They plan their growth decoupled from the organization’s growth. My advice is that you should contribute the organization to grow then normally you grow in a much better rate within the organization.
It is not the technical things that matters most: Many of my employees I see running behind the technologies blindly. But in my experience, the one who climb the ladder is not the best techie, but the one who is the most valuable in his context. For an example look at a team whose responsibility is to deliver a project. The one, who builds the team spirit, help others, takes responsibility, troubles less and make the delivery to happen is the most valuable one and will be given the best promotion. So never fall back if someone is more technical than you are! Technology is not the key to success.
Build and trust your second level: If you do not build your second level to take over from you, you will never be promoted, because no one is there to do your job. You have to start delegating your work as much as possible. You may not get the quality you expect at first, but close one eye and continue delegating. After sometime they will start performing better that you. Let your management style to be delegate-train-review.
Manage frustrations: As a leader never discuss or communicate your frustrations to your team. Bring up frustrations you have to your superior but not to the team. Also solve your team’s frustrations as soon as possible if any. I have seen leaders failing just because they spread their frustration to the team and then end up with a highly de-motivated team which doesn’t perform.
Never challenge other’s ego: There are times you need to put your foot down and be hard on your team. In such situations always go by logics and facts, never challenge the others ego. For an example if you say “How many times I have told you this? Why can’t you grasp this simple concept?” that may hurt the listeners ego. Instead you may explain the bad consequences of him not listening to you well and help him to improve the situation.
Argue but don’t fight: From my view point, the difference between a fight and an argument is how you conclude the discussion. If both of you end the conversation with respecting each other that will be an argument. But if any of you are hurt at the end of the conversation, then that is a fight. The good news is even if you fight with someone still you can end it as an argument before you finish with it.
Identify your greatest asset: Some of the leaders are afraid of their subordinates who are smarter than themselves. But they are the greatest assets you can use to climb your ladder. You may effortlessly get them to play your role and start acting at one level above your current role. This is the easiest way to grow. Never be afraid to hand over your current role to a smarter one in your team, because that is the best thing that can happen to an effective leader.

Software Architecture Attribute Checklist

Hasith Yaggahavita — Tue, 06 Feb 2007 03:10:00 +0000

After using RUP templates for software architecture specification for few years, we at Eurocenter have decided to build our own set of templates for Software Architecture and for Software Design specifications. The new templates needed to provide enough meta-information to the author for better analyze the system as well as not to miss any important design aspect. There I did some research to identify as much as possible architectural attributes (checklist) which may be important for a designer to consider in developing architecture. Before dive in to the checklist, as a background information let me explain the design documentation we prepare at various stages of a project.

In fact practically within Eurocenter we use 4 types of design documentations. At very initial stage of the project (proposal and scope stages) we develop ‘Architecture Overview Document’ which presents the very abstract view of our recommended architecture to solve the customer problem as well as several alternative architectures with merits/demerits. This is where we demonstrate our understanding of the problem to our customers (this is particularly helpful in our presale activities).

The next document on the pipeline is the ‘Software Architecture Specification’, describing the meta-structure of all software structures. Our approach in this document is based on 4+1 views described by Philippe Kruchten.

Next we perform a detail design analysis based on UML notations to bridge the gap between system architecture and implementation. We call this document the ‘Software Design Specification’.

The final design document we prepare is called ‘Developer Guideline Documentation’ which serves the purpose of documenting miscellaneous guidelines for the developers. This section may include some best practices, version controlling guide lines, project specific knowledge base, etc. This practice is very close to maintaining a project Wiki, but much organized and get delivered to the customer for reference at the end of the project (note that all our customers are software organizations).

Now, enough back ground. Let’s have a look at the key attributes in developing a software architecture specification. You may use the following as a checklist to verify that you do not miss any important architectural aspect. Basically these are related with the non-functional goals of your architecture. Perfect system architecture may describe the expected level and realization strategy for each of the following attributes.

Performance

Response time
Throughput
Scalability (supporting increasing loads – load balancing)

Operational

Availability
Manageability (How to manage executing components like Caches)
Upgradeability
Reliability
Recoverability (Fault Tolerance)
Flexibility (Ability to support multiple configurations, workflows, etc)
Transparency (Hide the complexities)
Distribution, Concurrency and Conflict resolution
Integration (Connectivity to other systems)
Resources (Constraints and requirements)
System configurations
Offline Operations
Stability, Consistency and Accuracy

Maintainability

Portability
Complexity
Understandability
Duplication
Fragility (possibility of breaking the system due to a change)
Extensibility
Debugging

Security

Integrity
Authentication
Authorization
Safety (System may not cause the Monitor to explode)
Secrecy
Accountability (who did what and when)
Verifications and Validations

Other

Internationalization
Configurations
Testability (No entity beans, lets use Hibernate)
Usability (effective HCI)

I hope this list may help you in designing and documenting your new architectures. If I have missed any attribute let me know so that I can add those to the list. Happy architecting…

Automating Database Connection Management

Hasith Yaggahavita — Thu, 23 Nov 2006 10:15:00 +0000

One of the most frustrating problems in software development is this database connection exceeding due to non closed db connection usage. Things getting worse especially when we do not discover this problem at the development/testing phase (as we may not be continuously running the development/testing systems for very long periods). But frustrating enough the problem is going to appear in the production site.

What can we do about this? One way is to thoroughly test the application for non closed connections. Yes this requires developer concentration and can be time consuming if you do not use proper tools. More effective solution is to use a database access platform that can automatically take care of closing the opened database connections. But what if our simple project requires use of direct database access technologies (such as pure JDBC or pure ADO.NET)?

Let’s look at a small code fragment which we can use to employ a very simple data access framework to take care of this problem. In short we are going to write a base database action class which can provide and manage database connection to its sub classes. We use sub classing to write data access code (data actions) we need. Below is an example for the base action class written on VB.NET.

Public MustInherit Class ActionBase(Of T)

Protected MustOverride Function Body(ByVal conn As DbConnection) As T

Public Function Execute() As T

' Create a database connection

Dim conn As DbConnection = ConnectionProvider.GetConnection()

' Call the body method by providing the connection

Body(conn)

' Close the connection

Conn.Close()

End Function

End Class

All the sub classes need to implement the abstract method “Body(ByVal conn As DbConnection)”. As a method parameter, a connection instance will be available to the implementing method and that can be used by the sub class method to perform data access logic. Above base class is written as a generic class and subclasses should provide the template class name which should be the expected return object type of the sub class. For an example, a data access action that retrieve a ‘User’ instance will like follows:

Friend Class RetrieveUserAction Inherits ActionBase(Of User)

Private _userId As String

Public Sub New(ByVal userId As String)

Me._userId = userId

End Sub

Protected Overrides Function Body(ByVal conn As DbConnection) As User

Dim user as User = new User()

' Use connection to fetch user info and populate ‘user’ instance

Return user

End Function

End Class

As you can see we write sub classes to execute database operation on the database. Developers will not have to look for database connections. It will be provided in the “Body()”method parameter. Also the developers are free from the burden of closing the connections as framework itself will take care of it.

Let’s look at how one can make use of the above written class to retrieve the user with id=12.

Dim action AS new RetrieveUserAction(12)

Dim user As User = Action.Execute()

Even though this simple framework is pretty useful in managing connection, this has some limitations also. Developers are not able to spawn threads and pass the connection in to it for further processing. Also the results will be always disconnected, you cant pass resultsets to upper layers for presessing (but I think this enforces a good practice).

Also one major thing to consider is whether actually database resources getting released by closing the connections. In Oracle with JDBC connection close will not release database resources and you need to close the ‘Statement’ instance to release database resources. In such situations we can ammend the framework to provide us statement objects instead of providing connections.

Estimating Estimations

Hasith Yaggahavita — Mon, 20 Nov 2006 05:13:00 +0000

During last few days I was doing a research on how to perform effective estimations. There are a few interesting points I have learnt in doing the research. One good theory I learnt was the “Cone of Uncertainty”.

The above picture depicts how the level of uncertainty changes during the project life cycle. As you can see the uncertainty reduces as project progresses. This may not be news to most of us, but the problem is how many of us use this instinctive knowledge in our estimation practices? For an example is we provide an estimation in the project agreement level, the deviation of actual effort can be 4 times the estimation. “Later you estimate, higher the accuracy will be”

Also if we look at the project failure (or overrun) rates, market research shows that the success probability is high for small projects.

Project Size (lines of code)	Early	On Time	Late	Failed
1000 LOC	11%	81%	6%	2%
10,000 LOC	6%	75%	12%	7%
100,000 LOC	1%	61%	18%	20%
1,000,000 LOC	<1%	28%	24%	48%
10,000,000 LOC	0%	14%	21%	65%

One reason for this fact is that it is difficult to estimate a large project than a small one (of course there are many other reasons also). If you have a large system to develop try to break it in to several small projects. “Smaller the project, higher the success probability”

Another common mistake we make is the bias towards so called ‘Expert Judgment’. How many of us believe our experience and knowledge is the best source for estimations? Probably many of us think so. But the researches have proven the converse. Expert judgment is one of the worst methodologies to derive an estimate. So what would give us a better estimate? Often the historical data provide us a better estimation than any other method. Historical data can be in 3 forms:

Industry average data
Organization average data
Project local data

Needless to say the data accuracy is higher in project data than industry average data. We should build an estimation process that uses historical data. It is better to use several industry accepted estimation methodologies (customized with historical data) instead of a one methodology. Simple models with less control knobs have proven to deliver effective results. “Never trust expert judgments, use count and compute methods empowered with historical data”

One other common mistake is forgetting many of the required software activities at the estimation time. It is very preferred to have omitted activity list and go through it at the estimation time. Also allow several individual estimators to work independently and then converge those to get the final estimate. The derived estimate should not be a single value but should be presented with a variance. Three point estimation techniques is a good place to start working on estimation probability.

I hope these hits will help you in making a proper estimation process for your organization. You probably need to build a customized process that match with your organization practices. Try to use estimations throughout the project life cycle after end of every phase. Also make sure to asses and improve the estimation process. Cost of over running a project for several times will be definitely higher than your estimation effort. “Invest on estimations. It always paybacks”

References:
Software Estimation: Demystifying the Black Art
Software Measurement and Estimation: A Practical Approach
CHAOS Report

SQLite as an Embeddable Database

Hasith Yaggahavita — Wed, 24 May 2006 09:33:00 +0000

These days I'm involved in designing a ‘Smart Client Application’ architecture for one of our customers. These smart client applications require fair amount of offline capabilities, fast data access and low network utilization. To achieve the objectives most of the data is cached and persisted on the client side. This required us to look at data manipulation, persisting and synchronization mechanisms at the client side.

The first alternative we had was to maintain XML serialized objects on the client side to cache data. This had the problems like Data querying difficulties, Security issues, Thread safe access issues, High memory consumption and Implementation difficulties.

Next we have decided to use an embeddable database for storing data. Our first candidate was MS SQL Express database, as our project is mainly based on VB.NET with MS technologies. Even though it has some limitations, SQL Express is a very feature rich database which can be invoked in the similar way you invoke SQL Server database (with SPs, Views, Functions, full ADO.NET support, etc…). This was a major plus point for us as we could reuse the same data access layer components on the client side. It also supported database replication so that the cached data syncing could be done even at the database level.

But the problem was the high installation requirements of the SQL Express database. Also configuration of the database seemed to be fairly complex for our requirements.

Next alternative we looked at was the open source database SQLite. It has an incredibly small foot print of less than 400Kb and performance was impressive as a small single user database. ADO.NET 2.0 data provider was also available for SQLite and can be integrated to Visual Studio 2005 easily. It has implemented most of the SQL92 standard with few exceptions. SQLite is really a zero configuration single-file database which runs in-process of your application.

Having said the plus points, it is now the time to look at the limitations also. One main problem is that, the only locking level it supports is at database level. This can be a problem when multiple processes accessing the database simultaneously. Since smart clients are generally used by a single user, this shouldn’t be a big problem as an embedded database. But in our case, the smart client application might be deployed on a Cytrix Server and may have many users connecting over consoles.

Having honored by “2005 Open Source Award from Google and O'Reilly”, I think we should seriously consider using SQLite as an embedded database for our smart client architectures.

Maintaining Database-Application Versioning

Hasith Yaggahavita — Fri, 12 May 2006 13:17:00 +0000

Maintaining Database-Application Versioning

One major challenge in product development is managing/tracking the database changes and versioning it. There are many things to consider developing a proper strategy for database version controlling.

Application should be able to identify the database version it is using
If the database is not up-to-date, the application should handle the problem gracefully with out bringing surprises to the users (either gracefully exit or upgrade the database).
Database also should be upgraded if the ‘Online Update’ (such as in ‘no-touch deployment’) strategy is used in the application.
If application has capabilities to upgrade the database then preferably it should be able to update the databases in various versions.
System should be able to track any improper database changes (e.g. uncontrolled manual changes by the customer)

After a little thought I came up with a model to solve the issue. Still I couldn’t experiment this in a real world project. Here is an outline of the model:

Obviously the database should contain a Metadata Table keeping the current version of it. This version is unique for a particular state of the (database schema + system data) in the database
Any version change (db upgrade) should increment the database version number and should be performed through a set of scripts. That is for any two adjacent versions are associated with a set of scripts performing that version upgrade.
For example bringing database from version 2.3.5 to version 2.3.7 requires running two sets of scripts, one to upgrade from 2.3.5 to 2.3.6 and another to upgrade from 2.3.6 to 2.3.7. At the end of this script, it should properly change the db version to indicate the current db version.
Now we need to have a database level trigger to alter the db version number if any uncontrolled change is done to the database schema in an improper way. Additionally system data tables can also be protected with the same trigger mechanism which changes the db version if any uncontrolled change is happened to the system tables.
This trigger will change the db version to a state such as “NOT-DEFINED” so that the application may inform the user regarding the uncontrolled database state.
Additionally if we need to make automatic db upgrades to happen with the application upgrades, then we can build a small class library which will check the database version at the application startup and if the database needs an upgrade run the necessary upgrade scripts.
For this we can use the ‘Command Pattern’ to have individual command classes which can upgrade the database from one version to the next adjacent version. For an example to upgrade database from version 2.3.5 to version 2.3.7, application will run two command objects one responsible for upgrading from 2.3.5 to 2.3.6 and another for upgrading from 2.3.6 to 2.3.7.

This may not be the ultimate model for managing database versions. But I feel this would do well if implemented and controlled consciously.

Composite UI Application Block

Hasith Yaggahavita — Wed, 15 Feb 2006 04:35:00 +0000

Composite UI Application Block

Another Microsoft Application Blocks is available for downloading now. This new ‘Composite UI Application Block’ provides a long awaited smart client application framework for .NET technologies. That is the ‘pluggable component architecture’ for smart client applications. This plugability is required in many applications to provide features such as:

providing multiple versions of application with different feature sets (e.g. enterprise version, standard version, etc)
creating the application as a composition of loosely coupled reusable modules
providing personalization capability for the application
Easily adding, updating, configuring sub modules of the application (e.g. online updating and versioning)

Few months ago, Microsoft has released a portal framework with the new ASP.NET 2.0 release, enabling web applications to enjoy many of the above features. Also several Java implementations provide similar frameworks for Smart Client application development. Eclipse framework is one of the major competitors in this area with a very powerful plug-in engine. Sun’s Netbeans application platform also provides a development framework with a pluggable architecture.

For some time, I have been waiting for a similar kind of framework to appear in the .NET arena as well. Few of our customers had similar kind of requirements for their smart client applications and I was wondering how to provide this plug-in capability in .NET platform. So I’m really excited with the .NET ‘Composite UI Application Block’ and hope that this will be the first step towards a feature rich .NET plugging framework for smart client applications.

Impressing Customers on their own ground

Hasith Yaggahavita — Sun, 29 Jan 2006 13:06:00 +0000

These days I’m back in Norway for some pre-project activities. This is really a different experience for me being in the winter time (last time it was the summer). We had some interesting ‘Skying’ sessions and as a result I have a broken knee now :D.

Alright… now back on the business… Last few days we had very interesting sessions with few of our prospective customers. It was really a challenging task to impress them on their own ground (specially as they are software companies who know their business and the technology very well). Also being a technical guy it was really difficult to just forget the technology and concentrate on the business requirements. Anyhow during this visit I have learned many tactics of how to impress customers on their own ground.

The most important thing to do is, we should make them feel that we understand their business very well. This is a challenge as all the customers were in the business for many years and they have knowledge which they think obvious (implicit knowledge) but being a third party that is not so for us. So here are few guidelines which we learned and used to capture the business domain knowledge during the few meetings we had.

Let them describe the business domain they are working on. This is not about their company processes but about the industry as a whole. Here we need to understand where our customer stands in the industry (E.g. Market share).
Take the main business process of the company and discuss it with the customer. We have to identify where in the process IT fits in to. Use of activity diagrams was very useful in our discussions. Also ask them to take few common variations of this process if any exists (This is effective if the customer is a vendor of a product which is used by several of his customers).
It is really important to keep a balance between automated and manual processes, as most of the companies add value to their process through manual processing (Use of human intelligence in the process). These manual processes may give them an edge in competing with other firms. So we need to be care full in proposing IT automations of the process components.

Discuss the importance of IT to their organization. Guide the discussion such a way it reveals lot of non functional requirements such as performance, robustness, error handling, and scalability of the proposed system.

Talk about the current IT infrastructure. Discuss the strengths and weaknesses of the current system.
Talk about the competitors. This generally led to a really interesting discussion and reveals most of the expectations of the customer. Discuss the IT infrastructure owned by the competitors, which processes they perform better and what to learn from those.
Discuss the income break down of the company. That is which part of the business is bringing more income to the company, which parts are weaker and how IT can help them to grow. One thing we need to remember is that, at the end of the day it is all about profit and incomes (providing cool and nice features which do not bring any profit to the company are out of the interest).

Most importantly discuss what make their business to grow. For an example attracting more suppliers, attracting more customers and making the internal processes efficient may be the growth factors of a business. So the proposed system needs to address those factors to make the business to grow.

Actually working with prospects is fairly different and difficult compared to a traditional requirement gathering session. To win the project we need to focus well and impress them on our capabilities and understanding of the business. Trust building is the key to win a project. Good concentration focused questions, professional answers and well thought improvement proposals helps to build this trust. Also never forget to talk about your dog and his cat to build a good personnel relationship…

War between typed datasets and custom entities

Hasith Yaggahavita — Thu, 05 Jan 2006 11:37:00 +0000

War between typed datasets and custom entities

In Eurocenter currently we are preparing for a new project-kickoff which is going to be based on the .NET platform. Among many other design decisions we are currently considering an implementation technology for our domain model. We have to decide between "typed datasets" and "custom entities". Both the methods has their own pros and cons. I have done a small research on this and I thought of sharing those information through this blog post.

Here are some major reasons where I think datasets are better in some situations.

Database integration of the domain objects is very simple. DataAdaptor classes take care of the most of the database operations of the dataset. Also dataset has ability to remember the original values making the dirty object identification very easy.
Easy serialization of data entities to XML
Easy binding to most of the UI components
Easy integration with other tools and components (E.g. Biztalk Server)
Very good documentation and community support

Antway saying all those plus points of datasets, there are very strong arguments why and where we should not use datasets but go with custom objects.

Datasets makes the system less object oriented (as business methods are separated from the domain model)
Too much of generated code of datasets causes code duplication and lesser maintainability.
Exposing datasets as SOA can make services less interoperable.
Custom class approach is much cleaner and clear.
Unit testing is pretty much easy with custom class approach as data is easily bound to the operations.
Use of DataReaders, SPs and Custom Entities seems to perform better in the sense of memory usage and data fetch time.

In my opinion, we need to make sure that we use right tool at the right scenario. In a complex application (lots of integrations), I would prefer going with custom entities. If the application is a data driven or when the application is a disconnected data centered desktop application I would consider using datasets.

Stateless Session beans kill Object Orientation??

Hasith Yaggahavita — Thu, 22 Dec 2005 03:15:00 +0000

Stateless Session beans kill Object Orientation??

Recently I had a chance to have close look at OO principles as I had to prepare to deliver a lecture for a ‘advanced OO development’ course. This allowed me to have a back look on OO after working in several commercial J2EE projects. I started thinking… in those projects, have we used proper OO with real encapsulation, inheritance and polymorphism principles??? Answer was “NOP”…

In most of the J2EE projects people tend to use more service oriented approach rather than going for a object oriented approach due to the J2EE technology limitations. For an example most of the business objects are just plain DTOs where Stateless session beans acts as service providers for those DTO business objects (as an EJB best practice, DTOs are used to transfer entity bean data across tiers :D ). Most of the so called “DTO business objects” are just having a bunch of getters setters encapsulating the attributes. Is this real encapsulation??? No… we need to have the object operations to encapsulate the attributes of our objects. But with DTO business objects we are separating out the object data from its operations causing less encapsulated objects. Making the things worse, stateful session beans did not proved itself in the industry, making OO much away from the J2EE projects.

At least do we get inheritance properly? Session beans do not support direct inheritance making polymorphism also to be impossible. Having no polymorphism in our business objects directly affects the design quality. We end up with lot of if-else clauses handling different variations of our business object types. It is true that there are tips and tricks to get around with EJB inheritance problems, but that is not we need as poor OO programmers. We need tools (J2EE) to help us; we don’t have much time here to help the tools…

So where are we heading??? EJB 3 seems a good improvement but still it doesn’t provide all we need. Some times it seems to be bias at J2EE server vendors but not at the java developers (may be as the spec committee is mostly driven by application server vendors). Anyway I wish to talk more on EJB 3 and OO principles in another post.

Considering all, for me it seems going for Spring Framework is still a better approach, especially when you do not need your system to be distributed across several VMs. In my opinion, stateless session beans still are a superior technology for a locally distributed system (When it is not local there are much better solutions like XML over HTTP). When we think of such locally distributed systems we should think those distributed components as services, not as objects. Internally within components we have to follow clean OO principles. In summery we can develop our logic on plain well object oriented object model and expose the operations (if needed) as services over stateless session bean. I believe that approach gives us a better, and maintainable system.

Add Business Intelligence to Your System

Hasith Yaggahavita — Thu, 01 Dec 2005 03:55:00 +0000

Add Business Intelligence to Your System

Recently I involved in designing a reporting system and hence have done fair bit of research on the available reporting mechanisms and frameworks. Although the requirement was to use .NET and Windows technologies, I thought of looking in to some other reporting platforms (in java and open source world) as well.

First thing I got in to my mind was 'Eclipse BIRT framework'. I remembered looking at the BIRT proposal and architecture about a year back. Also I remembered it had proposed a complete reporting life cycle management system. So I visited Eclipse site and saw that they have released the first stable version of the system and it seemed to be very promising. You can access Eclipse BIRT home page here and a very cool flash demo here. (java programmers must see this demo if Microsoft was able to surprisee you with their Visual Studio productivity features :D )

Eclipse BIRT architecture provided me with a very good understanding of the functionality of a report system. Then I thought of having a look at 'Jasper Reports'. Jasper also seems to be a rich reporting framework and I got a good understanding of reporting internals by looking at report samples and tutorials of Jasper reports.

So after looking in to couple of open source reporting frameworks then I was ready to get in to my real job, which is to explore .NET related reporting technologies. Visual Studio 2005 comes with two major reporting approaches. One approach is to use 'Crystal Reports' version shipped with Visual Studio. This is a limited version compared to 'Crystal Reports XI' but good enough for most of the reporting requirements. The 'CrystalReportViewer' component provides easy embedding of reports to web forms or win forms. Basically if you have a good 'Crystal Reports' experience and working knowledge on Visual Studio, you will be able to generate quality reports and embed them in to your web or windows application. One great this I noticed was that Crystal Reports engine is able to generate quality HTML reports having cross-browser support even in complex report navigation requirements (which ASP.NET 2.0 is failed to do in some cases).

The other approach is to use 'Microsoft Visual Studio 2005 Reporting' features with or without 'SQL Server Reporting Services'. In this approach Visual Studio provides a ÂReportViewerÂ component which can be used either in local-mode (no reporting server involved) or in remote-mode (with SQL Server Reporting Serviback-endning as a backend report server).

Local-mode is seems overlapping with the 'Visual Studio 2005 Crystal Reports' and good for simple reporting architectures. Local-mode provides you with more control over the data as you can bind datasets, custom objects in to your reports. If you want more complex/extensible system having report servers running you need to look in to 'SQL Server Reporting Services' and use 'ReportViewer' remote-mode. In this mode the data binding and report rendering is done within the report server and the report will be exposed as a web service consumable by 'ReportViewer' components embedded in web forms or win forms. Also SQL Server Reporting services can directly expose html reports with out a separate UI layer in place. Microsoft has done many improvements to their previous 'SQL Server 2000 Reporting Services' with their new releases of 'SQL Server 2005' and 'Visual Studio 2005'. Even though these technologies are not very matured as 'Crystal Reports', they seem to be very powerful and robust with the good industry impression they have on the 'SQL Server 200 Reporting Services'.

One thing I noticed after doing this research is that the reporting technologies have become much matured today with all sorts of user/developer friendly tools and complex/extensible architectural-styles. So it seems the true sense of the word 'Business Intelligence' is not so far from today...

Automate Your Builds and Tests

Hasith Yaggahavita — Mon, 07 Nov 2005 06:25:00 +0000

Automate Your Builds and Tests

This is going to be an ice-breaking post on my blog after a long silence. Couldn't write a post for some time as I was bit busy with a lot of work after coming back to Sri Lanka.

Last few days I was involved in automating build process of a J2EE project in Eurocenter. The project basically has a Struts based web front end, EJB/JDBC based db/business tier and a web service (XML over HTTP) interface for mobile devices. We have come up with an automation plan which will be implemented within next month and it seems progressing well curruntly.

So I thought of sharing some stuffs on the automation implementation of our project as it might help someone else in automating their products. As the first step we have to have a rough plan of our automation process. The plan used in our project goes as follows:

Write a one-step build script: Use Ant or Ant variants to come-up with a build file with will generate the complete build in one command. You may even consider using Ruby build scripts (Rake) for this purpose. (But we should be careful is selecting a immature technology for a core part like the build system of a project.

Add unit-test execution to the script: You should carefully select the unit tests which are properly written to run as stand alone tests. If you have followed my previous postings I have discussed some techniques explaining how to make your code testable. We can have an ant task which will run the unit test cases/suits on the compiled code.

Come up with a push button release: Once we have the build script we can extend it to produce single command releases. Basically this involves checking-out the code from a CVS tag and running the build task. This extended build script can be used by the build manager in producing builds for QA testing as well as producing production builds.

Schedule builds with automation engine: Here we go with automation. First we need to decide on a build engine. In the open source world there are several good automation engines such as ‘Cruise Control’, ‘AntHill’ and ‘BuildBot’. We have decided to go with ‘Cruise Control’ as it seems to be the most popular choice in the java community. In this step we configure our build engine to checkout all the code from CVS periodically (only if any modification is done after the last build) and build the product and run the unit tests. Once we have completed this step we track any build failures (such as compile errors) and unit test failures. In our project we have scheduled builds to run every one hour and this helps capturing introduced bugs within a period of one hour.

Establish reporting mechanisms: Through out the process we should be having basic notification mechanisms like Email and web console for reporting build status and failure notifications. All the build engines support these basic reporting mechanisms and all we have to do is to configure those services to suit the need. For an example if a build failed we can configure ‘Cruise Control’ to send notification emails to the developers who have committed code to CVS after the last successful build.

Automate integration testing: Here we attempt sub component integration in a test environment. In our project we wanted to make sure that the product components are integrating and deployable on the server. The integration testing will ensure that all the components are properly deployable (e.g. EJB testing) and the component communication channels (e.g. JMS queue testing, database connections) are established correctly. Most of the tests in this stage can be considered as in-container testing and test frameworks like ‘Cactus’ are good candidates to support the testing in this stage. In this stage we run “JBoss’ as our application server then deploy our application there and run integration test scripts.

Automate functional/acceptance testing: In this stage we add end-to-end testing to our automation. In our case basically this means performing tests on the web UI and web service interfaces. These test cases are written based on the end-user actions (based on use cases). Example test case is ‘user log-in to the system and changing his password’. In our product ‘HttpUnit’ test cases are written to simulate and validate user actions. Where ever response time is critical we may add performance test scripts written using ‘JMeter’, ‘JUnitPref’, etc… to ensure system response time.

Add code analyzers: Here with each build a code analyzer will inspect the code for “smelly code pieces”. With each build the team lead will get code analyzer report on any added/changed code fragments.

After coming up with an automation plan we can start implementing the plan. Once implemented, we get a set of actions which will run in a sequence in every scheduled build cycle. For you to get better understanding I will summarize the action steps we have in our project build cycle.

1. Checkout sources from CVS
2. Build the application from scratch
3. Execute build verification tests (unit tests)
4. Run code analyzer tool scripts
5. Install a ‘JBoss’ server instance
6. Configure the ‘JBoss’ instance
7. Create the test database
8. Populate test database with test data
9. Deploy the product over the JBoss/file system
10. Startup the server
11. Monitor log files for successful server startup
12. Execute integration test scripts
13. Execute acceptance test scripts
14. Shutdown the server
15. Clean up the build resources
16. Report build/test results

Once we got all the basics in place this is the time to play around extreme automation experiments. For example I have heard some project teams having flashing red bulbs in the development environment if a build failed. Trying these kind of extreme practices can bring team moral up towards testing/automation as well as can take attention of the rest of the project teams (in a good sense).

Security with Java Cryptography Architecture

Hasith Yaggahavita — Sun, 25 Sep 2005 22:55:00 +0000

Security with Java Cryptography Architecture

In the last week I was going through some discussions on how to provide security on our new application we are building at Scali AS. In our architecture the server is a C deamon which uses OpenSSL for security where as the client is a Java RCP which uses JSSE.

One observation I had in working with java security frameworks is that Java has lot of Security components, but most of the developers has no very clear understanding of where those components fits in to the application requirement they have. So I thought of having a blog post which introduces the Java security components and their usage.

Here we go... Broadly there are several components belonging to the Java Cryptography Architecture (JCA) (or simply Java Security). JCA can be divided in to several sub categories as follows :

a) Java Authentication and Authorization Services (JAAS)
b) Java Secure Socket Extension (JSSE)
c) Java Cryptography Extension (JCE)
d) General Security Services (Java GSS-API)
e) Java Certification Path API
f) Java Simple Authentication and Secure Layer (Java SASL)

JAAS is the Java solution (framework) for authenticating and authorization uses and services. JAAS provides user-centric as well as code-centric access control mechanisms. JAAS can be considered as a framework and an API focused mainly for non-connection oriented (stand-alone) application security. It is not about encrypting data but it is a implementation neutral API for authentication and authorization (For an example if we decide to move our usernames/passwords from a properties file to a database, still JAAS wil allow us to do this with out changing application code).

JAAS authentication mechanism is pluggable so vendors/users may write there own ‘login modules’ to read/validate authentication credentials from their preferred source (e.g. database, directory service). As I know, JDK does not provide many default ‘login modules’ but generally application servers come-up with several reusable login modules such for reading “prop-files”, “simple database tables”, etc...

JSSE provides a framework and implementation for securing your data in network communications by introducing security later on top of TCP/IP. J2SDK 1.4 onwards supports SSL 3.0 and TCL 1.0 protocols. The most popular SSL model used on internet is to authenticate only the server identity and use encryption on data. Client identity is authenticated using username/password provided by the client (e.g. a typical e-commerce web-site) JSSE provides a convenient API on top of java sockets for enabling your TCP connection security.

JCE consists of a framework defining java security components such as data encriptors, key generators, etc… Sun has provides a default implementation called “SunJCE” for this framework (For example RSA, DES encryption algorithms, DES key generators are provided with the ‘SunJCE’ provider). We may use this default SunJCE (any other third party provider) to provide some cryptographic operation such as encrypting a data stream using “DES symmetric key” encryption.

Java GSS-API allows java developers to make their programs to work with Kerberos based authentication. Similar to JSSE, GSS-API provides authentication, encryption and integrity protection on your data. The underline mechanism used is the widely known Kerberos security model. Kerberos provides strong token based security and supports single sign-on capabilities for the applications. (GSS API mechanism is defined as a Internet Standard under RFC2853)

Java Certification path consists of abstract framework classes which allows user to create, validate, manipulate ‘Certification Chains’. Certification chain is a ordered list of certificates which will establish a trustable mapping from ones identity to his ‘public key’. Java Certification Path API also consists of a default implementation for working with X.509 certificates.

Java SASL is the java implementation of the SASL internet standard defined under RFC2222. As the name suggest the SASL standard also provides authentication and secure transmission of data (so it also can be categorize with GSS and JSSE). SASL is defined to be a light weight protocol and used by LDAP and IMAP implementations. Thus any application intended to work with those applications may need Java SASL API for security integration.

Thus the bottom line is that, we should decide on the technology to be used based on our application context. As a rough example if my application is TCP socket based I may go for JSSE, if my organization security is Kerberos based then I go for GSS-API, if I need to communicate with a LDAP service I may need to consider Java SASL API and so on…

Ruby blocks, C# delegates, C function pointers and Java inner classes

Hasith Yaggahavita — Sun, 18 Sep 2005 21:34:00 +0000

Ruby blocks, C# delegates, C function pointers and Java inner classes

During the weekend I spent some time to look in to the Ruby 'Blocks'. Blocks are a Ruby language construct which adopts the concept of method references in a very useful manner. Even though this is not a new idea I haven’t seen a language adapting and using method references to the extent as Ruby does. Let’s look at how Ruby adapts this concept. One of the Ruby’s powers is that it takes fewer lines of codes compared to most of the other languages. For an example in Ruby iterating on an array and printing the elements will look like the following:

array.each { |animal| puts animal }

Actually what happens here is, to the "Array.each" method we pass the code block "{ |animal| puts animal }" so “array.each” method will call the passed code block for each element of the array (more information). As u may see this is not exactly passing methods as references. Instead of passing an external method as a parameter, in Ruby we write our code inline as a code block.

Let’s look at some other languages to see the way they allow developers to pass code blocks as parameters of a method.

In C# we may gain the same power with Delegates. Delegate is a type that describes a method signature. Methods with the same signature can be stored in a delegate and can be passed as parameters or be invoked. But delegates are more structured, complex and requires more coding compared to Ruby blocks.

Function pointers of C allows developers to pass functions as parameters to other functions. They are not type safe as C# delegates so the developers are able to pass functions with different signatures as parameters.

Java has no concept called method references and it does that using interfaces and classes. We need to define our code block inside a class and pass that class to the method as a parameter. For an example we use this in implementing listeners, callback routines, and to pass utility code blocks such as compare routines to ‘sorting’ algorithms. Mostly we see these code blocks are implemented as inner classes or anonymous inner classes in java.

If you are a regular reader of my blog you may remember the post “Intermediate Representations vs Software Performance”. There I have explained how to pass a code block to a DAO so that as and when the DAO encounters a database record it will invoke the passed code block. If you look at the way of Ruby’s “array.each” method works you can see both these implementations share the same concept to a some extent.

So I may say ruby has adopted the concept of passing code blocks to a method and made it a really useful construct in the language.

JMeter in Web Application Testing

Hasith Yaggahavita — Sun, 11 Sep 2005 23:10:00 +0000

JMeter in Web Application Testing

Last week I was discussing with Kartik on performing JMeter performance testing on the Ripple product. They were facing some difficulties in HTTP session handling in their JMeter test plan. This conversation allowed me to refresh my knowledge on this powerful testing framework. So I thought of making a note about JMeter on my blog. This will be a quick and short overview on some of the general JMeter capabilities.

When considering the non functional testing we often hear people talking about different types of tests. Some of those are “Load testing”, “Performance Testing” or “Stress Testing” on the application under test. First of all it is important to know the meaning and differences of these terms used in application testing.

a) Load testing is applying a heavy load on the application and monitors the behavior of it under a heavy load.

b) Performance testing may be carried out to ensure the application responses within acceptable timing on the requests it’s getting.

c) Stress testing is for ensuring fault tolerance capability of the application. For an example we may restart the database while application is in operation and check whether the application can recover under such a condition.

Let's have a quick look at JMeter. JMeter can be readily used in Load and performance testing. The test scripts we may create are called ‘Test Plans’. We can have create several test plans to test our application. These test plans can be used even as automated tests by running JMeter in non graphical mode.

There are several types of elements in a JMeter test plan. JMeter uses ‘Samplers’ to produce requests. It has ‘Logic controllers’ so we can control our test flow with them (e.g. to repeat a request). ‘Listeners’ are used to capture test results. Delays may be introduced by its ‘Timer’ elements.

Session handling is an important activity in any practical test plan. For general cookie based sessions we may use a Cookie manager to keep our cookie based session information across the requests. But when URL-Rewriting is used in session management we have to use “HTTP URL-Rewriting Modifier’ to keep session information across requests.

For me one of the most interesting elements in JMeter is its assertions. For an example returning HTTP 200 by our server doesn’t always mean that the request has been served successfully. That may be a user friendly error message returned by the server when a critical error occurred. We need to be more specific and probably wants to check the returned html to validate the server response. This is made possible with ‘Assertions’ in JMeter.

When configuring our test plan sometimes we face difficulties in simulating the exact browser behavior with JMeter. One handy tool that would help here is ‘TCPMON’ tool comes with Axis utilities. We can capture and compare http communications of JMeter with the communications of browser easily with this tool.

It is really interesting seeing JMeter as a part of a automated testing environment together with JUnit and HTTPUnit test frameworks…