Technology Loop Paralysis - Anti-Pattern

Last Sunday I was in the "Falando em Java 2009" talking with a friend when we got in the subject of new technologies. During the conversation I realized one common anti-pattern that I didn't know about, the Technology Loop Paralysis, TLP for now on.

The TLP has the following major characteristics:

• It's seems right, without attention can't be caught.
• Once started, break the loop is hard.
• Scrambles all current problems, so disguises itself.

So, what is TLP? TLP is just the change of a technology, maybe a framework, a language or something similar to avoid difficulties in development. The catch is, when a new technology is added all kinds of problems will arise.

TLP Examples:

Sometime ago me and another developer had problems in delivering changes to a legacy Java system, the system was poorly designed and any change could break a lot of unrelated features. When the difficulties were explained the manager asked "If we use PHP? Can we fix?". This is a classic TLP. The problem wasn't with the language, any effort had to be directed in refactoring not in language change. Experienced developers will notice that this is trap easy to fall, if the developers were inexperienced and more comfortable with PHP they could fall on this trap.

Another TLP example. In a bugtrack system, developers didn't fill the forms right. Status, ETA and other fields were empty, they also didn't close issues properly. Relations were poorly added and resolutions were not written accordingly. Then, some reports were created to control issues, SLA and so on, but the data was not good. The solution proposed? Change the bugtrack system, but the system wasn't the problem, change it would only disguise all current problems.

TLP Effect:

New technologies will not solve any team, managing structure, unrealistic expectations and similar problems. Once TLP started these problems will disguise as implementation problems.

TLP Name Explanation:

• Technology: Any support system, framework, language or build block to a system.
• Loop: The fact that when faced with problems, the team or management choose change the technology without know root cause of the problems faced.
• Paralysis: The problems can't be solved and despite the feeling that something is been solved it's not.

TLP Forces:

Technologies change and grow more rapidly than our systems, so before a team can finish one project is common that the supporting technologies have new and better versions.

Inexperience and anxiety for new features can lure the team and management to adopt every brand new technology around.

TLP Solution:

Experience to avoid fall in TLP in the first place. During a TLP, the team can stop looping and stick with a technology until it finishes the project. After finished, the upgrade should be considered harmful unless all implications are well understood.

Why Developers shouldn't work long hours.

In fact this post should be named "Why anyone shouldn't work long hours". Face a problem for long hours, fighting a deadline it will not help, this should common sense, if not, try read about "Sleep-dependent memory consolidation".

My point here is not about problem solving, performance and accidents, I think these topics are well documented. But I want to add just one little information on why "long hours" are a problem.

What happened to me this week is that I had to solve some personal matters, in fact, renew my drivers license. It took me almost a half of day and got me thinking. How come I can do this if my project was late? Stay with a expired driver license? Work until midnight? In either case I will get unsatisfied and worried, with this kind of mood how a good work can done?

Some arguments against a "long hour" work is that people have to live outside the cubicle, but in fact there's some things more practical. We can't cope with all things we need to do if we are in a "long hour work" policy, there's no time.

Developers have to work on a normal scheduling, if not, the consequences are terrible. A tired developer with fall in the easy solution to reach the deadline. This will lead to a less extensible code, that will lead to problems in module integration, that will make the final period of development a bug hunt and more long hours to find those. Well, what can happen if a you add tired developers, buggy code and deadline approach? Something has to give up. You can't make the developers more efficient with more long hours, the bugs will stay in the code until someone removes them, so the most common to give up is the deadline. The "long hour" solution turns to be longer that you expect.

That's it.

Inside Steve's Brain

I just finished the book Inside Steve's Brain, good book specially if you like Apple products like me. One tip about this book is not read the final notes of each chapter, mostly they are shallow summaries out of context.

The book detail some decisions and modus operandi of Steve Jobs that I personally found very interesting, first because in most of times I agree (in my humble opinion), and secondly because it works, for everyone who knows about the computer market can see that some actions made by him are "default" on computer industry nowadays. Here some points made in the book:

• Small and good teams


• Every one involved from begining to end


• Avoid the "more features" trap, keep it simple


• Prototype a lot


• Keep focused, your user is the focus, but he/she doesn't know what really want

Small and good teams

Probably is no news the that any team has to be assembled with good professionals, and adding more people don't help. And big teams will suffer with the n(n-1)/2 rule for communication channels, if your team is too big probably there's isn't a single view of what should be done.

Everyone involved from begin to end

Well, the schema software engineers make the software, the designers the design, the marketing people the marketing and so on, doesn't work. In fact, I really believe that didn't work forever. Just ask yourself, in any point of your profession if people told you the big picture, not all the business plan, but a little more, things would be done differently? Maybe a focus on modularity here and there, or a external configuration system or similar stuff. Probably, in my case, things will need less duct tape if a more broad view is attached with the requirements.

Avoid the more features trap, keep it simple

Too much people, including myself wrote about the KISS principle.

Prototype a lot

I believe this is a true way to get good products. No body will get it right on the first shot specially on a complex system. Software Prototyping is a valid approach in development, but it seems that in Apple every part of a product is prototyped. In case you're not convinced, they even prototyped the Apple Store.

Keep focused, your user is the focus, but he/she doesn't know what really want

Nobody will disagree that focus is important, but some will disagree that consumers don't know what they want. I'm part of a group that believes consumers (users in my case) feel what they want but are incapable of expressing it correctly, why? Because they don't know the tools you have, the possibilities you have in hand to build something. Is more safe to think they can feel a good experience and tell you "I'm really comfortable using this software", than "I need a checkbox here, a list there and etc."

People are subject to the Paradox of Choice and it seems that Steve Jobs knows this very well.

Conclusion

This is just a glimpse of what I learned from the book, somethings reinforced my convictions others made me think. It's a very light reading and a good book for who works in this area or not.

Running Selenium with TestNG

This week I had to rewrite a bunch of functional tests, mostly Selenium stuff. When I think about Java Unit tests the first thing that comes to my mind is the JUnit framework. All tests that I have written until past week were with JUnit, and was using it to run Selenium.

But I got one more requirement, since I was rewriting all tests I would like to change environments (development and pre-production) without dealing in the code where was running the tests. The problem was a simple one, just write configurations with URLs, expected values and so on, for each of my environments. A trivial example is using different URLs in selenium for the same tests. Well, how do this with JUnit?

First approach, using JUnit 3

In JUnit 3 (old one) you have Test Suites. Programatically you have a chance to read a properties file and give the information to your test classes. Well, this could work if I had not the principle: "Do not write test code if isn't a test." When I caught myself creating a "little" framework to test my application I realized "I'm in the wrong way". Not that you should never code a "little" framework to simplify even more the tests, but i hadn't the time to do it.

Second approach, using JUnit 4

Well, JUnit 4 is more flexible, newer and uses annotations; and I didn't explored it well. After some digging i found JUnit's Parameterized Tests, but again I had (bear with me in this one) "to code, code to test code". And one more thing, JUnit 4 have Test Suites but, AFAIK, you only have a annotation version to define the classes for a suite.

JUnit wasn't solving the problem

After some thought i had two choices, forget about environments, or spend some time coding new stuff that wasn't tests or my application. Justice has to be made, JUnit is a excellent test framework but I think my objectives weren't too "Unitwise".

TestNG, next generation?

I heard about TestNG about a year ago and didn't gave it enough attention, I was happy with JUnit. But, everyday is a new day and I had a problem, so I started to read about TestNG. I was skeptic at first, thinking "maybe this can do it, but I don't have time to learn it all". I was wrong, happily wrong.

TestNG, next generation!

I have to confess, there's nothing to write here, maybe just some code. Remember the problem? "Two environments, one test, test it all". First, let me show the test example.

package com.keepcoding.test;

import static org.testng.AssertJUnit.*;
import org.testng.annotations.*;
import com.thoughtworks.selenium.*;

public class Google {

 private Selenium selenium;
 
 @BeforeClass
 @Parameters({"selenium.host","selenium.port","selenium.browser","selenium.url"})
 public void startSelenium(String host, String port, String browser, String url) {
  
  this.selenium = new DefaultSelenium(host, Integer.parseInt(port), browser, url);
        this.selenium.start();
        this.selenium.open(url);
 }
 
 @AfterClass(alwaysRun=true)
 public void stopSelenium() {
  this.selenium.stop();
 }
 
 @Test
 @Parameters({"search","expected"})
 public void googling(String search, String expected) {
  try {
   selenium.type("name=q", search);
   selenium.click("name=btnG");
   selenium.waitForPageToLoad("3000");
   assertTrue(selenium.isTextPresent(expected));
   
  } catch (SeleniumException e) {
   fail(e.getMessage());
  }
 }
}

This is a simple test, will start Selenium, open a URL (to be defined), type something (to be defined) and finally check if a text is present. Some notes:

• @AfterClass is defined with "always=true" because Selenium has to stop, even on test a failure.


• Selenium commands are surrounded with a try/catch because I want to avoid ERROR statuses, better a FAILED status if Selenium fails.

Now let's look on TestNG configuration file:

Here I defined all parameters and what tests should run, as you probably guessed the test search on Google for "Keep Coding" and checks if this blog URL is in the results. The key thing is not the test itself, but it's setup. We have defined Selenium to open a certain URL and if we want change the URL is a simple case of editing the XML file.

Using Ant to test all environments

Finally, how test all environments? First, create another TestNG configuration file, change the parameters accordingly, and use Ant to run it all. Here's a build file example:

This is a simple ant file, the highlighted part shows where you can have as much environments you want, for each one create a file changing parameters, skipping (not in production) tests and so on. TestNG will run one file after another for you.

Conclusion

TestNG is really impressive, from very simple tests to complex environment tests, you have in hand a set of options to make it happen. When you mix TestNG with Selenium and Ant is possible to build very complex testing. TestNG goes beyond the "units" and provide what you need to think about "collections".

Web Services Versioning

I had a problem last week and it's about Web Services versioning. I read several posts about this issue but I didn't found a simple solution. Most solutions are somewhat complex using UDDI or proprietary based, but my problem was very simple stated like "Once published a Web Service you cannot change it in a way you will brake your clients". So, with this constraint how to evolve your application without break your API or getting too duplicated code?

Use URLs with wisdom

First develop a strategy for your services URLs, put the version of your service inside the URL. In my case:

• http://myapphost/services/v[MAJOR]_[MINOR]/[SERVICE]

Here's a example:

• http://myapphost/services/v1_0/SomeGoodService
• http://myapphost/services/v1_1/SomeGoodService

Note that you should keep the name of the service, this is very important! Same service same name. You don't want SomeGoodService, SomeGoodImprovedService, SomeGoodImprovedFastService to be the names.

For the WSDLs files I use the same reasoning.

• http://myapphost/wsdl/v1_0/SomeGoodService.wsdl

Or you can use dynamic WSDLs using functionality of the webservice framework, I use JBoss so I can provide a WSDL dynamically.

• http://myapphost/services/v1_0/SomeGoodService?wsdl

Using a URL strategy is a good way to tell your client what version of your web service is in use, it's a fixed endpoint and remember that this URL will exists for a very long time.

Expose simple versions of your objects

When you have complex structures or dependencies on your objects this will be reflected inside your client code, so provide a simplification to your client.


Such simplification pays off, avoiding complexities of XML type mapping is good because you don't know in what language your client will be developed. Making deserialization easy is a good thing.

For a example in my current app I had a object using some joda time types, if I exposed such object to a web service lot of unnecessary complexity would flow into the client. To fix this I created a simple version of it, here's a simple method signature simplification:

void setBeginHour(LocalTime beginHour);

to

void setBeginHour(Calendar beginHour);


Some dirty details

Well, enough with recommendations (language and application server agnostic), now let's look into some real implementation. My environment is Java 5 with a JBoss 4.2.2 as application server. I will use a simple web service called AgeService, given a birth date it tells how years old is the client.

Here's the package layout of my web service:

• br.com.svvs.model - Domain objects for internal use.
• br.com.svvs.service - Web Service implementation package.
  
• br.com.svvs.service.model - Simplified versions of domain objects.
• br.com.svvs.service.util - WSTypeAdapter should translate domain to ws objects and vice-versa.

Now let's look into the Birth class:

package br.com.svvs.model;

import org.joda.time.DateTime;

public class Birth {

 private DateTime birthDate;

 public DateTime getBirthDate() {
   return this.birthDate;
 }

 public void setBirthDate(DateTime birthDate) {
   this.birthDate = birthDate;
 }

}

It's a very simple class, the problem is it uses DateTime from joda-time and I do not want expose such implementation detail. To hide this detail let's create a simplified version just for use on my service. Here's is the WSBirthV1_0:

package br.com.svvs.service.model;
import java.util.Calendar;

import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlType;


@XmlRootElement(
   name="wsBirth",
   namespace="http://wsbirth_1_0.model.service.svvs.com.br"
)
@XmlType(name="wsBirth")
public class WSBirthV1_0 implements WSBirth {

 private Calendar birthDate;

 public void setBirthDate(Calendar birthDate) {
   this.birthDate = birthDate;
 }

 public Calendar getBirthDate() {
   return birthDate;
 }

}

WSBirthV1_0 changes the signature to a simple Calendar object which most XML mappers know how to convert. Note WSBirthV1_0 implements WSBirth, this a marker interface we will use it in WSTypeAdapter, let's look at the code:

package br.com.svvs.service.util;

import org.joda.time.DateTime;

import br.com.svvs.model.Birth;
import br.com.svvs.service.model.WSBirth;
import br.com.svvs.service.model.WSBirthV1_0;

public class WSTypeAdapter {

 public static <T extends WSBirth> T convertTo(Class<T> type, Birth wsbirth) {
 
   String typeClassName = type.getSimpleName();
 
   if(typeClassName.equals("WSBirthV1_0")) {
     return type.cast(convertToWSBirthV1_0(wsbirth));
   } else {
     throw new RuntimeException("Don´t know how to convert " + typeClassName);
   }      
 }

 private static WSBirthV1_0 convertToWSBirthV1_0(Birth birthDate) {
   WSBirthV1_0 wsbirth = new WSBirthV1_0();
   wsbirth.setBirthDate(birthDate.getBirthDate().toCalendar(null));
   return wsbirth;
 }

 public static <T extends WSBirth> Birth createFrom(T wsbirth) {
 
   String versionName = wsbirth.getClass().getSimpleName();
 
   if(versionName.equals("WSBirthV1_0")) {
     return createFromWSBirthV1_0((WSBirthV1_0) wsbirth);
   }
   else {
     throw new RuntimeException("Don´t know how to produce Birth from " + versionName);
   }  
 }

 private static Birth createFromWSBirthV1_0(WSBirthV1_0 wsbirth) {
   Birth birth = new Birth();
   birth.setBirthDate(new DateTime(wsbirth.getBirthDate().getTimeInMillis()));
   return birth;
 }


}

In this class, using generics, we can provide a simple API to convert to and from ws objects. Note the use of our marker interface WSBirth. It's a utility class with static methods, we can grow our private methods for more and more versions. And at last, here's the service:

package br.com.svvs.service;

import javax.ejb.Stateless;
import javax.jws.WebService;

import org.jboss.wsf.spi.annotation.WebContext;
import org.joda.time.DateTime;
import org.joda.time.Interval;

import br.com.svvs.model.Birth;
import br.com.svvs.service.model.WSBirthV1_0;
import br.com.svvs.service.util.WSTypeAdapter;

@Stateless
@WebService(
   name="AgeService",
   serviceName="AgeService",
   targetNamespace="http://age_1_0.service.svvs.com.br"
)
@WebContext(
   contextRoot="/services",
   urlPattern="/v1_0/AgeService"
)
public class AgeService_1_0 {

 public String tellAge(WSBirthV1_0 wsbirth) {
 
   Birth birth = WSTypeAdapter.createFrom(wsbirth);
 
   System.out.println(birth.getBirthDate());
 
   Interval i = new Interval(birth.getBirthDate(), new DateTime(System.currentTimeMillis()));
 
   return "You have " + i.toPeriod().getYears() + " years old.";
 }

}

Discussion

At first it seems to much work but I disagree. When you have to maintain old web services and did not planned for code evolution this could raise much more work, usually with a copy and paste solution (happened to me).

Some good points about this solution:

• You can evolve your code without fear of breaking the old service, just follow the conventions.
  
• Isolated conversion from domain to ws objects, you don't forward ws objects inside the domain.
• Control of what is exposed and how, simplified client development.
• It works.
  

Some bad points:

• Messy names like ServiceV1_0, ServiceV1_2, WSObjectV1_0 ...
• Care should be taken on annotations and namespaces.
• If you have complex domain objects, you have to create a simplified version of each object used in your service.
• Not simple enough, I still think that could be simpler.
  

Some thoughts about the real thing.

It's been some time that I'm feeling disappointed with software development, probably any developer felt this some day in his (her) career. Software is hard, hard to get it right, and things get pretty ugly when business requirements take place.

Despite all work done in better software management and development, still, get it right is hard. We surely have people who can make good software, I will cite ThoughtWorks as a example. But it seems to me that the distance from the day by day software from what is posted in PlanetTW is enormous, at least by my experience.

One fact is that every company has/need some in house software, the big ones and the small ones. They have IT managers which were elected to hire developers in the field to write some code. The managers don't know what correctness means for software, and the developers know that with enough tries something eventually will work. Both far from know who was Edsger Dijkstra and understand the reasons why he wrote:

Unfathomed misunderstanding is further revealed by the term "software maintenance", as a result of which many people continue to believe that programs - and even programming languages themselves - are subject to wear and tear. EWD1036.

This is not just about shooting stones on everybody roof since my is made from glass too, I consider myself one more developer trying to make better code everyday. I'm probably not alone when I mention that the "best things I wrote were in spare time", and that was not because it was off-work code, it was work code, real work code, but I knew what needed to be done and dedicated myself to discover "how" should be done. Time not allowed in many companies.

The real world of today's developer seems a schizophrenic one. In one hand writing, modeling, saying, in the other hand not knowing if wrote, modeled or spoken the true. Still amazes me how much "It works!" I hear in software development.

Best.

Ant Power, Cruise under Control and Selenium Fuel (part 4): Managing dependencies in Cruise Control

On my last post I mentioned about how to configure Cruise Control to build projects that depends on another project's artifacts. But I described only the configuration and how to find the latest artifact built. Now is time to show the build files of the projects and how they know what artifact to include. This can be simple as add one more classpath to Ant. But I have one more issue to solve, branches.

The problem here is, as many of you already know branches are used to split the development tree so you can have all sort of things developed in parallel, like new features, bug fixes and so on. In my case we usually have a main development tree and a bug fix tree. Of course I wanted that both trees were subject to Cruise Control, so the solution was call the bug fix project "project-release" and the main project just "project".

When you have such structure you have to merge both trees from time to time, the problem was if my Ant build file in the release points to a release artifact and the trunk points to a trunk artifact, when they get to merge we always will have to take care of the difference. Not a good thing to do. This was extremely easy to solve because Cruise Control has a build call layout very good.

Let's see how Cruise Control starts a build. First it sits and wait for a trigger, usually a commit on the SVN tree. After that if decides (it can subject to Queuing and Veto) to build the project, it calls a build file which has the name build-project.xml and inside that file another Ant call is made to actually run Ant on the project's build file. The key thing is, you should put the differences on a trunk build or branch build inside this first build file. There, in my case, I define which artifact will be included in the path. Let's see for one project the Cruise Control configuration file and both build files.

Here for the trunk project:

    <schedule interval="60">
    <ant anthome="/usr/local/cruisecontrol/apache-ant-1.7.0"
    buildfile="build-project.xml"
    target="build"
    uselogger="true"
    usedebug="false"/>
  </schedule>

And now for the branch project:

    <schedule interval="60">
    <ant anthome="/usr/local/cruisecontrol/apache-ant-1.7.0"
    buildfile="build-project-release.xml"
    target="build"
    uselogger="true"
    usedebug="false"/>
  </schedule>

The only difference if that Cruise Control calls build on different files for the branch project and for the trunk project. In these files I configure properties suited for each project. Here is the example of both files:

<!-- build-project.xml -->
<!-- for the project (trunk) -->
<target name="build">
<svn>
  <checkout url="http://localhost/svn/project/trunk" revision="HEAD" destPath="." />
</svn>
<ant antfile="build.xml" target="build-all">
  <property name="moduleA.dir"    value="/var/cc-sandbox/artifacts/moduleA/latest" />
  <property name="moduleB.dir"   value="/var/cc-sandbox/artifacts/moduleB/latest" />
</ant>
</target>

<!-- build-project-release.xml -->
<!-- for the project-release (branch)-->
<target name="build">
<svn>
  <checkout url="http://localhost/svn/project/branches/release_1_0" revision="HEAD" destPath="." />
</svn>
<ant antfile="build.xml" target="build-all">
  <property name="moduleA.dir"    value="/var/cc-sandbox/artifacts/moduleA-release/latest" />
  <property name="moduleB.dir"   value="/var/cc-sandbox/artifacts/moduleB-release/latest" />
</ant>
</target>

Now, since Ant inherits this properties I have to just create one single build file which is the same for branch and trunk. For example:

  <path id="internal-dep-classpath">
   <!-- depends on moduleA project -->
   <fileset dir="${moduleA.dir}" >
     <include name="*.jar" />
   </fileset>
   <!-- depends on module B libraries -->
   <fileset dir="${moduleB.dir}" >
     <include name="*.jar" />
   </fileset>
 </path>

That's it.