Skip to main content

Automation Testing and the Application Lifecycle

I have worked on two large projects now that both had grown large and complex testing frameworks.  When I work on projects, I like to see strong test suites; they help me feel more confident that any changes I make to the code base won't cause regressions/bugs.  This is important because in many legacy code bases, features become tangled and disorganized due to deadlines and scope creep.  We've all been there, so we know how such things come to be.

At first glance, the test suites looked pretty good.  Coverage was not documented well, but I could tell that the tests did a good job at covering most important cases and a lot of edge cases.  As I started to dig into the test suites, however, I noticed both projects had a common problem -- the complexity of the test suite grew in a different direction than the intention of application it was meant to test.  I get frustrated when this happens because such suites leave me confused about things like the true life cycle of an application.

Consider an application meant to administer surveys, for example.  The life cycle of such application probably looks like this:

  1. Administrator creates a new survey.
  2. Send survey to potential takers.
  3. Some takers complete the survey.
  4. Administrator reviews survey answers as more completions arrive.
  5. Administrator closes the survey.
  6. Administrator generates summary reports for survey.
Test suites that suffer from the problems I mention might have a test that run something like this:
  1. Generate a survey.
  2. Generate survey answers.
  3. Generate summary report for the survey.
  4. Verify summary is correct.
This is a simple example that highlights something very important: tests, usually automation tests, have a bad habit of skipping steps in the application life cycle.  In this example, no administrator generated the survey.  These kinds of things can confuse developers new to a project.  If a new developer has just started, he or she probably doesn't know that an administrator is supposed to create surveys.  The test tools just... make a survey.  The conditions for making that survey are not clear in the test.  What's worse... if we refactor the validation that requires an administrator create a survey down a bit, we might suddenly cause large sections of the test suite to fail in step one of the test because the tests all get errors saying "only an administrator can create a survey" or something like that.

This kind of stuff leaves me stumped.  If someone went to all the trouble of creating an automation test framework, why not set the automation to use very public APIs and honor application lifecycle?  Granted, there will always be situations where some data manipulation magic will be necessary (something that says "x happens after a week" can't be automated by waiting a week), but as a general principle, I think it makes so much more sense to create a test framework to both test and teach the application lifecycle correctly.  This doesn't require a choice between spock or cucumber or junit (though most of what I am saying here does not apply to unit tests)... all it requires is a commitment to test tools/frameworks at an integration or system level that work hard to express the lifecycle.
Labels:

Comments

Post a Comment

Popular posts from this blog

Spring Security - Authority vs Role

I have spent a lot of time recently trying to understand the difference between Authority and Role in Spring Security.  This is a brief review of what I found. When creating a UserDetailsService or overriding configure(AuthenticationManagerBuilder auth) in the security config class that extends WebSecurityConfigurerAdapter, I basically get complete control over what I populate inside of the UserDetails that is used/returned.  This is important because the UserDetails interface really only cares about how to return one thing: Collection<? extends GrantedAuthority> getAuthorities(); A GrantedAuthority just seems like a glorified String wrapper that names some thing.  The question is... what is that thing? This is where the subtle difference between Authority and Role comes into play. I think that Role is an older thought/construct that automatically gets plugged into Authority if we just create a user with a Role.  But completely forget about the code a...
Post a Comment
Read more

JHipster, Liquibase, MySQL, and initializing data, including booleans!

When generating a data model from JHipster JDL, we will often declare entities with Boolean fields.  I have so far abandoned H2 as a database because of liquibase issues, and both my dev and production databases will be MySQL.  This is relevant to the Boolean field desire there is a long history in software development of how to store Boolean data types in a SQL database whose standards classically do not support Boolean. In the current JHipster/Liquibase incarnation, tables in MySQL are generated for us, which is really nice.  The Boolean data types are stored as BIT  (1).  This is not a problem so far -- most developers seem to agree now that as a best practice, we should store values in databases as false = 0 and true = 1, and a BIT(1) is a great, simple way to do that. An issue arises when we try to use liquibase to set/update our database to the desired starting state.  For my project, I've chosen gradle instead of maven as a build tool, and gradle...
Post a Comment
Read more

SQL, Booleans, JPA, and Hibernate

For a long time, SQL and Booleans have not gotten along.  Standards for SQL never really addressed the need for boolean data -- it was assumed that some other data type could easily just step in and address this need.  The result was a lot of different data models for boolean values.  Here are some examples. TRUE or FALSE T or F Y or N 1 or 0 <any value> vs NULL The internet shows the debate has gone on , even as SQL standards have changed .  Coming from a professional background with Oracle, I struggled with this across my teams because everyone had a different opinion, which led to a lot of time wasted due to debate. This said, I appreciate working with native queries in hibernate's JPA implementation against MySQL.  MySQL supports a BIT data type I recently discussed .  When we represent data in MySQL with BIT and restrict the length to just 1 (ie 1 bit), Hibernate JPA magically knows to query and return this data as a Boolean in the ...
Post a Comment
Read more