Skip to main content

Automation Testing and the Application Lifecycle

I have worked on two large projects now that both had grown large and complex testing frameworks.  When I work on projects, I like to see strong test suites; they help me feel more confident that any changes I make to the code base won't cause regressions/bugs.  This is important because in many legacy code bases, features become tangled and disorganized due to deadlines and scope creep.  We've all been there, so we know how such things come to be.

At first glance, the test suites looked pretty good.  Coverage was not documented well, but I could tell that the tests did a good job at covering most important cases and a lot of edge cases.  As I started to dig into the test suites, however, I noticed both projects had a common problem -- the complexity of the test suite grew in a different direction than the intention of application it was meant to test.  I get frustrated when this happens because such suites leave me confused about things like the true life cycle of an application.

Consider an application meant to administer surveys, for example.  The life cycle of such application probably looks like this:

  1. Administrator creates a new survey.
  2. Send survey to potential takers.
  3. Some takers complete the survey.
  4. Administrator reviews survey answers as more completions arrive.
  5. Administrator closes the survey.
  6. Administrator generates summary reports for survey.
Test suites that suffer from the problems I mention might have a test that run something like this:
  1. Generate a survey.
  2. Generate survey answers.
  3. Generate summary report for the survey.
  4. Verify summary is correct.
This is a simple example that highlights something very important: tests, usually automation tests, have a bad habit of skipping steps in the application life cycle.  In this example, no administrator generated the survey.  These kinds of things can confuse developers new to a project.  If a new developer has just started, he or she probably doesn't know that an administrator is supposed to create surveys.  The test tools just... make a survey.  The conditions for making that survey are not clear in the test.  What's worse... if we refactor the validation that requires an administrator create a survey down a bit, we might suddenly cause large sections of the test suite to fail in step one of the test because the tests all get errors saying "only an administrator can create a survey" or something like that.

This kind of stuff leaves me stumped.  If someone went to all the trouble of creating an automation test framework, why not set the automation to use very public APIs and honor application lifecycle?  Granted, there will always be situations where some data manipulation magic will be necessary (something that says "x happens after a week" can't be automated by waiting a week), but as a general principle, I think it makes so much more sense to create a test framework to both test and teach the application lifecycle correctly.  This doesn't require a choice between spock or cucumber or junit (though most of what I am saying here does not apply to unit tests)... all it requires is a commitment to test tools/frameworks at an integration or system level that work hard to express the lifecycle.
Labels:

Comments

Post a Comment

Popular posts from this blog

JHipster, Liquibase, MySQL, and initializing data, including booleans!

When generating a data model from JHipster JDL, we will often declare entities with Boolean fields.  I have so far abandoned H2 as a database because of liquibase issues, and both my dev and production databases will be MySQL.  This is relevant to the Boolean field desire there is a long history in software development of how to store Boolean data types in a SQL database whose standards classically do not support Boolean. In the current JHipster/Liquibase incarnation, tables in MySQL are generated for us, which is really nice.  The Boolean data types are stored as BIT  (1).  This is not a problem so far -- most developers seem to agree now that as a best practice, we should store values in databases as false = 0 and true = 1, and a BIT(1) is a great, simple way to do that. An issue arises when we try to use liquibase to set/update our database to the desired starting state.  For my project, I've chosen gradle instead of maven as a build tool, and gradle has a plugin for liquiba
Post a Comment
Read more

Spring Security - Authority vs Role

I have spent a lot of time recently trying to understand the difference between Authority and Role in Spring Security.  This is a brief review of what I found. When creating a UserDetailsService or overriding configure(AuthenticationManagerBuilder auth) in the security config class that extends WebSecurityConfigurerAdapter, I basically get complete control over what I populate inside of the UserDetails that is used/returned.  This is important because the UserDetails interface really only cares about how to return one thing: Collection<? extends GrantedAuthority> getAuthorities(); A GrantedAuthority just seems like a glorified String wrapper that names some thing.  The question is... what is that thing? This is where the subtle difference between Authority and Role comes into play. I think that Role is an older thought/construct that automatically gets plugged into Authority if we just create a user with a Role.  But completely forget about the code and classes for a mi
Post a Comment
Read more

Gmail internal application, Two-Legged OAuth2, Server to Server authentication, and Google API versions

I am working on a little tool at home in my free time to put some skills into practice.  The general idea (nothing novel) is this: I have some financial alerts sent to a new email address I have spun up on my domain.  I am creating an AWS Lambda that will wake up on an hourly schedule, read those emails, and publish SNS messages with parsed financial transaction information.  I then will have an SQS queue listen to the SNS message topic that is consumed by a Step Function.  The Step Function will: store the financial transaction information into a database send an SMS to me if the transaction is above a certain threshold. I could later extend this to do some aggregation reporting, etc if I wanted, too.  This will only work for my own financial transactions, and the data being gathered/stored will be sufficiently vague, so I am not really concerned about financial security for this project. The biggest hurdle I have run into so far is connecting to Gmail securely.  I was ab
Post a Comment
Read more