Avoiding Repeated Test Behaviour Across Multiple Tests with Test Fixtures

Avoiding Repeated Test Behaviour Across Multiple Tests with Test Fixtures

Avoiding Repeated Test Behaviour Across Multiple Tests with Test Fixtures

For anyone who has had the pleasure of writing/maintaining tests for code that require the use of external dependencies, you will (hopefully!) have incorporated the use of a mock of a function/object. In gMock, you need to define the behaviour of a mock via the EXPECT_CALL & ON_CALL macros in each test before it is used. When multiple tests rely on the same use of a mock, the common method I see far too frequently is to either, copy-paste the mock’s behaviour across each test case that relies it, or just dump all your assertions into a single monolithic test case to avoid copy-pasting. Both are terrible habits to fall into that lead to tests that are brittle and harder to read.

In this guide, I want to demonstrate ways in which you can recognise repeated test setups, and where utilising a Test Fixture class to resolve this can simplify production & maintenance of tests.

Bad Examples of Tests

Consider the following procedural function written in C. Let’s say that we’ve been tasked to write a series of tests on some legacy code that when called retrieves some data from an external module (retrieveExternalData), before passing it on to a function that exists in another external module (sendMessage). For now, ignore the fact that this code is absolute trash and should have been refactored; it’s hard to think of a good example!

Source Code
int functionUnderTest( enum DataType data_type )
{
int error_code;
struct SourceDataStructure external_data;
struct DestinationDataStructure modified_data;
    error_code = retrieveExternalData( &external_data );
     if( error_code == 0 )
{
// Modify data according to the data type
switch( data_type )
{
case 0:
// Modify data one way
...
break;
            case 1:
// Modify data another way
...
break;
            default:
// Return error
error_code = -1;
break;
     if( error_code == 0 )
{
error_code = sendMessage( &modified_data );
}
}
    return error_code;
}

In between these 2 external function calls, some manipulation of data occurs that varies depending on the value of data_type passed into this function. For now our tests will focus on testing that the data passed into ‘sendMessage’ was performed to specification according to the value of data_type.

But because of the fact that there are external dependencies that we need to mock in order to take full control over how the tests will operate, for many of the tests the mocked behaviour may be identical. As a result, different tests may exhibit the same patterns. See for example the following:

Repeating Test Cases
TEST( ModifyingData, ShallSetXWhenDataTypeIs0 )
{
struct SourceDataStructure source_data;
struct DestinationDataStructure output_data;
int error_code;
      //----------------------------- ARRANGE ----------------------------
// Configure relevant variables
source_data.SomeProperty1 = FOO;
source_data.SomeProperty2 = BAR;
      // Initialise output_data to invalid values
output_data.SomeProperty1 = INVALID_VALUE;
output_data.SomeProperty2 = INVALID_VALUE;
      // Configure the mock behaviour of retrieveExternalData
EXPECT_CALL( *_MockObject, retrieveExternalData )
.Times( 1 )
.WillOnce( DoAll( SetArgPointee<0>( source_data ), // Provide the source data when it is called
Return( 0 ) ) ); // Return no error
      // Configure the mock behaviour of sendMessage to capture the modified data
// and store it in output_data to allow us to assert on its properties later
EXPECT_CALL( *_MockObject, sendMessage )
.Times( 1 )
.WillOnce( DoAll( SaveArg<0>( &output_data ), // Capture the modified data
Return( 0 ) ) ); // Return no error
      //------------------------------- ACT ------------------------------
// Call the function under test with data type 0
error_code = functionUnderTest( 0 );
      //------------------------------ ASSERT -----------------------------
// Check that no error was returned
ASSERT_THAT( error_code, 0 );
      // Check that the modified data captured is correct for this data type
ASSERT_THAT( output_data.SomeProperty1, EXPECTED_FOR_DATATYPE_0 );
ASSERT_THAT( output_data.SomeProperty2, EXPECTED_FOR_DATATYPE_0 );
}
TEST( ModifyingData, ShallSetXWhenDataTypeIs1 )
{
struct SourceDataStructure source_data;
struct DestinationDataStructure output_data;
int error_code;
      //----------------------------- ARRANGE ----------------------------
// Configure relevant variables
source_data.SomeProperty1 = FOO;
source_data.SomeProperty2 = BAR;
      // Initialise output_data to invalid values
output_data.SomeProperty1 = INVALID_VALUE;
output_data.SomeProperty2 = INVALID_VALUE;

// Configure the mock behaviour of retrieveExternalData
EXPECT_CALL( *_MockObject, retrieveExternalData )
.Times( 1 )
.WillOnce( DoAll( SetArgPointee<0>( source_data ), // Provide the source data when it is called
Return( 0 ) ) ); // Return no error

      // Configure the mock behaviour of sendMessage to capture the modified data
// and store it in output_data to allow us to assert on its properties later
EXPECT_CALL( *_MockObject, sendMessage )
.Times( 1 )
.WillOnce( DoAll( SaveArg<0>( &output_data ), // Capture the modified data
Return( 0 ) ) ); // Return no error
      //------------------------------- ACT ------------------------------
// Call the function under test with data type 1
error_code = functionUnderTest( 1 );
      //------------------------------ ASSERT -----------------------------
// Check that no error was returned
ASSERT_THAT( error_code, 0 );
      // Check that the modified data captured is correct for this data type
ASSERT_THAT( output_data.SomeProperty1, EXPECTED_FOR_DATATYPE_1 );
ASSERT_THAT( output_data.SomeProperty2, EXPECTED_FOR_DATATYPE_1 );
}

We can clearly see that these tests are almost identical in every way except for the values that we’re actually trying to test for. If the number of data types that change the behaviour of this function was to increase, this would add to the number of test cases that were needed. To make matters even worse, what happens if something changes to the external dependency that affects our code? It would probably lead to a lot of broken test cases and time would be wasted in trying to fix them all. Such pitfalls makes the process of refactoring code far more tedious than it ought to be.

Now the most natural conclusion to make when seeing code repeated would be to place that behaviour in its own function. That would be great, except for the fact that gMock will not allow the behaviour of the mocks to be defined in a function that can be called from a TEST macro. So how can this problem be solved? Thankfully, googletest provides us with…

Test Fixtures

In googletest, a Test Fixture is a class that allows us to control how a group of tests behave. We can use this to define a common behaviour for mocks and initialise data at the beginning of each test without having to continually repeat ourselves.

To begin with, the following is a bare-bones implementation of a test group derived from the TestFixture class:

TestFixture Implementation
class ModifyingData: public TestFixture
{
public:
// Constructor. Allows any data that needs to be created once, but
// used many times, to be initialised here.
ModifyingData() : TestFixture()
{
}
      // Test setup. This gets called before each test is run
void SetUp()
{
}
      // Test tear down. This gets called after each test has run
void TearDown()
{
}
}

With this class, we can now place the default behaviours of the mocks and initialise any data within the SetUp method as follows:

TestFixture with common behaviour
class ModifyingData: public TestFixture
{
public:
struct SourceDataStructure source_data;
struct DestinationDataStructure output_data;
      // Constructor. Allows any data that needs to be created once, but
// used many times, to be initialised here.
ModifyingData() : TestFixture()
{
}
      // Test setup. This gets called before each test is run
void SetUp()
{
// Configure relevant variables
source_data.SomeProperty1 = FOO;
source_data.SomeProperty2 = BAR;
          // Initialise output_data to invalid values
output_data.SomeProperty1 = INVALID_VALUE;
output_data.SomeProperty2 = INVALID_VALUE;
          // Configure the mock behaviour of retrieveExternalData
ON_CALL( *_MockObject, retrieveExternalData )
.WillByDefault( DoAll( SetArgPointee<0>( source_data ), // Provide the source data when it is called
Return( 0 ) ) ); // Return no error
          // Configure the mock behaviour of sendMessage to capture the modified data
// and store it in output_data to allow us to assert on its properties later
ON_CALL( *_MockObject, sendMessage )
.WillByDefault( DoAll( SaveArg<0>( &output_data ), // Capture the modified data
Return( 0 ) ) ); // Return no error
}
      // Test tear down. This gets called after each test has run
void TearDown()
{
}
}

Now for each test run, we no longer have to define the behaviour of the mocks, nor do we have to initialise commonly used data.

You may have also noticed that the mock behaviour is using the ON_CALL method instead of the EXPECT_CALL method. The reason that this is done is that I actually do not want to actually test whether a mocked function is called by default, I just want to define how that mocked function behaves if and when it is called. If I were to use EXPECT_CALL, then I am demanding that all tests for this group must always call these functions. This may not always be the case when tests focus on control flow that may result in sendMessage never being called, causing a test to fail unnecessarily. There’s an interesting read that goes into more detail on why you would use ON_CALL over EXPECT_CALL here: Knowing When to Expect.

With this common setup out of the way, we can refactor the tests so that they are easier to read and far more maintainable. This time, all tests will use the TEST_F macro instead:

Improved Test Cases
TEST_F( ModifyingData, ShallSetXWhenDataTypeIs0 )
{
int error_code;
      //----------------------------- ARRANGE ----------------------------
// Nothing to do here anymore...
      //------------------------------- ACT ------------------------------
// Call the function under test with data type 0
error_code = functionUnderTest( 0 );
      //------------------------------ ASSERT -----------------------------
// Check that no error was returned
ASSERT_THAT( error_code, 0 );
      // Check that the modified data captured is correct for this data type
ASSERT_THAT( output_data.SomeProperty1, EXPECTED_FOR_DATATYPE_0 );
ASSERT_THAT( output_data.SomeProperty2, EXPECTED_FOR_DATATYPE_0 );
}
TEST_F( ModifyingData, ShallSetXWhenDataTypeIs1 )
{
int error_code;
      //----------------------------- ARRANGE ----------------------------
// Nothing to do here anymore...
      //------------------------------- ACT ------------------------------
// Call the function under test with data type 1
error_code = functionUnderTest( 1 );
      //------------------------------ ASSERT -----------------------------
// Check that no error was returned
ASSERT_THAT( error_code, 0 );
      // Check that the modified data captured is correct for this data type
ASSERT_THAT( output_data.SomeProperty1, EXPECTED_FOR_DATATYPE_1 );
ASSERT_THAT( output_data.SomeProperty2, EXPECTED_FOR_DATATYPE_1 );
}

Already, these tests become far easier to read, and are far more maintainable. This simplifies the process of modifying tests when changes to the source code are planned, or allows for quick updates to the behaviour of external dependencies if they are changed.

On a side note, you have probably noticed that the tests still repeat themselves. The good news is that unlike gMock’s macros, googletest’s assertion macros can be placed into a function, with parameters allowing us to define what values we want to assert against. Bad news is that a very useful feature of googletest’s Test Explorer window in Visual Studio becomes slightly less useful. For any test that fails, the Test Explorer allows you to jump to the assertion that failed. If that assertion is within a function, it jumps to the line within that function, making it more difficult to see the overall context in which it failed. However, having less code overall is more beneficial in the long term than the slight annoyance introduced by having assertions within a function. In the interest of good coding practice, let’s refactor the tests to make them even more maintainable:

Improved & Refactored Test Cases
void assertModifiedDataIsCorrect( int error_code,
                                 int expected_value_property1,
                                 int expected_value_property2 )
{
      // Check that no error was returned
    ASSERT_THAT( error_code, 0 );
      // Check that the modified data captured is correct for this data type
    ASSERT_THAT( output_data.SomeProperty1, expected_value_property1 );
    ASSERT_THAT( output_data.SomeProperty2, expected_value_property2 );
}
TEST_F( ModifyingData, ShallSetXWhenDataTypeIs0 )
{
      //----------------------------- ARRANGE ----------------------------
      // Nothing to do here anymore...
      //------------------------------- ACT ------------------------------
      //------------------------------ ASSERT -----------------------------
    assertModifiedDataIsCorrect( functionUnderTest( 0 ),
                                 EXPECTED_FOR_DATATYPE_0,
                                 EXPECTED_FOR_DATATYPE_0 ); 
}
TEST_F( ModifyingData, ShallSetXWhenDataTypeIs1 )
{
      //----------------------------- ARRANGE ----------------------------
      // Nothing to do here anymore...
      //------------------------------- ACT ------------------------------
      //------------------------------ ASSERT -----------------------------
    assertModifiedDataIsCorrect( functionUnderTest( 1 ),
                                 EXPECTED_FOR_DATATYPE_1,
                                 EXPECTED_FOR_DATATYPE_1 ); 
}

Deviating a Mock’s Default Behaviour

It should be clear by now how we can avoid repeating ourselves when defining a mock’s behaviour, but what do you do when testing for edge cases that may need a mock to do something differently? Referring back to the source code, we can observe that there is control flow within the function that will not modify any data when retrieveExternalData returns an error code other than 0:

Source Code: different control flow
error_code = retrieveExternalData( &external_data );
if( error_code == 0 )
{
// Modify data
....
}

The great thing about gMock is that it allows us to create a new behaviour for a mock, and gMock will use the most recent definition that matches, allowing us to use the same Test Fixture. An example of a test case for testing this control flow could be as follows:

Overriding Default Mock Behaviour
TEST_F( ModifyingData, ShallNotOccurWhenExternalDataIsNotAvailable )
{
int error_code;
int expected_error_code = -1;
      //----------------------------- ARRANGE ----------------------------
// Configure the mock behaviour of retrieveExternalData
ON_CALL( *_MockObject, retrieveExternalData )
.WillByDefault( Return( expected_error_code ) ) ); // Return an error
      // Ensure that data is not passed to sendMessage
EXPECT_CALL( *_MockObject, sendMessage )
.Times( 0 );
      //------------------------------- ACT ------------------------------
error_code = functionUnderTest( 0 );
      //------------------------------ ASSERT -----------------------------
ASSERT_THAT( error_code, expected_error_code );
}

 

In the above example, the mock’s behaviour in ON_CALL is defined after the one in the Test Fixture’s SetUp method, so it will take precedence over the one in SetUp.

It is also interesting that ON_CALL and EXPECT_CALL both follow this rule. This means that the most recent EXPECT_CALL can take precedence over an ON_CALL for the same mock, and vice-versa. This is useful as demonstrated in the above example, where it is important that we assert that sendMessage is never called when the function under test fails to retrieve any data from retrieveExternalData. We achieve this by specifying the cardinality of the mocked function’s call as 0 ( .Times(0) ).

Other Tests to Consider

With this knowledge, consider how the default behaviours would differ when testing the following edge cases, and whether ON_CALL or EXPECT_CALL would be appropriate:

  • Passing a data_type outside of the range accounted for within this function. The function retrieveExternalData would still be called in the same way, but what happens to sendMessage?
  • How could a test be written in the same Test Fixture that forces sendMessage to return an error?

When Not To Deviate Default Mock Behaviour

If you ever find yourself having to deviate a mock’s default behaviour in the same way more than once, it’s probably a good time to ask yourself whether the Test Fixture has done everything that it can reasonably do. To avoid falling into the same trap and repeating yourself, it is probably a good idea to create a new Test Fixture and defining the default behaviour for those mock(s) in its SetUp method. At the end of the day, the less code there is to maintain, the better it is for everyone. If it helps, always pretend that the person lumped with the misfortune of maintaining your code is a psychopath who knows where you live…

More From The Blog

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?When I was a fledgling engineer, the company I worked for hired a new Technical Director.  I remember it vividly because one of his first presentations, to what was a very large engineering team, made the statement...

Standard[ised] Coding

Standard[ised] Coding

Standard[ised] CodingRecently I was handed a piece of code by a client “for my opinion”. A quick glance at the code and I was horrified! This particular piece of code was destined for a SIL0 subsystem of the safety critical embedded system that we were working on. Had...

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise ...Computers today are faster than at any time in history. For instance,  the PC on which this blog is being typed is running a multitude of applications, and yet, this concurrency barely registers as the characters peel their ways...

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?Test Driven Development (TDD) is a software development process that started around the early Noughties and is attributed to Kent Beck. The basic concept with TDD is that a test is written and performed first (obviously fails)...

Ticking The Box – Effective Module Testing

Ticking The Box – Effective Module Testing

Ticking The Box - Effective Module TestingIn the world of software development one of the topics of contention is Module Testing. Some value the approach whilst others see it as worthless. These diametrical opposed views even pervade the world of Safety Critical...

Ruminations on C++11

Ruminations on C++11

This is a blog about the new version of C++, C++11. It has been many years in the making and adds lots of new features to the C++ language. Some of those features are already supported by commonly used compilers, a description of these features follows.

Service-Oriented Architecture and the Enterprise Service Bus Model

Service-Oriented Architecture and the Enterprise Service Bus Model

Service-Oriented Architecture and the Enterprise Service Bus Model

With any service-oriented solution, there is a certain challenge to configuring a setup that scales well with workload. Implementing an Enterprise Service Bus (ESB) model helps to alleviate the difficulties in doing such, by allowing any connected application to act as a service provider.

  A visual comparison of ESB
  Peer-to-peer/Bit-Torrent communications

An Enterprise Service Bus model is a model in which any of the connected applications can be assigned a server or client role, in turns, allowing for full utilisation of all connected devices for the current problem being solved.

The ESB model promotes agility and flexibility in application communication, reducing the amount of pre-configuration for a high-performance setup.

Unlike similar systems which employ peer-to-peer communication for distribution, such as bit-torrent, the Enterprise Service Bus model allows for a greater range of possible functions for each client connected to the bus. Such functions may be:

  • Message routing
  • Calculation processing
  • Request fulfilment

Effectively, the ESB architecture allows for a simple plug-and-play system when connecting additional applications to a system, utilising a simple but well-defined messaging structure to allow any connected application to make full use of the resources available.

Scalability

One of the core advantages of implementing an ESB architecture is the ability of the system to be easily scaled up. As individual nodes on the ESB do not care about what other nodes are connected to the system, very little to no configuration is required when adding additional resources or applications. Should you need to add data processing to an ESB that has already handled the data that you wished to process, it would be as simple as adding a data processing application into the existing ESB network and reading the data output to process it.

Zircon’s Use of the ESB architecture

Zircon was asked by a client to look into implementing an ESB into one of their existing products. In its current state, each device connected to the product required much configuration in order to be considered reliable, as it utilised two separate networks for redundancy. This meant that engineer time was taken up simply creating different configurations to ensure that communication between devices was seamless.

The plan was to implement an ESB system to allow the following advantages over the existing solution:

  • To allow devices to be hot-swapped in with minimal configuration, which would also allow for easier maintenance of the solution’s hardware.
  • To allow easier scaling of the solution, by adding additional hardware that could be immediately leveraged.
  • To increase the redundancy of the system by replicating stored data across multiple nodes, to prevent data-loss due to hardware failure, power failure, etc.

The ESB architecture proposed would also allow for easier future expansions to the system, as the data was already available in a known format, and older systems would not need to be updated in order to support new additions.

Summary

The Enterprise Service Bus architecture is becoming increasingly popular in solutions that need high maintainability and scalability and is a cornerstone of modern service-oriented computing solutions. Zircon has knowledge and experience of the ESB architecture, and it is now part of the multi-faceted toolbox that we use in order to provide solutions to our clients going forward.

More From The Blog

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?When I was a fledgling engineer, the company I worked for hired a new Technical Director.  I remember it vividly because one of his first presentations, to what was a very large engineering team, made the statement...

Standard[ised] Coding

Standard[ised] Coding

Standard[ised] CodingRecently I was handed a piece of code by a client “for my opinion”. A quick glance at the code and I was horrified! This particular piece of code was destined for a SIL0 subsystem of the safety critical embedded system that we were working on. Had...

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise ...Computers today are faster than at any time in history. For instance,  the PC on which this blog is being typed is running a multitude of applications, and yet, this concurrency barely registers as the characters peel their ways...

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?Test Driven Development (TDD) is a software development process that started around the early Noughties and is attributed to Kent Beck. The basic concept with TDD is that a test is written and performed first (obviously fails)...

Ticking The Box – Effective Module Testing

Ticking The Box – Effective Module Testing

Ticking The Box - Effective Module TestingIn the world of software development one of the topics of contention is Module Testing. Some value the approach whilst others see it as worthless. These diametrical opposed views even pervade the world of Safety Critical...

Ruminations on C++11

Ruminations on C++11

This is a blog about the new version of C++, C++11. It has been many years in the making and adds lots of new features to the C++ language. Some of those features are already supported by commonly used compilers, a description of these features follows.

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

When I was a fledgling engineer, the company I worked for hired a new Technical Director.  I remember it vividly because one of his first presentations, to what was a very large engineering team, made the statement “There’s never enough time to do it right, but there’s always enough time to do it over” (Jack Bergman).

At this particular time, the organisation was suffering from a constant fire fighting method of working.  Work was constantly being shipped that was poorly engineered or darn right buggy.  The next project always started with an intent to “do it right” but problems from the delivery of the previous project rapidly eroded the time available and engineers slipped back into the “just ship it” attitude and the vicious cycle began again.

Sadly, the Technical Director was never able to pull the organisation out of that Venus fly trap – the way of working had become a “habit” and no time was spent on coaching the team into the right mindset.

Winding forward a couple of decades, I find myself with an opportunity to demonstrate the benefits of “doing things right”.

A client project required a critical module to be fundamentally reworked.  This particular module had actually been ported from a previous system by an offshore company.  Unfortunately, the module involved a sophisticated interaction with hardware which was not well understood by the offshore company.  What they delivered “worked” but lacked any real code quality.  In fact, the module was written like a “prototype” version despite it being a safety function within a SIL2 system. There were fragments of code that clumsily achieved the necessary function but formulated in an obfuscated way.  The client had attempted to fix a bug with this module but had failed due to the difficulty in understanding how it worked.

We set to work to fix the module.  Firstly we locked in the functional behaviour by creating a set of Unit Tests.  Fortunately, we had excellent domain knowledge for this module and were able to formulate a comprehensive suite of tests.  This was done one function and one Use Case at a time in a backward variation of Test Driven Development (TDD).  Our strategy was to separate out the actual hardware access from the processing element of the module (a good architectural principle). We were able to build stubs and therefore “pretend” to be hardware whilst running on a PC development environment.  These tests were applied to the code base and checked that we adequately captured all of the existing behaviour (including the undesirable bugs).  This test harness was to be our stable ground.

Continuing with the Test Driven Approach we began to refactor the module.  Each single change resulted in the suite of tests being run to confirm that behaviour had not changed.  Occasionally, the refactoring required tests to be created or changed.  The module eventually became understandable and many of the bad code smells removed. We were even able to clearly link the safety requirements to specific code functions making the assessment process smoother.

Having reached the point where the Unit Tests passed on our refactored code, we then formulated a set of integration tests. These were specifically to check that the right hardware interactions occurred and that data/calls to and from this module functioned correctly. This was a structured test in as much as we didn’t want to just plug it in and see if it worked, we wanted to ensure that it initialised correctly, setup its data and that processes got called in the right sequence.

I’ve seen many occasion where Integration tests are just simply a case of running the software.  When (if at the time) it fails the debug cycle begins.  However, that’s incredibly inefficient.  Just like with TDD, if you aim to check each step and a failure occurs then generally you know where the fault lies and you don’t need to spend much time debugging.  Also, just because the software ‘runs’ it doesn’t mean that there are no bugs.  Have a structured plan of integration tests provides a greater chance of finding them.

Fortunately, on our first Integration test we found a bug.  This test was checking that the initialisation occurred but one of the modules public functions was being called at the same time.  This was quickly fixed, the Unit test rerun and Integration continued.  These structured tests took less than half a day to complete but at the end of it we were confident that the system operated correctly.

Our last step was to perform a System Test. It’s important at this point to distinguish the different levels of testing. Unit Testing is aiming to remove the bugs associated with logical and functional behaviour.  Integration Testing is aiming to remove the problems with interacting with other bits of software/system such as missing or malformed data being passed. Finally, System testing is checking the performance of the system including whether it meets the timing constraints and whether it delivers the whole system behaviour.

I like to look at these as sieving the software for bugs – you are progressively removing problems. If you fail to undertake one of these levels of testing ( and the classic is to leave it all to a system test) then you can expect to spend a very long time testing and debugging.  This is because debugging on system test is generally very slow and lacks the granularity of control needed to ensure an adequate test.

Following this approach meant the module was refactored, fixed and tested rapidly. More importantly, other than one integration test that failed, the software worked perfectly on the final system, i.e. “First Time” and is still running without problem.

What this shows is that the investment in following best practise in testing really does pay off.  Making this a habit means that projects will get delivered on time and in budget more often.

More From The Blog

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?When I was a fledgling engineer, the company I worked for hired a new Technical Director.  I remember it vividly because one of his first presentations, to what was a very large engineering team, made the statement...

Standard[ised] Coding

Standard[ised] Coding

Standard[ised] CodingRecently I was handed a piece of code by a client “for my opinion”. A quick glance at the code and I was horrified! This particular piece of code was destined for a SIL0 subsystem of the safety critical embedded system that we were working on. Had...

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise ...Computers today are faster than at any time in history. For instance,  the PC on which this blog is being typed is running a multitude of applications, and yet, this concurrency barely registers as the characters peel their ways...

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?Test Driven Development (TDD) is a software development process that started around the early Noughties and is attributed to Kent Beck. The basic concept with TDD is that a test is written and performed first (obviously fails)...

Ticking The Box – Effective Module Testing

Ticking The Box – Effective Module Testing

Ticking The Box - Effective Module TestingIn the world of software development one of the topics of contention is Module Testing. Some value the approach whilst others see it as worthless. These diametrical opposed views even pervade the world of Safety Critical...

Ruminations on C++11

Ruminations on C++11

This is a blog about the new version of C++, C++11. It has been many years in the making and adds lots of new features to the C++ language. Some of those features are already supported by commonly used compilers, a description of these features follows.

Standard[ised] Coding

Standard[ised] Coding

Standard[ised] Coding

Recently I was handed a piece of code by a client “for my opinion”. A quick glance at the code and I was horrified! This particular piece of code was destined for a SIL0 subsystem of the safety critical embedded system that we were working on. Had it been destined for the more critical SIL2 control subsystem then I would immediately have halted its progress. So what gross indecency had this code committed?  well simply that it contravened the coding standard in place for the project. This got me thinking and I began to reflect on why we used coding standards in the first place.

Coding standards embody the norms and practices of an organisation with regard to programming style, rules and conventions but why have one? Why not allow each software engineer to use their maximum creativity, knowledge and experience to craft software modules?

To answer this question, it’s probably important to look at some of the things that can go wrong whilst writing software. For the sake of this article we will use the ‘C’ programming language. This language is a wonderful bare-bones programming experience giving the engineer flexibility to access the nuances of the hardware. However, there are some real dangers lurking beneath the waves.

By way of an example let’s look at the following:

int mycopy (char *dest, char *src, unsigned int len)
{
    while (len--)
    {
        *dest++ = *src++;
    }
}
int count = -1;
mycopy( buffer, source, count );
In this simple piece of code, the parameter ‘count’ would quietly be converted to an unsigned int and since the size of the types are the same the bit representation of the value remains the same (remember minus one is 0xFFFF on a sixteen-bit system). This would make the minus one value effectively a very large positive number in the unsigned ‘len’ parameter. Without doubt this would overrun the target buffer and trash its way through memory. Even though we know this, individuals can easily miss the type difference when writing large amounts of code and utilising modules from others.

Therefore, our coding standard should, as far as possible, prevent us from using constructs or operations in the language that are at risk of causing problems. Ideally we should use tools that automatically draw it to our attention. One such standard is ‘MISRA’ which has existed for many years now and has a multitude of tools available to check adherence to it (and in the case of the client above was actually mandated!).

What such a standard does it to help normalise the skill levels of the members of the team. Experienced engineers who might never make this mistake and junior engineers, who perhaps aren’t even aware of the type of problem, will now produce code with a lower probability of bugs. Any team is only as strong as its weakest link (the most junior engineer) and therefore finding ways for that individual not to make the mistake is well invested time – fewer bugs created at the source of the problem means time and cost savings!

The other area in which coding standards have a huge impact (but also brings the most arguments) is in style and naming conventions. Having a common style is more important than most engineers realise and I argue that this is because of productivity.  I will abstain from the discussion of whitespace other than to say that it is the consistency that is important so that, when reading code, the user can quickly judge blocks of code and should not have to hunt for the closing brace. This is most apparent when utilising another team members code, the ability to find ones’ way around it has a direct impact on the time taken to finish writing code. Common ways of organising and naming elements within the code all help to reduce the “learning and using” time of other people’s code.

Sometimes, coding standards will enshrine good programming principles such as keeping things small, modular, loosely coupled etc. Again, these principles should not be overlooked as they will have a direct bearing on the cost of development (and time to completion). Anything that reduces complexity will speed understanding and reduce the possibility of bugs – all enemies of productivity.

So, with all of these elements making up the coding standard one should expect that, generally speaking, each programmer would produce code that is similar or “standard code”.

The phrase “coding standard” is now a little warn-out, often has negative connotations OR is often overlooked. Using the phrase “Standard[ised] Coding”, I think, aptly describes the reason why we have these documents – not to restrict the flair, creativity and productivity of engineers but to ensure we get a consistent output from everyone who comes into contact with the code. This in time will save money and project duration which, in my book, is something that businesses should constantly strive for in order to be successful.

Finally, the biggest benefit of “Standardised code” is the maintainability of the software. Software lasts for a very long time and frequently the developers are not responsible for its maintenance. Having software that is consistent throughout will allow it to be far easier to maintain in the future for the engineers who will be responsible for corrective maintenance and feature enhancements.

So what about the piece of code that I “reviewed”? Well this will now have to be rewritten to comply to the standards that do exist in the client’s organisation. The code is riddled with the types of errors indicated above and is in a style totally different from that produced by any other engineer. Sadly, this will cost time for the project which will have an impact at a later date – not ideal.

More From The Blog

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?When I was a fledgling engineer, the company I worked for hired a new Technical Director.  I remember it vividly because one of his first presentations, to what was a very large engineering team, made the statement...

Standard[ised] Coding

Standard[ised] Coding

Standard[ised] CodingRecently I was handed a piece of code by a client “for my opinion”. A quick glance at the code and I was horrified! This particular piece of code was destined for a SIL0 subsystem of the safety critical embedded system that we were working on. Had...

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise ...Computers today are faster than at any time in history. For instance,  the PC on which this blog is being typed is running a multitude of applications, and yet, this concurrency barely registers as the characters peel their ways...

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?Test Driven Development (TDD) is a software development process that started around the early Noughties and is attributed to Kent Beck. The basic concept with TDD is that a test is written and performed first (obviously fails)...

Ticking The Box – Effective Module Testing

Ticking The Box – Effective Module Testing

Ticking The Box - Effective Module TestingIn the world of software development one of the topics of contention is Module Testing. Some value the approach whilst others see it as worthless. These diametrical opposed views even pervade the world of Safety Critical...

Ruminations on C++11

Ruminations on C++11

This is a blog about the new version of C++, C++11. It has been many years in the making and adds lots of new features to the C++ language. Some of those features are already supported by commonly used compilers, a description of these features follows.

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

Computers today are faster than at any time in history. For instance,  the PC on which this blog is being typed is running a multitude of applications, and yet, this concurrency barely registers as the characters peel their ways across the screen. Developers bask in the gigahertz of sheer power, rarely stopping to question whether the engine that sits purring beneath their glistening code will ever misfire. Students of Computer Science, however, will probably have come across Big-O as they tussled with the performance of a bubble sort versus a quick sort or such like.

As way of a simplified reminder, Big-O seeks to find the standard time of operation as a function of the size of the input. Values such O(n^2) and O(n log(n) [bubble sort and Quicksort respectively] provide an indication that, as the number of elements increases, the number of operations increase by a power of 2 (for Bubble sort). Of course astute developers will quickly select the algorithm closest to O(1), or realistically, some linear function at best. But for the rest, with their rippling muscular arrays of processors, never take, nor indeed need to care for, the time performance of their code.

The implementation of such Big-O driven algorithms, for most, is typically an academic exercise. However, the computing power that we’ve come to rely on once never existed beyond an 8 bit trickle of flip flops. Clearly, the art and science of computing drove such wild improvements. But there is still a place in today’s world where such Tera flops of computing power are simply unavailable – the embedded world. For whatever reason, Safety, Environmental or contractual obsolescence have demanded the use of smaller, less powerful CPU’s in the embedded world.

Enter the process of optimisation – the skinny sibling of the Big-O family and the big fat rain cloud of the embedded developer.

Those same computer scientists will remember that when an algorithm requires many thousands of repeat runs, the time taken to perform the run once becomes vitally important. However, on an embedded processor (running at just a few megahertz), performing an operation even a few hundred times means the single run time must be optimal. But in the embedded world there is a much closer harmony between High level code and CPU instructions. Most engineers will be familiar with how their respective compiler churns language such as ‘C’ into the myriad of instruction sequences. What can seem like a simple ‘for loop and if test’ in ‘C’ can generate many times the instructions on one compiler/processor when compared to another.

Let’s take a simple contrived example:

U8 sendDataToSPIBus( U8 data_byte[], U16 length,  U8 channel )
{
    U16 current_time;
    BOOLEAN success = TRUE;
    U16 i = 0;

    .... Defensive programming checks removed for conciseness....

    while ( ( length > 0 ) && ( success == TRUE ))
    {
        length--;
        ptr_hw[ channel ].TransmitBuffer = ( U32 ) data_byte[ i ];
        i++;

        /* wait for the Transmit Complete flag to be set */
        current_time = getHardwareTimerValue();

        while ( ptr_hw[ channel ].TransmitComplete == 0ul )
        {
            if ( (getHardwareTimerValue() - current_time) > TIMEOUT_VALUE )
            {
                success = FALSE;
                break;
            }
        }
    }

    /* clear the Transmit complete flag */

    ptr_hw[ channel ].TransmitCompleteClear == 0xffff;
    return success;
}

Actually, this fragment of code demonstrates an issue recently experienced by a client. The code essentially wrote data to an SD card over an SPI bus. However, the volume of data to be written to the card quickly showed an unacceptable amount of time taken. The answer seemed obvious: Optimisation! An engineer quickly did an assessment and found that this function could be improved to take less time. The engineer proceeded to amend the code and demonstrated the function had a time improvement.

The engineer in question decided to take the ‘channel’ aspect and place that in an ‘case’ statement (not the wisest of choices) and then accessed the hardware registers via a hardcoded address value. This removed the array access instructions. His last act of optimisation was to obtain the current time value directly from a hardware register rather than via the function getHardwareTimeValue() and to amend the use of the array index ‘i’. This yielded the following code:

case CHANNEL_0:

{
    while ( ( i < length ) && ( success == TRUE) )
    {
        *(( volatile U32 * const ) 0xF0001000ul) = data_byte[ i ];
        i++;
        current_time = (U16)*(( volatile U32 * const ) 0xf0003000ul);
        while ( *(( volatile U32 * const ) 0xF03100F8ul ) == 0ul )
        {
            if (  ((U16)*(( volatile U32 * const ) 0xf0003000ul) ) -
            current_time ) > TIMEOUT_VALUE )
            {
                success = FALSE;
                break;
            }
        }
    }
}

Job done! or was it?

In fact, this micro optimisation is not uncommon and just like in the field of economics (arguably the study of the effects of optimisation) often detrimental. In this particular case, the effect made very little difference  to the problem of storing a large volume of data to the SD card. The reason was quite simple, the bigger effect occurred at the system level where the Scheduler in use on this project pre-empted the task calling the function. The task was of the lowest priority and spent much of it’s time waiting for high priority tasks to complete. Another problem occurred with the optimisation whereby one of the changes broke another callee of the SPI function ( the case statement for the channel) and not least the lack of portability (amongst other things).

Therefore the message is clear, Optimisation is important and as developers we must always look to write simple but efficient code. However, optimisation needs to be done in the context of the whole system so that it makes the difference expected without other complications.

More From The Blog

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?When I was a fledgling engineer, the company I worked for hired a new Technical Director.  I remember it vividly because one of his first presentations, to what was a very large engineering team, made the statement...

Standard[ised] Coding

Standard[ised] Coding

Standard[ised] CodingRecently I was handed a piece of code by a client “for my opinion”. A quick glance at the code and I was horrified! This particular piece of code was destined for a SIL0 subsystem of the safety critical embedded system that we were working on. Had...

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise ...Computers today are faster than at any time in history. For instance,  the PC on which this blog is being typed is running a multitude of applications, and yet, this concurrency barely registers as the characters peel their ways...

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?Test Driven Development (TDD) is a software development process that started around the early Noughties and is attributed to Kent Beck. The basic concept with TDD is that a test is written and performed first (obviously fails)...

Ticking The Box – Effective Module Testing

Ticking The Box – Effective Module Testing

Ticking The Box - Effective Module TestingIn the world of software development one of the topics of contention is Module Testing. Some value the approach whilst others see it as worthless. These diametrical opposed views even pervade the world of Safety Critical...

Ruminations on C++11

Ruminations on C++11

This is a blog about the new version of C++, C++11. It has been many years in the making and adds lots of new features to the C++ language. Some of those features are already supported by commonly used compilers, a description of these features follows.

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development (TDD) is a software development process that started around the early Noughties and is attributed to Kent Beck. The basic concept with TDD is that a test is written and performed first (obviously fails) before code is developed to pass the test. For each additional feature or line of code to be developed another test case is written first. Each time, all previous test cases are rerun. Thus at its simplest, the approach assures a correctly operating feature and enables a regression test for each additional line of code.

To make Test Driven Development viable automated test systems are mandatory. These must also be fast systems since the developer will constantly be rerunning the test suite. No line of code should exist that does not have a corresponding test. This means that, by its very nature, not only is the feature tested but also each “defensive programming” element such as Null pointer checks and boundary checks are naturally included. This method of development is described as highly iterative since a developer is rapidly “refactoring code” as each new line (feature) is created and test cases reworked to suit.

One of the benefits of TDD is that code quality rises significantly. Indeed in a study conducted at Microsoft [1], projects running with a Test Driven Design Methodology found a 60-90% drop in defects. Defects, of course, are directly costly and time consuming, with each bug found at later stages costing significantly more than earlier in the lifecycle. This is not entirely unexpected since the code isn’t released for integration until it passes its test suite with each line of code having a corresponding test case.

This must of course come at a cost? Surely the effort required to build a test infrastructure around each code element must absorb the project timeframe? It’s easy to see this argument and even in the Microsoft study, managers subjectively claimed an increase of between 15% and 30% of project development time. However, one could argue that this method would still reduce overall time for development as the “back end” of the project becomes vastly more certain. In traditional projects it’s not uncommon to find the integration and system testing stages to be a never ending cycle of test, debug and fix. This never ending cycle becomes hard to plan for and thus code gets shipped complete with agreed outstanding bugs. With a much higher level of quality in TDD projects, the back end phases will be much shorter.

Another benefit of the TDD approach is that even when bugs are found later in the lifecycle, they can be fixed and the entire test suite rerun to ensure non regression of the software. A recent client demonstrated this fact clearly when, just days before a shipping date, a change was made to fix a bug. This particular change was extremely innocuous or so they thought, but this one bug stopped an unrelated aspect of the system from functioning which, fortunately, was detected moments before shipping and was rolled back. Had there been a test suite available, this simple change and its impact would have been detected instantly by the developer saving hours of engineer’s time in hunting for the culprit.

An additional benefit of TDD is that an evidential measure of progress can be established for the development programme. A feature doesn’t exist unless it works and it can’t be claimed to work unless a test case is written and executed for it. This is a gold mine for managers and planners alike and a positive step forward from the traditional claim from engineers that “it’s 90% done!” Features that are working are shippable thus if development time is running late decisions can be made about what can go in the release with certainty.

Thinking about how a particular test will be created also has the added benefits of improving the structure of developed code. Better interfaces, greater decoupling and lower complexity are the inevitable consequences of having to write tests first. The constant refactoring in TDD means that engineers are not frightened to make a change to lower complexity since everything will simply be retested anyhow.

However, it’s important to recognise the limitations of TDD.  What TDD doesn’t do (at least not in the established way) is remove the likelihood of usability errors, timing/real-time issues and other performance based aspects of a software system. When a developer implements a test, their assumptions are rolled into that case thus if they think the user wants a red button then the test case will reflect that. Confident that the test case has passed, the developer is ignorant of the fact that the user actually wanted a blue button. This type of test cannot be eliminated just because the project is using TDD. Similarly with Integration Testing, just because the individual units have been tested fully (via TDD) doesn’t mean they will function when connected together. This is so particularly in developments involving more than one person where differing assumptions can be made. Therefore, Integration, System and Acceptance testing are still crucial phases in a successful software development lifecycle.

Another argument often thrown in by critics of TDD is that more effort is required in the early stages of the development. This can be cumbersome on prototype or proof-of-concept projects and indeed it is – if the idea is simply to see if something is possible! However, many organisations have started out with a prototype that has evolved into the production version. Frederick Brook’s idea of “build one to throw away” is frequently seen as step too far! But let’s be clear, a prototype developed without TDD will not have the kind of quality discussed above and ergo the production version that it becomes will equally be afflicted by a swarm of bugs. The effort to tame this beast will be uncontrolled and it’s probably just too late to introduce TDD at this late stage. Let’s not forget, that TDD is an investment over the lifetime of the software not just during its initial development time (although opponents of TDD will be quick to point out the maintenance cost of the test suites).

So, is TDD the silver bullet? Obviously not, but it is a useful tool in the armoury of the Software Team that can reduce defects and produce working software swiftly and let’s face it – there is something quite satisfying (as an engineer) in getting something working!

 

References:

[1] research.microsoft.com (Accessed: April 2016)

More From The Blog

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?

Do It Right First Time? Or, Fight The Fire Later?When I was a fledgling engineer, the company I worked for hired a new Technical Director.  I remember it vividly because one of his first presentations, to what was a very large engineering team, made the statement...

Standard[ised] Coding

Standard[ised] Coding

Standard[ised] CodingRecently I was handed a piece of code by a client “for my opinion”. A quick glance at the code and I was horrified! This particular piece of code was destined for a SIL0 subsystem of the safety critical embedded system that we were working on. Had...

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise …

To Optimise Or Not To Optimise ...Computers today are faster than at any time in history. For instance,  the PC on which this blog is being typed is running a multitude of applications, and yet, this concurrency barely registers as the characters peel their ways...

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?

Test Driven Development: A Silver Bullet?Test Driven Development (TDD) is a software development process that started around the early Noughties and is attributed to Kent Beck. The basic concept with TDD is that a test is written and performed first (obviously fails)...

Ticking The Box – Effective Module Testing

Ticking The Box – Effective Module Testing

Ticking The Box - Effective Module TestingIn the world of software development one of the topics of contention is Module Testing. Some value the approach whilst others see it as worthless. These diametrical opposed views even pervade the world of Safety Critical...

Ruminations on C++11

Ruminations on C++11

This is a blog about the new version of C++, C++11. It has been many years in the making and adds lots of new features to the C++ language. Some of those features are already supported by commonly used compilers, a description of these features follows.