Continuous deployment for your tests

Source:- microfocus.com

Why would you want to continuously deploy your tests?

Release fast and release often. This core principle of modern software development provides numerous advantages, for example earlier customer feedback, faster response cycles, and overall higher customer satisfaction.

When trying to increase your release frequency, you will be faced with many testing issues, including the following:

Changing tests might lead to unexpected failures (false-positives) or unwanted passes (false-negatives).
The probability of introducing breaking changes to your automated test set increases with the size of the existing test set and the amount of people that are contributing changes to the test set.

How can you ensure that your testing can keep up with a faster release pace?

Simply adding more testers will probably not increase your ability to test your applications faster. To be able to continuously deliver functional and well tested software, you will have to apply continuous deployment for your automated tests!

You can combine Silk4J with a source control system (SCM) and a continuous integration (CI) server, for example Git and Jenkins, and build high-quality large-scale keyword libraries, which you can then schedule for automated testing in Silk Central. This enables you to create maintainable and scalable test sets which you can use to continuously and thoroughly test your software.

Structuring your tests

To be able to reap the benefits of continuous test deployment, you have to structure your testing efforts. Use the following best practices when working with large projects:

Separate the what and the how: Automation can get really complex. Automation experts have to consider many possible scenarios, while business experts know exactly what the desired business cases are. Separate the purely automation-related tasks from the business processes and you will make lives easier for both the automation expert and the business expert. You can easily achieve such a separation by using keyword-driven testing.
Give each user the optimal tool: Do not force technology affine staff to use a complicated web UI if they are faster with a text editor, and refrain from pushing Eclipse on your business experts if they do not want to get involved at that level. By using Silk Test with Silk Central, you can cleanly separate concerns and tools, and enable valuable contributions from both automation experts as well as business experts.
Test everything, even your tests: To save time and cost, avoid tests failing because of test-related issues, instead of spotting actual problems of the application under test. Ensure that you have tests for your tests so that you can find bugs with more confidence.

Defining the stages of the continuous delivery pipeline

The continuous delivery pipeline of our demo project includes the following stages:

Commit: Each tester contributes to a common version-controlled keyword library. You can use any SCM for this. We will use Git for our demo project.
Build on Jenkins: Each commit to the library triggers a build of the library in a separate build environment. By building the library often, any changes to local code, software, and files can be spotted immediately.
Validate: Test the keyword library. After the library has been built by Jenkins, run the following two types of tests:
- Unit-tests: These tests should validate that your keyword library does what it is supposed to do. Since we are using Silk4J, these tests are coded in Java.
- Integration or smoke tests: These are keyword-driven tests inside the library that make use of the library itself. Run these tests as part of your continuous integration build to ensure that the library can actually be used by Silk Central.
Deploy to Silk Central: If all integration tests were successful, you can automatically deploy the library to Silk Central. From that point on, any user of Silk Central can work with the updated library and immediately use newly created keywords.

Our example continuous deployment process

The architecture of our continuous deployment process looks like the following:

Prerequisites

To make use of this architecture, you need to have a Silk4J project that follows the keyword-driven testing methodology. Once you have all the pieces of your continuous deployment process in place, you will be able to use the keywords in this Silk4J project to create keyword-driven tests in Silk Central.

Additionally, ensure that the following infrastructure components are available:

A running Jenkins installation.
A Git repository to store the source code of your keyword library.
A Silk Central instance to store and schedule the keyword-driven tests that are based on the keyword library.

Building the project

The Silk4J project needs to be built automatically on Jenkins. Silk4J automatically creates an Ant buildfile, called build.xml, for newly created Silk4J projects. For our example workflow, we will use this project as our starting point.

By default, this project contains the following two tasks:

- compile: This task is used to compile the project into a JAR-file that can be used for running the tests
- build.keyword.library: This task creates a keyword library which can be uploaded to Silk Central.

Note: The compile task has to build the project before the build.keyword.library task can convert the project into a keyword library.

We will at a later point invoke the build.keyword.library task on Jenkins and use the Java Keyword Library Tool to upload the keyword library to Silk Central.

Configuring the source control system

Since we use Silk4J and Git in our example, we can use the integrated Git tooling in Eclipse to commit changes to the Git repository. For detailed information on how to set up and configure Git , see https://git-scm.com/book/en/v1/Getting-Started.

Creating the Jenkins job

We have to create a Jenkins job that checks out the source code of our Silk4J project from Git, runs the build.keyword.library task, and uploads the resulting keyword library to Silk Central. To make the job more generically usable we want the URL, the user name and password of Silk Central, as well as the keyword library name to be passed in as parameters.

In Jenkins, we will leverage the pipeline script feature to conveniently configure what exactly should happen whenever a tester commits changes to the library.

You can use the following script to create the pipeline job:

def gitCommit
def gitChangeLog

node("demonode") { // TODO: Replace with the name of the Jenkins node where the build should run
    stage("Preparation") {
        env.PATH = "${tool 'TODO PUT YOUR ANT TOOL NAME HERE'}/bin;${env.PATH}"
        git url: 'TODO: PUT YOUR GIT URL HERE'
        gitCommit = bat (returnStdout: true, script: 'git rev-parse HEAD').trim().split("\\n")[1]
        echo "${gitCommit}"
        keyworddrivenjar = "${env.OPEN_AGENT_HOME}/KeywordDrivenTesting/com.borland.silk.keyworddriven.jar"
        
        gitChangeLog = getChangeLog()
    }
    
    stage("Build") {
        bat 'ant compile'
    }
       
    stage("Upload keyword library") {
        if (currentBuild.result != 'FAILURE') {
            bat 'ant build.keyword.library'
            bat "java -jar \"${env.OPEN_AGENT_HOME}\\KeywordDrivenTesting\\com.borland.silk.keyworddriven.jar\" -upload ${libraryName} output/library.zip ${scURL} ${scUser} ${scPassword} \"Git commit: ${gitCommit}, Changes: ${gitChangeLog}\""
        }
    }
    
    stage("Archive keyword library") {
        archiveArtifacts artifacts: 'output/library.zip'
    }
}

@NonCPS
def getChangeLog() {
  def log = ""
  def changeLogSets = currentBuild.rawBuild.changeSets
  for (int i = 0; i < changeLogSets.size(); i++) {
     def entries = changeLogSets[i].items
     for (int j = 0; j < entries.length; j++) {
          def entry = entries[j]
          log += "${entry.commitId} by ${entry.author} on ${new Date(entry.timestamp)}: ${entry.msg}"
          def files = new ArrayList(entry.affectedFiles)
          for (int k = 0; k < files.size(); k++) {
              def file = files[k]
              log += "  ${file.editType.name} ${file.path}"
          }
      }
  }
  return log

}

Let us go through the script step by step:

The Preparation stage of the pipeline script sets up the required tools and checks out the source code of the keyword library from our Git repository. Additionally, this stage parses the Git commit history of all changes since the last checkout. This information will be used as a history comment in Silk Central, so that you can track changes made to the library from Silk Central to the source code consistently.
The Build stage invokes Ant to compile the library into a JAR file. Any compile errors are detected here.
The Upload keyword library stage invokes the build.keyword.library task to generate a keyword library named library.zip and to update this library to Silk Central. The upload is done with the Java Keyword Library tool, which is part of the com.borland.silk.keyworddriven.jar, from the Silk Test installation folder.
Finally, the Archive keyword library stage archives the library.zip on Jenkins, so that you could always manually restore the library of a particular build.

Note: Replace the highlighted place holders with values that apply to your configuration. For details on the Ant tool configuration refer to the documentation of the Jenkins Ant plug-in.

Add the following four parameters to the Jenkins job:

scURL (String parameter): The URL of the Silk Central installation to which the keyword library should be uploaded.
scUser (String parameter): The name of the Silk Central user that is used for uploading.
scPassword (String parameter): The password of the Silk Central user that is used for uploading.
libraryName (String parameter): The name of the keyword library in Silk Central.

Executing the job

Run the job manually by clicking Build with Parameters in Jenkins.

After the job successfully builds, the Stage View shows an overview of what has actually happened:

When you run the job, the current version of your keyword library is uploaded to Silk Central. The library is listed in the Libraries view in Silk Central, along with a consistent history of all changes made to the library through Git:

Based on that library, you can now create keyword-driven tests in Silk Central and schedule them for execution!

Improving the process

We can now automatically build and upload our keyword library to Silk Central on demand. The following enhancements improve our process even more.

Automatically uploading the keyword library

To automatically trigger your build and upload job on every single commit to the repository, so that you have the confidence that the library in Silk Central is always up to date, you can use the “Poll SCM” build trigger in Jenkins.

Validating

As soon as every commit leads to a build, and potentially an update of the keyword library in Silk Central, unwanted or even breaking changes might occur.

Apply the following three strategies to avoid such breaking changes:

Automatically build the library on Jenkins before uploading to ensure that compile errors and missing files are spotted early.
By using the Java Keyword Library Tool, you can ensure that an error is raised when trying to upload a library that would delete or rename keywords that are still used by keyword-driven tests in Silk Central.
Run a small set of smoke tests against the newly built keyword library before actually uploading it.

Smoke testing

Before uploading your keyword library to Silk Central, run a set of keyword-driven tests with that new library, and only trigger the upload if the tests pass. That way, you can ensure that the library that is released to the Silk Central users is actually usable.

To create the smoke test set, create new keyword-driven tests in your Silk4J project. Then use the Junit-way of running keyword-driven tests to launch the tests through Ant from Jenkins.

In our example, the JUnit test suite is called SmokeTestSuite and is executed by the keyword-driven test QuoteTest:

To call the keyword-driven test from Jenkins, create an additional task called test.keyword.library in your build.xml buildfile with the following content:

<target name="test.keyword.library">
  <mkdir dir="./reports"/>
    
  <junit printsummary="true" showoutput="true" fork="true">
      
    <classpath>
      <fileset dir="${output}">
        <include name="**/*.jar" />
      </fileset>
      <fileset dir="${buildlib}">
        <include name="**/*.jar" />
      </fileset>
      <fileset dir="C:/Program Files (x86)/Silk/SilkTest/ng/KeywordDrivenTesting">
        <include name="**/*.jar" />
      </fileset>
    </classpath>
      
    <test name="silktest.demos.insurance.SmokeTestSuite" outfile="./reports/TEST-silktest.demos.insurance.SmokeTestSuite-${silktest.configurationName}.xml">
      <formatter type="xml"/>
    </test>
  </junit>

</target>

You have now ensured that changes to the keyword library are only uploaded to Silk Central if the smoke test set passes with the changes in place.

With the tests included, the Stage View looks like the following:

Note: The Jenkins Junit plug-in enables you to drill down on the tests that were ran, allowing you to analyze any errors that might have occurred:

Summary

In this blog post, we have shown how large teams of testers can work together in building and maintaining stable keyword libraries. You can use such keyword libraries to create automated tests in Silk Central. We have also shown how you can use Jenkins and Git to enable multiple contributors to work on the same keyword library, and how you can use Silk Central as the interface between test developers and library maintainers.