Use TestLink and Jenkins to automate tests results logging

The purpose of this page is to explain how to create a Jenkins job that will run an automated test project an save test results in Testlink.

Prerequisite

Three things are required :

  • A Jenkins server
  • A TestLink server
  • A maven java project with automated tests

Step 1 : Configure TestLink

We start by configuring Testlink. The goal is to add some informations and configuration that will be used by Jenkins. We consider that we starts with an already existing Testlink project that contains the tests that are automated in the java project.

Add custom fields

We add a custom field to every test scenario. Those fields will be used to bind one test to one java class.

Select the right Project ("Test Management") and Test Plan ("System testing (Automated)"). From the main page, go to the "Define custom fields" section at the left of the screen". Click on "Create". Fill in informations :

  • Name : "java_class"
  • Label : "TestNG class name"
  • Available for : "Test Case"
  • Type : "string"
  • Enable on : "Test Spec Design
  • Display on test execution : "No"

Fill in custom fields and configuring test scenarios

Go to the "Test Specifications" section. The list of every test case is displayed. For every test case for whom you have an automated test :

  • Click on this test in the list
  • On the right part of the screen, click on "Add to test plan".
  • Tick "System testing (Automated)" and click on "Add"
  • For every step of the scenario, click on the line on the table, then select "Automated" in the "Execution" column then click on "Save"
  • Click on "Save & exit"
  • On the right part of the screen, click on the Edit button
  • Fill in the field "java_class" with the complete class name, including packages. For example : "net.ihe.gazelle.test.test_case.TM321".
  • Check that the Execution Type select box is at "Automated". Chose Automated if it's not.
  • Click on the Save button.

Create a developper key

Teslink has a special user which corresponds to the automated tester. He is called "Jenkins tester". To identify as this user, Jenkins uses its developper key. Currently it is "3f216bbb9926913a3373274d33c04242". If the key changed, it is accessible in the "My setting" menu when logged-in as the Jenkins tester user in Testlink.

At the home page of testlink, with an admin account, click on "Test project management". Clik on our project ("Test Management") and make sure that "Enable Test Automation (API key)" and "Enable inventory" are properly checked.

We are now done with TestLink. Every test case is considered as automated and is able to tell which java class should run it.

Step 2 : Configure Jenkins

Install the TestLink plugin

Click on "Administrate Jenkins" > "Plugins Management" > "Available" tab > at the "TestLink Plugin" line, check the box. At the bottom of the page, click on "Download now and install after restart". Wait a few seconds.

Go back to jenkins' homepage. Click on "Administrate Jenkins" > "Configure System".

At the TestLink section, fill in informations :

  • Name : the name of the installation
  • URL : the url of the xmlrpc.php page of the TestLink installation. In our case "http://gazelle.ihe.net/testlink/lib/api/xmlrpc/v1/xmlrpc.php"
  • Developper key : the key generated at the end of the TestLink section above
  • Click on "Save" at the bottom of the page

Installing the Extensible Choice plugin

Click on "Administrate Jenkins" > "Plugins Management" > "Available" tab > at the "Extensible Choice" line, check the box. At the bottom of the page, click on "Download now and install after restart". Wait a few seconds.

Installing the Xvfb plugin

Click on "Administrate Jenkins" > "Plugins Management" > "Available" tab > at the "Xvfb" line, check the box. At the bottom of the page, click on "Download now and install after restart". Wait a few seconds.

This plugin is used to create a temporaty virtual frame buffer for the job.

Installing the SSH plugin

Click on "Administrate Jenkins" > "Plugins Management" > "Available" tab > at the "SSH Plugin" line, check the box. At the bottom of the page, click on "Download now and install after restart". Wait a few seconds.

Go back to jenkins' homepage. Click on "Administrate Jenkins" > "Configure System".

At the "SSH Remote Hosts" section, Fill in information for the ssh access to the server that host the tested Gazelle-TM.

  • Hostname : GAZELLE_APPLICATION_SERVER_IP_ADDRESS
  • Port : 22
  • User name : gazelle
  • Keyfile : /home/jenkins/.ssh/id_rsa  (A pair of key must be generated for the jenkins user and the server must be configured to accept him)

This way jenkins can perform some initialization commands in the Gazelle-TM server.

Configure global choice parameters

Go to "Manage Jenkins" menu then click on "Configure Jenkins".

Then scroll to :

Extensible Choice: Available Choice Providers

 

In this section, make sure that Global Choice Parameter is ticked. Then click on "Add New Choice List" button.

Fill-in the new container like this :

  • Name : BROWSERS
  • Choices : List every browser you want to use. (firefox, chrome, ie choices only are supported)

Then click on "Add New Choice List" button.Fill-in the new container like this :

  • Name : XML_FILES
  • Choices :
    • testng
    • institutionmanagement
    • invoicemanagement
    • testingsessionmanagement
    • systemmanagement
    • useraccountmanagement
    • preconnectathonworkflow
    • testng-failed

Then click on "Add New Choice List" button.Fill-in the new container like this :

  • Name : SERVERS
  • Choices :
    • Ip adress or domain name of Gazelle-TM server

Create the job

Go back to Jenkins' home page. Click on "New Job", select "Free Style Project" and fill-in the name ("Gazelle-TM-GuiTesting").

In "source code management" select how your test project will be fetched by Jenkins. If it is on a SVN repository, select "Subversion" and fill-in repository informations. If the project is locally stored select "none".

The currend location is : "https://scm.gforge.inria.fr/anonscm/svn/gazelle/Maven/gazelle-gui-testing/Automated-Test-testNG/branches/idr_testing"

In the "General" section, tick "This build has parameters" then select "Add parameter" and chose "Extensible Choice". Fill-in informations :

  • Name : selenide.browser (Don't worry about selenide word, it's for future updates)
  • Choice Provider : Global Choice Parameter
    • Name : BROWSERS (Global choice parameter defined previously)
    • Default choice : As your convenience

Click another time on "Add parameter" and chose "Extensible Choice" again. Fill-in like this :

  • Name : XML_FILE
  • Choice Provider : Global Choice Parameter
    • Name : XML_FILES (Global choice parameter defined previously)
    • Default choice : testng.xml or as your convenience

Click another time on "Add parameter" and chose "Extensible Choice" again. Fill-in like this :

  • Name : server.ip
  • Choice Provider : Global Choice Parameter
    • Name : SERVERS (Global choice parameter defined previously)
    • Default choice : As your convenience

In the "build" section, select "Add a step" and chose "Invoke TestLink". Fill-in informations :

  • TestLink Version : select the one that has the name defined above
  • Test Project Name : the TestLink project name
  • Test Plan Name : the TestLink Plan name
  • Build name : the name of every test execution in testlink. You can use variables to have dynamic naming. Here :"build-$BUILD_NUMBER"
  • Custom field : the custome field we defined above : "java_class"

In the "Test execution subsection", for "single build test", select "Add action". Then fill-in :

  • Maven Target : "-P jenkins clean test"

In the "Result seeking strategy" select Add Strategy > TestNG class name. Fill-in :

  • Include pattern : "target/surefire-reports/testng-results.xml". It is the path of the xml containing the test results.
  • Key custom field : "java_class", same as above
  • Select "Attach TestNG xml"

Click on save at the bottom of the page

 

Create restore database and application job

Go back to Jenkins' home page. Click on "New Job", select "Free Style Project" and fill-in the name ("Gazelle-Deployment").

In the "build" section, select "Add a step" and choose "Execute shell script on remote host using ssh". Fill-in informations :

  • SSH Site : select the one that has been defined in Administrate Jenkins above
  • Command : 

cd /home/gazelle/unit_tests_selenium/
./restore_database_J7.sh

In the "Actions after build" section, select "Add a step" and choose "Trigger parametrized build on other projects".

Fiil-in informations : 

  • Project to build : Gazelle-TM-GuiTesting
  • Trigger when build is : Stable

Click on save at the bottom of the page

Step 3 : Execute the tests

The jenkins jobs are now completely configured and should work properly.

A sum up of an execution would be :

  • Run a new build of Gazelle-Deployment
  • He execute restore_database.sh to reset the database
  • After build Success, trigger build on Gazelle-TM-GuiTesting
  • Jenkins gather the sources
  • He starts the xvfb instance
  • He asks Testlink to know which test are automated
  • He build the project and start the maven target test
  • TestNG checks the testng.xml file in the java project
  • This file indicates him to run every class in the net.ihe.gazelle.test.test_case package
  • The run() method of all class are executed
  • An xml file cointaining test results is generated by TestNG
  • Jenkins parses this file and logs the result in TestLink. He knows which class corresponds to which test with the java_class field.