The Butler did it! Jenkins/JIRA tricks and tips

1. Automatic fixVersions in JIRA

We use Bitbucket and Jira. Pretty much awesome Atlassian products, rather common at companies.

Our process requires developers to move JIRA tickets from the To Do column to the Ready for demo column and eventually move them to the Done column. The development process is pretty much straightforward. When you start working on a story you can create a branch from your terminal. If you make sure that the branch name contains the issue ID, when you push that branch up to the remote repository, even without changes, JIRA can automatically move it to the In Progress column. When we open a pull-request, JIRA automatically moves the ticket to the Code Review column, and after merging it moves to the Exploratory column where another developer or a tester checks the functionality, confirms that the acceptance criteria are met and everything is fine. From there we move it to the Ready for demo column, and at the end of the iteration, we demo the changes. If they are accepted by the Product Owner they are moved to the done column and we can trigger a release build.

Now the only thing not automated in that process is that we do our releases using the Jira versions. When the release build runs successfully it tags the actual commit with that version registered in JIRA. The problem is that we are developers and we are lazy. We always forgot to add the last unreleased version to the stories that we worked on.

And in rare cases when we didn’t forget it, it happened that the unreleased version of in-progress tickets was released and we needed to change that manually to add the new fixVersion. This was unacceptable, and since I happen to be a developer who can do something about it, I took it upon myself to automate this task, using the Jira Steps Plugin and tweak our Jenkinsfile for pull requests. So let’s check how to do it.

At the top of the Jenkinsfile, we need to set up some methods and variables. We work on the front-end of our application, therefore, our release versions always start with FE, such as FE-2.3.12. So first we get the project versions, and then we use some groovy magic to get the only unreleased version which contains FE in its name. Finally, we apply the result to a variable.

def getFrontEndReleaseVersions() {
  def versions = jiraGetProjectVersions idOrKey: 'YOUR-PROJECT-KEY', site: 'YOUR-CONFIGURED-JIRA-SITE'
  return versions.data.findAll { it.released == false && it.name.contains('FE') }
}

FE_RELEASE_VERSIONS = getFrontEndReleaseVersions()

So the FE_RELEASE_VERSIONS variable now contains a list of version objects. Well, a list of exactly one version object. Then, when we start our build, before checkout we run a script in a try-catch block. The reason why it is inside a try-catch block is that if whenever the JIRA site goes down but the build runs, the build won’t fail.

pipeline {
  // ...
  stages {
    steps {
      script {
        try {
          def issuekey = (GIT_BRANCH.toUpperCase() =~ /JIRAPROJECTID-\d*/)[0]

          def editedIssue = [fields: [fixVersions: [FE_RELEASE_VERSIONS.first()]]]

          def response = jiraEditIssue idOrKey: issuekey, issue: editedIssue, site: 'YOUR-CONFIGURED-JIRA-SITE'

          echo response.successful.toString()
          echo response.data.toString()
        } catch (Exception e) {
          echo "Could not update fixVersions. Please do it manually"
        }
      }
    }

    // ...
  }
}

First we call .toUpperCase() on the GIT_BRANCH variable. This is because some of our developers like to create their branches from the terminal. They are usually too lazy to press shift when writing the JIRA issue number into the branch name. After that, we use the =~ operator to extract a substring using the following regex: /JIRAPROJECTID-\d*/. Use your own project’s letters and it should work for you. The operator returns an array of matches and we get the first one using bracket notation. After that, we define a variable with the fields that we want to change in our JIRA issue. And finally, we use the jiraEditIssue to update it in our JIRA.

2. Affected pipeline for PR builds

Another project that I work on is a rare delicacy. The back-end part consists of several JAVA SpringBoot microservices. With multiple front-ends. One back-office front-end, one customer-facing, with the chance of changing requirements there might be a third front-end as well in the future.

Our problem is, that because of our limited resources we cannot run the front-end and back-end builds in parallel. Now, the good thing is that everything is tested thoroughly. Which makes the back-end build run for 17 minutes and the front-end builds would run for another 14 minutes. That is simply too much for a pull-request build.

The first step for reducing this build time is that we implemented our two front-end applications using the NX workspace. That has reduced the front-end build time radically, however, installing node and npm, then running npm ci still takes a lot when there are only back-end changes. Similarly building three SpringBoot microservices with all their unit and functional tests take some time.

So, in order to deal with this problem we came up with the idea to make the front-end and back-end pull-request builds conditional. Yes, we are mimicking the affected builds of the NX workspace, but in our pull-request pipelines. After a small research, I have run into this blog post. (Thank you, Mr Becker!) The command that would allow us to condition our build steps is:

git diff --quiet HEAD origin/master -- web-ui || echo changed

This command consists of two parts. The first part before the OR operator returns with an exit code. If the output fails (nonzero exit-code) then the echo will run. We need two variables in our Jenkinsfile. FE_IS_DIRTY and BE_IS_DIRTY. We need to define them at the top of the Jenkinsfile, outside of our pipeline declaration.

def FE_IS_DIRTY
def BE_IS_DIRTY

pipeline { //...

Then, we need to assign the values to these empty variables. We need to run checkout scm first, so we have the actual changes that we can run our commands against. The variable assignment should run inside a script scope.

stage('Init build') {
  steps {
    // ...
    checkout scm
    script {
      FE_IS_DIRTY = sh (
        script: "git diff --quiet HEAD origin/master -- web-ui || echo changed",
        returnStdout: true
      )
      echo "Frontend isDirty: ${FE_IS_DIRTY}"
      BE_IS_DIRTY = sh (
        script: "git diff --quiet HEAD origin/master -- server || echo changed",
        returnStdout: true
      )
      echo "Backend isDirty: ${BE_IS_DIRTY}"
    }
  }
}

Now we just have to decorate our build steps with an appropriate when block:

stage('FE init') {
  when {
    expression { return FE_IS_DIRTY }
  }
  steps {
    sh './gradlew npmInstall'
    sh './gradlew generateGitHash'
  }
}
// ...
stage('BE Unit tests') {
  when {
    expression { return BE_IS_DIRTY  }
  }
  steps {
    echo 'Execute SpringBoot Application unit tets'
    sh './gradlew server:test'
  }
}

This way when there are only changes on the back-end the front-end tests won’t run. Similarly, when the changes only involve the front-end, the back-end build steps are not executed unnecessarily.

3. Notify DevOps

In our process, deploying to the dev and the test environment is automatic. New changes are immediately pushed to the dev environment whenever a master merge occurs. Every night at 21:00 the nightly build runs all the tests, builds the artefacts and publishes to the test (staging) environment. Deploying to production is a protected part of this process. Only the DevOps team can (and should) deploy to production.

The process is that whenever a release should be made, we run the release Jenkins job, which runs every test and builds the artefacts. The same version should be on the test environment, and the manual tests should run against it. After that, we need to create an issue for the DevOps team and send a notification e-mail to the DevOps team coordinator. As you have guessed already, the Jira Steps Plugin comes to the rescue. Creating looks like the following:

    stage('Notify DevOps') {
      steps {
        script {
          try {
            def devopsTicket = [fields: [
                project: [key: 'YOUR_PROJECT_KEY'],
                summary: "Release PROJECT ${params.RELEASE_VERSION} to production",
                description: "See Jenkins job: '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})"
              ]
            ]
            response = jiraNewIssue site: 'https://your.jira.com/' issue: devopsTicket

            echo response.successful.toString()
            echo response.data.toString()

            mail bcc: '', body: "<b>Release</b><br> Project: ${env.JOB_NAME} <br>Build Number: ${env.BUILD_NUMBER} <br> Build URL: ${env.BUILD_URL} <br> New version: ${params.RELEASE_VERSION}", cc: '', charset: 'UTF-8', from: 'jenkins@yourcompany.com', mimeType: 'text/html', replyTo: '', subject: "Deploy new release ${params.RELEASE_VERSION}", to: 'devops@devops.com';
          } catch (Exception e) {
            // if for some reason the JIRA ticket could not be created, send a notification to slack or an e-mail
            // or something that could benefit your process
          }
        }
      }
    }

We put the whole step into a try-catch block, so if something fails during issue creation, a fallback notification could be sent. We don’t want to fail the build on a technicality. Since this is the last step of the build if a flaky connection to the JIRA server fails the release build that is a false alarm. In the try block, we define the issue that needs to be created and populate it with the available data. Then we can also send out an e-mail to the DevOps team, so they can see that this task has priority.

Jira with its plugins can be a powerful tool to automate your builds and processes. I hope you enjoyed this article, and maybe these small tricks and tips could help you in the long run.