Automating your API Testing with Postman and Testrail On AWS Fargate

Let’s automate some API testing. The testing strategy uses AWS ECS Fargate, AWS CodeBuild and CodePipeline for continuous integration and delivery, Postman for testing, and TestRail for reporting. At the end of this article, I’ll give a Terraform example to make it easy to reproduce this test-automation solution on your AWS infrastructure. If you like to read code more than you like to read blog posts, you can find the full implementation here.

AWS ECS is a fully managed container orchestration service. We like to use AWS ECS Fargate, which is a serverless compute service for containers that removes the need to provision and manage servers.

To manage builds and continuously deliver testing automation, we use AWS CodePipeline and AWS CodeBuild.

• CodeBuild is a fully-managed continuous integration (CI) service that compiles source code.
• CodePipeline is a continuous delivery (CD) pipeline that automates builds with CodeBuild.

Postman is an API development tool that makes it easy to test and develop your APIs. Postman supports a command-line tool called Newman which allows you to run and test a Postman Collection directly from the command-line. It is built with extensibility in mind so that you can easily integrate it with your continuous integration servers and build systems.

Finally, in this example, I will demonstrate how to use TestRail for reporting results. TestRail is a web-based test case management tool. Geocene is not particularly attached to any particular reporting tool, but TestRail is used by many testers, developers, and team leads to manage, track, and organize software testing efforts.

What is behind Test-Automation?

The goal behind this project is to Run thousands of Postman tests automatically on a daily schedule and reports all the results in TestRail. This could be useful for a QA team to monitor milestones and test plan progress and to inform the backend team about any failing tests.

TestRail Configuration

I’ll be using Newman to report results to TestRail, but first we need to programmatically access TestRail. I’ll use the TestRail API, which allows us to integrate automated testing frameworks, submit test results, and automate various other aspects of TestRail reporting via HTTP requests.

Go to TestRail’s Administration > Site Settings > API and click on Enable API.

Then create a new api key:

On your Profile > My Settings > API KEYS tab and generate a new API key.

Linking Postman API Tests with TestRail Test Cases

I assume here that you already know how to use TestRail, and you have created some test cases.

The rules to associate your Postman tests with you TestRail test cases are simple: in Postman, just prefix test assertions with TestRail Case ID including the letter C.

This linking method can work in different ways, you can assign one TestRail test case to many Postman tests, and you can even assign many Test Rail test cases to one Postman test.

Newman Configuration

To execute Postman tests, we need to use the command-line Collection Runner Newman, but to report the results to TestRail we need also newman-reporter-testrail.

RUN npm install -g newman
RUN npm install -g newman-reporter-testrail

Once Newman and newman-reporter-testrail are installed, all we need is to export the following environment variables, here is a full example:

TESTRAIL_DOMAIN=example.testrail.io  # your own custom TestRail subdomain
TESTRAIL_USERNAME=john.doe@example.com # username, usually your email address.
TESTRAIL_APIKEY=your_api_key # This is the API key you have previously created.
TESTRAIL_PROJECTID=1
TESTRAIL_RUNID=1 
TESTRAIL_TITLE="My Postman API Tests" 
newman run collection.json --reporters cli,testrail

Test-Automation

At this point, we know how to create Test Cases, TestRuns, and how to use Newman to execute tests and report results in TestRail.

But Imagine you have a big Postman collection with 20 folders, and each folder contains hundreds if not thousands of tests, and you have a QA team that works together on the same collection from their local machine. Geocene has been in this situation several times! Here are are some blockers and questions that we have faced in the past:

• How to deal with speed and performance?
• The QA team has to spend several hours to run the whole Postman collection.
• What about TestRail? Can TestRail’s API handle thousands of test results?
• newman run collection.json --reporters cli,testrail requires using a JSON export of the collection that might change and be different from QA team member to team member.
• sharing a public link to the collection might be an option, but this has some disadvantages
  • each time the collection changes, you have to manually sync it.
  • sharing a public link has securities issues.

The details matter, and your implementation might depend on your project, QA team practices, how Postman tests are written by the QA team, and the number of test scenarios you are trying to report with TestRail.

• We have noticed that TestRail can crash unexpectedly when the test results get bigger (tens of thousands of test cases).
• If the prefix rules described earlier are malformed in one of your assertions, TestRail’s API may crash. As the Postman collection gets bigger, expect to encounter more issues.
• From a security perspective, the JSON export of the Postman collection could contain an internal token to your backend, so using a collection.json file or even a public link in your test-automation repo is risky.

To address almost all of these questions, you can use a Testrail/Postman structure that we developed that uses an Amazon ECS task that runs inside of VPC. For additional security I’ll demonstrate how to use only API Keys for both Postman and Testrail.

• To execute Newman commands using Postman’s API.
• To retrieve all details about each TestRun using TestRail’s API

Postman and TestRail Structure

To solve most of the issues we have encountered with integrating TestRail, we created a TestRun for each subfolder of our Postman collection. Let’s call these “modules.” Let’s say that each module corresponds to a feature of our backend. So instead of running Newman against the whole collection and taking the risk of failure, we run Newman on each module as a separate TestRun. This structure dramatically reduces TestRail API failure rates and increases the performance of testing automation by allowing us to parallelize by the number of subfolders/modules.

The final script looks like this :

run_newman.sh

#!/bin/bash


COLLECTION_ID=$(curl --location --request GET "https://api.getpostman.com/collections?apikey=${POSTMAN_API_KEY}" | \
jq --arg COLLECTION_NAME "$COLLECTION_NAME" '.collections[] | select(.name==$COLLECTION_NAME) | .uid')

ENVIRONMENT_ID=$(curl --location --request GET "https://api.getpostman.com/environments?apikey=${POSTMAN_API_KEY}" | \
jq --arg ENVIRONMENT_NAME "$ENVIRONMENT_NAME" '.environments[] | select(.name==$ENVIRONMENT_NAME) | .uid')

COLLECTION_ID=${COLLECTION_ID%\"}
COLLECTION_ID="${COLLECTION_ID#\"}"
ENVIRONMENT_ID=${ENVIRONMENT_ID%\"}
ENVIRONMENT_ID="${ENVIRONMENT_ID#\"}"

curl --location --request GET "https://api.getpostman.com/collections/$COLLECTION_ID?apikey=${POSTMAN_API_KEY}" \
>> collection.json

curl --location --request GET "https://api.getpostman.com/environments/$ENVIRONMENT_ID?apikey=${POSTMAN_API_KEY}" \
>> postman_environment_variables.json

#report Newman results to Testrail by chunks (Newman runs tests on folders and updates the existing test runs)

curl -H "Content-Type: application/json" -u "${TESTRAIL_USERNAME}:${TESTRAIL_PASSWORD}" \
 "https://${TESTRAIL_DOMAIN}/index.php?/api/v2/get_runs/${TESTRAIL_PROJECTID}" >> test_runs.json

jq -c '.[]' test_runs.json | while read i; do

  TEST_RUN_NAME=$(echo $i | jq '.name')
  TEST_RUN_ID=$(echo $i | jq '.id')

  echo "start processing ${TEST_RUN_NAME}"

  newman run collection.json \
  --folder "$TEST_RUN_NAME" -e postman_environment_variables.json \
  --reporters cli,testrail \
  --timeout-request 30000 \
  --timeout-script 30000

  echo "processing ${TEST_RUN_NAME} finished"


done

Pulling Everything Together with AWS ECS Fargate

Now we are able to execute thousands of tests created with Postman, and we have a full visibility of results in TestRail. Now let’s keep all the reporting fresh and relevant by scheduling the testing automation on AWS.

Let’s say that we want our project, test-automation, to start an ECS task that runs Newman commands in a Docker container and execute tests created with Postman. Then we want to report results in TestRail using newman-reporter-testrail dependency. We use CodeBuild as well to build our Docker image and automate the release each time the test automation repository is updated.

First, we need to Dockerize test-automation. Postman has its own official Docker Image on Dockerhub.

Our Dockerfile looks like this :

Dockefile

FROM node:12

RUN npm install -g newman
RUN npm install -g newman-reporter-testrail

# install jq to parse json within bash scripts
RUN curl -o /usr/local/bin/jq http://stedolan.github.io/jq/download/linux64/jq && \
  chmod +x /usr/local/bin/jq

COPY * /
RUN chmod +x /run_newman.sh

ENTRYPOINT ["/run_newman.sh"]

We use a basic buildspec.yml file that contains the commands that CodeBuild runs to build the Docker image.

buildspec.yml

version: 0.2

phases:
  pre_build:
    commands:
      - $(aws ecr get-login --no-include-email)
  build:
    commands:
      - docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION .
  post_build:
    commands:
      # push
      - docker tag $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:latest

Scheduling an Amazon ECS Task to Run the test-automation Container

AWS Fargate supports the ability to run tasks on a schedule and in response to CloudWatch Events. This makes it easier to launch and stop container services that you need to run only at certain times.

All we need to do now is to run our repo on a scheduled task using AWS Fargate and Cloudwatch.

As a reminder, the structure of test-automation looks like this :

test-automation
  |- buildspec.yml
  |- Dockerfile
  |- run_newman.sh

Here is a minimal example of how to set up in Terraform an ECS scheduled task using Cloudwatch events. This example is just for demonstration, so I’m ignoring details like CodePipeline, CodeBuild, ECR, IAM, and other infrastructure code. These these details are out of the scope of this post, but you can find all the infrastructure in this repository to reproduce the same solution on your AWS account.

resource "aws_ecs_task_definition" "test-automation-scheduler" {
  family                   = "test-automation"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 1024
  memory                   = 8192
  execution_role_arn = aws_iam_role.ecs-task-execution-role.arn
  task_role_arn      = aws_iam_role.ecs-test-automation-task-role.arn
  container_definitions = <<DEFINITION
[
  {
    "image": "${aws_ecr_repository.test-automation.repository_url}:latest",
    "name": "test-automation-scheduler",
    "networkMode": "awsvpc",
    "environment": [
        {
          "name": "TESTRAIL_DOMAIN",
          "value": "exampletestrail.testrail.io"
        },
        {
          "name": "TESTRAIL_USERNAME",
          "value": "name@example.com"
        },
        {
          "name": "TESTRAIL_PASSWORD",
          "value": "secret"
        },
        {
          "name": "TESTRAIL_APIKEY",
          "value": "TestRail Api Key"
        },
        {
          "name": "TESTRAIL_PROJECTID",
          "value": "1"
        },
        {
          "name": "NODE_OPTIONS",
          "value": "--max-old-space-size=8192"
        },
        {
          "name": "TESTRAIL_INCLUDEALL",
          "value": "false"
        },
        {
          "name": "POSTMAN_API_KEY",
          "value": "Postman Api Key"
        },
        {
          "name": "COLLECTION_NAME",
          "value": "Collection Name"
        },
        {
          "name": "ENVIRONMENT_NAME",
          "value": "Postman Environment name"
        }        
    ],
    "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
            "awslogs-group": "${aws_cloudwatch_log_group.test-automation-scheduler.name}",
            "awslogs-region": "${var.AWS_REGION}",
            "awslogs-stream-prefix": "ecs"
        }
    }

  }
]
DEFINITION
}

## Cloudwatch event role

resource "aws_iam_role" "scheduled_task_cloudwatch" {
  name               = "test-automation-st-cloudwatch-role"
  assume_role_policy = file("${path.module}/policies/scheduled-task-cloudwatch-assume-role-policy.json")
}

data "template_file" "scheduled_task_cloudwatch_policy" {
  template = "${file("${path.module}/policies/scheduled-task-cloudwatch-policy.json")}"

  vars = {
    task_execution_role_arn = aws_iam_role.scheduled_task_cloudwatch.arn
  }
}

resource "aws_iam_role_policy" "scheduled_task_cloudwatch_policy" {
  name   = "test-automation-st-cloudwatch-policy"
  role   = aws_iam_role.scheduled_task_cloudwatch.id
  policy = data.template_file.scheduled_task_cloudwatch_policy.rendered
}

resource "aws_cloudwatch_log_group" "test-automation-scheduler" {
  name = "/ecs/test-automation"
}

resource "aws_cloudwatch_event_rule" "test-automation-rule" {
  name        = "test-automation"
  description = "Run every day at 5am"
  schedule_expression = "cron(0 5 * * ? *)"
}

resource "aws_ecs_cluster" "test-automation" {
  name = "test-automation"
}

resource "aws_cloudwatch_event_target" "test-automation-target" {
  target_id = "test-automation-scheduler"
  arn  =      aws_ecs_cluster.test-automation.arn
  rule      = aws_cloudwatch_event_rule.test-automation-rule.name
  role_arn  = aws_iam_role.scheduled_task_cloudwatch.arn

  ecs_target {
    task_count          = 1
    task_definition_arn = aws_ecs_task_definition.test-automation-scheduler.arn
    launch_type         = "FARGATE"

 network_configuration {
    subnets          = slice(module.vpc.public_subnets, 1, 2)
    security_groups  = [aws_security_group.ecs-demo.id]
    assign_public_ip = true
  }

  }
}

# security group
resource "aws_security_group" "ecs-demo" {
  name        = "ECS demo"
  vpc_id      = module.vpc.vpc_id
  description = "ECS demo"

  ingress {
    from_port   = 3000
    to_port     = 3000
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port   = 0
    protocol  = "-1"
    cidr_blocks = [
      "0.0.0.0/0"
    ]
  }
}

Conclusion

So, we have looked at how to set up, develop, and Dockerize automated API testing with Postman and TestRail, then deploy it on AWS with Terraform. Feel free to check out the full implementation in Geocene’s test-automation demonstration repo. I hope you enjoyed this and found some valuable insights. If you have more questions, need help automating your own testing system, or just want to drop me a line, contact Geocene here.