Automate Your Tests Effortlessly with Allure TestOps

Why Use Allure TestOps

The world of automation testing is constantly evolving, increasing the need for tools specifically tailored to manage its unique demands. Working with my client, I’m using Selenium with Python for our automation tests. Until recently, our CI/CD tool – GitLab CI – was our sole tool for running and visualizing these tests. However, I found that GitLab CI lacked some essential features designed for the specific needs of automation testing.

While it’s pretty simple to integrate test jobs into the CI and store and access the test results, that is not enough. Our automation tests lacked some very important abilities:

1. A main place to configure tests, trigger them and see the logs in real time.
2. Aggregate, categorize, and filter test history.
3. Advanced visualizations and statistics about test results.
4. Easy way to rerun only failed tests.  

GitLab CI does offer the option to create artifacts for jobs and store them inside the platform, but there is no concept of ‘history.’ I can’t see in one place all the runs of a specific test. I can only go one by one into each artifact and look. For visualization, Gitlab can display a report if it’s in a Junit report format, but it’s a very simple display. Other limits in using only GitLab CI are having no way to rerun only failed tests, create visual separation for different test categorizations, or create simple click-and-run test templates.  All in all, I felt that GitLab CI, while a great tool for CI/CD pipelines, is not a tool for automation testing. 

That set me to search for the right tool to help us with all our pain points. That’s when I came upon AllureTestOps, a test management platform that helps automate testing processes, manage test cases, and analyse test results. It also integrates with CI/CD tools such as GilabCI and many more, so it can still easily run as part of the continuous deployment process.

How hard is it to integrate existing tests with Allure TestOps

From my own experience with GitLab CI, integrating an existing CI/CD pipeline with Allure TestOps is a pretty simple process. Allure has a two-week free trial, so you can create an account in minutes and play around with the different features Allure offers.

Also, Allure TesOps has documentation with detailed guides on how to integrate Allure into different CI tools. That was my starting point when I went to configure Gitlab, Pytests and Allure, but I felt that the documentation lacked some very important points that I had to figure out myself. In the next section, I’ll guide you step by step so you can easily set everything up. 

Note: As I said, our tests are written in Python, and our CI/CD runs in GitLab CI, so the step-by-step guide refers to integrating AllureTestOps with these two. However, the concept is the same for different languages and CI/CD tools, so you will benefit from reading on even if your environment is not exactly the same. 

Understanding the concept

Two tools are required to set up AllureTestOps: Allure Reports and allurectl.

Allure Reports:

Allure Reports is a framework for generating test execution reports in a format that can integrate with Allure TesOps. Allure Reports supports many testing frameworks, you can check out here the full list.

For Python, there is the allure-pytest plugin that enables the generation of Allure Reports. This plugin does not require you to change any code in your tests (though it does offer additional features that may be useful for writing better tests), only a simple flag added to pytest command:

python -m pytest --alluredir <allure-results-dir>.  

allurectl:

A CLI tool provided by Allure TestOps to manage integration within a CI/CD pipeline. It’s used to upload the test results to the Allure TestOps server, and manage other communication between AllureTestOps and the CI job. You can either use the CLI to upload the test results files once the tests finish, or you can wrap your test execution command with an allurectl watch command and stream the test results as they run.

How to integrate Allure TestOps with GitLab CI for Python tests

Prerequisites:

1. An account in AllureTestops (you have a 2-week trial with no credit card needed)
2. Your automation test code (Pytest)
3. A working CI job that runs your Pytest scripts (GitLab CI)

Steps:

1. Create a ‘Project’ in Allure TestOps

2. Create an access token in gitlab:

1. In the Gitlab repository, click on settings -> Access Tokens.
2. Click ‘add new token’
3. Fill in the fields:
   Token name –  a name to help you recognize the token, e.g. “AllureTestopsToken”

Expiration date –  any date within a year from the current date. After this date, the integration will stop working, and you will need to create a new token to continue using the integration.

4. Select a role –  “Maintainer”
5. Select scopes – “api” and “read_api”
6. Click ‘Create project access token”
7. Copy and save the access token for the next step.

3. Setup gitlab integration in AllureTestOps:

1. Log into Allure Testops as admin.
2. Click on the user Avatar and go to Administration -> Integrations
3. Click Add integration
4. Search for Gitlab and select it
5. In the popup window, fill in the fields:

Name – e.g. “MyGitLabRepo”

Endpoint – the URL of the GitLab server. If it’s not self-hosted, just fill in https://gitlab.com/.

6. Click Add integration
7. Go back to the projects page and select your project
8. Go to Settings -> Integration
9. Under the Available integrations, find the GitLab integration and click Add integration next to it.

10. In the pop-up window that appears, add the fields:

Project ID – The ID of your gitlab project. It can be found in the Gitlab UI in Settings -> General -> Naming, topics, avatar.

Token – Paste the access token you created in step 2.

h.  Click on Test connection to test connectivity, and then click Add integration.

4. Add environment variables in Gitlab 

In your Gitlab project, go to Settings -> CI/CD -> Variables and add the following env vars:

5. Add in CI/CD job:

# Install allure-pytest
pip install allure-pytest

# Install allurectl
	Wget https://github.com/allure-framework/allurectl/releases/latest/download/allurectl_linux_amd64 -O ./allurectl
    	chmod +x ./allurectl

	# Send test results in real time as the tests are being executed
./allurectl watch python pytest --alluredir=build/allure-results

This basic setup should allow you to run the tests as usual and view the test results in the AllureTestops UI. 

The next level of integration is triggering test runs from AllureTestops:

6. Triggering CI/CD run from AllureTestops:

1. In AllureTestops, go to your project page.
2. In the left menu, go to Jobs. If you successfully run the tests from GitLab CI once, you should see a new job that was automatically added. You will see a red X sign under the server icon.  
3. Click the job’s three dots menu, then select Configure.
4. In the dialog that appears, edit the fields:

Name – a name to help you recognize the job.

Build server – the name of the integration that you added in step 2.e.

Job can be used to run tests — if checked, users will be able to trigger this job from Allure Testops.

Parameters — parameters that should be passed to GitLab via environment variables, see Environment.

e. Click on Submit, and you will see the red X is replaced by a green V

Conclusion

AllureTestOps is a powerful tool that can add a lot of value and is very easy to set up and configure with an existing CI/CD pipeline. There is no need for dramatic changes to the test code; everything is simple and straightforward. Of course, from here, you can dive deeper, use more capabilities of allure-pytest, and enrich your test result data.  

I hope this simple guide helped you start out with AllureTesOps, and that it will help you to quickly and efficiently test your applications and speed up your development cycle.