Unit testing with Scala: Libraries landscape

If you are confused with a variety of test libraries in Scala ecosystem like me, then this is article for you.

In this article, I’d like to do a quick tour among Scala frameworks and libs and show capability of each of them on one same example to be able to compare approaches and how final test will look at very high level.

IMPORTANT NOTES:

• This article doesn’t cover tests for particular technologies like akka, Play Framework, cats-effect - only general-purpose unit testing technologies - but they will be mentioned in the end in a short wrap-up table.
• Perhaps not all features of every testing library will be covered, but rather main functionality, like tests structure, assertions and test results matches.
• ScalaCheck is not considered a separate library for testing in the scope of this post, because the majority of libs have integration with it.

System under the test example

In order to unify further comparison somehow, let’s consider next example of system under test (SUT) which we are going to use.

This is a simple abstract authentication service, which decides whether some User has access to perform Action over some ResourceId. The domain model is next: User can be assigned to multiple Roles. Each role has associated ResourceId and max Action it can perform. Ordering goes next: Delete > Update > Read.

Since we need to access external resources result will be wrapped in Future.

Apart from that, let’s introduce a bug in our system to show how every library shows error result in the test report.

Let’s consider test setup like:

Let’s consider next 5 abstract scenarios we want to test:

1. For given userId, which does not exist in UserStorage it is expected to receive Future.successfull(Left(NoUserFound)) result
2. For given userId, exists user in UserStorage, but for which one of rolesId does not exits in RoleStorage it is expected to receive Future.successfull(Left(NoRoleFound)) result
3. For given userId, exists a role which does not have specified resourceId it is expected to receive Future.successfull(Right(false))
4. For given userId, exists a role which has specified resourceId, but with lower level action it is expected to receive Future.successfull(Right(false))
5. For given userId, exists a role which permits to execute action it is expected to receive Future.successfull(Right(true))

ScalaTest

Perhaps one of the most popular libs in Scala world for unit testing, which provide rich set of styles and approaches For the past years, I’ve meat AnyWordSpec perhaps the most among other, so let's look how this unit test will look like. 3.2.7 version was used for next example:

sbt test produces next result:

Specs2

Yet another popular Scala testing solution. Like ScalaTest it also provides different styles of testing For sake of this example let's use Acceptance specification, because Unit specification looks pretty similar to other ScalaTest styles: 4.10.0 version was used for next example:

sbt test produces next result:

Munit

New testing library written by Scalameta. It’s reuse JUnit and tests structure looks pretty similar to it by the structure:

Unlike ScalaTest and Specs2, Munit provides only assertions functionality, instead of extensive result matching DSL. Also, as you notice — supports Async style of testing so instead await for the Future result - you await for test result.

sbt test produces next result:

uTest

GutHub. One more minimalistic testing library, but which provides rich functionality.

sbt test produces next result:

Minitest

Yet another minimalistic testing library written by Monix creator.

As you may see unlike ScalaTest and Specs2 it does not provide await-like functionality, but like Munit and uTest alternatevly it provides async test style.

sbt test produces next result:

Conclusion

As you may see among usual and pretty well know test frameworks like ScalaTest and Specs2, there are other solutions which might fit for your needs better, especially if you want to have something small and simple.

Short Comparison table

In the table below you can find a briefly described topic, which was not covered in the main post:

Your feedback is more than welcome and very appreciated, in case if you noticed that something important missed, please let me know.

Thank you very much for your time and attention!