Scala implicits

Versions: Scala 2.12.1

Scala implicits have many drawbacks but in some cases their use is justified. They remain though one of the advanced concepts for a lot of newcomers and it's worth explaining them a little bit more in details.

Data Engineering Design Patterns

Looking for a book that defines and solves most common data engineering problems? I'm currently writing one on that topic and the first chapters are already available in 👉 Early Release on the O'Reilly platform

I also help solve your data engineering problems 👉 contact@waitingforcode.com 📩

Implicits will be explained in this post in 6 short sections. The first one defines them from the global point of view by showing the use cases and resolution rules. It also contains a short example. The next 5 parts cover more in details each of the use cases shortly mentioned in the beginning. Please notice that one of the uses isn't detailed here because it was already explained in the post about context bounds in Scala.

Implicits definition

Simply speaking Scala implicits remove the explicitness requirement about passing the arguments to some functions and invoking some other functions. For instance, if somewhere in the code we've defined an implicit conversion from String to Seq[String] called convertStringToSeqString(text: String), we can simply write val textFromSeq: Seq[String] = "test" instead of ... = convertStringToSeqString("test").

The implicits resolution occurs at compilation time so it doesn't penalize the application's execution. However it may penalize the programmers because often it's harder to work with the code based on the implicits. Thus, they should be used cautiously and according to the context. Among the legitimate use cases of implicits we can distinguish:

less redundancy - sometimes we may need to use the same object in multiple places. One of great examples of such use is implicit scala.concurrent.ExecutionContext widely present in Future's methods, as for instance onComplete or andThen. For that specific case creating an implicit ExecutionContext is legitimate since most of time one instance's pool can be freely shared by different Future objects and having it to declare only once is less error-prone.
type safety - implicits increase it by providing constraint evidence.
extending 3rd part libraries - implicits are one of the fundamental components of Pimp My Library pattern explained in the next part of the post. Thanks to them we can easily extend the features of these libraries by defining, for instance, implicit conversion methods.

dependency injection - the implicits can also be useful for dependency injection:

    class UserService()
    class OrderService()
    implicit val userService = new UserService()
    implicit val orderService = new OrderService()

    class Application()(implicit val userService: UserService, val orderService: OrderService) {
    }
    val prodApplication = new Application()

    val mockedUserService = new UserService() {}
    val mockedOrderService = new OrderService() {}

    val testApplication = new Application()(mockedUserService, mockedOrderService)

    prodApplication.userService shouldBe theSameInstanceAs(userService)
    prodApplication.orderService shouldBe theSameInstanceAs(orderService)
    testApplication.userService shouldBe theSameInstanceAs(mockedUserService)
    testApplication.orderService shouldBe theSameInstanceAs(mockedOrderService)

In the other side among the drawbacks of implicits we can find:

difficulty to debug - even though modern IDEs support implicits (e.g. Ctrl + Shift + Q on IntelliJ to show a list of implicit candidates), it's still less obvious to work than with explicit code.
Understanding difficulties are pretty well visible in the implicit conversions between similar datatypes as lists or streams. Totally transparent conversion hides the specificity of converted types and the user can not be aware of them.
don't fit everywhere - implicits can't be used everywhere. The post Scala Patterns To Avoid: Implicit Arguments With Default Values shows how it's bad to use implicits with default values as method arguments

mutable state - the point is related to the previous one about the context. Implicits, if must be used, should be immutable. A simple illustration shows why:

      implicit var mutableImplicit = "."
      def writeSentence(text: String)(implicit termination: String) = s"${text}${termination}"
    
      // User#1 writes the sentence
      val sentenceUser1 = writeSentence("Test")
      sentenceUser1 shouldEqual "Test."
    
      // User#2 changes the value
      mutableImplicit = "!"
      val sentenceUser2 = writeSentence("Test")
      sentenceUser2 shouldEqual "Test!"
    
      // Here User#1 will be surprised that within the same session he sess 2 different outputs
      val nextSentenceUser1 = writeSentence("Other test")
      nextSentenceUser1 shouldEqual "Other test!"

Resolution rule

Implicits are resolved according to some specific rules. First, the compiler looks for matching objects or methods in the current scope. If it doesn't find any, it verifies if the implicits are defined in one of imported items. If the implicit is still not found, the compiler checks the companion object of given type. Finally it looks for the definitions in other scopes: of an argument's type, of type arguments or in outer object for nested types. Obviously if a definition is not found, the code can't be compiled.

Some of these rules are illustrated in the following tests:

class ImplicitResolutionTest extends FlatSpec with Matchers {

  behavior of "implicits resolution rules"

  it should "chose the implicit from the same scope" in {
    implicit val sentenceTerminationSign = "!"
    def testMethod(): String = {
      implicit val sentenceTerminationSign = "."

      def writeSentence(sentence: String)(implicit termination: String): String = s"${sentence}${termination}"
      writeSentence("Test")
    }

    val sentence = testMethod()

    sentence shouldEqual "Test."
  }

  it should "get the implicit from the explicit import statement" in {
    def writeSentence(sentence: String)(implicit termination: String): String = s"${sentence}${termination}"
    object ImplicitsContainer {
      implicit val sentenceTerminationSign = "!"
    }
    object WildcardImplicitContainer {
      implicit val sentenceTerminationSign = "?"
    }
    // Explicit imports are preferred over the wildcard ones
    import WildcardImplicitContainer._
    import ImplicitsContainer.sentenceTerminationSign
    // If we put the wildcard import after the explicit one, the compiler will fail with this error:
    // Error:(37, 33) could not find implicit value for parameter termination: String
    //    val sentence = writeSentence("Test")
    val sentence = writeSentence("Test")

    sentence shouldEqual "Test!"
  }

  it should "find implicit in wildcard import" in {
    def writeSentence(sentence: String)(implicit termination: String): String = s"${sentence}${termination}"
    object WildcardImplicitContainer {
      implicit val sentenceTerminationSing = "..."
    }
    import WildcardImplicitContainer._
    val sentence = writeSentence("Test")

    sentence shouldEqual "Test..."
  }

  it should "find the implicit in the companion object" in {
    object Sentence {
      implicit def convertToLetters(sentence: Sentence): Seq[String] = sentence.content.split("")
    }
    class Sentence(val content: String) {}
    val sentence = new Sentence("Test")

    // Normally the Sentence doesn't have a mkString method. But with the implicit conversion to a list
    // we can easily access it
    val lettersFromSentence = sentence.mkString("::")

    lettersFromSentence shouldEqual "T::e::s::t"
  }

  it should "find the implicit in the children's companion object" in {
    // The implicit method will be inherited by any Sentence sublcass if their companion objects don't implement the conversion
    // themselves
    object Sentence {
      implicit def convertToLetters(sentence: Sentence): Seq[String] = sentence.content.split("")
    }
    class Sentence(val content: String) {}

    object Sentence2 {
      implicit def convertToLetters(sentence: Sentence2): Seq[String] = sentence.content.split(",")
    }
    class Sentence2(val content2: String) extends Sentence(content2)

    val sentence = new Sentence2("Te,st")

    val lettersFromSentence = sentence.mkString("<->")

    lettersFromSentence shouldEqual "Te<->st"
  }

  it should "find the implicit in the scope of the argument type" in {
    class Sentence(val content: String) {
      def +(otherSentence: Sentence): Sentence = new Sentence(s"${content}${otherSentence.content}")
    }
    object Sentence {
      implicit def convertToSentenceFromString(text: String) = new Sentence(text)
    }
    // If we define a new implicit converter here:
    // implicit def convertToSentenceFromString(text: String) = new Sentence("")
    // It'll be used instead the one defined in the companion object and it's normal regarding the resolution rules
    val sentence1 = new Sentence("Test")
    
    val concatenatedSentence = sentence1 + " and another test"

    concatenatedSentence.content shouldEqual "Test and another test"
  }

}

Pimp My Library pattern

After providing such general point of view for implicits we'll focus on different use cases more in details. The first interesting use of implicits is Pimpy My Library pattern. This facility helps to extend already existent types with new methods or fields, as for instance in the following snippet:

package implicits

import org.scalatest.{FlatSpec, Matchers}

class PimpMyLibraryPatternTest extends FlatSpec with Matchers {

  behavior of "pimpy my library pattern"

  it should "add new behavior to already existent class" in {
    // Let's imagine we use a HttpContext object from some public library. Meantime HTTP protocol
    // evolved and added some new features. We can extend it with the help of Pimp My Library pattern
    case class HttpContext(ip: String, browser: String)
    case class EnrichedHttpContext(basicHttpContext: HttpContext, version: String) {
      def checkIfVersionIsSet: Boolean = version.nonEmpty
      def getFlatVersion: String = version.replace(".", "")
    }
    var createdInstances = 0
    implicit def convertToEnrichedHttpContext(basicHttpContext: HttpContext): EnrichedHttpContext = {
      createdInstances += 1
      EnrichedHttpContext(basicHttpContext, "1.0")
    }

    val httpContext = HttpContext("1.2.3.4.5", "unknown")

    val isVersionSet = httpContext.checkIfVersionIsSet
    val flattenVersion = httpContext.getFlatVersion

    isVersionSet shouldBe true
    flattenVersion shouldEqual "10"
    // As you can see, the number of created instances is 2 even though we operate on a single one object
    // It's one of the drawbacks
    createdInstances shouldEqual 2
  }

  it should "apply to already basic types" in {
    val number = 1

    // Scala's Int doesn't have .until(...) method but RichInt that is its extended
    // ("pimped") version does
    val rangeFromRichInt = number.until(5)

    rangeFromRichInt should contain allOf(1, 2, 3, 4)
  }
}

As you can see, the pattern implicitly converts our basic object to its extended version. It's achieved pretty easily with a simple conversion method. However the pattern is not a silver bullet since it has some important drawbacks. As you can see in the first test case, it creates a new instance of enriched object for every operation. It can be problematic for time-sensitive applications, especially if such object resists somehow to the GC. Of course, we could avoid that by explicitly converting the basic object once that is though against the idea of this pattern.

Implicit conversion

Thanks to Pimp My Library pattern we could see the specific use of another implicit's case called implicit conversion. As shown in the example, implicit conversion from type A to type B is done by an implicit method or an implicit value with the function of type A => B. You can see that in following 2 test cases:

class ImplicitConversionTest extends FlatSpec with Matchers {

  behavior of "implicit conversion"

  case class Person(name: String, city: String)

  it should "apply with conversion method" in {
    case class RichPersonFromMethod(name: String, city: String) {
      def getAverageSalaryPerCity: Double = 50.0d
    }
    implicit def convertPersonToRichPerson(person: Person): RichPersonFromMethod =
      RichPersonFromMethod(person.name, person.city)

    val person = Person("test", "test_city")
    val averageSalary = person.getAverageSalaryPerCity

    averageSalary shouldEqual 50.0d
  }

  it should "apply with implicit value" in {
    case class RichPersonFromValue(name: String, city: String) {
      def getAverageSalaryPerCity: Double = 50.0d
    }
    implicit val personToRichPersonConverter: (Person) => RichPersonFromValue =
      (person) => RichPersonFromValue(person.name, person.city)

    val person = Person("test", "test_city")
    val averageSalary = person.getAverageSalaryPerCity

    averageSalary shouldEqual 50.0d
  }

}

The tests pass without problems but the code compiles with a warning about the implicit conversions:

Warning:(15, 18) implicit conversion method convertPersonToRichPerson should be enabled
by making the implicit value scala.language.implicitConversions visible.
This can be achieved by adding the import clause 'import scala.language.implicitConversions'
or by setting the compiler option -language:implicitConversions.
See the Scaladoc for value scala.language.implicitConversions for a discussion
why the feature should be explicitly enabled.
    implicit def convertPersonToRichPerson(person: Person): RichPersonFromMethod =

By showing that Scala prevents us against an over-use of the implicit conversion. They can largely complicate the code understanding. Thankfully the compiler prevents us against an important problem where one implicit conversion already exists for given type. Then if we want to add a new conversion, for instance for RichInt, the code won't compile:

    implicit def convertIntToRichInt(nr: Int): RichInt = {
      println("converting here")
      new RichInt(nr)
    }

    val nrInt: Int = 3
    val nr: RichInt = nrInt

Because of:

Error:(46, 23) type mismatch;
 found   : Int
 required: scala.runtime.RichInt
Note that implicit conversions are not applicable because they are ambiguous:
 both method intWrapper in class LowPriorityImplicits of type (x: Int)scala.runtime.RichInt
 and method convertIntToRichInt of type (nr: Int)scala.runtime.RichInt
 are possible conversion functions from Int to scala.runtime.RichInt
    val nr: RichInt = nrInt

Implicit classes

Another kind of implicits use are implicit classes. A class can be prefixed with implicit keyword too. If so, it can be implicitly constructed from the type defined in its primary constructor:

import org.scalatest.{FlatSpec, Matchers}

class ImplicitClassesTest extends FlatSpec with Matchers {

  behavior of "implicit classes"

  it should "apply to an implicit class with constructor having converted type" in {
    case class Person(name: String)
    var createdInstances = 0
    implicit class RichPerson(person: Person) {
      createdInstances += 1
      def hasName: Boolean = person.name.nonEmpty
    }

    val person = Person("test")
    val personHasName = person.hasName
    val personHasNameOtherTest = person.hasName

    personHasName shouldBe true
    personHasNameOtherTest shouldBe true
    createdInstances shouldEqual 2
  }

}

However this apparently flexible behavior brings some constraints:

Implicit class can have only 1 parameter. Otherwise the compilation error is produced:

case class Person(name: String)
    var createdInstances = 0
    implicit class RichPerson(person: Person, city: String) {
      createdInstances += 1
      def hasName: Boolean = person.name.nonEmpty
    }

    val person = Person("test")
    val personHasName = person.hasName
    val personHasNameOtherTest = person.hasName

For error:

  Error:(29, 20) implicit classes must accept exactly one primary constructor parameter
    implicit class RichPerson(person: Person, city: String) {

The implicit class can't be a top-level class. The compilation error is also produced:
```
Error:(47, 16) `implicit' modifier cannot be used for top-level objects
```

Implicit class name can't shadow the name of any object or value present in the scope. The following code won't compile because of that:

    case class Person(name: String)

    val RichPerson = Person("test")

    implicit class RichPerson(person: Person)

It will fail with this error:

    Error:(49, 20) RichPerson is already defined as value RichPerson
    implicit class RichPerson(person: Person)

Implicit parameters

Implicit parameters are other type of implicits supported in Scala. When a parameter is defined as implicit, the compiler will first look if some of its implicit definition exists somewhere in the scope. If it's not the case, we can still define the parameter explicitly. Only when none of them happen the compiler will throw a compilation error:

import org.scalatest.{FlatSpec, Matchers}

class ImplicitParametersTest extends FlatSpec with Matchers {

  behavior of "implicit parameters"

  case class Configuration(key: String)

  it should "find the implicit parameter in the same scope" in {
    implicit val prodConfiguration = Configuration("prod")

    def configureService(implicit configuration: Configuration): String = {
      s"configured ${configuration.key}"
    }

    val configurationMessage = configureService

    configurationMessage shouldEqual "configured prod"
  }

  it should "not fail when missing implicit parameter is replaced with explicit parameter" in {
    def configureService(implicit configuration: Configuration): String = {
      s"configured ${configuration.key}"
    }

    val configurationMessage = configureService(Configuration("dev"))

    configurationMessage shouldEqual "configured dev"
  }

  it should "use 1 implicit parameter for all methods" in {
    def configureService(method: Configuration => String): String = {
      val configuration = Configuration("prod")
      method(configuration)
    }
    def configurationAction1(implicit configuration: Configuration): String = "action1"
    def configurationAction2(implicit configuration: Configuration): String = "action2"
    val configuredActions = configureService { implicit configuration => {
      s"${configurationAction1} ${configurationAction2}"
    }}


    configuredActions shouldEqual "action1 action2"
  }
}

The compilation error would be produced for the following code:

    def configureService(implicit configuration: Configuration): String = {
      s"configured ${configuration.key}"
    }

    val configurationMessage = configureService

And the error would be:

  Error:(28, 32) could not find implicit value for parameter configuration: ImplicitParametersTest.this.Configuration
    val configurationMessage = configureService
    Error:(28, 32) not enough arguments for method configureService: (implicit configuration: ImplicitParametersTest.this.Configuration)String.
Unspecified value parameter configuration.
    val configurationMessage = configureService

Implicit conversions as implicit parameters

In some cases the conversion and parameters can be used together:

import org.scalatest.{FlatSpec, Matchers}

class ImplicitConversionAsImplicitParameterTest extends FlatSpec with Matchers {

  behavior of "implicit conversion as implicit parameter"

  it should "be used for implicit parameter" in {
    def getLength[T](text: String)(implicit converter: T => String): Int = text.length
    implicit def convertSeqToString(sequence: Seq[Int]): String = sequence.mkString(",")

    val numbersLength = getLength(Seq(1, 2, 3, 4))

    numbersLength shouldEqual 7
  }

}

This category is called implicit conversion as implicit parameter. Unlike previous concepts, we don't need to define the parameter as implicit. Instead we must declare a conversion method as implicit and provide its implementation for common types. Under-the-hood the compiler executes first the conversion with corresponding method and only later applies the extended method on converted object.

Throughout this post we could learn about the implicits in Scala. As we saw, it applies for parameters, classes and conversions. One of their common use case is extending already existent code base (e.g. 3rd part libraries) with customized code. It's what happens with Scala's wrappers for Java primitives (e.g. RichInt created with implicit conversion scala.LowPriorityImplicits#intWrapper). The implicits let us to save a lot of space for sometimes redundant operations (e.g. conversions). However we'd keep in mind that they add an extra complexity in code understanding and when over-used they can largely increase learning curve of a project.

TAGS: #One Scala feature per week