Compliance Tests

The Smithy prelude has support for compliance testing for services that use HTTP as their protocol. It is built on top of regular traits, you can read more about it over here.

Basically, you annotate operations and/or structures (that depends on the type of test being defined) and protocol implementors can generate tests cases to ensure their implementation behaves correctly.

Smithy4s publishes a module that you can use to write compliance test if you're implementing a protocol. Add the following to your dependencies if you use sbt:

import smithy4s.codegen.BuildInfo._

libraryDependencies ++= Seq(
  "com.disneystreaming.smithy4s" %% "smithy4s-compliance-tests" % smithy4sVersion.value
)

If you use mill:

import smithy4s.codegen.BuildInfo._

def smithy4sIvyDeps = Agg(
  ivy"software.amazon.smithy:smithy-protocol-test-traits:$smithyVersion"
)

def ivyDeps = Agg(
  ivy"com.disneystreaming.smithy4s::smithy4s-compliance-tests:${smithy4sVersion()}"
)

The rest of this document will demonstrate how to write a Smithy specification that specify HTTP compliance tests, and then how to use the module mentioned above to run the tests.

Note: currently, only httprequesttests-trait are supported. other traits support will be integrated soon.

Example specification

$version: "2"

namespace smithy4s.example.test

use smithy.test#httpRequestTests
use smithy.test#httpResponseTests
use alloy#simpleRestJson

@simpleRestJson
service HelloService {
    operations: [SayHello, Listen, TestPath]
}

@http(method: "POST", uri: "/")
@httpRequestTests([
    {
        id: "say_hello",
        protocol: simpleRestJson,
        params: {
            "greeting": "Hi",
            "name": "Teddy",
            "query": "Hello there"
        },
        method: "POST",
        uri: "/",
        queryParams: [
            "Hi=Hello%20there"
        ],
        headers: {
            "X-Greeting": "Hi",
        },
        body: "{\"name\":\"Teddy\"}",
        bodyMediaType: "application/json"
    }
])
@httpResponseTests([
    {
        id: "say_hello"
        protocol: simpleRestJson
        params: { payload: { result: "Hello!" }, header1: "V1" }
        body: "{\"result\":\"Hello!\"}"
        headers: { "X-H1": "V1"}
        code: 200
    }
])
operation SayHello {
    input: SayHelloInput,
    output: SayHelloOutput
    errors: [SimpleError, ComplexError]
}

@input
structure SayHelloInput {
    @httpHeader("X-Greeting")
    greeting: String,

    @httpQuery("Hi")
    query: String,

    name: String
}

structure SayHelloOutput {
    @required
    @httpPayload
    payload: SayHelloPayload

    @required
    @httpHeader("X-H1")
    header1: String
}
structure SayHelloPayload {
    @required
    result: String
}


@http(method: "GET", uri: "/listen")
@readonly
@httpRequestTests([
    {
        id: "listen",
        protocol: simpleRestJson,
        method: "GET",
        uri: "/listen"
    }
])
operation Listen { }

@http(method: "GET", uri: "/test-path/{path}")
@readonly
@httpRequestTests([
    {
        id: "TestPath",
        protocol: simpleRestJson,
        method: "GET",
        uri: "/test-path/sameValue"
        params: { path: "sameValue" }
    }
])
operation TestPath {
    input := {
        @httpLabel
        @required
        path: String
    }
}

// The following shapes are used by the documentation
@simpleRestJson
service HelloWorldService {
  version: "1.0.0",
  operations: [Hello]
}
@httpRequestTests([
    {
        id: "helloSuccess"
        protocol: simpleRestJson
        method: "POST"
        uri: "/World"
        params: { name: "World" }
    },
    {
        id: "helloFails"
        protocol: simpleRestJson
        method: "POST"
        uri: "/fail"
        params: { name: "World" }
    }
])
@http(method: "POST", uri: "/{name}", code: 200)
operation Hello {
  input := {
    @httpLabel
    @required
    name: String
  },
  output := {
    @required
    message: String
  }
}

@httpResponseTests([
    {
        id: "simple_error"
        protocol: simpleRestJson
        params: { expected: -1 }
        code: 400
        body: "{\"expected\":-1}"
        bodyMediaType: "application/json"
        requireHeaders: ["X-Error-Type"]
    }
])
@error("client")
structure SimpleError {
    @required
    expected: Integer
}
@httpResponseTests([
    {
        id: "complex_error"
        protocol: simpleRestJson
        params: { value: -1, message: "some error message", details: { date: 123, location: "NYC"} }
        code: 504
        body: "{\"value\":-1,\"message\":\"some error message\",\"details\":{\"date\":123,\"location\":\"NYC\"}}"
        bodyMediaType: "application/json"
        requireHeaders: ["X-Error-Type"]
    },
    {
        id: "complex_error_no_details"
        protocol: simpleRestJson
        params: { value: -1, message: "some error message" }
        code: 504
        body: "{\"value\":-1,\"message\":\"some error message\"}"
        bodyMediaType: "application/json"
        requireHeaders: ["X-Error-Type"]
    }
])
@error("server")
@httpError(504)
structure ComplexError {
    @required
    value: Integer
    @required
    message: String
    details: ErrorDetails
}

structure ErrorDetails {
    @required
    @timestampFormat("epoch-seconds")
    date: Timestamp
    @required
    location: String
}

We have a very simple specification: one operation with basic input and output shapes. We've added a httpRequestTests to define a compliance test for protocol implementors.

Testing the protocol

The service in the specification is annotated with the alloy#simpleRestJson protocol definition. We'll use the compliance-tests module to make sure this protocol can handle such an operation.

Note: the following code and the compliance-tests module do not depend on a specific test framework. If you want to hook it into your test framework, it is easy to do so but it's outside the scope of this document. Refer to this example to see how we did it for Weaver in this project.

First, some imports:

import cats.effect._
import org.http4s._
import org.http4s.client.Client
import smithy4s.compliancetests._
import smithy4s.example.test._
import smithy4s.http.HttpMediaType
import smithy4s.http4s._
import smithy4s.kinds._
import smithy4s.Service
import smithy4s.schema.Schema

Then, you can create and instance of ClientHttpComplianceTestCase and/or ServerHttpComplianceTestCase while selecting the protocol to use and the service to test:

object SimpleRestJsonIntegration extends Router[IO] with ReverseRouter[IO] {
    type Protocol = alloy.SimpleRestJson
    val protocolTag = alloy.SimpleRestJson

    def expectedResponseType(schema: Schema[_]) = HttpMediaType("application/json")

    def routes[Alg[_[_, _, _, _, _]]](
        impl: FunctorAlgebra[Alg, IO]
    )(implicit service: Service[Alg]): Resource[IO, HttpRoutes[IO]] =
      SimpleRestJsonBuilder(service).routes(impl).resource

    def reverseRoutes[Alg[_[_, _, _, _, _]]](app: HttpApp[IO],testHost: Option[String] = None)(implicit
        service: Service[Alg]
    ): Resource[IO, FunctorAlgebra[Alg, IO]] = {
      import org.http4s.implicits._
      val baseUri = uri"http://localhost/"
      val suppliedHost = testHost.map(host => Uri.unsafeFromString(s"http://$host"))
      SimpleRestJsonBuilder(service)
        .client(Client.fromHttpApp(app))
        .uri(suppliedHost.getOrElse(baseUri))
        .resource
    }
  }

val tests: List[ComplianceTest[IO]] = HttpProtocolCompliance
    .clientAndServerTests(SimpleRestJsonIntegration, HelloWorldService)

Now, you can iterate over the test cases and do what you want. This is where you would hook in the test framework of your choice, but in the following example, we're just going to print the result:

import cats.syntax.traverse._
import cats.effect.unsafe.implicits.global

val runTests: IO[List[String]] = tests
  .map { tc =>
    tc.run.map(_.toEither).map {
      case Left(value) =>
        s"Failed ${tc.show} with the following message: $value"
      case Right(_) => s"Success ${tc.show}"

    }
  }
  .sequence

Will produce the following when executed:

Success smithy4s.example.test#Hello(client|Request): helloSuccess
Failed smithy4s.example.test#Hello(client|Request): helloFails with the following message: NonEmptyList(path test : the result value: Vector("World") was not equal to the expected TestCase value Vector("fail").)
Success smithy4s.example.test#Hello(server|Request): helloSuccess
Failed smithy4s.example.test#Hello(server|Request): helloFails with the following message: NonEmptyList( the result value: HelloInput(name = "fail") was not equal to the expected TestCase value HelloInput(name = "World").)