Dynamic module

Introduction

It is highly recommended to learn about the library's design before going into this section.

Smithy4s is first and foremost a code generation tool for the Smithy language in Scala. Although it does provide interpreters for the Smithy services, which can be used to derive e.g. HTTP clients and servers, the codegen way can only get you so far - there are some situations when it's not sufficient for the job.

Code generation works well if your Smithy model changes no more often than your service's implementation - as long as you run your build whenever you make a code change, codegen will also be triggered, and the Scala compiler will ensure you're in sync with the Smithy model in its present state. But what if your Smithy model changes are more frequent than the service? Or what if you simply don't have access to all the Smithy models your code might have to work with?

These cases, and possibly others, are why Smithy4s has the dynamic module.

(Why) do we need codegen?

As you know by now, Smithy4s's codegen is static - it requires the model to be available at build-time, so that code can be generated and made available to you at compile-time.

In short, what happens at build-time are the following steps:

1. Read the Smithy files available to your build
2. Build a semantic Smithy model, which is roughly a graph of shapes that refer to each other
3. Generate files for each relevant shape in the model (e.g. a service, a structure, an enum...), including metadata (services and schemas).

Then, there's the runtime part. Let's say you're building an HTTP client - in that case, what you see as a Smithy4s user is:

SimpleRestJson(WeatherService)
  .client(??? : org.http4s.client.Client[IO])
  .make

or more generically:

interpretToRestClient(WeatherService)

The steps that the HTTP client interpreter performs to build a high-level client are:

1. Capture a Smithy4s service representing the service you wrote in Smithy. This was generated by Smithy4s's codegen.
2. Analyze the service's endpoints, their input/output schemas, the Hints on these schemas...
3. Transform the service description into a high-level proxy to the underlying client implementation.

Turns out that interpreters like this aren't actually aware of the fact that there's code generation involved. As long as you can provide a data structure describing your service, its endpoints and their schemas (which is indeed the Service type), you can use any interpreter that requires one: code generation is just a means to derive such a data structure automatically from your Smithy model.

This all is why you don't need code generation to benefit from the interpreters - you just need a way to instantiate a Smithy4s Service (or Schema, if that's what your interpreter operates on).

The Dynamic module of smithy4s was made exactly for that purpose.

The Dynamic way

In the previous section, we looked at the steps performed at build time to generate code:

• Read Smithy files
• Build a Smithy model
• Generate Scala files with Smithy4s schemas.

Don't be fooled - although we had Smithy files as the input and Scala files as the output, the really important part was getting from the Smithy model to the Service and Schema instances representing it. The Dynamic module of smithy4s provides a way to do this at runtime.

And the runtime part, where the interpreter runs? It's the same as before! The Service and Schema interfaces are identical regardless of the static/dynamic usecase, and so are the interpreters¹.

Loading a dynamic model

First of all, you need the dependency:

libraryDependencies ++= Seq(
  // version sourced from the plugin
  "com.disneystreaming.smithy4s"  %% "smithy4s-dynamic" % smithy4sVersion.value
)

Now, you need a Smithy model. There are essentially three ways to get one:

1. Load a model using the awslabs/smithy library's ModelAssembler
2. Load a serialized model from a JSON file (example), or
3. Deserialize or generate the smithy4s.dynamic.model.Model data structure in any way you want, on your own.

The ModelAssembler way only works on the JVM, because Smithy's reference implementation is a Java library. We'll use that way for this guide:

import software.amazon.smithy.model.Model

val s = """
$version: "2"

namespace weather

use alloy#simpleRestJson

@simpleRestJson
service WeatherService {
    operations: [GetWeather]
}

@http(method: "GET", uri: "/weather/{city}")
operation GetWeather {
    input := {
        @httpLabel
        @required
        city: String
    }
    output := {
        @required
        weather: String
    }
}

structure Dog {
    @required
    name: String
}
"""

val model = Model
  .assembler()
  .addUnparsedModel(
    "weather.smithy",
    s,
  )
  .discoverModels()
  .assemble()
  .unwrap()

The entrypoint to loading models is DynamicSchemaIndex.

import smithy4s.dynamic.DynamicSchemaIndex
val dsi = DynamicSchemaIndex.loadModel(model)
// dsi: DynamicSchemaIndex = smithy4s.dynamic.internals.DynamicSchemaIndexImpl@59ea158f

For alternative ways to load a DSI, see DynamicSchemaIndex.load.

Using the DSI

Having a DynamicSchemaIndex, we can iterate over all the services available to it:

dsi.allServices.map(_.service.id)
// res0: Iterable[smithy4s.ShapeId] = List(
//   ShapeId(namespace = "weather", name = "WeatherService")
// )

as well as the schemas:

dsi.allSchemas.map(_.shapeId).filter(_.namespace == "weather")
// res1: Iterable[smithy4s.ShapeId] = List(
//   ShapeId(namespace = "weather", name = "GetWeatherInput"),
//   ShapeId(namespace = "weather", name = "GetWeatherOutput"),
//   ShapeId(namespace = "weather", name = "Dog")
// )

You can also access a service or schema by ID:

import smithy4s.ShapeId

dsi.getService(ShapeId("weather", "WeatherService")).get.service.id
// res2: ShapeId = ShapeId(namespace = "weather", name = "WeatherService")
dsi.getSchema(ShapeId("weather", "Dog")).get.shapeId
// res3: ShapeId = ShapeId(namespace = "weather", name = "Dog")

Note that you don't know the exact type of a schema at compile-time:

import smithy4s.Schema

dsi.getSchema(ShapeId("weather", "Dog")).get
// res4: Schema[_] = StructSchema(
//   shapeId = ShapeId(namespace = "weather", name = "Dog"),
//   hints = Impl(memberHintsMap = Map(), targetHintsMap = Map()),
//   fields = Vector(
//     Field(
//       label = "name",
//       schema = PrimitiveSchema(
//         shapeId = ShapeId(namespace = "smithy.api", name = "String"),
//         hints = Impl(
//           memberHintsMap = Map(
//             ShapeId(namespace = "smithy.api", name = "required") -> DynamicBinding(
//               keyId = ShapeId(namespace = "smithy.api", name = "required"),
//               value = DObject(value = ListMap())
//             )
//           ),
//           targetHintsMap = Map()
//         ),
//         tag = PString
//       ),
//       get = Accessor(index = 0)
//     )
//   ),
//   make = <function1>
// )

It is very similar for services. This is simply due to the fact that at compile-time (which is where typechecking happens) we have no clue what the possible type of the schema could be. After all, the String representing the model doesn't have to be constant - it could be fetched from the network, and even vary throughout the lifetime of our application!

This doesn't forbid us from using these dynamic goodies in interpreters, though.

Case study: dynamic HTTP client

Let's make a REST client for a dynamic service. We'll start by writing an interpreter.

"But wait, weren't we supposed to be able to use the existing interpreters?"

That's true - the underlying implementation will use the interpreters made for the static world as well. However, due to the strangely-typed nature of the dynamic world, we have to deal with some complexity that's normally invisible to the user.

For example, you can write this in the static world:

import cats.effect.IO
import weather._

// imagine this comes from an interpreter
val client: WeatherService[IO] = ??? : WeatherService[IO]

client.getWeather(city = "hello")

but you can't do it if your model gets loaded dynamically! You wouldn't be able to compile that code, because there's no way to tell what services you'll load at runtime.

This means that we'll need a different way to pass the following pieces to an interpreter at runtime:

• the service being used (HelloWorldService)
• the operation being called (Hello)
• the operation input (a single parameter: name = "Hello")

Let's get to work - we'll need a function that takes a service and its interpreter, the operation name, and some representation of its input. For that input, we'll use smithy4s's Document type.

import smithy4s.Document
import smithy4s.Endpoint
import smithy4s.Service
import smithy4s.kinds.FunctorAlgebra
import smithy4s.kinds.FunctorInterpreter
import cats.effect.IO

def run[Alg[_[_, _, _, _, _]]](
  service: Service[Alg],
  operationName: String,
  input: Document,
  alg: FunctorAlgebra[Alg, IO]
): IO[Document] = {
  val endpoint = service.endpoints.find(_.id.name == operationName).get

  runEndpoint(endpoint, input, service.toPolyFunction(alg))
}

def runEndpoint[Op[_, _, _, _, _], I, O](
  endpoint: Endpoint[Op, I, _, O, _, _],
  input: Document,
  interp: FunctorInterpreter[Op, IO],
): IO[Document] = {
  // Deriving these codecs is a costly operation, so we don't recommend doing it for every call.
  // We do it here for simplicity.
  val inputDecoder = Document.Decoder.fromSchema(endpoint.input)
  val outputEncoder = Document.Encoder.fromSchema(endpoint.output)

  val decoded: I = inputDecoder.decode(input).toTry.get

  val result: IO[O] = interp(endpoint.wrap(decoded))

  result.map(outputEncoder.encode(_))
}

That code is a little heavy and abstract, but there's really no way to avoid abstraction - after all, we need to be prepared for any and all models that our users might give us, so we need to be very abstract!

To explain a little bit:

FunctorAlgebra[Alg, IO] is Alg[IO] (for a specific shape of Alg). This could be HelloWorldService[IO], if we knew the types (which we don't, because we're in the dynamic, runtime world). Related to that, FunctorInterpreter[Op, IO] is a different way to view an Alg[IO], which is as a higher-kinded function. See this document for more explanation.

The steps we're taking are:

1. Find the endpoint within the service, using its operation name
2. In runEndpoint, decode the input Document to the type the endpoint expects
3. Run the interpreter using the decoded input
4. Encode the output to a Document.

Let's see this in action with our actual service! But first, just for this guide, we'll define the routes for a fake instance of the server we're going to call:

import org.http4s.HttpApp
import org.http4s.MediaType
import org.http4s.headers.`Content-Type`
import org.http4s.dsl.io._

val routes = HttpApp[IO] { case GET -> Root / "weather" / city =>
  Ok(s"""{"weather": "sunny in $city"}""").map(
    _.withContentType(`Content-Type`(MediaType.application.json))
  )
}
// routes: HttpApp[IO] = Kleisli(
//   run = org.http4s.Http$$$Lambda$28780/0x00007f35f079eaa0@b369c76
// )

Now we'll build a client based on the service we loaded earlier, using that route as a fake server:

import org.http4s.client.Client
import smithy4s.http4s.SimpleRestJsonBuilder

// first, we need some Service instance - we get one from the DynamicSchemaIndex we made earlier
val service = dsi.getService(ShapeId("weather", "WeatherService")).get

val client =
  SimpleRestJsonBuilder(service.service)
    .client(Client.fromHttpApp(routes))
    .make
    .toTry
    .get

And finally, what we've been working towards all this time - we'll select the GetWeather operation, pass a Document representing our input, and the client we've just built.

import cats.effect.unsafe.implicits._

run(
  service = service.service,
  operationName = "GetWeather",
  input = Document.obj("city" -> Document.fromString("London")),
  alg = client,
).unsafeRunSync().show
// res6: String = "{weather=\"sunny in London\"}"

Enjoy the view! As an added bonus, because we happen to have this service at build-time, we can use the same method with a static, compile-time service:

import weather._

val clientStatic =
  SimpleRestJsonBuilder(WeatherService)
    .client(Client.fromHttpApp(routes))
    .make
    .toTry
    .get
// clientStatic: WeatherServiceGen[[I, E, O, SI, SO]IO[O]] = weather.WeatherServiceOperation$Transformed@1c43b34c

run(
  service = WeatherService,
  operationName = "GetWeather",
  input = Document.obj("city" -> Document.fromString("London")),
  alg = clientStatic,
).unsafeRunSync().show
// res7: String = "{weather=\"sunny in London\"}"

Again, this is equivalent to the following call in the static approach:

clientStatic.getWeather(city = "London").unsafeRunSync()
// res8: GetWeatherOutput = GetWeatherOutput(weather = "sunny in London")

---

1. That is, assuming they're written correctly to make no assumptions about the usecase.↩