Getting started with ATD and Melange

Edit: this article was updated by Javier Chávarri on Oct 2023 to replace the usage of BuckleScript with Melange.

ATD is a project to create types and data structures that can be serialized to JSON. It is very convenient when communicating between multiple processes, creating a REST API or consuming JSON objects from other tools. It can be compared to JSON schema or Protocol Buffers, but with richer types and more features.

The idea is to write a list of types in a specification file, an .atd file. Then by running atdgen, it is possible to generate code in many languages like OCaml, DLang, Java, Python, Scala, TypeScript, or interface description languages like JSON Schema, to serialize/deserialize values of those types to/from the corresponding JSON.

Originally, ATD could generate code only for native OCaml. But nowadays it also supports Melange, a backend for the OCaml compiler that emits JavaScript. atdgen the cli tool is still a native OCaml binary. But it can output some OCaml code that can be compiled using Melange.

The work to implement the compatibility with Melange in ATD was funded by Ahrefs. We highly appreciate open source tools. And as much as possible, we prefer to contribute to existing open source projects rather than to re-invent the wheel internally.

Installation

To install ATD we first need to install opam (OCaml package manager). The procedure is simple and documented here: https://opam.ocaml.org/doc/Install.html

Then we need to initialize opam and create a switch. To work with Melange 2.0 or newer versions, we will need at least the version 5.1.0 of the OCaml compiler.

opam init -a
opam switch create . 5.1.0 -y

Once it is done, we have to install ATD:

opam install atdgen

Make sure that atdgen is available.

$ which atdgen
(current $PWD)/_opam/bin/atdgen

Of course, we need Melange.

opam install melange

We also need the Melange runtime, as it is not currently provided by ATD itself. This runtime is also open-source: https://github.com/ahrefs/melange-atdgen-codec-runtime.

This runtime is responsible for the conversion between JSON values and OCaml values. The JSON values are based on the standard Js.Json.t type provided by Melange to be sure that it is easy to interoperate with the rest of the ecosystem.

It is also published on opam:

opam install melange-atdgen-codec-runtime

Project configuration

Next, we will add a dune-project file to the root folder. This file tells Dune how our project is configured:

(lang dune 3.8)
(using melange 0.1)
(name melange-atdgen-example)

After this, we can add a dune file inside the src folder that describes how to compile the code using Melange:

(melange.emit
 (target example)
 (alias example)
 (libraries melange.node melange-atdgen-codec-runtime)
 (promote (until-clean)))

(rule
 (targets meetup_bs.ml meetup_bs.mli)
 (deps meetup.atd)
 (action
  (run atdgen -bs %{deps})))

(rule
 (targets meetup_t.ml meetup_t.mli)
 (deps meetup.atd)
 (action
  (run atdgen -t %{deps})))

For more information about Dune and melange.emit stanza, refer to the Dune documentation site.

First ATD definitions

It is time to create a first .atd file, containing our types. This part is also documented on https://atd.readthedocs.io/en/latest/atdgen.html#getting-started

For this example, I decided to go with a meetup event. Put the type definitions in src/meetup.atd.

(* This is a comment. Same syntax as in ocaml. *)

type access = [ Private | Public ]

(* the date will be a float in the json and a Js.Date.t in ocaml *)
type date = float wrap <ocaml module="Js.Date" wrap="Js.Date.fromFloat" unwrap="Js.Date.valueOf">

(* Some people don't want to provide a phone number, make it optional *)
type person = {
  name: string;
  email: string;
  ?phone: string nullable;
}

type event = {
  access: access;
  name: string;
  host: person;
  date: date;
  guests: person list;
}

type events = event list

We use the atdgen binary (compiled previously) to generate the ocaml types and the code to serialize/deserialize those types.

atdgen -t meetup.atd # generates an ocaml file containing the types
atdgen -bs meetup.atd # generates the code to (de)serialize

The generated files are:

• meetup_t.ml(i) which contain the ocaml types corresponding to our ATD definitions.
• meetup_bs.ml(i) which contain the ocaml code to transform from and to json values.

At this point we can compile our project.

dune build

If everything worked properly, we now have two .js files in the src/example/src directory, which is where Dune places the generated JavaScript files:

$ tree src/example/src
src/example/src
├── meetup_bs.js
└── meetup_t.js
0 directories, 3 files

At this point, we can create new OCaml/Reason files in the src directory and use all the code atdgen generated for us. Two examples to illustrate that.

Query a REST API

A common usage of atdgen is to decode the JSON returned by a REST API. Here is a short example, using the reason syntax and melange-fetch.

let get = (url, decode) =>
  Js.Promise.(
    Fetch.fetchWithInit(url, Fetch.RequestInit.make(~method_=Get, ()))
    |> then_(Fetch.Response.json)
    |> then_(json => json |> decode |> resolve)
  );
let v: Meetup_t.events =
  get(
    "http://localhost:8000/events",
    Atdgen_codec_runtime.Decode.decode(Meetup_bs.read_events),
  );

Read and write a JSON file

Atdgen for Melange doesn’t take care of converting a string to a JSON object. Which allows us to use the performant JSON parser included in Node or the browser.

let read_events = filename => {
  /* Read and parse the json file from disk, this doesn't involve atdgen. */
  let json = Node.Fs.readFileAsUtf8Sync(filename) |> Js.Json.parseExn;
  /* Turn it into a proper record. The annotation is of course optional. */
  let events: Meetup_t.events = (
    Atdgen_codec_runtime.Decode.decode(Meetup_bs.read_events, json): Meetup_t.events
  );
  events;
};

The reverse operation, converting a record to a JSON object and writing it in a file is also straightforward.

let write_events = (filename, events) =>
  /* turn a list of records into json */
  Atdgen_codec_runtime.Encode.encode(Meetup_bs.write_events, events)
  /* convert the json to a pretty string */
  ->(Js.Json.stringifyWithSpace(2))  
  /* write the json in our file */
  |> Node_fs.writeFileAsUtf8Sync(filename);

Full example

Now that we have our functions to read and write events, we can build a small cli to pretty print the list of events and add new events.

The source code of the full example is available on github.

You can run it like this:

$ echo "[]" > events.json
$ node src/example/src/cli.js add louis louislouis@nospam.com
$ node src/example/src/cli.js add bob bobbob@nospam.com
$ node src/example/src/cli.js meetup print
=== OCaml/Reason Meetup! summary ===
date: Tue, 11 Sep 2018 15:04:16 GMT
access: public
host: bob <bob@nospam.com>
guests: 1
=== OCaml/Reason Meetup! summary ===
date: Tue, 11 Sep 2018 15:04:13 GMT
access: public
host: louis <louis@nospam.com>
guests: 1
$ cat events.json
[
  {
    "guests": [
      {
        "email": "bob@nospam.com",
        "name": "bob"
      }
    ],
    "date": 1536678256177,
    "host": {
      "email": "bob@nospam.com",
      "name": "bob"
    },
    "name": "OCaml/Reason Meetup!",
    "access": "Public"
  },
  {
    "guests": [
      {
        "email": "louis@nospam.com",
        "name": "louis"
      }
    ],
    "date": 1536678253790,
    "host": {
      "email": "louis@nospam.com",
      "name": "louis"
    },
    "name": "OCaml/Reason Meetup!",
    "access": "Public"
  }
]