/

tl;dr - This post contains examples of a folder structure & examples of a manually-edited OpenAPI v3 schema setup that works well with openapi-generator. At the end I go the relevant config to set up automated client generation with Gitlab CI

Eons ago, before GraphQL became the new hotness (I’m very bearish on GraphQL), the fast-moving frontend world had just started to settle in to a bunch of tools for standardizing REST-ish and kinda-HATEOAS-y backends with a few tools (in no particular order):

• HAL
• JSONAPI
• JSON Schema & JSON Hyper-Schema
• Swagger (which merged with/became OpenAPI)
• RAML
• API blueprint
• JSON LD

These technologies helped make it possible to automate and standardize more and more of the API-calling experience – these days no one (should) be worrying about writing HTTP clients for their REST-ful HTTP backends, because you can generate them. Of course, there may be a few bugs (and lack of compatibility with some more advanced HTTP schemes/patterns), but by and large, for the simple POST request to your /users endpoint, people are well served by these tools/standards/methodologies.

One of the best things about these tools is that a bunch of them (the “lower level” or more standards-like ones) were actually interoperable – you could absolutely use JSON schema (a validation spec) with Swagger (an API description specification, which uses a slightly modified JSON schema spec), and things wouldn’t look too out of place/conflict very heavily. As a dedicated seller of the particular brand of broken dreams that is the semantic web, I like these protocols (over an approach like GraphQL) because they encourage meta-programming and reusability, along with the flexibility of JSON. I won’t get into it too much (in this post at least) but there are a few reasons I prefer these technologies to GraphQL and think GraphQL is distracting from true progress in the front/backend harmony endeavor.

In this post, I want to focus on Swagger/OpenAPI v3 – I’ve adopted it for most of my personal and client projects of significant size, and I find myself often dealing with a few shortcomings/painpoints:

• Swagger/OpenAPI is a bit behind on jsonschema spec (though to be fair they churn slower)
• The transition from Swaggerv1 -> Swaggerv2/OpenAPIv1 -> OpenAPIv3 is mess to explain every time
• Implementation & auto-generation support can vary a lot across languages
• NodeJS basically has no unified libraries that support auto-generating OpenAPIv3 for a web server*

Of those shortcomings, the last one is what I want to zone in on – twice now on bigger projects I’ve had to resort to writing the API specification manually, becuase there were no mature libraries (at the time) that would work together properly to automatically generate proper . This will likely get better with Typescript’s decorators support, but I looked and couldn’t find anything compelling, especcially considering I wanted to target OpenAPI v3 (it’s much improved over v2). Of course, if you’re using a language like Ruby with well-known batteries-included frameworks like Rails (or Python with Django), you’re going to have much better support than what NodeJS can offer, with it’s more spread out libraries.

While 99% of the time you should absolutely avoid manually maintaining your OpenAPI schema if it can be done through decorators/code-inspection, this post is going to contain a fully worked, working example (with decent good generated code, cutting down on spurrious InlineResponse200x classes), as a reference for others (and myself) who possibly just got assigned some ticket “set up swagger” and were messing around for the last ~5 hours doing no work but now need to get something done for a demo meeting.

NOTE I’m going to be assuming that you’re writing a schema and generating Typescript APIs – so some of this post might not be applicable if you’re working in Python for example, but the schema itself should still be the same.

Directory Structure

Here is the file structure for the swagger/ folder in a recent project:

swagger
├── components.yaml
├── openapi.yaml
├── paths.yaml
├── responses.yaml
├── generated
│   ├── clients
│   │   └── ts
│   │       ├── api.ts
│   │       ├── configuration.ts
│   │       ├── custom.d.ts
│   │       ├── git_push.sh
│   │       └── index.ts
│   └── openapi.json
└── routes
    ├── addresses
    │   ├── byUuid.yaml
    │   ├── index.yaml
    │   └── search.yaml
    ... lots more folders & yaml

Here are some of the important files and what they do:

File content examples

Here are some examples of the various files in the set up that should be instructive of how they should look:

components.yaml:

#########
# Enums #
#########

Country:
  type: string
  enum:
    - "japan" # 日本
    - "usa" # USA

# ... lots more code ...

##########
# Models #
##########

Address:
  type: object
  description: Address
  properties:
    uuid:
      type: string
      format: uuid
    addressLine1:
      type: string
    addressLine2:
      type: string
      nullable: true
    prefecture:
      type: string
    city:
      type: string
    country:
      $ref: "#/Country"
    postalCode:
      type: string
    lat:
      type: number
      format: double
    lon:
      type: number
      format: double
    creatorUuid:
      type: string
      format: uuid
    creator:
      $ref: "#/User"
    createdAt:
      type: string
      format: date-time

# ... lots more code ...

###########################
# Request/Response Bodies #
###########################

JSONError:
  type: object
  properties:
    status:
      description: HTTP status
      type: number
    code:
      description: Server-side error code
      type: number
    errors:
      description: Error(s) that occurred
      type: array
      items:
        type: string

# ... lots more code ...

openapi.yaml:

openapi: "3.0.2"
servers:
  - url: http://localhost:3000/v1
    description: local
  - url: https://example.com/api/v1
    description: production
info:
  version: "1.0.0"
  title: Your Project API
  description: Your Project API client
paths:
  $ref: "paths.yaml"
components:
# Schemas *have* to be redefined here... see:
# https://github.com/OpenAPITools/openapi-generator/issues/8
# https://github.com/OpenAPITools/openapi-generator/issues/222
  schemas:
    $ref: "components.yaml"
  responses:
    $ref: "responses.yaml"
  securitySchemes:
    authorization_header:
      type: http
      scheme: bearer
      bearerFormat: JWT
security:
  - authorization_header: []

paths.yaml:

/: {$ref: "routes/root.yaml"}
/addresses/search: {$ref: "routes/addresses/search.yaml"}
/addresses/{uuid}: {$ref: "routes/addresses/byUuid.yaml"}
/addresses: {$ref: "routes/addresses/index.yaml"}
# ... lots more paths ...

responses.yaml:

Unauthenticated:
  description: Unauthenticated request
  content:
    application/json:
      schema:
        type: object
        $ref: "./openapi.yaml#/components/schemas/JSONError"

# ... other very frequently used responses ...

routes/addresses/index.yaml:

# POST /addresses
post:
  operationId: createAddress
  description: Create an address

  requestBody:
    content:
      application/json:
        schema:
          $ref: "../../openapi.yaml#/components/schemas/NewAddress"

  responses:
    '200':
      description: Success
      content:
        application/json:
          schema:
            $ref: "../../openapi.yaml#/components/schemas/EnvelopedAddress"

    '401':
      $ref: "../../openapi.yaml#/components/responses/Unauthenticated"
    '403':
      $ref: "../../openapi.yaml#/components/responses/Unauthorized"
    '500':
      $ref: "../../openapi.yaml#/components/responses/UnexpectedServerError"

# GET /addresses
get:
  operationId: getAllAddresses
  description: Get all addresses from the backend
  responses:
    '200':
      description: Success
      content:
        application/json:
          schema:
            $ref: "../../openapi.yaml#/components/schemas/EnvelopedPaginatedAddressList"
    '401':
      $ref: "../../openapi.yaml#/components/responses/Unauthenticated"
    '403':
      $ref: "../../openapi.yaml#/components/responses/Unauthorized"
    '500':
      $ref: "../../openapi.yaml#/components/responses/UnexpectedServerError"

NOTE Don’t worry if it takes you some time to understand $ref and schema and how they interplay exactly – I stumbled on it a little bit, particularly in the different places you could use $refs and what happened when nesting them.

Gotcha: Unsighltly InlineSchema objects everywhere

If you didn’t follow the file contents/structure closely, you’d have to worry (as I did) about InlineSchemaxxxx objects popping up all over your generated JS client. Here’s a comment some unfortunate soul will have to read in the YAML I wrote someday:

# Schemas *have* to *all* be defined here in detail because $ref isn't usable under components @ top level
# request bodies have to be defined as components @ top level because inline request bodies setting don't work properly:
# https://github.com/OpenAPITools/openapi-generator/issues/8
# https://github.com/OpenAPITools/openapi-generator/issues/222
#
# This sucks, but is the only work around right now (outside of reverting to Swagger/OpenAPI v2)
#

Unfortunately you can’t do much about the InlineResponse objects as far as I can tell but that will likely change in the future (or I’m just not using the generator(s) right).

Tooling: Write yourself a swagger-watch target

One of the best things I did was write myself a swagger-watch target (I’m pre-supposing you’re using make here):

YARN ?= yarn

swagger-watch:
    find swagger -name *.yaml | entr $(YARN) swagger

This make target makes use of the always-excellent entr tool to run a npm scripts task which runs swagger-cli:

  ... other JSON ...
  "scripts": {
    ... other JSON ...
    "swagger": "./node_modules/.bin/swagger-cli validate swagger/openapi.yaml",
    "swagger-bundle": "./node_modules/.bin/swagger-cli bundle swagger/openapi.yaml -o swagger/generated/openapi.json"

Gotcha: Rewrites/fixes for mistakes made systemically are brutal

Consistency is good, except for when a past you was consistently wrong. Theoretically, YAML should enable really good programmatic manipulation (though I’ve never seen a compelling all-in-one tool for it), and maybe this won’t apply to you if you’re a sed/general command line master, but I found making changes after systematically doing things the wrong (or just different) way really brutal. Making tons and tons of changes in files is not hard but pretty draining when you need to do it a lot, and it’s hard to tell at which point you should be writing a script to make this one-time task quicker.

BONUS: Automatically generating client libraries with Gitlab CI

While this has less to do with the project itself, one of the awesome things you can do when you have CI in your toolbox (most of the time that means Gitlab CI for me), is that you can wire up some processes to generate and commit client libraries automatically when versions of your API are released. We’re going to use a few pieces of tech together:

• Gitlab CI runners
• Gitlab CI runner’s support for Docker In Docker (DinD)
• Gitlab CI support for cross-project pipelines
• Makefile shenanigans

The first step is to make another repository – I normally call this something like <project>-client-ts to represent the Typescript client for a certain project. That repository should only contain the generated client code, at the top level, commits to the actual <project> repository will be used to cross-commit to the <project>-client repository.

The step by step process ends up being something like this:

1. A release is made in the <project> repository, with the appropriate tag applied (ex. v0.2.2)
2. The act of tagging the release kicks off a docker image build and push of the new version
3. A cross-project job is triggered on the <project>-client repository
4. The job pulls the created version of the image <project> for project pulls out the relevant code and uses it to generate the swagger client
5. The swagger client is copied to the top level of the <project>-client repository, and committed right from CI into a branch with the appropriate version

Here’s what the gitlab-ci.yml and Makefile would look like in the <project> repository:

gitlab-ci.yml

stages:
  - lint
  - build
  - test
  - publish
  - client_generation

# ... other code ...

publish:
  stage: publish
  image: docker
  services:
    - docker:dind
  # Only publish when a release tag (vX.X.X) is pushed
  only:
    - /v[0-9|\.]+/ # release tags
    - /release-v[0-9|\.]+/ # pre-release branches
  except:
    - branches
  script:
    - apk add --update ca-certificates make nodejs nodejs-npm
    - npm install
    - docker login -u gitlab-ci-token --password $CI_BUILD_TOKEN registry.gitlab.com
    - make docker-image publish-docker-image

trigger_ts_client_generation:
  stage: client_generation
  # Only re-generate the ts client when a change to master is made
  only:
    - /v[0-9|\.]+/ # release tags
    - /release-v[0-9|\.]+/ # pre-release branches
  script:
    - apk add make curl
    - export VERSION=`make print-version`
    - echo -e "Triggering automatic generation of TS client for version [$VERSION]"
    - |
      curl --request POST \
      --form "token=$TS_CLIENT_TRIGGER_TOKEN" \
      --form ref=master \
      --form "variables[VERSION]=$VERSION" \
      https://gitlab.com/api/v4/projects/xxxxxxxxx/trigger/pipeline      

Makefile

VERSION ?= $(shell $(NODE) some-script-that-gets-the-current-project-version.sh)

print-version:
    @echo $(VERSION)

DOCKER ?= $(shell which docker)
DOCKER_IMAGE_FULL_NAME ?= whatever/your/docker/image/name/is:version

docker-image: check-tool-docker
    $(DOCKER) build \
    -f infra/docker/Dockerfile \
    -t ${DOCKER_IMAGE_FULL_NAME} \
    .

publish-docker-image: check-tool-docker
    $(DOCKER) push ${DOCKER_IMAGE_FULL_NAME}

And in the <project>-client repository things get a little more complicated but hopefully it’s still pretty easy to follow:

gitlab-ci.yml:

image: node:10-alpine

stages:
  - generate_and_tag

# build TS client lib based on the version in package.json
# (job should download docker image, run generation, copy out files locally
build_job:
  stage: generate_and_tag
  only:
    - trigger
    - web
    - pipelines
  services:
    - docker:dind
  variables:
   DOCKER_HOST: tcp://docker:2375/
   DOCKER_DRIVER: overlay2
  script:
    - apk --update add make docker openssh git
    - yarn install
    # Set version to current if not passed in through pipeline var
    - |
            if [ -z "${VERSION}" ]; then export VERSION=`make print-version`; fi
    - echo -e "Creating client for version [$VERSION]"
    # Bump version, if a new version is passed in
    - npm version $VERSION || true
    # Copy out swagger stuff with docker
    - "docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com"
    # Run the swagger generation build in docker
    - make
    # Log in w/ SSH agent
    - eval $(ssh-agent -s)
    - mkdir -p /root/.ssh/ && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
    - echo "$CI_PUSH_SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - git config --global user.email "user@example.com"
    - git config --global user.name "CI Automation"
    # Add repo remote
    - git remote add upstream git@gitlab.com:your/project.git
    # Add code
    - git add .
    - git commit -am "Update code (automerge)"
    # Tag & push
    - git tag -v $VERSION || git tag -d $VERSION && git push upstream :$VERSION # delete tag if it exists
    - git tag $VERSION
    - git push upstream master
    - git push upstream $VERSION # push up new tag for current version

Makefile:

extract: extract-swagger-from-api fix-swagger-file-permissions
generate: swagger-gen-js-client move-files-to-root

print-version:
    @echo "$(VERSION)"

clean-swagger-folder:
    rm -rf swagger/

extract-swagger-from-api: clean-swagger-folder
    docker run --rm \
    --entrypoint cp \
    -v ${PWD}/swagger:/swagger-extraction \
    registry.gitlab.com/your/project/image:$(VERSION) \
    -r /home/node/api/swagger /swagger-extraction

fix-swagger-file-permissions:
    @echo "\n[info] Fixing permissions of swagger folder..."
    $$CI || (sudo chown -R $(USER) $(SWAGGER_FOLDER) && chgrp -R $(USER) $(SWAGGER_FOLDER))
    mv -f swagger/swagger/* swagger/
    rm -r swagger/swagger

setup:
    $(YARN) install

swagger:
    $(YARN) swagger

swagger-bundle:
    $(YARN) swagger-bundle

swagger-watch:
    find swagger -name *.yaml | entr $(YARN) swagger

swagger-server: swagger-bundle
    docker run --rm \
    -p 80:8080 \
    -v ${PWD}/swagger:/swagger \
    -e SWAGGER_JSON=/swagger/generated/openapi.json \
    swaggerapi/swagger-ui

swagger-gen-js-client: swagger-bundle
    mkdir -p $(SWAGGER_JS_CLIENT_OUTPUT_DIR)
    @echo "Generating swagger Typescript client to [$(SWAGGER_JS_CLIENT_OUTPUT_DIR)]"
    docker run --rm \
        -e TS_POST_PROCESS_FILE="/usr/local/bin/prettier --write" \
        -v ${PWD}/swagger:/local \
        openapitools/openapi-generator-cli generate \
        -i /local/generated/openapi.json \
        -g typescript-fetch \
        -o /local/generated/clients/ts
    $$CI || (sudo chown -R $(USER) $(SWAGGER_FOLDER) && chgrp -R $(USER) $(SWAGGER_FOLDER))

move-files-to-root:
    mv -f swagger/generated/clients/ts/* .
    chmod +x ./git_push.sh

It might take some fiddling, but a day’s worth of work is well worth it to have such nice automation around for everyone to use – no more waiting on backend developers to explain or write new documentation on endpoints or for frontend developers to add endpoints to some predistributed client SDK.

Wrapup

Hopefully you’ve enjoyed this quick introduction to manually curating/editing your own swagger API schema. Again, this is something you very likely shouldn’t do, if you can get appropriate code-level support via decorators or processing, but if you can’t, hopefully it’s clear how to get it done.