DDG search drop in powered by ddg.patdryburgh.com

Awesome FOSS Logo
Discover awesome open source software
Launched 🚀🧑‍🚀
Mon, Dec 2, 2019 by vados

Python E2E Test Config on Circle CI

Categories
python testing circleci ci tools devops
Python logo + CircleCI logo

tl;dr - A while back I had to set up Python E2E tests on CircleCI. It took lots of experimentation so I thought I’d share

While working with a previous client I had the distinct misfortune of not working with Gitlab’s CI system but instead with CircleCI’s. Alright, CircleCI isn’t that bad, but it is distinctly more complicated and less well documented than GitLab’s and as such is harder to use. In particular I was trying to wire up some E2E tests to run and couldn’t figure out how to do so easily – where GitLab supports “services” that effectively let you run just about any container alongside your main test container, CircleCI required quite a bit of other configuration. In particular I had to:

  • Switch to the “machine” runner on CircleCI
  • Figure out what is installed on the CircleCI VM images by default (at the time that was the circleci/classic:201808-01 image)
  • Do the trial and error of finding out what commands would work like I expect, installing docker.

E2E Test configuration

I’ll get to it, here’s the working configuration that I ended up with:

version: 2.1

workflows:

  e2e_test:
    jobs:
      - e2e_test

commands:
  restore_machine_cache:
    steps:
      - restore_cache:
          # https://discuss.circleci.com/t/add-mechanism-to-update-existing-cache-key/9014/12
          # when restoring, the prefix will be used, and most recent epoch selected
          key: machine-local-package-cache
      - run:
          name: Restore pyenv opt cache
          command: |
            test -f ~/machine-cache-pyenv.tar.gz && tar -C / -zxvf  ~/machine-cache-pyenv.tar.gz
            test -f ~/machine-cache-pyenv.tar.gz || echo "achived /opt contents (machine-cache-pyenv.tar.gz) not present"            

  update_machine_cache:
    steps:
      - run:
          name: Copy pyenv /opt cache
          command: |
             tar -zcvf ~/machine-cache-pyenv.tar.gz /opt/circleci/.pyenv/versions/3.7.3
             ls ~/machine-cache-pyenv.tar.gz
             du -hs ~/machine-cache-pyenv.tar.gz             
      - save_cache:
          # https://discuss.circleci.com/t/add-mechanism-to-update-existing-cache-key/9014/12
          # when saving, later epoch will ensure newer version is restored on next run
          key: machine-local-package-cache-{{ epoch }}
          paths:
            - "~/.cache/pip"
            - "~/.pyenv"
            - "~/.pip-cache"
            - "~/.local/share/virtualenvs"
            - "~/machine-cache-pyenv.tar.gz"

  install_docker:
    steps:
      - run:
          name: install docker client
          command: |
            set -x
            VER="18.09.1"
            test -f /tmp/docker-client.tgz || sudo curl -L -o /tmp/docker-client.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
            sudo tar -xz -C /tmp -f /tmp/docker-client.tgz
            sudo mv /tmp/docker/* /usr/bin            

jobs:
  e2e_test:
    # Machine executor needs to be used here -- otherwise docker networking won't work correctly
    machine:
      image: circleci/classic:201808-01
    steps:
      - checkout
      - run:
          name: Install python prereqs
          command : |
            sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
            libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
            xz-utils tk-dev libffi-dev liblzma-dev python-openssl git libpq-dev python3-dev            
      - restore_machine_cache
      - run:
          name: Update pyenv & install pipenv
          command: |
            git clone git://github.com/pyenv/pyenv-update.git $(pyenv root)/plugins/pyenv-update
            pyenv update
            pyenv global 3.7.3
            sudo pip install pipenv            
      - run:
          name: setup & test-e2e
          command: |
            cd home-agent
            PIPENV_YES=true make setup test-e2e            
      - update_machine_cache

BONUS: Building and pushing images with docker

On the same project, I also had to manage the building and publishing of docker containers that we’d end up running in production to AWS’s Elastic Container Registry (ECR). That also took more time than I expected – so here’s what it took to get it done.

version: 2.1

workflows:

  publish:
    jobs:
      - build_and_publish_image:
          filters:
            branches:
              # publish images to ECR on pre-release branches
              only: /release-v([0-9]+\.*)+/
            tags:
              # only publish images to ECR when a version is tagged
              only: /v([0-9]+\.*)+/

commands:
  restore_pip_cache:
    steps:
      - restore_cache:
          key: pip-local-package-cache

  update_pip_cache:
    steps:
      - save_cache:
          # https://discuss.circleci.com/t/add-mechanism-to-update-existing-cache-key/9014/12
          # when saving, later epoch will ensure newer version is restored on next run
          key: pip-local-package-cache-{{ epoch }}
          paths:
            - "~/.cache/pip"
            - "~/.pip-cache"
            - "~/.local/share/virtualenvs"

  install_docker:
    steps:
      - run:
          name: install docker client
          command: |
            set -x
            VER="18.09.1"
            test -f /tmp/docker-client.tgz || sudo curl -L -o /tmp/docker-client.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
            sudo tar -xz -C /tmp -f /tmp/docker-client.tgz
            sudo mv /tmp/docker/* /usr/bin            

jobs:
  build_and_publish_image:
    docker:
      # NOTE: The circleci/python image is only used for quick access to 'pip',
      # build is done in docker via the 'docker-image' Makefile target called below
      - image: circleci/python:3.7.3
    steps:
      - checkout
      - install_docker
      - setup_remote_docker
      - restore_pip_cache
      - run:
          name: install awscli
          command: |
            sudo pip install --upgrade pip
            pip install --user awscli            
      - update_pip_cache
      - run:
          name: build docker image
          command: |
            cd home-agent && make docker-image            
      - run:
          name: Push to ECR
          command: |
            cd home-agent
            export AWS_ACCESS_KEY_ID=$ECR_PUSH_AWS_ACCESS_KEY_ID
            export AWS_SECRET_ACCESS_KEY=$ECR_PUSH_AWS_SECRET_ACCESS_KEY
            export PATH="${PATH}:${HOME}/.local/bin"
            export VERSION=`make print-version`
            export IMAGE_FULL_NAME=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$REPO_AND_IMAGE_NAME:$VERSION
            eval $(aws ecr get-login --region $AWS_REGION --no-include-email)
            docker push $IMAGE_FULL_NAME

As you might imagine, you’ll need to fill in a few variables via the CircleCI console here:

variable example description
ECR_PUSH_AWS_ACCESS_KEY_ID AKIAIOSFODNN7EXAMPLE The access key Id for an IAM accoun tthat is (hopefully) restricted in permissions
ECR_PUSH_AWS_SECRET_ACCESS_KEY wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY The secret access key for an IAM account that is (hopefully) restricted in permissions
AWS_ACCOUNT_ID 111111111111 The AWS account ID under which the ECR resides
AWS_REGION us-east-1 The AWS region you’re using
REPO_AND_IMAGE_NAME your-project/component The full name of your image
VERSION 0.1.0 The version of the image (in this example generated by the print-version Makefile target

Wrapup

Hopefully it’s not too difficult for poeple to work out but it sure did take me more time than expected! There’s a lot more that could be done to increase the speed and efficiency of these builds, but for now, this should still work for those starting from nothing.

I don’t often use python in my own projects, but hopefully this information helps out some that do.