tl;dr - Undestand & use the short Ansible playbook at the bottom of this post

UPDATE end_play is broken in the latest version of Ansible 2.3, more details at the end of the blog pos

[core@coreos-101 ~]$ sudo rpm-ostree install python3
[core@coreos-101 ~]$ sudo systemctl reboot

After rebooting, you should be able to query the python version:

[core@coreos-101 ~]$ which python
/usr/bin/python
[core@coreos-101 ~]$ python --version
Python 3.7.7

[core@coreos-101 ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200505.3.0 (2020-05-19T13:39:46Z)
                BaseCommit: 01f074cc6cd88d8d2b43f821da692f2367c101eb4377802cb35092bde0ef02f7
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
           LayeredPackages: python3

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200505.3.0 (2020-05-19T13:39:46Z)
                    Commit: 01f074cc6cd88d8d2b43f821da692f2367c101eb4377802cb35092bde0ef02f7
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

Pat shared that he was in the process of moving from RancherOS to FedoraCoreOS and while I don't have any particular opinion ATM on RancherOS, I've long since moved on from CoreOS after the acquisition, I run regular Ubuntu for my cluster now.
Hopefully this correciton works out for anyone else who happens upon this article.
</p>

I have a bunch of posts that are backlogged (raw versions written but not pulled together & edited in anyway), but I’m going to skip the queue to get this post I think is highly useful out. If you’re out there messing with Kubernetes (‘k8s’), Ansible, and CoreOS, and trying to make them all work together you might find this post useful. There’s a HUGE post series coming on how I set up my Kubernetes cluster, so stay tuned for that.

Containers?

If you’re new to containers, this is almost certainly not the post to cut your teeth on the concepts, but I’ll try and sum it up for you:

• A “Container” are basically like big zip files with EVERYTHING it takes to run (and even build) software inside, whether that’s just one executable (if you have stuff that statically compiles), of a full linux OS like ubuntu and a bunch fo stuff installed.

• Containers were brought to the mainstream by Docker, but they depend on features that have been available at the OS-level for a while, depending on which OS you use. There’s also another often-used container implementation called rkt, it adheres to the standards of the Open Container Initiative (OCI)

• Containers make a new cross-language and cross-platform delivery option – no matter how you write your app, database, whatever, get it to run in a container and you can now run it anywhere. Imagine picking a fresh ubuntu install as a base, and codifying all the commands you need to get your app to run, from git pulls to file copies to apt-get installs. Now open your eyes, because that’s pretty much how it works – developers can build an image (the filesystem as a result of running all those commands) on their machine, and ship it to production and those steps AREN’T re-run – the production servers just load up the world that exists in the image, and does whatever you tell it to – for example running /usr/bin/node.

CoreOS

CoreOS is a operating system for the container-based future. It’s a minimal linux distribution without much except tools centered around running containers. It doesn’t even have a package manager (so no apt-get or yum), or GCC (or any tools to build code) bundled with it. No Python, no Ruby, no Perl, almost none of the tools you’re used to, except of course core utilites like mkdir, chmod, vi (no emacs :< ).

Some OSes go even further, like RancherOS which tried to actually run core system-level services as docker containers. RancherOS is a little extreme for my tastes, but it’s management tools (called Just “Rancher”) look pretty intuitive and interesting so I’ll be giving them a look soon.

CoreOS + Ansible

That very minimalism that makes CoreOS a great choice for the container-based future is the exact reason this blog post exists – CoreOS is so minimal that Python isn’t even installed, which means it’s hard to manage it with Ansible, a popular provision-by-ssh tool.

Up until now, setting up CoreOS and Kubernetes on a new dedicated server I purchased has been a lot of manual hacking and fixing – Outside of an initial Ignition configuration (a configuration that CoreOS uses to set up, once), nothing about my infrastructure is reproducible with code. That’s a big no-no for me these days, as I like to live a stress-free life with minimal time applying manual fixes to a server from the console, and you probably should too.

It’s paramount to use tools (like Ansible) which will SSH and do the hard work for me, when just blowing away the machine isn’t an option (yet). This concept is generally referred to as treating servers as pets or treating them as cattle. Pets you take care of, and you love, cattle less so. The goal these days is to treat your servers as cattle, which means you can easily off them, and replace them with another one in perfect condition (of course, this means you nail the process for creating one in perfect condition) that can do the work the removed one was doing.

Ideally, I would be able to very easily teardown my server and restart it, but I’m not on a cloud provider, I’m just using dedicated hardware so I can’t reach that level just yet. The next best thing is orchestrating as much as I can with Ansible.

CoreOS + Ansible + Kubernetes

In addition to just managing the system, what I want to do is manage Kubernetes running on the system. Since Kubernetes’s job is to abstract the system away (for the most part), it’s imperative that Ansible be able to work with it. This means I need to make sure Ansible can communicate the with server (running CoreOS), and influence the Kubernetes cluster (running as a service on the server itself). Ansible, however, requires Python (on the target server) to run the more advanced commands (basically every module except raw).

This means I need to install Python on CoreOS. Of course, it was off to google to see what others had done and I found some options:

1. [A guide to bootstrapping ansible on CoreOS] (https://coreos.com/blog/managing-coreos-with-ansible.html)
   • This didn’t really work… Not a huge fan of pulling in a bunch of code that just doesn’t work and hasn’t been updated in a year
2. [A script that installed Python on CoreOS] (https://github.com/ziozzang/python-on-coreos/blob/master/install-python-on-coreos.sh)
   • No HTTPS on the download link for ActivePython is a huge red flag
3. [Another repo that had CoreOS Python installation code] (https://github.com/judexzhu/Install-Python-on-CoreOs)
   • No HTTPS here either, basically the same as (2)

At this point, I started asking myself what ActivePython actually was – hopefully not some Python with an alternative backend (CPython works just fine for Ansible)? Luckily for me StackOverflow came to my rescue. So it’s basically just like python+stuff-you’re-gonna-download-later-anyways.

WARNING - In going with ActivePython, you’re actively choosing to download and run binaries built by someone else on your system. This is dangerous.

The scripts took a little work and weren’t quite right (probably just code rot, as links changed, etc), so I tried to install python myself using JUST the raw module. This was a big mistake because I realized at this point that CoreOS doesn’t even come with GCC! Gonna be real hard to build Python (a program) without a compiler like Clang or GCC for obvious reasons… Now I get why people are using ActivePython…

After a few seconds I inevitably asked myself “Why just use some bash script when I could make an Ansible task?”, and started writing the Ansibel playbook, after lots of googling, trying to figure out how to do very specific things in Ansible. Here are the fruits of my labor, hopefully it’ll save you some time!

The results

Ansible config changes

Since this method installs python to a different spot than ansible normally expects (/usr/bin/python is the default), you’ll need to tell ansible know where to find the python interpreter. I’ve also included some variables that will be used by the ansible play later.

The code below for me goes in group_vars/coreos.yaml, to target the coreos named hosts in my hosts.conf. If that statement confuses you, you should probably read up a tiny bit more on Ansible.

group_vars/coreos.yaml

---
ansible_ssh_user: core # whoever your root user is, coreos recommends 'core' so I used that instead of the usual 'root'
ansible_python_interpreter: "/opt/bin/python" # This is the path python will get installed to

# ap = ActivePython
ap_releases: https://downloads.activestate.com/ActivePython/releases
ap_python2_version: 2.7.13.2715
ap_python2_package_name: "ActivePython-{{ap_python2_version}}-linux-x86_64-glibc-2.12-402695"
ap_python2_package_file_name: "{{ap_python2_package_name}}.tar.gz"
ap_python2_package_download_url: "{{ap_releases}}/{{ap_python2_version}}/{{ap_python2_package_file_name}}"

Ansible Play

playbook.yaml

---
- name: setup python for coreos
  hosts: coreos
  gather_facts: False # would fail since Python isn't installed

  tasks:
    # Check if the python has already been downloaded, skip this whole play
    - block:
        - name: check for installed python2 (in opt/bin)
          raw: "[ -f /opt/bin/python2 ] && echo 'installed' || echo 'missing'"
          register: py2_check

        - meta: end_play
          when: py2_check.stdout_lines[0] == "installed"

    - name: check for downloaded ActivePython package
      raw: "[ -f {{ap_python2_package_file_name}} ] && echo 'present' || echo 'missing'"
      register: package_check

    - name: download ActivePython
      raw: wget {{ap_python2_package_download_url}}
      when: package_check.stdout_lines[0] == "missing"

    - name: check for unzipped ActivePython package folder
      raw: "[ -d {{ap_python2_package_name}} ] && echo 'present' || echo 'missing'"
      register: package_folder_check

    - name: unzip ActivePython
      raw: tar -xvf {{ap_python2_package_file_name}}
      when: package_folder_check.stdout_lines[0] == "missing"

    - name: create the resulting directory for ActivePython
      become: true
      raw: 'mkdir -p /opt/bin/active_python'

    - name: install ActivePython
      become: true
      raw: 'cd {{ap_python2_package_name}} && ./install.sh -I /opt/bin/active_python'

    # Setup links
    - name: link easy_install
      become: true
      raw: ln -sf /opt/bin/active_python/bin/easy_install /opt/bin/easy_install
    - name: link pip
      become: true
      raw: ln -sf /opt/bin/active_python/bin/pip /opt/bin/pip
    - name: link python
      become: true
      raw: ln -sf /opt/bin/active_python/bin/python /opt/bin/python
    - name: link python2
      become: true
      raw: ln -sf /opt/bin/active_python/bin/python /opt/bin/python2
    - name: link virtualenv
      become: true
      raw: ln -sf /opt/bin/active_python/bin/virtualenv /opt/bin/virtualenv

#
# After this, add the plays that do the actual stuff you need to do
#

UPDATE A reader emailed to tell me that end_play is actually broken in the latest version of Ansible 2.3, so it looks like this script won’t work as well as it should. One of the only ways I can think of to fix this is to just use only whens in the script, so this means moving the initial block to be just a regular command, andt hen making sure every subsequent command has a when: that checks package_check.stdout_lines[0] == "missing". I haven’t tried this yet, but wanted to at least forward the heads up I received (thanks again to the reader that submitted this!).

And that’s all it takes! These commands also do their bit to check and skip work when they can, which took a little time to set up properly. Note that I’m using features like block and meta: end_play which are only supported on relatively recent versions of Ansible, so you may need to backport if you’re on a significantly older version.