I first learned about Travis-CI through venturing out into GitHub to checkout several open source projects. I've seen how they have continuous integration workflows set up test their code on every commit, branch, and pull request.
That seemed so convenient so I went off to try it myself. I've documented what I've learned below to help get you started!
My learning started with Travis-CI and am using it since I am familiar with it. I haven't done a detailed comparison of the tools and it fits the needs of simpler python CI/CD projects. I want to look into GitHub actions, now that it’s coming out of Beta.
You are free to use what you prefer, feel is right, or would like to start with. This article is not a comparison, but rather an introduction into what the process looks like with one CI service.
Start small and build up what you learn. You will make progress, run into challenges, figure some out. Remember to sleep!!!! Photo credit: 4geeksacademy.co/feelings-learning-coding/
It's an exhilirating, rewarding, and sometimes stressful day to day. 😬
There isn't a wrong option on your path to learning more about these tools and process. Where you start doesn't have to be where you are in the future. My recommendation is that you pick one, don't over think it, and get it up and working!
Put your work out there and connect with others. You don't have to create something from scratch. You can help your favorite existing projects set up or improve their CI/CD pipeline. It's better when we build together. 🌎🌈🌍🚀🌏
Knowledge compounds and you'll marvel at where you're at soon. 🗺️ 🚀 💻
You should have basic knowledge about Github, Python, testing frameworks, and how to use the Terminal. Knowledge of Pipenv (Pipfiles) and pytest is a plus, but not required. CI/CD tools are a great way to learn both. We will set up a Travis-CI account as well.
Most CI services interact with your repositories using a YAML configuration file.
In case you need a tune up on YAML, here is a recommended link:
Note: Ansible uses the YAML format, and is is another useful tool. A more detailed guide is an encouraged topic for another time.
Here is a full example
.travis.yml file from one of my projects. We will break down each section.
# .travis.yml dist: xenial language: python cache: pip python: - "3.6" - "3.7" - "3.8" - "nightly" matrix: allow_failures: - python: "nightly" install: - pip install pipenv --upgrade-strategy=only-if-needed - pipenv install --dev script: - bash scripts/test.sh after_script: - bash <(curl -s https://codecov.io/bash)
dist is where the Ubuntu Release codename is specified.
xenial is Ubuntu 16.04 LTS codename - see: http://releases.ubuntu.com/ for a full list.
This specifies the base operating system used for the rest of workflow.
language is the programming language used. This tutorial uses python, but it very well could be
cache allows for python package versions to be stored between runs, to speed up sequential builds. Cache can apply to more than just python packages.
python: - "3.6" - "3.7" - "3.8" - "nightly"
python, given the above language specification, is a key for a sequence of python versions to perform builds against. Generally most CI tools use the latest bug release version for each minor version. The build logs will tell you the specific versions.
matrix: allow_failures: - python: "nightly"
matrix allows for modifications in the above build sequence. In this case, the allow failures key is specifies a reference to the python sequence above, and has the value of
"nightly", meaning that that version is allowed to fail. Think of it as early detection and future python development. *Note: there are also development versions of each python version, such as
"3.8-dev". These are used when python version are in release candidate or beta stages.
install: - pip install pipenv --upgrade-strategy=only-if-needed - pipenv install --dev
install is the first section that is performed inside each of the build targets (virtual environments for each of the python versions above).
Each item in the sequence is executed in order!
In this example, we are using
pipenv and a
Most Pipfiles have production dependencies and development dependencies. You will often find the packages required to run test cases or build documentation under the
dev-packages section. Below is an example:
[[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] mypy = "*" black = "*" jupyter = "*" better-exceptions = "*" pytest = "*" pytest-cov = "*" isort = "*" flit = "*" autoflake = "*" mkdocs = "*" mkdocs-material = "*" [packages] pydantic = "==0.32.2"
pip install --dev installs all packages under
pip install would only install the packages under
Note: It is also common to use
pip install -r requirements.txt
Python.org now recommends
pipenv as part of its general tool recommendations.
Pipenv has the advantage of providing a mechanism to install your packages into a virtual environment without another package. There are many alternative - one simple alternaive is
A newer alternative is
poetry aims to facilitate what both requirements.txt and Pipfiles do not bundle, and that is released dependencies when deploying to PyPi. You will typically see
flit and a
pyproject.toml file, or a
setup.py file with redundant depedency information. That redundancy is what
poetry is aiming to improve upon.
Ensure you have a compatible python 🐍 environment on your computer.
python3 --version python3 -m pip --version python3 -m pytest --version
If you are missing any of the above, you can add the packages to your main python install.
Recommended reading on Python main install vs. altinstall. If you only have or want one python3 version on your machine, there is no harm to run the following commands. There are nuances between Ubuntu versions (16.04 vs. 18.04 vs. latest) on which python3 version is the default install.
sudo apt-get update sudo apt-get install python3-pip sudo apt-get install python3-pytest
Now that we have confirmed your base python setup, let’s go ahead and ensure the pipenv and cookiecutter python packages are installed.
python3 -m pip install --user pipenv python3 -m pip install --user cookiecutter
# make sure your path finds --user installs ## add `export PATH="$HOME/.local/bin:$PATH"` ## to your ~/.bashrc, ~/.zshrc file on linux cookiecutter https://github.com/iancleary/pypackage # For the sake of brevity, repos on GitHub can just use the 'gh' prefix cookiecutter gh:iancleary/pypackage
Fill out the wizard!
cd new-directory to change into the new directory you just created (replace
pipenv to display it's options. Take a look at what’s offered.
pipenv install --dev to install the production and development dependecies specified in the
pipenv shell to load the virtual environment.
I recommend you open up VS code or your text editor of choice to view the folder structure
./scripts/tests.sh from within the directory's Pipenv.
This command executes a bash script that does several things:
pytestto check test cases and check test coverage
These are all things that should be relegated to scripts to allow you to free your mind from simple formatting and other draining tasks. Let it do that work for you!
I recommend you follow the guide on the Github Help page:
Again, the documentation of both GitHub and Travis-CI are great to follow for this step.
Head over to Travis-CI and sign in with your GitHub account:
Once you login to Travis-CI and Enable GitHub Access, you'll want to configure Travis from your Github Settings page.
Make a simple change to the repo, either a commit, or open a new branch to start a build!
Grab your 🍿 and let's watch the CI process start and complete! 😎
It’s very cool to watch the process. The key to remember is the repeatability of the test suite.
Once complete, you will have your new project added to the Travis-CI dashboard. The three red boxes below are:
Your setup should give you and others confidence that the code will work under those conditions. If it’s an open source repo, there is no taking your word for it. The logs are right there!
You will not want to go back to your life before it!
It is such a quality improvement!
I haven't done much in this space yet, but I've seen other projects successfully use the flit python package.
If that is your next step, checkout
flit. I know there are some other good tutorials out there to get started inside and outside of a CI/CD pipeline.
I've been using Jupyter notebooks and Docker more and more recently. I significantly improved the maintenance model of my personal image by setting up an automated testing and deployment pipeline using Travis CI. Check out my personal Jupyter notebook repository!
Checkout the Docker image is iancleary/personal-notebook.
docker pull iancleary/personal-notebook
I also am in the process of learning how to leverage continuous deployment onto Amazon EC2 instances.
You can find my notes on connecting AWS CodeDeploy and Travis CI using a Flask project.
This example is a proof of concept, and I'm excited to use this workflow more in the future.