Automatically building, previewing, and pushing your book with CircleCI

CircleCI is a continuous integration service that lets you run various commands every time a new change is made to a repository. This can be used to build your book, preview changes, and even push live HTML as you update your book content.

In order to accomplish each of these, we'll use a CircleCI configuration file. This is a YAML file that is used to tell Circle what to do with your repository.

In each case, the expectation is that your master branch holds your book content.

We'll step through each piece of a sample CircleCI configuration to show you how to accomplish this.

Preparing CircleCI

First of all, you should set up your CircleCI account to start running CI jobs for your book repository. Follow these steps:

Tell CircleCI to build your repository. To do so, follow the CircleCI github integration instructions.
Tell CircleCI to build pull-requests to your repository. To do so, go to
```
https://circleci.com/gh/{{YOUR-GITHUB-ORG}}/{{YOUR-GITHUB-REPO}}/edit#advanced-settings
```
Find the "build forked pull requests" section, and switch it to ON.

Now, CircleCI will start watching your repository. If it finds a Circle configuration file (more information on this below), it'll run a CI job according to the configuration it finds.

You can copy/paste the empty CircleCI configuration here. This won't actually do anything but we'll add to it later:

# Tell CircleCI which version of its API you wish to use
version: 2.1
jobs:
    # Jobs define the different parts of your CircleCI workflow
    # They can depend on one another, use pieces from one another, etc.
    # We'll fill them in later
workflows:
    # Workflows tell CircleCI the order in which to run your jobs
commands:
    # Commands are re-usable chunks of steps that can be shared across jobs

Let's start filling out this template with a few jobs. In each case, we'll use a Python Docker image to both build each page's HTML, and build the book's HTML.

Step 1: Build each page's HTML

First you'll build each page's HTML. This is the initial conversion from ipynb, md, etc files. We'll use a Python container for this in order to use the Jupyter Book command-line interface with jupyter-book build.

You can build your book's HTML files and preview them using CircleCI artifacts. To do this, you'll need to use two CircleCI jobs:

We'll need to persist the results of this step so that they are available in subsequent steps. Here's the CircleCI configuration that will accomplish this, which you can add to the skeleton configuration you've created above:

jobs:
  build_page_files:
    docker:
      - image: circleci/python:3.6-stretch
    steps:
      # Get our data and merge with upstream
      - checkout

      # Install the packages needed to build our documentation
      # This will depend on your particular package!
      - run: pip install --user -r requirements.txt

      # Build the page intermediate HTML files
      - run:
          name: Build page intermediate HTML files
          command: jupyter-book build .

      # Persist the specified paths to be used in subsequent jobs
      # (see https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs)
      - persist_to_workspace:
          root: .
          paths:
            - ./_build/

Note that, at the end of this job, we've persisted the contents of the _build/ folder. This allows us to re-use these contents in subsequent jobs.

Step 2: Install Jekyll with Anaconda

Now that our page HTML files have been built, we can use Jekyll to build the HTML for our entire book. This is useful for two purposes:

previewing your Jupyter Book using CircleCI artifacts.
publishing the HTML of your book to someplace online

In both cases we need to install Jekyll and build the HTML for the book, so let's first define a CircleCI command to do this.

The following command will copy over the built page HTML from the previous job, install Miniconda which we'll use for environment management, then install Jekyll and the dependencies needed to build your book's HTML using conda-forge.

commands:
  prepare_jekyll_installation:
    steps:
      - checkout
      - attach_workspace:
          # Must be absolute path or relative path from working_directory
          at: /tmp/workspace

      # Grab the the built intermediate files from the last step
      - run:
          name: Copy over built site files
          command: |
            rm -rf _build
            cp -r /tmp/workspace/_build .

      # Install miniconda to test install
      - run:
          name: install miniconda
          command: |
            export MINICONDA=$HOME/miniconda
            echo "export PATH=$MINICONDA/bin:$PATH" >> $BASH_ENV
            source $BASH_ENV
            hash -r
            wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
            bash miniconda.sh -b -f -p $MINICONDA
            conda config --set always_yes yes
            conda update conda
            conda info -a
            conda create -n testenv python=3.7.0
            source activate testenv
            rm miniconda.sh

      # Install Ruby/Jekyll dependencies
      - run:
          name: Installing Ruby/Jekyll from conda-forge
          command: conda install -c conda-forge rb-github-pages

      # Build the book's HTML w/ the base_url for CircleCI artifacts
      - run:
          name: Install book Ruby dependencies
          command: bundle install

Step 3: Build and preview your book's HTML with Circle artifacts

We'll re-use the command from above in order to preview our built site using Circle artifacts. Add the job below to your CircleCI configuration file.

jobs:
  doc:
    docker:
      - image: circleci/python:3.7-stretch
    steps:
      - prepare_jekyll_installation
      - run:
          name: Build the website
          command: bundle exec jekyll build --baseurl /0/html/

      # Tell Circle to store the documentation output in a folder that we can access later
      - store_artifacts:
          path: _site/
          destination: html

After the Jekyll installation command, it then runs the build command from Jekyll, which outputs all the HTML for your site. We add the --baseurl /0/html because this is the prefix for the Jekyll artifact URL. Finally, we use the store_artifacts command to tell Jekyll to keep these artifacts for later. Once this job completes, you'll be able to click the "Artifacts" tab to preview your book HTML.

Step 4: Automatically publish live HTML from your master branch

You can also choose to automatically push built HTML from your master branch to a live textbook. This lets you automatically deploy changes to your book so that they go live online. Again, we'll use the page HTML command defined above. The job will be very similar to the HTML artifacts preview job, with an extra step to actually push the book's HTML online.

This step assumes that you are hosting your live book on a Git repository using GitHub Pages. We'll need the include a security key that allows CircleCI push access to your GitHub repository. You should first create an SSH key with write access to the repository that will be hosting your live site. Then, use the below configuration to tell Circle to automatically push to this repository.

jobs:
  deploy:
    docker:
      - image: circleci/python:3.7-stretch
    steps:
      # Add deployment key fingerprint for CircleCI to use for a push
      - add_ssh_keys:
          fingerprints:
            - "{{ YOUR SSH KEY FINGERPRINT }}"

      - prepare_jekyll_installation
      - run:
          name: Build the website for deploy
          command: bundle exec jekyll build

      # Deploy the built site with ghp-import
      - run:
          name: Deploying site using ghp-import
          command: |
            pip install ghp-import
            ghp-import -p -f -n ./_site/

In this case we've used the excellent ghp-import tool for pushing to github pages. The command pushes the contents of ./_site (your book's HTML) to the gh-pages branch of the repository. The -n flag adds a .nojekyll file to the built HTML, which ensures that Jekyll will treat it as raw HTML.

Step 4: Tying these workflows together

Now that we've defined several jobs above, we need to tell CircleCI how to use them sequentially (or in parallel). In particular, we want the job that builds each page's HTML to run first so that the each page's HTML can be stitched together into a book. Here's the configuration for this:

workflows:
  version: 2
  default:
    jobs:
      - build_page_html:
          filters:
            branches:
              ignore:
                - gh-pages
      - doc:
          requires:
            - build_page_html
          filters:
            branches:
              ignore:
                - gh-pages
      - deploy:
          requires:
            - build_page_html
          filters:
              branches:
                only:
                  - master
                ignore:
                  - gh-pages

Appendix: The full configuration file

Below is a full CircleCI configuration file that covers each of the steps above. Note that the syntax may be slightly different because we're putting each step above in a single file.

# NOTE: This is an example CircleCI configuration that
# will build your book and preview its HTML content.
# You will probably have to modify it in order to get it working
# just the way you want. See https://jupyterbook.org/advanced/circleci.html
# for more information
version: 2.1
jobs:
  build_page_html:
    docker:
      - image: circleci/python:3.7-stretch
    steps:
      - checkout
      - run: pip install --user -r requirements.txt
      - run:
          name: Build site intermediate files
          command: jupyter-book build .

      # Persist the built files for the deploy step
      - persist_to_workspace:
          root: .
          paths:
            - _build/

  doc:
    docker:
      - image: circleci/python:3.7-stretch
    steps:
      - prepare_jekyll_installation
      - run:
          name: Build the website
          command: bundle exec jekyll build --baseurl /0/html/

      # Tell Circle to store the documentation output in a folder that we can access later
      - store_artifacts:
          path: _site/
          destination: html

  # Deploy the built site to jupyter-book.github.io
  deploy:
    docker:
      - image: circleci/python:3.7-stretch
    steps:
      # Add deployment key fingerprint for CircleCI to use for a push
      - add_ssh_keys:
          fingerprints:
            - "{{ YOUR SSH FINGERPRINT }}"

      - prepare_jekyll_installation
      - run:
          name: Build the website for deploy
          command: bundle exec jekyll build

      # Deploy the built site with ghp-import
      - run:
          name: Deploying site using ghp-import
          command: |
            pip install ghp-import
            ghp-import -p -f -n ./_site/


# Tell CircleCI to use this workflow when it builds the site
workflows:
  version: 2
  default:
    jobs:
      - build_page_html:
          filters:
            branches:
              ignore:
                - gh-pages
      - doc:
          requires:
            - build_page_html
          filters:
            branches:
              ignore:
                - gh-pages
      - deploy:
          requires:
            - build_page_html
          filters:
              branches:
                only:
                  - master
                ignore:
                  - gh-pages

commands:
  prepare_jekyll_installation:
    steps:
      - checkout
      - attach_workspace:
          # Must be absolute path or relative path from working_directory
          at: /tmp/workspace

      # Grab the the built intermediate files from the last step
      - run:
          name: Copy over built site files
          command: |
            rm -rf ./_build
            cp -r /tmp/workspace/_build .

      # Install miniconda to test install
      - run:
          name: install miniconda
          command: |
            export MINICONDA=$HOME/miniconda
            echo "export PATH=$MINICONDA/bin:$PATH" >> $BASH_ENV
            source $BASH_ENV
            hash -r
            wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
            bash miniconda.sh -b -f -p $MINICONDA
            conda config --set always_yes yes
            conda update conda
            conda info -a
            conda create -n testenv python=3.7.0
            source activate testenv
            rm miniconda.sh

      # Install Ruby/Jekyll dependencies
      - run:
          name: Installing Ruby/Jekyll from conda-forge
          command: conda install -c conda-forge rb-github-pages

      # Build the book's HTML w/ the base_url for CircleCI artifacts
      - run:
          name: Install book Ruby dependencies
          command: bundle install