Introduction to Concourse

You probably wonder if I am this bored when I am about to write an article about Concourse but my main goal today is going to be to show you how easy, safe, and smooth it is and maybe why is it better than Jenkins? (I bet 90% of you closed this page after reading this). All jokes aside, let’s scratch the surface and see what is Concourse all about.

Why Concourse?

Concourse is an open-source-based CI pipeline system. Its common usage is, of course, CI/CD and it’s built originally to scale to any kind of automation pipeline you might want to make, doesn’t matter if it’s a simple pipeline or a complex one. It is written in GoLang and is using PostgreSQL as a backend. For the task that are being run Concourse is using containers. You can read more about how this mechanism works here.

If you compare it to Jenkins for instance which seems to be the most popular and the most used CI/CD tool out there the advantages are pretty much obvious instantly. Jenkins is plugin-based (often causing headaches and asking for a lot of maintenance), it doesn’t have that much strict isolation, it is very dependant and well, it is written in java.

It’s important to mention that Concourse can be configured as code. Nothing is required on the user interface level. Here is a good example:

resources:
- name: example
  type: git
  source: {uri: "https://github.com/example"}

jobs:
- name: unit
  plan:
  - get: example
    trigger: true
  - task: test
    file: example/ci/test.yml

What i also like about Concourse is a fact that CI is under source control. For example:

$ fly -t ci set-pipeline -p example -c pipeline.yml
$ vim pipeline.yml
$ fly -t ci set-pipeline -p example -c pipeline.yml
$ git add pipeline.yml
$ git commit -m "initial pipeline"

It is also worth mentioning that local iteration is incredibly fast. Not only that but you could bring in your own integrations from.

Architecture

So looking at the image above, let’s get into talking about the architecture of the Concourse. We have two types of nodes available, there is a Web and there is a Worker node. You can also see PostgreSQL that contained all the backend data as pipeline data, logs, etc… You will notice that in a Web node there are two components ATC and TSA. ATC is the heart of the Concourse basically. It runs a web user interface, API and it’s also responsible for pipeline scheduling as well. You can see in the picture that is connected with a database, of course, being “the heart” of the architecture logically. Further below we can see there are multiple components inside the ATC and each of those components have their own responsibility. There is a checker which is responsible for checking continuously whether there are changes in the pipeline. For example, if you push some changes in your repository, the checker should detect them and notify the pipeline about new changes. There is also a scheduler that as its own name says schedules the actions like build jobs for example. We also have a build tracker that is tracking each build uniquely. Last but not least is the famous garbage collector. Every modern application has a garbage collector nowadays so I am assuming you would know what it does but in case you don’t, it’s a cleanup mechanism that removes old, unused objects and resources like containers, or volumes.

Now moving on next component in our Web node, TSA. TSA is basically a custom built SSL server that is used for securely registering workers with ATC.

Now going further into a Worker node. The worker node as you can see has two components Garden and Baggageclaim. You will notice those components are connected to ATC via HTTP protocol. Garden is an API for container creation and management tool, how originally named ha? There is also a Baggageclaim that is a volume management API used for container management.

Fly CLI

This is in my opinion the most important part after “the heart” of course. This is “just another CLI tool” that is being used in the world of bazillion CLIs but nonetheless, it’s very useful. You can download the CLI tool from the Concourse homepage and get right into making your first pipeline. You will have to the login of course but the general workflow goes something like this:

Login:

$ fly -t knoldus login -c http://localhost:8080 -u admin -p admin

Create Pipeline:

$ fly -t knoldus set pipeline -p spring-boot-service -c pipeline.yml

Validate pipeline:

$ fly validate-pipeline -c pipeline.yaml

Trigger a job from a pipeline:

$ fly -t {target} trigger-job --job pipelineName/jobName

Delete a pipeline:

$ fly -t {target} destroy-pipeline -p pipelineName

Pipeline

The pipeline is what we get as the result of configured Jobs and Resources together. When you configure a pipeline, it gets created in order to continuously detect resource versions and queue new builds for the jobs as they get new inputs.

As you can see above, the pipeline is combined of Resources, Jobs and Tasks.

Resources are the kinda the main part of Councourse. They represent all the possible external inputs to and of course outputs of jobs in the pipeline.

Jobs decide the actions that will be taken in our pipeline. They decide how resources go through our actions and how the pipeline is visualized.

Tasks in Concourse are the smallest possible configurable units in our pipelines. They can be thought of as a function from task.inputs to task.outputs that can always either be successful or fail.

Hands-on

So let’s go on and start with the simplest task, running a Concourse introduction pipeline. We can start that very quickly with docker following steps here. Clone the repo and run these commands:

$ ./keys/generate
$ docker compose up -d
$ docker run concourse/concourse

Open localhost:8080 after that and you should see something like this:

If you haven’t already download Fly CLI by clicking on your operating system icon on the page. Simply follow all the instructions on the screen. I am on MacBook so all I did was:

$ brew install fly

Hint: some of you might have fly blocked:

It’s not virus so no worries, go to System Preferences -> Security & Privacy -> General and you will see this:

Click Allow Anyway and try running the command again. You will get notified again but this time just click Open.

So click on login or run:

$ fly -t tutorial login -c http://localhost:8080 -u test -p test

As you can see default username and password are “test”. So go on make a hello_world_task.yml file and paste this:

---
jobs:
- name: job
  public: true
  plan:
  - task: simple-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: { repository: busybox }
      run:
        path: echo
        args: ["Hello world!"]

Now go on and run:

fly -t tutorial execute -c hello_world_task.yml

After that you will see pipeline in your web interface at localhost:8080.

So that’s about it, this was a short introduction to Concourse. I hope you liked it and realized that it can be a serious competition for big players like Jenkins. Make sure you check some deployment strategies that could work great with these tools.