Installing a self-hosted GitLab server and testing Continuous Integration

Intro

In this tutorial we'll install the GitLab Community Edition repository manager in a virtual machine, setup the continuous integration tool that comes with it (GitLab CI) and push through Git a minimal test suite from the host OS to see how a successful/failed commit works.

To follow this you need to be comfortable with the Bash *nix shell and to have an idea about version control systems (VCS) and continuous integration (CI) through behavior-driven and/or test-driven development.

There are other ways to install GitLab but I've done it in the recommended way and tried to automate as much as I could. The idea is to create in the end an unattended install through Vagrant provisioning.

Let's see where this goes through the next steps!

Step 1 - create Vagrant virtual machine

To create the Ubuntu Linux 16.04 LTS (Xenial Xerus) 64-bit virtual machine where GitLab CE (and the other tools) are going to be installed we'll use Vagrant.

I'm using a Windows 7 host OS with Git Bash, here we go:

1. $ mkdir gitlab

2. $ cd gitlab

3. $ vagrant init ubuntu/xenial64

4. edit the generated 'Vagrantfile':

-> set the virtual machine RAM to 2GB minimum, I put 4GB for this test

config.vm.provider "virtualbox" do |vb|
 # # Display the VirtualBox GUI when booting the machine
 # vb.gui = true
 #
 # # Customize the amount of memory on the VM:
 vb.memory = "4096"
 end

-> add port forwarding for GitLab server - you can skip this step and leave port 80 if it's not in use by some other process on your localhost, if you do that don't forget to remove port 8080 from the next steps 🙂

config.vm.network "forwarded_port", guest: 80, host: 8080

5. $ vagrant up

6. $ vagrant ssh

Now we should be logged in the virtual machine and we can go to the next step.

Step 2 - add a swap file to our virtual machine

Even if you set more than 2GB of memory to the VM (4GB like I did) it is possible to run into 'out of memory' situations under heavy-load. Adding a 4GB swap file is a simple way to extend the total available memory.

1. create swap file:

$ sudo fallocate -l 4G /swapfile
 sudo: unable to resolve host ubuntu-xenial

If you encounter the annoying error above running sudo (don't worry, the command was executed even with this message!) you need to add the 'ubuntu-xenial' hostname to '/etc/hosts' for 127.0.0.1 IP:

a) $ sudo nano /etc/hosts
b) edit first line:

127.0.0.1 localhost ubuntu-xenial

c) save & exit

OR

we can automate this in a one-liner (and we will see during this tutorial how important is this ability in *nix systems to automate everything!):

$ sudo sed -i 's/127.0.0.1 localhost/127.0.0.1 localhost ubuntu-xenial/' /etc/hosts

2. check if swap file was created:

$ ls -lh /swapfile
 -rw-r--r-- 1 root root 4.0G Jun 1 20:20 /swapfile

3. change permission to swap file to be accessible only by 'root' user:

$ sudo chmod 600 /swapfile

4. verify new permissions:

$ ls -lh /swapfile
 -rw------- 1 root root 4.0G Jun 1 20:21 /swapfile

OR (because we are not logged in as 'root'):

$ cat /swapfile
 cat: /swapfile: Permission denied

5. set swap file as Linux swap space:

$ sudo mkswap /swapfile
 Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
 no label, UUID=ff93a772-4f06-48e8-b8fe-bc0ae3a732fc

6. make swap available to system:

$ sudo swapon /swapfile

7. check if swap was enabled:

$ sudo swapon --show
 NAME TYPE SIZE USED PRIO
 /swapfile file 4G 0B -1

8. check total available system memory:

$ free -m -t
 total used free shared buff/cache available
 Mem: 3951 42 3694 5 215 3861
 Swap: 4095 0 4095
 Total: 8047 42 7790

9. add swap to system init:

a) we can do this the manual way

$ sudo nano /etc/fstab

append this line to the end of file:

/swapfile none swap sw 0 0

save & exit

OR

b) we could do this without firing the nano editor (like a true 'enterprise automation junkie devops evangelist' - yes, such a funny job title really exists... 😛 )

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Step 3 - install GitLab Community Edition

1. install Postfix and other useful packages (setting before the Postfix configuration so we can do an unattended setup):

$ sudo debconf-set-selections <<< "postfix postfix/mailname string localhost"
 $ sudo debconf-set-selections <<< "postfix postfix/main_mailer_type string 'Internet Site'"

$ sudo apt-get install -y curl openssh-server ca-certificates postfix

2. run remote installation script of the GitLab CE repositories used by Ubuntu's 'apt-get install' command:

$ curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

3. $ sudo apt-get install gitlab-ce -y

4. $ sudo gitlab-ctl reconfigure

5. check that GitLab CE is installed and running with curl:

$ curl http://ubuntu-xenial

<html><body>You are being <a href="http://ubuntu-xenial/users/sign_in">redirected</a>.</body></html>

It's OK, we can access GitLab from the virtual machine!

9. set GitLab external url as 'ubuntu-xenial' instead of 'localhost':

$ sudo sh -c "sed -i 's|external_url '\''http://localhost'\''|external_url '\''http://ubuntu-xenial'\''|' /etc/gitlab/gitlab.rb"

10. reload GitLab configuration:

$ sudo gitlab-ctl reconfigure

11. edit 'hosts' file for host operating system, add this line:

127.0.0.1 ubuntu-xenial

12. access in host OS:

http://ubuntu-xenial:8080

13. set up 'root' password for GitLab - different from the 'root' user of the guest OS, for this example I've used something simple like 'rootgitlab':

gitlab_root_passwordThis can be automated in the virtual machine running:

$ sudo gitlab-rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=rootgitlab GITLAB_ROOT_EMAIL=test@email.com force=yes

After this you need to reconfigure GitLab again with:

$ sudo gitlab-ctl reconfigure

14. sign in with user 'root' and password 'rootgitlab' and you will see the GitLab Admin Area homepage:

gitlab_home

Step 4 - install GitLab Runner

To use continuous integration on your GitLab server you need to install a runner. This piece of software runs your tests and sends the results to GitLab. GitLab CI is the open-source continuous integration service included with GitLab that coordinates the testing.

1. $ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash

2. $ sudo apt-get install -y gitlab-ci-multi-runner

3. $ sudo gitlab-ci-multi-runner register

a) set gitlab-ci coordinator url to http://ubuntu-xenial/ci
b) enter gitlab-ci token from host OS page http://ubuntu-xenial:8080/admin/runners
c) leave gitlab-ci description as 'ubuntu-xenial', press [ENTER]
d) add a gitlab-ci tag like 'runner'
e) finally select the executor, we will use 'shell' for this tutorial

After completing all data you will receive this message:

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

4. refreshing the Runners page of the GitLab Admin Area in your host OS should look like this:

gitlab_runners

Step 5 - set up a Git project

1. access from host OS http://ubuntu-xenial:8080/projects/new to add your new project.

Let's name it 'test' and choose 'Public' for visibility to keep things simple.

The project will be accesible by Git on http://ubuntu-xenial:8080/root/test.git (make sure that you use the correct forwarded port).

2. open a Git console in the host OS and clone the 'test' project above:

$ git clone http://ubuntu-xenial:8080/root/test.git project
 Cloning into 'project'...
 warning: You appear to have cloned an empty repository.
 Checking connectivity... done.

3. $ cd project

4. add a file using your favorite editor, for example 'test.txt' with content '123abc'

5. $ git add test.txt

6. $ git commit -m 'first commit'

7. $ git push -u origin master (enter user: 'root', password: 'rootgitlab')

Counting objects: 3, done.
 Writing objects: 100% (3/3), 222 bytes | 0 bytes/s, done.
 Total 3 (delta 0), reused 0 (delta 0)
 To http://ubuntu-xenial:8080/root/test.git
 * [new branch] master -> master
 Branch master set up to track remote branch master from origin.

8. alright, let's check out first commit in the GitLab Admin Area on http://ubuntu-xenial:8080/root/test

gitlab_first_commit

Step 6 - setup Continous Integration

1. create a file named '.gitlab-ci.yml' in the root folder of the Git project created in the step above with content:

stages:
 - test

test:
 script:
 - bash test.sh

2. create a bash script called 'test.sh' that will be used to test exit codes. First we will use a command that returns the exit status 0 which is for successful execution.

content:

date

This is a simple and valid Linux command that will execute with no error.

3. add these 2 new files to Git:

$ git add .

4. commit files:

$ git commit -m 'continous integration - success'
 [master 12ad840] continous integration - success
 2 files changed, 7 insertions(+)
 create mode 100644 .gitlab-ci.yml
 create mode 100644 test.sh

5. push to GitLab server:

$ git push -u origin master
 Counting objects: 4, done.
 Delta compression using up to 4 threads.
 Compressing objects: 100% (3/3), done.
 Writing objects: 100% (4/4), 378 bytes | 0 bytes/s, done.
 Total 4 (delta 0), reused 0 (delta 0)
 To http://ubuntu-xenial:8080/root/test.git
 196d832..12ad840 master -> master
 Branch master set up to track remote branch master from origin.

6. we can see the successful commit at this address http://ubuntu-xenial:8080/root/test

gitlab_ci_passed7. edit 'test.sh' and use a command that will fail (invalid parameter):

date --fail

8. commit 'test.sh':

$ git commit -a -m 'continous integration - fail'
 [master 6c858eb] continous integration - fail
 1 file changed, 1 insertion(+), 1 deletion(-)

9. push to GitLab server:

$ git push -u origin master
 Counting objects: 3, done.
 Delta compression using up to 4 threads.
 Compressing objects: 100% (2/2), done.
 Writing objects: 100% (3/3), 336 bytes | 0 bytes/s, done.
 Total 3 (delta 0), reused 0 (delta 0)
 To http://ubuntu-xenial:8080/root/test.git
 12ad840..6c858eb master -> master
 Branch master set up to track remote branch master from origin.

10. we can see the failed commit refreshing the http://ubuntu-xenial:8080/root/test page:

gitlab_ci_failedYou can also find a list of these tests on the Builds page and access details with a click on status or build ID:

gitlab_ci_build_details

Step 7 - create Vagrant provisioning script

Vagrant comes with the powerful provisioning feature to automatically install software and make configurations in your virtual machine. In this step we will write a Bash script to install and setup GitLab as much as we can without user intervention (this is called an unattended installation).

Before we start this step we need to destroy the GitLab VM created above (don't worry it will be back up in short time) OR create a new folder and copy there the 'Vagrantfile'. 🙂

To destroy the VM run 'vagrant halt && vagrant destroy' in your host OS and confirm destruction.

1. create a file called 'gitlab.sh' in the same directory as your 'Vagrantfile' with this content:

#!/usr/bin/env bash

# create swap file

sudo sed -i 's/127.0.0.1 localhost/127.0.0.1 localhost ubuntu-xenial/' /etc/hosts
 sudo fallocate -l 4G /swapfile
 ls -lh /swapfile
 sudo chmod 600 /swapfile
 ls -lh /swapfile
 sudo mkswap /swapfile
 sudo swapon /swapfile
 sudo swapon --show
 free -m -t
 echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

# install GitLab

sudo debconf-set-selections <<< "postfix postfix/mailname string localhost"
 sudo debconf-set-selections <<< "postfix postfix/main_mailer_type string 'Internet Site'"
 DEBIAN_FRONTEND=noninteractive sudo apt-get install -y curl openssh-server ca-certificates postfix
 curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
 sudo apt-get install gitlab-ce -y
 sudo gitlab-ctl reconfigure
 sudo gitlab-rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=rootgitlab GITLAB_ROOT_EMAIL=test@email.com force=yes
 sudo sh -c "sed -i 's|external_url '\''http://localhost'\''|external_url '\''http://ubuntu-xenial'\''|' /etc/gitlab/gitlab.rb"
 sudo gitlab-ctl reconfigure

# install GitLab Runner

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh | sudo bash
 sudo apt-get install -y gitlab-ci-multi-runner

2. setup provisioning in 'Vagrantfile', we can add it under the 'config.vm.box' setting:

config.vm.box = "ubuntu/xenial64"
config.vm.provision :shell, path: "gitlab.sh"

4. $ vagrant up

5. access http://localhost:8080/ from host OS and login with user: 'root', password: 'rootgitlab'

OR

access http://ubuntu-xenial:8080/ if you didn't forgot to setup the hostname in your host OS like I did when writing this tutorial... 😀

6. setup the GitLab runner settings using the correct token, see Step 4.3

7. create your project and go to Steps 5 & 6 in this tutorial to setup Git and continuous integration

The 'Vagrantfile' and related Bash script 'gitlab.sh' are available on my GitHub, you can get them with:

$ git clone https://github.com/mihailj/gitlabci-ubuntu1604.git

Then enter the project directory, run $ vagrant up and you're good to go!

Outro

Alright, this is the end of this tutorial!

I've tried to keep it very simple but the opportunities and advantages of using Continuous Integration are huge. We can add unit testing to every modern language and make sure that our code pushed to Git passes these tests with visual confirmation in the GitLab Admin Panel.

THE END!

Links:

https://www.vagrantup.com/

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

https://about.gitlab.com/downloads/#ubuntu1604

https://about.gitlab.com/gitlab-ci/

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/install/linux-repository.md

http://www.thegeekstuff.com/2010/03/bash-shell-exit-status/

https://semaphoreci.com/community/tutorials/continuous-integration

https://www.vagrantup.com/docs/getting-started/provisioning.html

https://github.com/mihailj/gitlabci-ubuntu1604

Leave a Reply

Your email address will not be published. Required fields are marked *