VOBox Container (Legacy)¶

This guide describes how to create a networked Docker container for VO-Box use. If possible, please refer to this guide instead.

Update: 10/01/20

Please be aware that the instructions listed below are outdated. A priviledged CentOS 6 container could previously be set up the same way as a traditional VOBox, as long as it was a MACVLAND bridge. Due to the introduction of systemd in CentOS 7, the setup procedure VOBoxes is now different for containers. To simplify setup, the following Dockerfiles can instead be user to quickly deploy preconfigured container VOBoxes:

HTCondor: https://gitlab.cern.ch/mstoretv/dockervobox
ARC/Generic: https://gitlab.cern.ch/mstoretv/dockervobox_arc

Requirements¶


CentOS	7.0 or later
Docker	1.12 or later (Tested on 17.03.1-CE a.k.a. "1.14")
CVMFS	Installed on the host

Hint

The commands listed below must be executed as root.

Setup Networking¶

Create a new MACVLAN bridge named docknet using the following command:

~# docker network create -d macvlan \
    --subnet=137.138.47.192/26 \
    --gateway=137.138.47.193 \
    --ipv6 \
    --subnet=2001:1458:201:b50e::/64 \
    --gateway=2001:1458:201:b50e::1 \
    -o parent=eth0 docknet

Command details

subnet/gateway must be replaced with the settings applicable to your network, while parent must be the network interface.
Why MACVLAN? Docker's normal approach to bridging is simply a NAT translation scheme. MACVLAN bridges give the containers direct access to the network.
Containers on this bridge will be assigned their own MAC addresses, and appear as conventional computers on the local network.
However, due to the way MACVLAN works, the host will not be able to ping the containers (and vice versa).

Create Container¶

Download the latest CentOS from Docker Hub (for CentOS 6.9, replace centos:latest with centos:centos6.9):
```
~$ docker pull centos:latest
```
Warning

Be sure to change the default root password! This is simply root in the above image.
Create and launch a new CentOS container connected to the MACVLAN bridge:
```
~$ docker run -it -v /cvmfs:/cvmfs \
-h myalienvo.cern.ch \
--name=myvocontainer \
--net=docknet \
--ip=137.138.47.251 \
--ip6=2001:1458:201:b50e::100:3e \
centos /bin/bash
```
Command details

-v mounts the directory /cvmfs from the host as /cvmfs within the container
-h sets the hostname myalienvo.cern.ch
--name sets the container name (as seen from Docker) to myvocontainer
--net attaches the container to the previously created MACVLAN bridge docknet
--ip/ip6 sets the desired network IPs for the container

Info

Docker gives containers limited access to the host kernel, which may prevent some tools from working properly. A possible workaround is to give the container additional privileges (see advanced topics for details).

As opposed to mounting CVMFS from the host, it is also possible to have it installed normally within the container. However, this will only work on a container with full access privileges. Be aware that unless otherwise configured, this will give each container its own CVMFS cache.

Container Usage¶

The following table lists useful container-specific commands:

Command	Description
`root@myalienvo: ~$ exit`	Exit running container Note: Docker will stop the container once this bash session is closed
`~$ docker start myvocontainer`	Start the container Note: This way, the container will remain active until you type `~$ docker stop myvocontainer`
`~$ docker exec -it myvocontainer /bin/bash`	Start a bash session within the container

From this point onward, proceed with configuring the VO-Box as you normally would on a VM, e.g. as a WLCG VOBOX. Once again, note that depending on the level of access given to the container, some tools may not function, and using service start may produce errors (see advanced topics for more information).

Useful Commands¶

The following table lists commands that may be of use when working with the VoBox containers.
Refer to command line Docker documentation for more details.

Command	Description
`~$ docker ps`	List active containers
`~$ docker ps -a`	List all containers
`~$ docker images`	List all images cached locally
`~$ docker commit myvocontainer myvoimagename`	Save container state as an image. With myimagename being the desired image name
`~$ docker export mycontainer > /home/myexportcontainer.tar`	Save the container as a tar file. With /home/myexportedcontainer.tar being the directory and name for the exported container

Docker commit vs export

The commit command will save the container as an image in Docker's internal format. It will also preserve the container's history, layers and metadata, and also create a new layer on top. Beware that an image will rapidly grow in size if it is continuously being committed, as a result of all the stacked layers.

The export command will save the container to a tar file, that can be reimported and used to launch new containers on the same machine, or uploaded and/or transferred elsewhere. However, it does not store any history or metadata, and will delete all previous layers except the current. Exporting and reimporting an image can thus be used to flatten a container that has gained a large file size, e.g. as a result of having previously used commit several times.

Advanced Topics¶

Setting DNS¶

When a container is restarted, Docker will overwrite its DNS configuration with a default one.
To avoid having to reapply the DNS settings on every container restart, this can be changed either by adding --dns=YOUR_DNS_IP to the run command in the create container step, or by changing the default DNS used by Docker. This can be done by creating the file /etc/docker/daemon.json with the following contents:

{
    "dns": ["YOUR_DNS_IP1", "YOUR_DNS_IP2"]
}

Hint

Be aware that Docker must be restarted for the changes to take effect.

Automated Service Start¶

Unlike a full OS in a virtual machine, Docker containers have no init system that can be used to automatically start services. One possible way to achieve this at container launch, is by using a Dockerfile. In essence, a Dockerfile contains a set of commands and arguments that will be performed on a given image. This file can thus contain a full list of commands to execute at launch, or simply point to a bash script, for example:

ROM mycontainerimage
ENTRYPOINT /etc/init.sh && /bin/bash

Here, mycontainerimage is the image to be used, and init.sh is a bash script to run at launch.

Example init.sh

#! /bin/bash

d=`date '+%y%m%d-%H%M%S'`
log=/tmp/init-$d.log

> $log && exec > $log 2>&1 < /dev/null

f=/etc/sysconfig/iptables
[ -r $f ] && iptables-restore < $f

service dnsmasq start

#
# hack: work around PID file corruptions...
#

service rsyslog stop

for i in 1 2
do
    echo == rsyslog $i
    sleep 5
    service rsyslog status || service rsyslog start
    perl -ni -e 'print if $. < 2' /var/run/syslogd.pid
done

svcs='
    crond
    sshd
    gsisshd
    alice-box-proxyrenewal
    condor
    autofs
'

#
# random ones have been found absent...
# we try up to 3 times for now
#

for i in 1 2 3
do
    for svc in $svcs
    do
     echo == $svc $i
     sleep 1
     service $svc status || service $svc start
    done
done

service fetch-crl-boot start > /tmp/fetch-crl-$$.log 2>&1 < /dev/null &

exit 0

Info

By running ~$ docker build -t mynewcontainerimage Docker will in this case produce a new image mynewcontainerimage based on mycontainerimage, where the contents of /etc/init.sh will be run at start. Note that the dockerfile must be named Dockerfile, and the previous build command executed in the same directory. The contents of Dockerfile will run independently of a container's access privileges, thus the /etc/init.sh file can contain commands such as service start, despite having no additional privileges.

Alternatively, all commands can be given within the Dockerfile, without having to reference a separate bash script. However, this will require the image to be rebuilt every time changes are added to the Dockerfile.

Privileged Mode¶

For the sake of isolation, access to the kernel is restricted within the containers. To enable some access for networking tools, such as iptables, add --cap-add=NET_ADMIN when creating the container. More access can be given through --cap-add=SYS_ADMIN, and full access can be given with --privileged.

Warning

Be aware that this will create pathways for potentially breaking container isolation.

Size Limits¶

The maximum container size can be limited by adding --storage-opt size=XXG when creating the container (replace "XX" with the desired storage capacity). However, note that Docker will most likely ask you to change its current storage driver to perform this.

Warning

Be aware that doing this requires deleting all stored containers. Before continuing, back up any containers you want to keep using docker export.

Access from Host¶

As mentioned earlier, the host will be unable to reach/ping the containers, and vice versa. A workaround is to create an additional ("normal") Docker bridge, and have the host / containers connect to it.

AutoFS Bug¶

CVMFS may respond with a "too many symbolic links" error when accessed from a new container or after a reboot. This is known as an autofs bug, and can be avoided by disabling autofs on the host and mounting your CVMFS directory manually. If you prefer to leave autofs enabled, you can remove the error by accessing a directory within CVMFS on the host (e.g /cvmfs/alice.cern.ch), and then restarting the container.

Warning

Be aware that this error will likely return on the next reboot.

HTCondor Error¶

HTCondor may throw the following error: Failed DISCARD_SESSION_KEYRING_ON_STARTUP.
This can fixed by setting DISCARD_SESSION_KEYRING_ON_STARTUP = false in the Condor config file.