LXC containers

Linux LXC containers

In this small howto, we'll setup LXC containers with networking via a bridge provided by virsh on a Debian Stretch host. Now, what exactly is LXC?

To quote linuxcontainers.org:

LXC is a userspace interface for the Linux kernel containment features.
Through a powerful API and simple tools, it lets Linux users easily create
and manage system or application containers.

Yeah, still don't get it? It's a technology allowing you to run several lightweight virtual machines (LXC container) on another machine (host). You could run a Fedora container on a Debian host for instance.

Is it similar to Docker? Well, Docker makes use of LXC containers.

Why would I need it? Some usecases:

  • Testing out another distribution
  • Testing software, installation, operation, distribution, ...
  • Isolating system services to one machine. For instance, 1 container for the mailserver, 1 for your http server, DNS, ...

1. Install LXC

We need lxc, and the libvirt-clients (previously libvirt-bin) for networking:

apt-get install lxc libvirt-clients libvirt-daemon-system

We are going to setup a bridge used by the lvm containers to connect to the internet. For this, we allow forwarding by setting ip_forward in sysctl and reloading it:

vi /etc/sysctl.conf
sysctl -p

To check the config:

grep ip_forward /etc/sysctl.conf
net.ipv4.ip_forward=1

2. Networking

We already installed the necessary packages. We'll use the default bridge for networking. See the default networksettings:

virsh net-info default

Mark the bridge as autostart:

virsh net-autostart default

To edit the network:

virsh net-edit default

To start:

virsh net-start default

More info on the default network:

cat /etc/libvirt/qemu/networks/default.xml

Logging can be found here:

cat /var/log/libvirt/libvirtd.log

Check if the default bridge is up. Using net-info:

virsh net-info default

Name:           default
UUID:           <long uuid>
Active:         yes
Persistent:     yes
Autostart:      yes
Bridge:         virbr0

Using ip:

ip a | grep virbr0

21: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
22: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000

3. Create a container

We'll create a stretch container:

sudo MIRROR=https://httpredir.debian.org/debian lxc-create -n stretch2 -t debian -- -r stretch -a amd64

where:

-n: name of the lxc container
-t: starting template to use, debian distribution
-r: release, stretch in this case
-a: amd64 architecture

This will invoke debootstrap to create the machine in /var/lib/lxc/stretch2. Edit the configuration to match the following:

# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template: -r stretch -a amd64
# Template script checksum (SHA-1): 2ad4d9cfe8988ae453172bd4fe3b06cf91756168
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

#lxc.network.type = empty
#lxc.cgroup.use = @all
lxc.rootfs = /var/lib/lxc/stretch2/rootfs
lxc.rootfs.backend = dir

# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf

# Container specific configuration
lxc.tty = 4
lxc.pts = 1024
lxc.utsname = stretch2
lxc.arch = amd64

lxc.autodev = 1
lxc.kmsg = 0
lxc.start.auto = 1

# If using apparmor
lxc.aa_allow_incomplete = 1

# Network
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = virbr0
lxc.network.hwaddr = 00:AA:BB:CC:00:04
#lxc.network.ipv4.gateway = auto

# Use virsh network bridge as gateway
lxc.network.ipv4.gateway = 192.168.122.1
lxc.network.ipv4 = 192.168.122.24

# Devices
# /dev/null and zero
lxc.mount.auto = cgroup:mixed
lxc.mount.entry = tmpfs dev/shm tmpfs rw,nosuid,nodev,create=dir 0 0
lxc.mount.entry = tmpfs run tmpfs rw,nosuid,nodev,mode=755,create=dir 0 0
lxc.mount.entry = tmpfs run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,create=dir 0 0
lxc.mount.entry = debugfs sys/kernel/debug debugfs rw,relatime 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir 0 0
lxc.mount.entry = hugetlbfs dev/hugepages hugetlbfs rw,relatime,create=dir 0
lxc.cgroup.devices.allow = c 1:3 rwm # dev/null
lxc.cgroup.devices.allow = c 1:5 rwm # dev/zero

# consoles
lxc.cgroup.devices.allow = c 5:1 rwm # dev/console
lxc.cgroup.devices.allow = c 5:0 rwm # dev/tty
lxc.cgroup.devices.allow = c 4:0 rwm # dev/tty0
lxc.cgroup.devices.allow = c 4:1 rwm # dev/tty1
lxc.cgroup.devices.allow = c 4:2 rwm # dev/tty2
lxc.cgroup.devices.allow = c 4:3 rwm # dev/tty3
lxc.cgroup.devices.allow = c 4:4 rwm # dev/tty4
lxc.cgroup.devices.allow = c 4:5 rwm # dev/tty5
lxc.cgroup.devices.allow = c 4:6 rwm # dev/tty6

# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm # dev/urandom
lxc.cgroup.devices.allow = c 1:8 rwm # dev/random
lxc.cgroup.devices.allow = c 136:* rwm # dev/pts/*
lxc.cgroup.devices.allow = c 5:2 rwm # dev/pts/ptmx

# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
lxc.cgroup.devices.allow = c 10:229 rwm

#lxc.cap.drop = sys_module
#lxc.cap.drop = mac_admin
#lxc.cap.drop = mac_override
#lxc.cap.drop = sys_time
#lxc.cap.drop = sys_admin

# Limits
lxc.cgroup.memory.limit_in_bytes = 512M
#lxc.cgroup.memory.mewsw.limit_in_bytes = 1024M
#lxc.cgroup.cpuset.cpus = 1

4. Start the container

To start the container as a daemon:

lxc-start -n stretch2 -d

If you encounter errors, use the logfile and logpriority parameter to get more info:

lxc-start -n stretch2 -d --logfile=lxc.log --logpriority=DEBUG

See the state of the container:

lxc-ls --fancy

NAME     STATE   AUTOSTART GROUPS IPV4                           IPV6
stretch1 RUNNING 1         -      192.168.122.23                 -
stretch2 RUNNING 1         -      192.168.122.24                 -

Check to see if the container interface connects to the bridge.

brctl show virbr0

bridge name bridge id       STP enabled     interfaces
virbr0      8000.725300141932   yes         veth1415A5
                                            veth1
                                            veth2
                                            virbr0-nic

We set the initial root passw to be able to login:

lxc-attach -n stretch2 passwd

Connect to the container. If you don't see a prompt, hit enter:

lxc-console -n stretch2 -t 1

Connected to tty 1
              Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself

Debian GNU/Linux 9 stretch2 tty1

stretch2 login:

Login and check the network, ping, etc:

ip a

To ping you might first have to install an additional package but defeats the purpose because if you can update the machine, the connection to the internet works, but here we go:

apt-get update
apt-get install iputils-ping

Check connectivity to other lxc containers, the bridge, local machine, ... Further update and customize the machine:

apt-get upgrade

5. Further info

The following information could come in handy if you encounter problems with your container(s).

5.1. fstab

You could create an fstab file in the container directory, in this case we don't. An example could be:

none /srv/sidtest/dev/pts devpts defaults 0 0
none /srv/sidtest/sys     sysfs  defaults 0 0
none /srv/sidtest/dev/shm tmpfs  defaults 0 0

5.2. console

If you have console problems, check the following. The devices list from a newly created container, handy in case of tty problems:

ls -la /dev

drwxr-xr-x  6 root root     540 jul  7 12:07 .
drwxr-xr-x 22 root root    4096 jul  7 12:07 ..
crw--w----  1 root tty  136, 12 jul  7 12:08 console
lrwxrwxrwx  1 root root      11 jul  7 12:07 core -> /proc/kcore
lrwxrwxrwx  1 root root      13 jul  7 12:07 fd -> /proc/self/fd
crw-rw-rw-  1 root root   1,  7 jul  7 12:07 full
drwxr-xr-x  2 root root       0 jul  7 12:07 hugepages
lrwxrwxrwx  1 root root      25 jul  7 12:07 initctl -> /run/systemd/initctl/fifo
lrwxrwxrwx  1 root root      28 jul  7 12:07 log -> /run/systemd/journal/dev-log
drwxrwxrwt  2 root root      40 jul  7 12:07 mqueue
crw-rw-rw-  1 root root   1,  3 jul  7 12:07 null
lrwxrwxrwx  1 root root      13 jul  7 12:07 ptmx -> /dev/pts/ptmx
drwxr-xr-x  2 root root       0 jul  7 12:07 pts
crw-rw-rw-  1 root root   1,  8 jul  7 12:07 random
drwxrwxrwt  2 root root      40 jul  7 12:07 shm
lrwxrwxrwx  1 root root      15 jul  7 12:07 stderr -> /proc/self/fd/2
lrwxrwxrwx  1 root root      15 jul  7 12:07 stdin -> /proc/self/fd/0
lrwxrwxrwx  1 root root      15 jul  7 12:07 stdout -> /proc/self/fd/1
crw-rw-rw-  1 root root   5,  0 jul  7 12:07 tty
crw-------  1 root tty  136,  0 jul  7 12:13 tty1
crw--w----  1 root tty  136,  1 jul  7 12:08 tty2
crw--w----  1 root tty  136,  2 jul  7 12:08 tty3
crw--w----  1 root tty  136,  3 jul  7 12:08 tty4
crw--w----  1 root tty  136,  4 jul  7 12:07 tty5
crw--w----  1 root tty  136,  5 jul  7 12:08 tty6
crw-rw-rw-  1 root root   1,  9 jul  7 12:07 urandom
crw-rw-rw-  1 root root   1,  5 jul  7 12:07 zero

If you have problems with tty, you might check /etc/securetty in the container:

...
# LXC (Linux Containers)
lxc/console
lxc/tty1
lxc/tty2
lxc/tty3
lxc/tty4

You should see agetty processes on the lxc container:

root        73  0.0  0.0  12668  1720 pts/3    Ss+  13:28   0:00 /sbin/agetty --noclear tty4 linux
root        75  0.0  0.0  12668  1568 pts/2    Ss+  13:28   0:00 /sbin/agetty --noclear tty3 linux
root        76  0.0  0.1  14308  2084 console  Ss+  13:28   0:00 /sbin/agetty --noclear --keep-baud console 115200,38400,9600 vt220
root        77  0.0  0.0  12668  1696 pts/1    Ss+  13:28   0:00 /sbin/agetty --noclear tty2 linux

Basic inittab settings:

grep ^[^#] /etc/inittab

id:3:initdefault:
si::sysinit:/etc/init.d/rcS
l0:0:wait:/etc/init.d/rc 0
l1:1:wait:/etc/init.d/rc 1
l2:2:wait:/etc/init.d/rc 2
l3:3:wait:/etc/init.d/rc 3
l4:4:wait:/etc/init.d/rc 4
l5:5:wait:/etc/init.d/rc 5
l6:6:wait:/etc/init.d/rc 6
z6:6:respawn:/sbin/sulogin
1:2345:respawn:/sbin/getty 38400 console
c1:12345:respawn:/sbin/getty 38400 tty1 linux
c2:12345:respawn:/sbin/getty 38400 tty2 linux
c3:12345:respawn:/sbin/getty 38400 tty3 linux
c4:12345:respawn:/sbin/getty 38400 tty4 linux
p6::ctrlaltdel:/sbin/init 6
p0::powerfail:/sbin/init 0

I found some info pointing to systemd if no getty is running. In the LXC container:

cp /lib/systemd/system/getty@.service /etc/systemd/system

Adjust getty@.service: comment the line "ConditionPathExists=/dev/tty0" in the copied getty@.service Restart the container.

5.3. iptables

When rebooting the host, the iptables firewall script was run when the network devices were up. Afterwards the LXC machines are booting in a particular order. This caused some service to not be avaiable until the firewall script was run again. A simple solution is to add a cron job after reboot to reload the firewall rules.

After running your iptables script or set-up, save the rules:

iptables-save > /etc/iptables.up.rules

Next, make a cron job to reload the iptables rules @reboot, but first wait some time to allow the LXC servers to boot:

cat /etc/cron.d/firewall_after_lxc_boot

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

@reboot   root    sleep 10;iptables-restore < /etc/iptables.up.rules

Off course, you still want to load the iptables rules from the moment the interfaces are up. This just reloads them to make the services available.