======KVM and Docker on one host======
=====Background=====
I would like to have a single server which serves as Docker host and KVM at the same time using the same network bridges.
I am not going to use NAT in Docker and will use the same L2 bridges used by KVM.
I am going to use Ubuntu 18.04 LTS
======Base OS======
=====Prerequisites=====
  * Download Ubuntu 18.04 LTS from [[http://www.ubuntu.com]]

=====Installation=====
Choose Ubuntu minimal install and only select SSH server to be installed additionally.

=====Configure=====
We are going to use netplan for network configuration. In our example we have one network interface with multiple VLANs on it. We create bridge interfaces using these VLAN interfaces.

<code - /etc/netplan/01-netcfg.yaml>
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      accept-ra: no
      dhcp4: no
      dhcp6: no
  vlans:
    vlan8:
      id: 8
      link: eno1
      accept-ra: no
      dhcp4: no
      dhcp6: no
    vlan9:
      id: 9
      link: eno1
      accept-ra: no
      dhcp4: no
      dhcp6: no
    vlan10:
      id: 10
      link: eno1
      accept-ra: no
      dhcp4: no
      dhcp6: no
  bridges:
    mgmt:
      accept-ra: no
      addresses:
        - x.x.2.x/24
        - x:x:x:2::x/64
      gateway4: x.x.2.1
      gateway6: x:x:x:2::1
      nameservers:
        addresses: [ "x.x.x.x", "208.67.222.222" ]
        search: [ mgmt ]
      interfaces: [eno1]
    internet:
      accept-ra: no
      dhcp4: no
      dhcp6: no
      interfaces: [vlan8]
    dmz:
      accept-ra: no
      dhcp4: no
      dhcp6: no
      interfaces: [vlan9]
    local:
      accept-ra: no
      dhcp4: no
      dhcp6: no
      interfaces: [vlan10]
</code>
=====Remove unnecessary crap=====
  apt purge snapd

======KVM/libvirt======  
=====Installation=====
  apt install qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils virt-manager
  systemctl start libvirtd
  systemctl enable libvirtd
=====Configure=====
====Network====
Because we already have the bridges created with netplan, we only need to assign them to networking names in libvirt.
<code - /etc/libvirt/qemu/networks/internet.xml>
<network>
  <name>internet</name>
  <forward mode='bridge'/>
  <bridge name='internet'/>
</network>
</code>
<code - /etc/libvirt/qemu/networks/dmz.xml>
<network>
  <name>dmz</name>
  <forward mode='bridge'/>
  <bridge name='dmz'/>
</network>
</code>
<code - /etc/libvirt/qemu/networks/local.xml>
<network>
  <name>local</name>
  <forward mode='bridge'/>
  <bridge name='local'/>
</network>
</code>
<code - /etc/libvirt/qemu/networks/mgmt.xml>
<network>
  <name>mgmt</name>
  <forward mode='bridge'/>
  <bridge name='mgmt'/>
</network>
</code>
Now we have to activate the xmls
<code>
virsh net-destroy default
virsh net-undefine default
virsh net-define internet.xml
virsh net-define dmz.xml
virsh net-define local.xml
virsh net-define mgmt.xml
virsh net-start internet
virsh net-start dmz
virsh net-start local
virsh net-start mgmt
virsh net-autostart internet
virsh net-autostart dmz
virsh net-autostart local
virsh net-autostart mgmt
</code>
Check if everything is ok
  virsh net-list
<code>
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 dmz                  active     yes           yes
 internet             active     yes           yes
 local                active     yes           yes
 mgmt                 active     yes           yes
</code>
====Systemd====
Because we are going to use Docker with the same bridges, we have to wait for libvirt to start after Docker is started. Else the bridges are not available and autostart of VMs will fail.
<code>
cd /etc/systemd/system
rm libvirt-bin.service
cp /lib/systemd/system/libvirtd.service .
vi libvirtd.service
</code>
<code - /etc/systemd/system/libvirtd.service>
.
.
.
After=network.target
After=dbus.service
After=iscsid.service
After=apparmor.service
After=local-fs.target
After=remote-fs.target
After=systemd-machined.service
After=docker.service
Documentation=man:libvirtd(8)
Documentation=https://libvirt.org

[Service]
.
.
.
</code>
Now we need to reload systemd
  systemctl daemon-reload

======Docker======
=====Installation=====
  apt install docker.io
  systemctl enable docker
We need docker-compose and are going to install it via pip
  apt install python3-pip
Now install docker-compose
  pip3 install docker-compose
  
=====Configure=====
Docker is using iptables to mitigate traffic between containers. Because we are using VLANs with bridges, we do not want Docker to intervene network traffic.
<code - /etc/docker/daemon.json>
{
"iptables": false
}
</code>
When we stop docker to manipulate iptables it will not add the masquerade rule to the docker0 bridge. We need this iptables rule to make docker build work again. So first we are going to add the iptables rule manually. Find out which subnet is connected to docker0

  ip a s docker0
<code>
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b5:d4:a8:53 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:b5ff:fed4:a853/64 scope link 
       valid_lft forever preferred_lft forever
</code>  
In this case the subnet is 172.17.0.0/16. Add the rule
  iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -j MASQUERADE

Now we have the rule added. Make it persistent. First install iptables-persistent
  apt install iptables-persistent
It will tell you it created rules in: /etc/iptables/rules.v4 and /etc/iptables/rules.v6.
Check the rules.v4 file for the masquerade rule

  cat /etc/iptables/rules.v4
<code>
# Generated by iptables-save v1.6.1 on Fri Jul 26 18:19:47 2019
*nat
:PREROUTING ACCEPT [468:72707]
:INPUT ACCEPT [22:10893]
:OUTPUT ACCEPT [8:601]
:POSTROUTING ACCEPT [8:601]
-A POSTROUTING -s 172.17.0.0/16 -j MASQUERADE
COMMIT
# Completed on Fri Jul 26 18:19:47 2019
# Generated by iptables-save v1.6.1 on Fri Jul 26 18:19:47 2019
*filter
:INPUT ACCEPT [2328:191223]
:FORWARD ACCEPT [3836:8412380]
:OUTPUT ACCEPT [1473:233542]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri Jul 26 18:19:47 2019
</code>
====ZFS====
When you have a ZFS pool you want to use for storage
<code>
systemctl stop docker
rm -rf /var/lib/docker
zfs create -o mountpoint=/var/lib/docker data/docker
</code>
<code - /etc/docker/daemon.json>
{
  "storage-driver": "zfs",
  "iptables": false
}
</code>
<code>
systemctl start docker
</code>
Check with docker info
<code>docker info</code>
<code>
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.8
 Storage Driver: zfs
  Zpool: data
  Zpool Health: ONLINE
  Parent Dataset: data/docker
  Space Used By Parent: 98304
  Space Available: 965292744704
  Parent Quota: no
  Compression: off
.
.
.
</code>
====Networks====
In our example we are not going to use containers in the routed subnet bridge called internet
<code>
docker network create -d macvlan --subnet=x.x.0.0/24 --gateway=x.x.0.1 --ip-range=x.x.0.128/28 -o parent=local local
</code>
<code>
docker network create -d macvlan --subnet=x.x.1.0/24 --gateway=x.x.1.1 --ip-range=x.x.1.128/28 --subnet=x:x:x:1::/64 --gateway=x:x:x:1::1 --ip-range=x:x:x:1::128/124 -o parent=dmz --ipv6 dmz
</code>
<code>
docker network create -d macvlan --subnet=x.x.2.0/24 --gateway=x.x.2.1 --ip-range=x.x.2.128/28 --subnet=x:x:x:2::/64 --gateway=x:x:x:2::1 --ip-range=x:x:x:2::128/124 -o parent=mgmt --ipv6 mgmt
</code>
Check it
  docker network ls
<code>
NETWORK ID          NAME                DRIVER              SCOPE
6ae8a5b99638        bridge              bridge              local
700edcbcc28b        dmz                 macvlan             local
06d5b0706837        host                host                local
cb04700d3481        local               macvlan             local
b0bdff0a6d53        mgmt                macvlan             local
320e38274915        none                null                local
</code>

{{tag>linux}}