Eviltuzki

一线码农一枚

0%

Arm64麒麟V10安装Kata

发表于 分类于

前置条件检查是否支持虚拟化

  • 通过dmesg查看是否支持虚拟化，arm不同于x86，lscpu看不出来
    1
    2
    3
    4
    5
    6
    7
    8
    9
    # dmesg |grep kvm
    [    0.499391] kvm [1]: Hisi ncsnp: enabled
    [    0.499605] kvm [1]: 16-bit VMID
    [    0.499606] kvm [1]: IPA Size Limit: 48bits
    [    0.499644] kvm [1]: GICv4 support disabled
    [    0.499645] kvm [1]: vgic-v2@9b020000
    [    0.499647] kvm [1]: GIC system register CPU interface enabled
    [    0.500407] kvm [1]: vgic interrupt IRQ1
    [    0.501131] kvm [1]: VHE mode initialized successfully
  • 如果是下面这样，就不用继续了，浪费时间
    1
    2
    # dmesg |grep kvm
    [    0.136111] kvm [1]: HYP mode not available
  • 注意：Arm架构不支持嵌套虚拟化，也就是说只能在物理机下运行kata，无法在虚拟机下运行

下载&编译&安装

gcc (yum源7以上无需编译)

  • 编译qemu需要
  • 下载源码 
    1
    wget https://github.com/gcc-mirror/gcc/archive/refs/tags/releases/gcc-8.5.0.zip
  • 解压后，检查依赖项
    1
    ./contrib/download_prerequisites
  • yum安装依赖
    1
    # yum -y install bzip2 gcc gcc-c++ gmp-devel mpfr-devel libmpc-devel make zlib-devel flex bison-devel
  • 进入gcc8.5.0目录进行编译安装, make -j 后面的数字是编译的并行数，可适当调整
    1
    2
    3
    # mkdir build && cd build
    # ../configure --prefix=/opt/gcc-8.5.0 --enable-languages=c,c++ --disable-multilib
    # make -j8 && sudo make install
  • 进行软链接,软链前删除/usr/bin/下的cc和c++
    1
    2
    # ln -s  /opt/gcc-8.5.0/bin/gcc /usr/bin/cc
    # ln -s  /opt/gcc-8.5.0/bin/c++ /usr/bin/c++

rust

  • 编译kata需要
  • 直接在线安装
    1
    curl --proto '=https' --tlsv1.2 -sSf sh.rustup.rs | sh

golang

  • 编译kata需要
  • 直接下载二进制
    1
    wget https://studygolang.com/dl/golang/go1.19.3.linux-arm64.tar.gz
  • 解压后将bin目录添加至PATH即可

kata

  • 下载源码 
    1
    git clone https://github.com/kata-containers/kata-containers.git
  • 在kata-containers目录编译安装
    1
    2
    3
    4
    5
    $ pushd kata-containers/src/runtime
    $ make && sudo -E "PATH=$PATH" make install
    $ sudo mkdir -p /etc/kata-containers/
    $ sudo install -o root -g root -m 0640 /usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers
    $ popd

containerd

  • 直接下载对应的rpm包
    1
    2
    # yum install 	http://mirror.centos.org/altarch/7/extras/aarch64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm
    # yum install 	https://download.docker.com/linux/centos/7/aarch64/stable/Packages/containerd.io-1.5.11-3.1.el7.aarch64.rpm

python3.7 （yum源3.7以上无需编译）

  • 编译re2c需要

  • 下载源码

    1
    # wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz

  • 安装依赖

    1
    # yum install libffi-devel

  • 解压并编译

    1
    2
    3
    4
    5
    6
    7
    # tar -zxvf Python-3.7.0.tgz
    # mv Python-3.7.0 /usr/local
    # rm -rf /usr/bin/python
    # cd /usr/local/Python-3.7.0/
    # ./configure
    # make
    # make install

re2c

  • 编译ninja需要
  • 下载源码 
    1
    git clone https://github.com/skvadrik/re2c.git
  • 安装依赖 
    1
    # yum install automake libtool gcc gcc-c++
  • 编译安装,进入re2c目录 
    1
    2
    # autoreconf -i -W all
    # ./configure && make && make install

ninja

  • 编译qemu需要
  • 下载源码
    1
    git clone https://github.com/ninja-build/ninja.git
  • 编译，进入ninja目录
    1
    ./configure.py --bootstrap
  • 将ninja拷贝至PATH
    1
    mv ninja /usr/bin/

yq

  • 直接下载二进制就行
    1
    2
    https://github.com/mikefarah/yq/releases
    https://github.com/mikefarah/yq/releases/download/v4.30.4/yq_darwin_arm64

qemu

  • 可以通过tests脚本下载(是网络情况决定)
  • 提前下载源码,后续直接进行编译
    1
    2
    3
    4
    5
    git clone https://gitlab.com/qemu-project/qemu.git
    git clone https://gitlab.com/qemu-project/dtc.git
    git clone https://gitlab.com/qemu-project/meson.git
    https://gitlab.com/qemu-project/keycodemapdb.git

  • 这里采用kata-tests的脚本进行编译安装，具体见[kata-tests]

kata-tests

  • 进入kata-containers目录,下载源码tests

    1
    git clone https://github.com/kata-containers/tests.git

  • 调整目录结构（方便执行install_qemu脚本）

    1
    2
    3
    4
    5
    6
    7
    将kata-containers移动到/opt/kata/src/github.com/kata-containers

    /opt/kata/src/github.com/kata-containers/
        - tests
        - kata-containers (软连接至当前目录ln -s /opt/kata/src/github.com/kata-containers kata-containers)
        - 其他kata-containsers文件

    1
    将上面下载的qemu目录mv到/opt/kata/src/github.com/qemu
    1
    2
    /opt/kata (下面的GOPATH)路径下增加bin目录
    将yq二进制文件拷贝至bin目录下（跳过install_yq）

  • 修改脚本(必须)

    • 修改./kata-containers/tests/.ci/lib.sh, 增加如下
    • export GOPATH=上面调整的目录，我这里是/opt/kata

  • 修改脚本(可选，网速不好参考)

    • ./kata-containers/tests/.ci/install_qemu.sh
      • 方便调试增加set -x
    • ./kata-containers/tests/.ci/lib.sh
    • ./kata-containers/tests/.ci/aarch64/lib_install_qemu_aarch64.sh
      • 如果上面已经移动了qemu目录,执行下面的注释，否则不用执行
      • 注释掉clone_qemu_repo
      • 注释掉 sudo -E git fetch
    • ./kata-containers/tools/packaging/scripts/configure-hypervisor.sh
      • 如果提前clone了目录，增加如下，这样可以在config qemu的时候忽略子模块校验 
        1
        qemu_options+=' --with-git-submodules=ignore'

  • 执行安装

    1
    # ./kata-containers/tests/.ci/install_qemu.sh

kata-agent(可选)

kernel (除非你有arm内核，否则还是需要编译)

  • 下载对应版本的内核并解压
  • 进入内核目录执行
    make -j 8
  • 编译完成后拷贝./arch/arm64/boot/Image至对应目录
    • 可软链，默认/usr/share/kata-containers/vmlinux.container
  • 编译模块部分（需要内核开启模块，自己根据版本调整，如果guestImage需要使用则需要编译） 
    1
    2
    3
    4
    修改makefile的EXTRAVERSION 适配自己的版本，然后执行编译
    mkdir -p ../build/lib/modules/5.4.160-1.el7.aarch64
    make modules -j64
    make modules_install INSTALL_MOD_PATH=../build

guest Image (同kernel，如果有则无需编译)

  • 进入kata-containers/tools/osbuilder/rootfs-builder/centos, 根据自己rootfs选择

  • copy kernel modules （按需）

    1
    cp -r -d ${kernel}/../build/lib/modules/5.4.160-1.el7.aarch64/ lib/modules/5.4.160-1.el7.aarch64

  • 修改 config.sh(主要将yum或者dnf源修改为适配aarch64的，如果使用官方无需修改)

    1
    BASE_URL="https://mirrors.aliyun.com/centos/8-stream/BaseOS/aarch64/os/"

  • 增加rust加速config, 打到Docker镜像中

    1
    2
    3
    4
    5
    [source.crates-io]
    registry = "https://github.com/rust-lang/crates.io-index"
    replace-with = 'ustc'
    [source.ustc]
    registry = "git://mirrors.ustc.edu.cn/crates.io-index"

  • 修改Dockerfile(我这里rust加速，copy了config)

    1
    2
    增加
    COPY config /root/.cargo/config

  • 回到进入kata-containers/tools/osbuilder，执行编译(如果哪一步有超时，自己修改dns或者修改代理)

    1
    2
    make DISTRO=centos OS_VERSION=stream8 SECCOMP=no DEBUG=true USE_DOCKER=true AGENT_INIT=yes rootfs
    make USE_DOCKER=true image-centos -j 16

  • 拷贝编译好的Image(我这里是kata-containers-image-centos.img)

配置

kata

  • /etc/kata-containers/configuration.toml //过滤空行和注释后，我这里开启了debug模式 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    [hypervisor.qemu]
    path = "/usr/bin/qemu-system-aarch64"
    kernel = "/usr/share/kata-containers/vmlinux.container"
    image = "/usr/share/kata-containers/kata-containers.img"
    machine_type = "virt"
    enable_annotations = []
    valid_hypervisor_paths = ["/usr/bin/qemu-system-aarch64"]
    kernel_params = " initcall_debug"
    firmware = ""
    machine_accelerators=""
    cpu_features="pmu=off"
    default_vcpus = 1
    default_maxvcpus = 1
    default_bridges = 1
    default_memory = 2048
    disable_block_device_use = false
    shared_fs = "virtio-9p"
    virtio_fs_daemon = "/usr/libexec/kata-qemu/virtiofsd"
    valid_virtio_fs_daemon_paths = ["/usr/libexec/kata-qemu/virtiofsd"]
    virtio_fs_cache_size = 0
    virtio_fs_extra_args = ["--thread-pool-size=1"]
    virtio_fs_cache = "auto"
    block_device_driver = "virtio-blk"
    enable_iothreads = false
    enable_vhost_user_store = false
    vhost_user_store_path = "/var/run/kata-containers/vhost-user"
    valid_vhost_user_store_paths = ["/var/run/kata-containers/vhost-user"]
    valid_file_mem_backends = [""]
    pflashes = []
    enable_debug = true
    disable_image_nvdimm = true
    valid_entropy_sources = ["/dev/urandom","/dev/random",""]
    [factory]
    [agent.kata]
    enable_debug = true
    enable_tracing = true
    kernel_modules=[]
    debug_console_enabled = true
    [netmon]
    path = "/usr/libexec/kata-containers/kata-netmon"
    enable_debug = true
    [runtime]
    enable_debug = true
    internetworking_model="tcfilter"
    disable_guest_seccomp=true
    disable_selinux=false
    sandbox_cgroup_only=false
    sandbox_bind_mounts=[]
    vfio_mode="guest-kernel"
    experimental=[]
    [image]

containerd

  • /etc/containerd/config.toml 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    disabled_plugins = []

    [debug]
    #  address = "/run/containerd/debug.sock"
    #  uid = 0
    #  gid = 0
    level = "debug" # 我这里为了debug观察用

    [plugins]
    [plugins.cri.cni]
        conf_dir = "/etc/cni/net.d"
    [plugins.linux]
        shim_debug = true
    [plugins.cri]
        [plugins.cri.containerd]
        [plugins.cri.containerd.runtimes]
            [plugins.cri.containerd.runtimes.kata]
            runtime_type = "io.containerd.kata.v2"
            privileged_without_host_devices = true
            [plugins.cri.containerd.runtimes.kata.options]
            ConfigPath = "/etc/kata-containers/configuration.toml"   # 指定kata配置文件

cni

  • /etc/containerd/config.toml 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    {
        "cniVersion": "0.2.0",
        "name": "mynet",
        "type": "bridge",
        "bridge": "cni0",
        "isGateway": true,
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "subnet": "172.19.0.0/24",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ]
        }
    }

验证

check

1
2
3
4
# kata-runtime check
INFO[0000] IOMMUPlatform is disabled by default.
System is capable of running Kata Containers
System can currently create Kata Containers

启动容器

1
2
$ sudo ctr image pull docker.io/library/busybox:latest
$ sudo ctr run --cni --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh

Q&A 编译阶段

python

  1. ModuleNotFoundError: No module named ‘_ctypes’ 
    1
    2
    yum install libffi-devel 
    然后重新configure make make install

gcc

  1. g++: 错误：gengtype-lex.c：没有那个文件或目录 
    1
    yum install flex

qemu

  1. ERROR: glib-2.56 gthread-2.0 is required to compile QEMU 
    1
    yum install glib2-devel
  2. ERROR: Dependency “pixman-1” not found, tried pkgconfig 
    1
    yum install pixman-devel
  3. ERROR: Dependency “libseccomp” not found, tried pkgconfig 
    1
    yum install libseccomp-devel
  4. ERROR: C header ‘cap-ng.h’ not found 
    1
    yum install libcap-ng-devel
  5. ERROR: C shared or static library ‘rados’ not found 
    1
    yum install libcephfs-devel librbd-devel librados-devel

Q&A 运行阶段

  1. ctr: failed to create shim: failed to launch qemu: exit status 1, error messages from qemu log: qemu-system-aarch64: -device nvdimm,id=nv0,memdev=mem0: memory hotplug is not enabled: missing acpi-ged device : unknown 

    1
    2
    修改kata-container的configuration.toml
    disable_image_nvdimm = true      
    1
    2
    或者qemu应用下面补丁
    https://patchwork.kernel.org/project/qemu-devel/cover/20181018143042.29588-1-eric.auger@redhat.com/

  2. ctr: failed to create shim: Failed to Check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 2680247850:1024: unknown

    1
    内核文件问题，参考上面内核文件编译

  3. Err:Could not create the sandbox resource controller cgroups: cgroup mountpoint does not exist

    1
    2
    sudo mkdir /sys/fs/cgroup/systemd
    sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd