500 行代码搞定 Linux 容器

# 500 行代码实现 Linux 容器 Source: https://blog.lizzie.io/linux-containers-in-500-loc.html --- 1 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.subverting-capabilities) Erica Windisch 的 "Linux User Namespaces Might Not Be Secure Enough" (https://medium.com/@ewindisch/linux-user-namespaces-might-not-be-secure-enough-a-k-a-subverting-posix-capabilities-f1c4ae19cad#.3lbw4loa7) 说： > 如果一个（真正的）root 用户的 SYS_CAP_ADMIN 能力被移除了，但他随后创建了一个用户命名空间，那么该能力会被恢复给（虚假的）root 用户。也就是说，在创建命名空间之前，'mount' 操作会被拒绝，但在创建用户命名空间之后，'mount' 系统调用又会神奇地恢复工作——尽管是以一种受限的方式。虽然功能上有限制，但其严重程度足以说明：如果拥有（真正的）root 用户权限且内核启用了用户命名空间，Linux capabilities 可能会被完全绕过。 而 `man 7 user_namespaces` (http://man7.org/linux/man-pages/man7/user_namespaces.7.html) 则说： > 使用 CLONE_NEWUSER 标志通过 clone(2) 创建的子进程，在新的用户命名空间中会拥有一整套完整的 capabilities。 在 "Understanding and Hardening Linux Containers" (https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/april/ncc_group_understanding_hardening_linux_containers-1-1.pdf) 中又提到： > 用户命名空间还允许安全模型之间产生"有趣的"交叉，因为新的命名空间会被授予完整的 root capabilities。这可以使 CLONE_NEWUSER 有效地对其他（已暴露的）网络命名空间使用 CAP_NET_ADMIN，前提是容器并未使用（相应网络）。此外，正如我们多次看到的那样，拥有 CAP_NET_ADMIN 的进程具有巨大的攻击面，已经导致了许多不同的内核漏洞。这可能使得一个非特权的用户命名空间能够针对巨大的攻击面（内核网络子系统）进行攻击，而一个能力受限的特权容器则不会有这样的权限。详见第 39 页的 5.5 节，可获得更深入的讨论。 我们可以在一台启用了用户命名空间编译的主机上，通过清单 1 来演示这种行为：`subverting_networking.c` ``` /* Local Variables: */ /* compile-command: "gcc -Wall -Werror -static subverting_networking.c \*/ /* -o subverting_networking" */ /* End: */ #define _GNU_SOURCE #include <stdio.h> #include <sys/ioctl.h> #include <sys/socket.h> #include <sys/types.h> #include <sys/un.h> #include <unistd.h> int main (int argc, char **argv) { if (unshare(CLONE_NEWUSER | CLONE_NEWNET)) { fprintf(stderr, "++ unshare failed: %m\n"); return 1; } /* this is how you create a bridge... */ int sock = 0; if ((sock = socket(PF_LOCAL, SOCK_STREAM, 0)) == -1) { fprintf(stderr, "++ socket failed: %m\n"); return 1; } if (ioctl(sock, SIOCBRADDBR, "br0")) { fprintf(stderr, "++ ioctl failed: %m\n"); close(sock); return 1; } close(sock); fprintf(stderr, "++ success!\n"); return 0; } ``` ``` alpine-kernel-dev:~$ whoami lizzie alpine-kernel-dev:~$ ./subverting_networking ++ success! alpine-kernel-dev:~$ ``` 但我们实际上并没有那么强大。清单 2：`subverting_setfcap.c` ``` /* Local Variables: */ /* compile-command: "gcc -Wall -Werror -lcap -static subverting_setfcap.c \*/ /* -o subverting_setfcap" */ /* End: */ #define _GNU_SOURCE #include <stdio.h> #include <sys/capability.h> #include <unistd.h> int main (int argc, char **argv) { if (unshare(CLONE_NEWUSER)) { fprintf(stderr, "++ unshare failed: %m\n"); return 1; } cap_t cap = cap_from_text("cap_net_admin+ep"); if (cap_set_file("example", cap)) { fprintf(stderr, "++ cap_set_file failed: %m\n"); cap_free(cap); return 1; } cap_free(cap); return 0; } ``` ``` alpine-kernel-dev:~$ whoami lizzie alpine-kernel-dev:~$ touch example alpine-kernel-dev:~$ ./subverting_setfcap ++ cap_set_file failed: Operation not permitted ``` --- 2 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.turned-off-in-linux) `init/Kconfig:1207@c8d2bc` (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?id%3Dc8d2bc9bc39ebea8437fd974fdbc21847bb897a3#n1207) ``` config USER_NS bool "User namespace" default n help This allows containers, i.e. vservers, to use user namespaces to provide different user info for different servers. When user namespaces are enabled in the kernel it is recommended that the MEMCG option also be enabled and that user-space use the memory control groups to limit the amount of memory a memory unprivileged users can use. If unsure, say N. ``` --- 3 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.distros-userns) Ubuntu 启用了 `CONFIG_USER_NS`，但对其进行了补丁修改，使其非特权使用可以通过 sysctl（`unpriviliged_userns_clone`）来禁用。 清单 3：`92e575e769cc50a9bfb50fb58fe94aab4f2a2bff` (http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id%3D92e575e769cc50a9bfb50fb58fe94aab4f2a2bff) ``` commit 92e575e769cc50a9bfb50fb58fe94aab4f2a2bff Author: Serge Hallyn Date: Tue Jan 5 20:12:21 2016 +0000 UBUNTU: SAUCE: add a sysctl to disable unprivileged user namespace unsharing It is turned on by default, but can be turned off if admins prefer or, more importantly, if a security vulnerability is found. The intent is to use this as mitigation so long as Ubuntu is on the cutting edge of enablement for things like unprivileged filesystem mounting. (This patch is tweaked from the one currently still in Debian sid, which in turn came from the patch we had in saucy) Signed-off-by: Serge Hallyn [bwh: Remove unneeded binary sysctl bits] Signed-off-by: Tim Gardner ``` Debian 有相同的行为： 清单 4：`debian/patches/debian/add-sysctl-to-allow-unprivileged-CLONE_NEWUSER-by-default.patch` (https://anonscm.debian.org/git/kernel/linux.git/tree/debian/patches/debian/add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by-default.patch) ``` From: Serge Hallyn Date: Fri, 31 May 2013 19:12:12 +0100 (+0100) Subject: add sysctl to disallow unprivileged CLONE_NEWUSER by default Origin: http://kernel.ubuntu.com/git?p=serge%2Fubuntu-saucy.git;a=commit;h=5c847404dcb2e3195ad0057877e1422ae90892b8 add sysctl to disallow unprivileged CLONE_NEWUSER by default This is a short-term patch. Unprivileged use of CLONE_NEWUSER is certainly an intended feature of user namespaces. However for at least saucy we want to make sure that, if any security issues are found, we have a fail-safe. Signed-off-by: Serge Hallyn [bwh: Remove unneeded binary sysctl bits] --- ``` Grsecurity 则彻底禁用了非 `CAP_SYS_ADMIN`、`CAP_SETUID` 和 `CAP_SETGID` 用户的这项功能。 清单 5：https://grsecurity.net/test/grsecurity-3.1-4.7.9-201610200819.patch ``` --- a/kernel/user_namespace.c +++ b/kernel/user_namespace.c @@ -84,6 +84,21 @@ int create_user_ns(struct cred *new) !kgid_has_mapping(parent_ns, group)) return -EPERM; +#ifdef CONFIG_GRKERNSEC +/* + * This doesn't really inspire confidence: + * http://marc.info/?l=linux-kernel&m=135543612731939&w=2 + * http://marc.info/?l=linux-kernel&m=135545831607095&w=2 + * Increases kernel attack surface in areas developers + * previously cared little about ("low importance due + * to requiring "root" capability") + * To be removed when this code receives *proper* review + */ +if (!capable(CAP_SYS_ADMIN) || !capable(CAP_SETUID) || + !capable(CAP_SETGID)) + return -EPERM; +#endif ``` Arch Linux 则完全关闭了这项功能。 清单 6：{linux} 3.13 add CONFIG_USER_NS (https://bugs.archlinux.org/task/36969) ``` Comment by William Kennington (Webhostbudd) - Sunday, 06 October 2013, 03:55 GMT I agree with Florian, allowing non-root users to take advantage of elevating themselves to a local root seems like a huge attack surface. Preferably this would be a sysctl with a huge warning attached to it when it is switched on. Comment by Daniel Micay (thestinger) - Monday, 24 November 2014, 03:55 GMT [...] Arch doesn't add new features via patches. If you want to see this feature enabled, then land something like this upstream. Note that CONFIG_USER_NS is already enabled in the linux-grsec package because it fully removes the ability to have unprivileged user namespaces. ``` 如果能在这里包含 Red Hat 的补丁就好了，但我没有找到。 --- 4 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.man-clone) 本节大部分内容摘自 `man 2 clone` (http://man7.org/linux/man-pages/man2/clone.2.html) 底部的示例。 --- 5 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.clone-stack-temporary) 清单 11：`clone_stack.c` ``` /* -*- compile-command: "gcc -Wall -Werror clone_stack.c -o clone_stack" -*- */ #define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/wait.h> #include <unistd.h> #define STACK_SIZE (1024 * 1024) int child (void *_) { int stack_value = 0; fprintf(stderr, "pre-execve, stack is ~%p\n", &stack_value); execve("./show_stack", (char *[]) {",/show_stack", 0}, NULL); return 0; } int main (int argc, char **argv) { void *stack = malloc(STACK_SIZE); clone(child, stack + STACK_SIZE, SIGCHLD, NULL); wait(NULL); return 0; } ``` 清单 12：`show_stack.c` ``` /* -*- compile-command: "gcc -Wall -Werror -static show_stack.c -o show_stack" -*- */ #include <stdio.h> int main (int argc, char **argv) { int stack_value = 0; fprintf(stderr, "post-execve, stack is ~%p\n", &stack_value); return 0; } ``` ``` [lizzie@empress linux-containers-in-500-loc]$ ./clone_stack pre-execve, stack is ~0x7f3f98deefec post-execve, stack is ~0x7ffd14d2291c ``` 栈在 x86 上是向下增长的，因此 execve 后地址数值上变高这一事实表明已经分配了一个新的栈。 --- 6 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.pointer-addition-ub) 我原以为这可能是未定义行为，因为 `stack + STACK_SIZE` 确实指向了数组最后一个元素之后的位置。但 ISO-9899 (http://www.iso-9899.info/n1570.html) 中 6.5.6 节 [加法运算符] 的第 8 条涵盖了这一点： > 如果指针操作数和结果都指向同一数组对象的元素，或者指向数组对象最后一个元素之后的位置，则求值不得产生溢出；否则，行为是未定义的。如果结果指向数组对象最后一个元素之后的位置，则不得将其用作一元 * 运算符求值时的操作数。 也就是说，指针加法是合法的，但对它进行解引用则不行。 --- 7 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.pidns-sigkill) 我不确定 `waitpid` 是否足以等待该进程及其所有子进程，但当 pid 命名空间的根进程退出时，它的所有子进程都会收到 `SIGKILL`。`man 7 pid_namespaces` (http://man7.org/linux/man-pages/man7/pid_namespaces.7.html) 说： > 如果 PID 命名空间的 "init" 进程终止，内核将通过 SIGKILL 信号终止该命名空间中的所有进程。这一行为反映了 "init" 进程对 PID 命名空间正常运行是必不可少的这一事实。 在我找到这些之前，我自己也验证过了这一点： 清单 18：`persistent_child.c` ``` /* -*- compile-command: "gcc -Wall -Werror -static persistent_child.c -o persistent_child" -*- */ #include <stdio.h> #include <stdlib.h> #include <sys/stat.h> #include <sys/wait.h> #include <unistd.h> int main (int argc, char **argv) { switch (fork()) { case -1: fprintf(stderr, "++ fork failed: %m\n"); return 1; case 0:; int fd = 0; if ((fd = open("persistent_child.log", O_CREAT | O_APPEND | O_WRONLY, S_IRUSR | S_IWUSR)) == -1) { fprintf(stderr, "++ open failed: %m\n"); return 1; } size_t count = 0; while (count < 100) { if (dprintf(fd, "%lu\n", count++) < 0) { fprintf(stderr, "++ dprintf failed: %m\n"); close(fd); return 1; } sleep(1); } close(fd); return 0; default: sleep(2); return 0; } } ``` ``` [lizzie@empress l-c-i-500-l]$ touch persistent_child.log [lizzie@empress l-c-i-500-l]$ chmod 666 persistent_child.log [lizzie@empress l-c-i-500-l]$ sudo strace -f ./contained -m . -u 0 -c ./persistent_child execve("./contained", ["./contained", "-m", ".", "-u", "0", "-c", "./persistent_child"], [/* 15 vars */]) = 0 brk(NULL) = 0x605490 # ... [pid 736] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x6b68d0) = 2 strace: Process 746 attached [pid 736] nanosleep({2, 0}, [pid 746] open("persistent_child.log", O_WRONLY|O_CREAT|O_APPEND, 0600) = 3 [pid 746] fstat(3, {st_mode=S_IFREG|0666, st_size=4, ...}) = 0 [pid 746] lseek(3, 0, SEEK_CUR) = 0 [pid 746] write(3, "0\n", 2) = 2 [pid 746] nanosleep({1, 0}, 0x3fee2d718d0) = 0 [pid 746] fstat(3, {st_mode=S_IFREG|0666, st_size=6, ...}) = 0 [pid 746] lseek(3, 0, SEEK_CUR) = 6 [pid 746] write(3, "1\n", 2) = 2 [pid 746] nanosleep({1, 0}, [pid 736] <... nanosleep resumed> 0x3fee2d718d0) = 0 [pid 736] exit_group(0) = ? [pid 746] +++ killed by SIGKILL +++ [pid 736] +++ exited with 0 +++ # ... ``` 清单 19：`<>` += ``` close(sockets[1]); sockets[1] = 0; if (handle_child_uid_map(child_pid, sockets[0])) { err = 1; goto kill_and_finish_child; } goto finish_child; kill_and_finish_child: if (child_pid) kill(child_pid, SIGKILL); finish_child:; int child_status = 0; waitpid(child_pid, &child_status, 0); err |= WEXITSTATUS(child_status); clear_resources: free_resources(&config); free(stack); ``` 一个进程设置自己的用户命名空间能力是相当有限的⁸ (https://blog.lizzie.io/linux-containers-in-500-loc.html#fn.self-userns-limited)，因此父进程会等待子进程进入用户命名空间，然后向其 `uid_map` 和 `gid_map` 写入映射。 --- 8 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.self-userns-limited) 清单 20：`man 7 user_namespaces` (http://man7.org/linux/man-pages/man7/user_namespaces.7.html) ``` In order for a process to write to the /proc/[pid]/uid_map (/proc/[pid]/gid_map) file, all of the following requirements must be met: 1. The writing process must have the CAP_SETUID (CAP_SETGID) capability in the user namespace of the process pid. 2. The writing process must either be in the user namespace of the process pid or be in the parent user namespace of the process pid. 3. The mapped user IDs (group IDs) must in turn have a mapping in the parent user namespace. 4. One of the following two cases applies: * Either the writing process has the CAP_SETUID (CAP_SETGID) capability in the parent user namespace. + No further restrictions apply: the process can make mappings to arbitrary user IDs (group IDs) in the parent user namespace. * Or otherwise all of the following restrictions apply: + The data written to uid_map (gid_map) must consist of a single line that maps the writing process's effective user ID (group ID) in the parent user namespace to a user ID (group ID) in the user namespace. + The writing process must have the same effective user ID as the process that created the user namespace. + In the case of gid_map, use of the setgroups(2) system call must first be denied by writing "deny" to the /proc/[pid]/setgroups file (see below) before writing to gid_map. Writes that violate the above rules fail with the error EPERM. ``` --- 9 (https://blog.lizzie.io/linux-containers-in-500-loc.html#fnr.setgroups-setresuid) `gid`、`sgid` 和 `egid` 与 `struct cred` 中的 `group_info` 是分离的： 清单 22：`include/linux/cred.h:95@c8d2bc` (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/cred.h?id%3Dc8d2bc9bc39ebea8437fd974fdbc21847bb897a3#n95) ``` /* * The security context of a task * * The parts of the context break down into two categories: * * (1) The objective context of a task. These parts are used when some other * task is attempting to affect this one. * * (2) The subjective context. These details are used when the task is acting * upon another object, be that a file, a task, a key or whatever. * * Note that some members of this structure belong to both categories - the * LSM security pointer for instance. * * A task has two security pointers. task->real_cred points to the objective * context that defines that task's actual details. The objective part of this * context is used whenever that task is acted upon. * * task->cred points to the subjective context that defines the details of how * that task is going to be acted upon. */ ```