With qemu-user and binfmt you can even chroot into foreign CPU architectures, which is a handy thing to have when you mount your phone's eMMC to fix a hacking session gone wrong.
Though these days you may want to look into things like systemd-nspawn instead of plain chroot.
> you can even chroot into foreign CPU architectures, which is a handy thing to have when you mount your phone's eMMC
This sounds very interesting! What's the scenario where you'd do this? Would you be, for example, emulating an ARM processor with qemu on an x86 computer and chrooting into Android on an eMMC?
These days (with recent kernels) you don't even have to copy the qemu binary into the rootfs nor use a static binary - these used to be workarounds for things that kernel now handles on its own.
Well technically you can't, in the sense that chroot has no idea you're doing it... the magic of binfmt support in the kernel (setup for qemu by the post-install script for qemu-user-static I believe) basically lets you run any architecture's binary that qemu-user-* supports just by trying to execute it natively, it basically just translates the execution of "./other-arch-cmd" into "qemu-user-static ./other-arch-cmd".
Thank you. That makes sense. Even though there is a translation layer via qemu-user-static, still having the facilities to have that transparently is very fantastic. And also very fascinating and a revelation to learn about for a bearded old timer like me who has never seen it before.
Yeah, that’s how we customize the NVidia Orin BSP before flashing it. Untar the rootfs onto a fast x86-64 machine, chroot into it, and use qemu-user to run a bunch of Arm commands and build some stuff. Way easier than trying to set up a cross-compiler toolchain.
chroot can be a alternative to setting up fussy cross-compilers. It'll be slower, but that slowness pays off as long as you spend less time on extra CPU cycles than you do on getting cross compilers to work.
It's also useful for reverse-engineering router/IoT firmware.
Indeed you can skip most of the unnecessary additional mount requirements with systemd-nspawn. If /dev/sde1 (your root partition) is mounted to /tmp/rescue just run:
This brings back memories - I owned a large web hosting company and we had thousands of machines. When hardware issues came up, or machines wouldn't boot, using this method was our first line of defense - we'd boot the machine from a burned copy of "Recovery Is Possible" which was an all-in-one Linux distro for recovery, then mount the partitions and chroot in to figure out what is going on - or use rsync to migrate data off as needed.
Just looked and it looks like "Recovery Is Possible" hasn't been updated in a dozen years which dates my story, but I fondly remember overnight phone calls from panicked new sysadmins and telling them to be calm and "RIP it and get chrooted in" and then waking up to help them troubleshoot.
System Rescue is what i use these days, but to be honest i can’t recall using it in the last couple of year. Virtualization and containers have taken over.
System Rescue used to be Gentoo based and then switched to Arch. You can install both Arch and Gentoo with SR and obviously you can install Arch with a Gentoo install CD and vice versa.
SR also has some rather handy SAM database editing facilities. Mount Windows at /mnt and then enable and reset the Administrator password. Jolly handy for getting super duper user access on a Windows box.
Its been a while since I installed Gentoo but you can probably quite easily add more stuff to your Gentoo install CD. I don't know if Gentoo have added a script to do all the bind mounts but "arch-chroot /bin/bash" is very convenient. I used to forget about /sys.
There seems to be a fundamental mismatch between how sane people think about sandboxing, and how linux manages namespaces.
A linux-naive developer would expect to spawn a new process from a payload with access to nothing. It can't see other processes, it has a read only root with nothing in it, there are no network devices, no users, etc. Then they would expect to read documentation to learn how to add things to the sandbox. They want to pass in a directory, or a network interface, or some users. The effort goes into adding resources to the sandbox, not taking them away.
Instead there is this elaborate ceremony where the principal process basically spawns another version of itself endowed with all the same privileges and then gives them up, hopefully leaving itself with only the stuff it wants the sandboxed process to have. Make sure you don't forget to revoke anything.
A lot of things break if there's no /proc/self. A lot more things break if the terminfo database is absent. More things break if there's no timezone database. Finally, almost everything breaks if the root file system has no libc.so.6.
When you write Dockerfiles, you can easily do it FROM scratch. You can then easily observe whether the thing you are sandboxing actually works.
> no users
Now you are breaking something as fundamental as getuid.
The modern statically linked languages (I'm thinking of Go and Zig specifically) increasingly need less and less of the cruft you mentioned. Hopefully, that trend continues.
> no users
I mean running as root. I think all processes on Linux have to have a user id. Anything inside a sandbox should start with all the permissions for that environment. If the sandbox process wants to muck around with the users/groups authorization model then it can create those resources inside the sandbox.
The things that break in C if /proc/self or the terminfo DB are missing will break in Go and Zig too.
What I think you might mean is something like: "in modern statically linked applications written with languages like Go and Zig, it is much less likely for the them to call on OS services that require these sorts of resources".
Or capabilities. Additive security has been known for decades; Linux really dropped the ball here. Linux file descriptors (open file descriptions, whatever) are close to a genuine capability model, except there's plenty of leakage where you can get at the insecure base.
> Instead there is this elaborate ceremony where the principal process basically spawns another version of itself endowed with all the same privileges and then gives them up
The flags to unshare are copies of clone3 args, so you're actually free to do this. There's some song and dance though, because it's not actually possible to exec an arbitrary binary will access to nothing.
But I think the big discrepancy is that there is inherently a two step process to "spawn a new process with a new executable." Doesn't work that way - you clone3/fork into a new child process, inheriting what you will from the parent based on the clone args/flags (which could be everything, could be nothing), do some setup work, and then exec.
> There seems to be a fundamental mismatch between how sane people think about sandboxing, and how linux manages namespaces.
What bothers me most about sandboxing with linux namespaces is that edge cases keep turning up that allow them to trick the kernel into granting more privileges than it should.
I wonder if Landlock can/will bring something more like FreeBSD jails to the table. (I haven't made time to read about it in detail yet.)
There is the later added posix_spawn, which could be implemented with a system call, even if on Linux it is emulated with clone + exec.
posix_spawn can do much, but not all, of what is possible with clone + exec. Presumably the standard editors have been scared to add too complex function parameters for its invocation, though that should not have been a problem if all parameters had reasonable default values.
yup! FreeBSD jails are essentially what OP wants with chroot++.
I was pretty puzzled when Docker and LXC came around as this whole new thing believed to have "never been done before"; FreeBSD had supported a very similar concept for years before security groups were added in Linux.
Jails and ezjail were stellar to make mini no-overhead containers when running various services on a server. Being able to archive them and expand them on a new machine was also pretty cool (as long as the BSD version was the same.)
this whole new thing believed to have "never been done before";
Nobody with knowledge of sandboxing believed this, Virtuozzo and later OpenVZ had been on Linux for a long time after all. Virtuozzo was even from a similar time frame as FreeBSD jails (2000-ish).
The key innovation of Docker was to provide a standardized way to build, distribute, and run container images.
Virsh had worked for a long time before docker came around, but yeah… you essentially had to build your own Docker-like infrastructure that only you were using
Plan9 had a proper solution for this. New processes don't get access to any files by default - you have to explicitly mount directories for them, capability style.
It has nothing to do with weirdness; Unix itself was plenty weird for its time. The relevant difference between Unix™ and Plan 9 is that Unix source code was given away (or cheaply licensed) to hardware companies which all wrote their own operating systems on top (SunOS, Ultrix, HP-UX, etc. etc.). This made Unix the common factor of very many commercial workstation environments. Plan 9? It was sold directly as a commercial product, for no hardware platform in particular. Nobody wanted to buy it.
People liked Unix because it was free – either really free, via BSD, or as a Unix derivative provided at no cost when people bought their workstations. A new revolutionary operating system had absolutely no reason for anybody to buy it: No commercial developers wanted to develop to a platform without users, and no users wanted a platform without software.
Plan 9 only changed their license many years later, when it was too late for anybody to care, and Unix had become the established standard.
As a number of comments have noted, there are a bunch of different axes that chroot could be 'better' on - e.g. security and sandboxing.
I wrote https://github.com/aidanhs/machroot (initially forked from bubble wrap) a while ago to lean into the pure "pretend I see another filesystem" aspect of chroot with additional conveniences (so no security focus). For example, it allows setting up overlay filesystems, allows mounting squashfs filesystems with an overlay on top...and because it uses a mount namespace, means you don't need to tear down the mount points - just exit the command and you're done.
The codebase is pretty small so I just tweaked it with whatever features I needed at the time, rather than try and make it a fully fledged tool.
(honestly you can probably replicate most of it with a shell script that invokes unshare and appropriate mount commands)
ironically docker never gave you true network isolation because there's no way to make it user friendly. plus the many exploits on the all powerful daemon.
but most professional world use systemd to bootstrap isolated processes nowadays, which is kinda if what you are hinting at. cgroups2 and namespaces are what you want.
We should get a better change root yes, and the LD_LIBRARY_PATH gets autoupdated with respect to the new root. A few flags here and there to set permissions of the child process and we're off too the races.
Oh wow, wow. That all sounded so intensely complex, incomprehensible. What we are going to need to do is build a program to handle all that, highly formalized. Let's make it so formalized it's one of those things like taxes or AWS where people can just make a living from understanding the beast. It can be like systemd meets multics meets java. have it's own various complicated commands, complicated file formats, and so on. The chroot() is only historically understood by everyone, so let's steal a page from the java playbook and just rename everything with our own terminology. The product will be so outstanding, wow, I call it "Shocker"
I'm not GP, but if I were to hazard a guess, they want something more than just mount space isolation. Something akin to BSD jails, without the bells and whistles of OCI containers like overlay filesystem, network virtualization, resource management, etc.
That requirement is pretty legitimate, since its easier and suitable enough for many applications for which we currently use OCI containers. For example, isolated builds, development environments, sandboxes etc. (I have an isolated build tool for Gentoo).
But Linux already has multiple solutions that fit the bill, like systemd-nspawn, LXC, bubblewrap, etc. Too bad, they aren't as widely known as chroot.
None of those things do what chroot does but many of them involve chroot - so I'm still not grasping what "better chroot" is, other than "not chroot, but something completely different."
One annoying part of using chroot if you're creating them on the fly is teardown - you have to manually invoke umount, and also take care to get this right for partially created chroots (maybe you detected an error after mounting proc, in the process of getting other files in place).
This was my original motivation in creating machroot (mentioned elsewhere in this thread) and having it use namespaces.
I remember first installing Gentoo…printed the whole manual and followed along. After a failed attempt and second successful install I learned enough to know that I could use a LiveUSB of my choosing and chroot into the install. Good times!
On Gentoo, it is also simple to install a completely new system in a chroot environment, even if it is intended for a computer with a different, but compatible architecture.
This is frequently very convenient if you want to install Gentoo by compiling everything from sources on a cheap and small computer, e.g. one with some Intel Atom CPU.
Instead of compiling anything on the resource-constrained computer, you install a fresh Gentoo system for it, in a chroot environment on a fast desktop computer, which supports a superset of the ISA of the small computer, so you can still execute the programs intended for the target computer.
Then you just copy the installation result over the SSD/HDD of the destination computer. If you have many identical computers, you can copy the installed Gentoo over all of them without any problems, removing the need for multiple installations.
If desired, you can keep the chroot environment with the installation result and perform any later updates on it. Then you synchronize the updated Gentoo from the chroot with the one or more target computers.
Debootstrap is for installing a minimal system into a dir from another running system (so only useful in the setup phase), whereas arch-chroot is used to access an already installed system (e.g. when booting off a USB stick to fix something). They're not the same thing. The debootstrap-like tool in arch is called pacstrap.
i was very happy when that arch-chroot + pacstrap simplified my arch setup not having to run those mounts (i remember doin the same setup process for arch for centos/rhel just to not use anaconda and removing bloat, or rather installing what i wanted)
for btrfs, if you use a consistent volume mapping on your systems, its pretty easy. in arch setup i typically enable ssh and have a pretty simple set of bashisms for target device, compression, and mount points. then its copy-pasta since its a boiler plate for when i need to recover or fresh install
not glanced at manjaro so not familiar with its install-methods
This is a little extra. What you can generally do is immediatelly after chroot just run 'mount -a' to mount everything from the chroot's fstab. The empty `/boot` probably already exists.
I just installed Gentoo from WSL2 this way, on my secondary NVME on my Framework 16. Went without a hitch besides some minor things you need to be aware of:
-the lack of access to efi subsystem from wsl means you need to pass some extra flags to help grub/etc along, and you may need to set it as the boot partition in the bios manually
-you'll have to mount the drive to wsl with `wsl --mount <DiskPath> --bare`, after finding the right DiskPath with `Get-CimInstance -query "SELECT * from Win32_DiskDrive"`, and you might have to offline the disk first in Windows disk manager
Currently working on scripts to provision a drive outside of the machine it’s meant to go on using chroot. I’ve so far accidentally unmounted /dev/pts several times from the host system and running docker inside the chroot caused a hard lock. Fun stuff.
Used this or at least a very similar technique a few times to fix boot issues.
I've got a version of the mounting command that I think is easier to use:
for f in proc sys dev run dev/pts ; do mount --bind /$f /mnt/$f ; done
Change the "/mnt/$f" to whatever mountpoint that you're using which would be "/rescue/$f" to align with TFA.
I don't know what difference it makes to have /run mounted, but once you chroot into the mountpoint you can mount the boot partition etc and run whatever grub or mkinitramfs command you need to fix stuff.
I would leave the /boot mounting to later in the process - after you chroot. This way you can easily check /etc/fstab for where the boot partition lives (or if there is one), so you only need to locate the root partition initially which is generally easy to figure out from the disk sizes.
There's extra steps needed however if the system uses LVM.
chroot was (is?) the recommended way of installing gentoo which is pedagogical.
There are various handy (chroot) techniques that are probably considered "old school" now. For example, having a "rescue partition" which can be booted into remotely, and from there reinstall or repair the "main os". This is necessary when repartitioning remotely, for example.
That’s still how most Linux OS installers install. They just do that for you.
It’s still used for recovery but recovery partitions have kinda gone out of fashion as the ability to consistently boot has gotten better. Additionally, thumb drives and net boot make the partition a little less necessary.
It's kind of funny to see this being called a technique. It has been something I've done so many times to impressive results. To me, it's just something you can do, among a million other things, if you know enough about Linux.
The tools shouldn't require the use of chroot. It is quite common on Solaris to be able to specify a path that should be considered the root directory for the tool, usually -R. This was super useful in a netboot world using nfs, and also for crash recovery.
If a system is screwed up enough, then a chroot strategy won't work because it relies on the path you are chrooting to to be generally valid and functional. If it's missing libraries you may well be screwed.
Interesting question and I don't know how you'd do that - maybe using some statically linked binaries for fdisk and mkfs. However, why would you ever want to do that? If you want to remove data from the disk, then just running a suitable "dd" command would wipe the disk. I can't think of any other use case that wouldn't subsequently involve booting from a live iso which would let you format the disk anyway.
One use case is: what if the computer is at a remote location and you'd want to put a new image on the main disk. If Linux is currently using that disk as its root disk, then I suppose you'd have to somehow unmount it, or use a different instance of Linux that runs entirely in ram (so you can kill the original Linux instance and overwrite it).
I guess you could boot into a live installer ISO and get that to install over the root disk, though I don't know what problems you'd face with trying to unmount the original root partition.
If you're wanting to do something like that fairly regularly, then it'd be easier to just run virtual machines (e.g. QEMU/KVM) to do that kind of thing.
It's definitely possible. nixos-anywhere [0] implements this in a neat package for NixOS specifically. I am relatively certain you could use the same script to boot into a different distribution, though the installation part itself would obviously need to be adapted.
Came here to suggest this. pivot_root has the advantage that it can remount the existing root. It's essentially meant for going from an initramfs using linuxrc to boot to a full-on init in the other root filesystem. If things are horribly broken you could use a shell or some other process as init.
Ah - my attempt at doing this was almost there, except for the few bits that kept erroring out because I didn't mount the proc,dev,etc. mountpoints correctly :) Something to give another whirl on. At the time, I wanted something simple without docker to store everything development-related using different libraries. This is where debootstrap+chroot comes in - to build against various versions of libraries in Debian (if I remember correctly - it's been awhile.)
arch-chroot [1], despite its name pretty much does all the `mount -t proc` stuff the post says. It's also available on other distros like debian [2]. I have used it in the past to chroot into fedora as well.
Once I had to rescue a SunOS system and edit /etc/vfstab but I only had ed to my disposal. At that point I had never touched ed before, so that was a bit of a learning exercise, for sure.
In a computer lab I hung out in back in college...
We used rcp to keep passwords in sync. Add the account on the main machine, rcp the password file to the other machine. sudo rcp /etc/password other:/etc/passwd was muscle memory.
One day, someone was getting added to the groups file to be able to work in the server web project. sudo rcp /etc/group other:/etc/passwd
Ooops. Couldn't log in to fix it.
"Is anyone logged into the other machine?" (someone said yes). "Type while 1 sync" ... (ok) ... And we flipped the power switch and brought it up in single user mode (since the password file was invalid). Next, need to establish a minimal /etc/passwd ... emacs /etc/passwd (nope) vi /etc/passwd (nope - invalid terminal 300h not in termcap). "Uhm... cat > /etc/passwd ?" (possible, but a PITA when there is a typo in transcription)
I was a wizard on a lpmud. "I know ed".
And we got a minimal password file restored while reading the hashed values over (no way where we going to have root::0:0:... as the file even for a second) and then rcp'ed the proper /etc/passwd and /etc/group file over to the other machine.
Sure, chroot can be useful in a pinch, but it would not be the first thing I'd reach to. If you got the partitions mounted, you can already do quite a lot of things without needing to chroot.
Definitely weird, or perhaps I'm just not imaginative enough. I have a hard time figuring out what some KeyKOS constructs mean in familiar terms, particularly whether a given feature is an instance of essential or accidental complexity. I'm more familiar with today's seL4, which has carried on many of the principles of KeyKOS. Learning about the state of the art today is one thing, but learning history is quite another. I rely on simplicity in kernel design to avoid hindering more creative ideas that others have had.
This is how Linux From Scratch is done, and I've used the same technique to install missing video drivers that the Ubuntu install didn't include for whatever reason.
Which raises the question of why they need chroot if they have the VM.
Another semi-related but equally odd view: Docker is an operating system and containers are processes.
The difference between Docker and chroot, by the way, is that Docker does a bunch more system calls to ch the root of several other things that are not the filesystem. It also sets up things inside those roots for you by default, so your containers can access the internet, for example.
Basically, it runs docker containers using chroot to prove that it's possible.
Docker uses Layered Filesystems (Multiple Filesystems mounted under the same folder on top of one another), something that used to be mostly used when you have a read only Filesystem, like a cdrom, then mount a writeable folder over the same mount point to make that folder writeable.
Bocker does a similar mounting of the layers with chroot to run docker containers.
With qemu-user and binfmt you can even chroot into foreign CPU architectures, which is a handy thing to have when you mount your phone's eMMC to fix a hacking session gone wrong.
Though these days you may want to look into things like systemd-nspawn instead of plain chroot.
> you can even chroot into foreign CPU architectures, which is a handy thing to have when you mount your phone's eMMC
This sounds very interesting! What's the scenario where you'd do this? Would you be, for example, emulating an ARM processor with qemu on an x86 computer and chrooting into Android on an eMMC?
Here is an example of preparing a debian ARM image on x86 with debootstrap, qemu and chroot:
These days (with recent kernels) you don't even have to copy the qemu binary into the rootfs nor use a static binary - these used to be workarounds for things that kernel now handles on its own.
This is very cool! I had no idea you could chroot into different architectures.
Well technically you can't, in the sense that chroot has no idea you're doing it... the magic of binfmt support in the kernel (setup for qemu by the post-install script for qemu-user-static I believe) basically lets you run any architecture's binary that qemu-user-* supports just by trying to execute it natively, it basically just translates the execution of "./other-arch-cmd" into "qemu-user-static ./other-arch-cmd".
Thank you. That makes sense. Even though there is a translation layer via qemu-user-static, still having the facilities to have that transparently is very fantastic. And also very fascinating and a revelation to learn about for a bearded old timer like me who has never seen it before.
I've done this to build custom RPi images. Way faster than trying to build on a low power ARM platform, and way less fragile than cross compilers.
Same here, and it is blazingly fast for me running on a M2 macbook air using macOS built in virtulization framework to run an arm64 Debian.
Probably even faster on Asahi Linux, but having both macOS and a fast Debian at the same time is soo neat :)
Perhaps it's fast because M2 is already arm64.
Yeah, that’s how we customize the NVidia Orin BSP before flashing it. Untar the rootfs onto a fast x86-64 machine, chroot into it, and use qemu-user to run a bunch of Arm commands and build some stuff. Way easier than trying to set up a cross-compiler toolchain.
I've done this on some tinker boards we use at work.
systemd-nspawn is a great tool for this.
chroot can be a alternative to setting up fussy cross-compilers. It'll be slower, but that slowness pays off as long as you spend less time on extra CPU cycles than you do on getting cross compilers to work.
It's also useful for reverse-engineering router/IoT firmware.
I used it with Debian on my phone, but yes.
yeah exactly that. your laptop is x86 and your phone or raspberry pi or other hardware is not.
Indeed you can skip most of the unnecessary additional mount requirements with systemd-nspawn. If /dev/sde1 (your root partition) is mounted to /tmp/rescue just run:
systemd-nspawn --directory /tmp/rescue --boot -- --unit rescue.target
It should automatically find the boot partition and mount it as well.
Of course you don't need it, but if you're into docker images, docker or podman can be configured to use QEMU when running a container too.
This brings back memories - I owned a large web hosting company and we had thousands of machines. When hardware issues came up, or machines wouldn't boot, using this method was our first line of defense - we'd boot the machine from a burned copy of "Recovery Is Possible" which was an all-in-one Linux distro for recovery, then mount the partitions and chroot in to figure out what is going on - or use rsync to migrate data off as needed.
Just looked and it looks like "Recovery Is Possible" hasn't been updated in a dozen years which dates my story, but I fondly remember overnight phone calls from panicked new sysadmins and telling them to be calm and "RIP it and get chrooted in" and then waking up to help them troubleshoot.
System Rescue is what i use these days, but to be honest i can’t recall using it in the last couple of year. Virtualization and containers have taken over.
https://www.system-rescue.org/
I use my Gentoo installation usb when this comes up
System Rescue used to be Gentoo based and then switched to Arch. You can install both Arch and Gentoo with SR and obviously you can install Arch with a Gentoo install CD and vice versa.
SR also has some rather handy SAM database editing facilities. Mount Windows at /mnt and then enable and reset the Administrator password. Jolly handy for getting super duper user access on a Windows box.
Its been a while since I installed Gentoo but you can probably quite easily add more stuff to your Gentoo install CD. I don't know if Gentoo have added a script to do all the bind mounts but "arch-chroot /bin/bash" is very convenient. I used to forget about /sys.
Yes, they do have such script.
Hetzner also chroots you into a recovery linux and mounts your drive for access. Same thing different scale.
I actually wish that instead of docker & etc we had just gotten a better chroot... Or maybe just a new kernel syscall that is chroot()++.
There seems to be a fundamental mismatch between how sane people think about sandboxing, and how linux manages namespaces.
A linux-naive developer would expect to spawn a new process from a payload with access to nothing. It can't see other processes, it has a read only root with nothing in it, there are no network devices, no users, etc. Then they would expect to read documentation to learn how to add things to the sandbox. They want to pass in a directory, or a network interface, or some users. The effort goes into adding resources to the sandbox, not taking them away.
Instead there is this elaborate ceremony where the principal process basically spawns another version of itself endowed with all the same privileges and then gives them up, hopefully leaving itself with only the stuff it wants the sandboxed process to have. Make sure you don't forget to revoke anything.
> a read only root with nothing in it
A lot of things break if there's no /proc/self. A lot more things break if the terminfo database is absent. More things break if there's no timezone database. Finally, almost everything breaks if the root file system has no libc.so.6.
When you write Dockerfiles, you can easily do it FROM scratch. You can then easily observe whether the thing you are sandboxing actually works.
> no users
Now you are breaking something as fundamental as getuid.
The modern statically linked languages (I'm thinking of Go and Zig specifically) increasingly need less and less of the cruft you mentioned. Hopefully, that trend continues.
> no users
I mean running as root. I think all processes on Linux have to have a user id. Anything inside a sandbox should start with all the permissions for that environment. If the sandbox process wants to muck around with the users/groups authorization model then it can create those resources inside the sandbox.
The things that break in C if /proc/self or the terminfo DB are missing will break in Go and Zig too.
What I think you might mean is something like: "in modern statically linked applications written with languages like Go and Zig, it is much less likely for the them to call on OS services that require these sorts of resources".
That is pretty much what jails are in FreeBSD, especially thin jails.
Or capabilities. Additive security has been known for decades; Linux really dropped the ball here. Linux file descriptors (open file descriptions, whatever) are close to a genuine capability model, except there's plenty of leakage where you can get at the insecure base.
> Instead there is this elaborate ceremony where the principal process basically spawns another version of itself endowed with all the same privileges and then gives them up
The flags to unshare are copies of clone3 args, so you're actually free to do this. There's some song and dance though, because it's not actually possible to exec an arbitrary binary will access to nothing.
But I think the big discrepancy is that there is inherently a two step process to "spawn a new process with a new executable." Doesn't work that way - you clone3/fork into a new child process, inheriting what you will from the parent based on the clone args/flags (which could be everything, could be nothing), do some setup work, and then exec.
> There seems to be a fundamental mismatch between how sane people think about sandboxing, and how linux manages namespaces.
What bothers me most about sandboxing with linux namespaces is that edge cases keep turning up that allow them to trick the kernel into granting more privileges than it should.
I wonder if Landlock can/will bring something more like FreeBSD jails to the table. (I haven't made time to read about it in detail yet.)
This is why I would still rather isolate using QEMU, docker, or Virtually Box rather than a very think chroot-like environment
Docker uses namespaces by default. Are you using an add-on that makes it use a hypervisor instead?
I believe this is because on POSIX systems the only way to create a new process is fork().
There is the later added posix_spawn, which could be implemented with a system call, even if on Linux it is emulated with clone + exec.
posix_spawn can do much, but not all, of what is possible with clone + exec. Presumably the standard editors have been scared to add too complex function parameters for its invocation, though that should not have been a problem if all parameters had reasonable default values.
Come to FreeBSD, we have just that - jails.
yup! FreeBSD jails are essentially what OP wants with chroot++.
I was pretty puzzled when Docker and LXC came around as this whole new thing believed to have "never been done before"; FreeBSD had supported a very similar concept for years before security groups were added in Linux.
Jails and ezjail were stellar to make mini no-overhead containers when running various services on a server. Being able to archive them and expand them on a new machine was also pretty cool (as long as the BSD version was the same.)
this whole new thing believed to have "never been done before";
Nobody with knowledge of sandboxing believed this, Virtuozzo and later OpenVZ had been on Linux for a long time after all. Virtuozzo was even from a similar time frame as FreeBSD jails (2000-ish).
The key innovation of Docker was to provide a standardized way to build, distribute, and run container images.
Virsh had worked for a long time before docker came around, but yeah… you essentially had to build your own Docker-like infrastructure that only you were using
Solaris Zones too. Absolute magic, many years before Docker and friends.
Or DragonFlyBSD with vkernels
https://www.dragonflybsd.org/docs/handbook/vkernel/
We kind of did but its all put in the context of containers. Check out the unshare command.
unshare --mount
Most examples you'll find put it in the context of containers, like https://www.redhat.com/en/blog/mount-namespaces
Apples and oranges.
Among many other things, Docker (and Podman etc) has
1. Images and OverlayFS
2. Networking
3. User namespace mappings
4. Resource management
---
If all you want is file system isolation, then docker (and postman, etc) is massive overkill chroot is correct.
*podman, etc
Plan9 had a proper solution for this. New processes don't get access to any files by default - you have to explicitly mount directories for them, capability style.
Shame Plan9 blew its weirdness budget.
It has nothing to do with weirdness; Unix itself was plenty weird for its time. The relevant difference between Unix™ and Plan 9 is that Unix source code was given away (or cheaply licensed) to hardware companies which all wrote their own operating systems on top (SunOS, Ultrix, HP-UX, etc. etc.). This made Unix the common factor of very many commercial workstation environments. Plan 9? It was sold directly as a commercial product, for no hardware platform in particular. Nobody wanted to buy it.
People liked Unix because it was free – either really free, via BSD, or as a Unix derivative provided at no cost when people bought their workstations. A new revolutionary operating system had absolutely no reason for anybody to buy it: No commercial developers wanted to develop to a platform without users, and no users wanted a platform without software.
Plan 9 only changed their license many years later, when it was too late for anybody to care, and Unix had become the established standard.
rfork it's easy :D
Pids and cgroups all the way down (also why the wise greybeards rejected docker)
As a number of comments have noted, there are a bunch of different axes that chroot could be 'better' on - e.g. security and sandboxing.
I wrote https://github.com/aidanhs/machroot (initially forked from bubble wrap) a while ago to lean into the pure "pretend I see another filesystem" aspect of chroot with additional conveniences (so no security focus). For example, it allows setting up overlay filesystems, allows mounting squashfs filesystems with an overlay on top...and because it uses a mount namespace, means you don't need to tear down the mount points - just exit the command and you're done.
The codebase is pretty small so I just tweaked it with whatever features I needed at the time, rather than try and make it a fully fledged tool.
(honestly you can probably replicate most of it with a shell script that invokes unshare and appropriate mount commands)
systemd-nspawn is probably what you want.
I built https://github.com/jrz/container-shell as a simple chroot on docker. I use it for both development and sandboxing.
Besides that I have a simple script that starts an ephemeral docker with debian-full + tools.
I also have some scripts that leverage macOS's 'sandbox-exec'
Isn't LXC more or less an unsupervised chroot in an isolated process?
Yes. And so is bubblewrap - if security through isolation is a priority.
Working on proot-docker, a bash script on top of skopeo and proot:
https://github.com/mtseet/proot-docker
We need more people to improve it!
How do you imagine a "better chroot" would differ from a Docker container? Seems to me like that's exactly what Docker is.
ironically docker never gave you true network isolation because there's no way to make it user friendly. plus the many exploits on the all powerful daemon.
but most professional world use systemd to bootstrap isolated processes nowadays, which is kinda if what you are hinting at. cgroups2 and namespaces are what you want.
We should get a better change root yes, and the LD_LIBRARY_PATH gets autoupdated with respect to the new root. A few flags here and there to set permissions of the child process and we're off too the races.
Oh wow, wow. That all sounded so intensely complex, incomprehensible. What we are going to need to do is build a program to handle all that, highly formalized. Let's make it so formalized it's one of those things like taxes or AWS where people can just make a living from understanding the beast. It can be like systemd meets multics meets java. have it's own various complicated commands, complicated file formats, and so on. The chroot() is only historically understood by everyone, so let's steal a page from the java playbook and just rename everything with our own terminology. The product will be so outstanding, wow, I call it "Shocker"
what would "better chroot" do?
I'm not GP, but if I were to hazard a guess, they want something more than just mount space isolation. Something akin to BSD jails, without the bells and whistles of OCI containers like overlay filesystem, network virtualization, resource management, etc.
That requirement is pretty legitimate, since its easier and suitable enough for many applications for which we currently use OCI containers. For example, isolated builds, development environments, sandboxes etc. (I have an isolated build tool for Gentoo).
But Linux already has multiple solutions that fit the bill, like systemd-nspawn, LXC, bubblewrap, etc. Too bad, they aren't as widely known as chroot.
None of those things do what chroot does but many of them involve chroot - so I'm still not grasping what "better chroot" is, other than "not chroot, but something completely different."
It sounds like people want "better exec"
One annoying part of using chroot if you're creating them on the fly is teardown - you have to manually invoke umount, and also take care to get this right for partially created chroots (maybe you detected an error after mounting proc, in the process of getting other files in place).
This was my original motivation in creating machroot (mentioned elsewhere in this thread) and having it use namespaces.
what Solaris and now illumos zones do[1]
[1] https://www.usenix.org/legacy/event/lisa04/tech/full_papers/...
Docker is not using chroot in any way. Escaping chroot is only a matter of calling chdir('..') and poof, you are out of the "sandbox"
The arch linux install has a little wrapper around chroot, used to configure the installed system without booting it.
https://wiki.archlinux.org/title/Installation_guide#Chroot
Gentoo’s stage3 would like to say a word
I remember first installing Gentoo…printed the whole manual and followed along. After a failed attempt and second successful install I learned enough to know that I could use a LiveUSB of my choosing and chroot into the install. Good times!
On Gentoo, it is also simple to install a completely new system in a chroot environment, even if it is intended for a computer with a different, but compatible architecture.
This is frequently very convenient if you want to install Gentoo by compiling everything from sources on a cheap and small computer, e.g. one with some Intel Atom CPU.
Instead of compiling anything on the resource-constrained computer, you install a fresh Gentoo system for it, in a chroot environment on a fast desktop computer, which supports a superset of the ISA of the small computer, so you can still execute the programs intended for the target computer.
Then you just copy the installation result over the SSD/HDD of the destination computer. If you have many identical computers, you can copy the installed Gentoo over all of them without any problems, removing the need for multiple installations.
If desired, you can keep the chroot environment with the installation result and perform any later updates on it. Then you synchronize the updated Gentoo from the chroot with the one or more target computers.
Yes, there’s a few gentoo developers and bug hunters that do exactly that but to find bugs and test updates 8D!
Debian's debootstrap would also like to participate in this discussion
Debootstrap is for installing a minimal system into a dir from another running system (so only useful in the setup phase), whereas arch-chroot is used to access an already installed system (e.g. when booting off a USB stick to fix something). They're not the same thing. The debootstrap-like tool in arch is called pacstrap.
Manjaro has the same, I'd assume inherited from arch and modified.
I just wish the script could figure out a BTRFS drive without me manually mounting volumes :(
i was very happy when that arch-chroot + pacstrap simplified my arch setup not having to run those mounts (i remember doin the same setup process for arch for centos/rhel just to not use anaconda and removing bloat, or rather installing what i wanted)
for btrfs, if you use a consistent volume mapping on your systems, its pretty easy. in arch setup i typically enable ssh and have a pretty simple set of bashisms for target device, compression, and mount points. then its copy-pasta since its a boiler plate for when i need to recover or fresh install
not glanced at manjaro so not familiar with its install-methods
I've actually used this to unbrick my laptop before. Very useful
> sudo mkdir /rescue/boot
> sudo mount /dev/nvme0n1p3 /rescue/boot
This is a little extra. What you can generally do is immediatelly after chroot just run 'mount -a' to mount everything from the chroot's fstab. The empty `/boot` probably already exists.
Good idea!
I just installed Gentoo from WSL2 this way, on my secondary NVME on my Framework 16. Went without a hitch besides some minor things you need to be aware of:
-the lack of access to efi subsystem from wsl means you need to pass some extra flags to help grub/etc along, and you may need to set it as the boot partition in the bios manually
-you'll have to mount the drive to wsl with `wsl --mount <DiskPath> --bare`, after finding the right DiskPath with `Get-CimInstance -query "SELECT * from Win32_DiskDrive"`, and you might have to offline the disk first in Windows disk manager
That's pretty incredible, that it works from Windows too!
Now I'm just wondering if it would work from a Linux running in the browser, accessing the USB through WebUSB ;D
Currently working on scripts to provision a drive outside of the machine it’s meant to go on using chroot. I’ve so far accidentally unmounted /dev/pts several times from the host system and running docker inside the chroot caused a hard lock. Fun stuff.
Used this or at least a very similar technique a few times to fix boot issues.
I've got a version of the mounting command that I think is easier to use:
Change the "/mnt/$f" to whatever mountpoint that you're using which would be "/rescue/$f" to align with TFA.I don't know what difference it makes to have /run mounted, but once you chroot into the mountpoint you can mount the boot partition etc and run whatever grub or mkinitramfs command you need to fix stuff.
I would leave the /boot mounting to later in the process - after you chroot. This way you can easily check /etc/fstab for where the boot partition lives (or if there is one), so you only need to locate the root partition initially which is generally easy to figure out from the disk sizes.
There's extra steps needed however if the system uses LVM.
chroot was (is?) the recommended way of installing gentoo which is pedagogical.
There are various handy (chroot) techniques that are probably considered "old school" now. For example, having a "rescue partition" which can be booted into remotely, and from there reinstall or repair the "main os". This is necessary when repartitioning remotely, for example.
That’s still how most Linux OS installers install. They just do that for you.
It’s still used for recovery but recovery partitions have kinda gone out of fashion as the ability to consistently boot has gotten better. Additionally, thumb drives and net boot make the partition a little less necessary.
It's kind of funny to see this being called a technique. It has been something I've done so many times to impressive results. To me, it's just something you can do, among a million other things, if you know enough about Linux.
"Technique" is a term for "something you can do, among a million other things, if you know enough about X".
The tools shouldn't require the use of chroot. It is quite common on Solaris to be able to specify a path that should be considered the root directory for the tool, usually -R. This was super useful in a netboot world using nfs, and also for crash recovery.
If a system is screwed up enough, then a chroot strategy won't work because it relies on the path you are chrooting to to be generally valid and functional. If it's missing libraries you may well be screwed.
I can highly recommend: https://github.com/fsquillace/junest
What if I'm ssh'd into a Linux box and want to (say) format its root disk. Can I boot a new instance of Linux in memory and then access the root disk?
Interesting question and I don't know how you'd do that - maybe using some statically linked binaries for fdisk and mkfs. However, why would you ever want to do that? If you want to remove data from the disk, then just running a suitable "dd" command would wipe the disk. I can't think of any other use case that wouldn't subsequently involve booting from a live iso which would let you format the disk anyway.
One use case is: what if the computer is at a remote location and you'd want to put a new image on the main disk. If Linux is currently using that disk as its root disk, then I suppose you'd have to somehow unmount it, or use a different instance of Linux that runs entirely in ram (so you can kill the original Linux instance and overwrite it).
I guess you could boot into a live installer ISO and get that to install over the root disk, though I don't know what problems you'd face with trying to unmount the original root partition.
If you're wanting to do something like that fairly regularly, then it'd be easier to just run virtual machines (e.g. QEMU/KVM) to do that kind of thing.
It's definitely possible. nixos-anywhere [0] implements this in a neat package for NixOS specifically. I am relatively certain you could use the same script to boot into a different distribution, though the installation part itself would obviously need to be adapted.
[0] https://github.com/nix-community/nixos-anywhere
Creating a mount namespace and using pivot_root seems like a safer solution.
I don’t think pivot_root is necessary for something like this, but a new mount namespace will definitely help avoid creating a mess on accident
Came here to suggest this. pivot_root has the advantage that it can remount the existing root. It's essentially meant for going from an initramfs using linuxrc to boot to a full-on init in the other root filesystem. If things are horribly broken you could use a shell or some other process as init.
Ah - my attempt at doing this was almost there, except for the few bits that kept erroring out because I didn't mount the proc,dev,etc. mountpoints correctly :) Something to give another whirl on. At the time, I wanted something simple without docker to store everything development-related using different libraries. This is where debootstrap+chroot comes in - to build against various versions of libraries in Debian (if I remember correctly - it's been awhile.)
Is there already a wrapper utility to automate all of this process laid out in the article?
There is!
arch-chroot [1], despite its name pretty much does all the `mount -t proc` stuff the post says. It's also available on other distros like debian [2]. I have used it in the past to chroot into fedora as well.
[1] https://man.archlinux.org/man/arch-chroot.8 [2] https://packages.debian.org/arch-install-scripts
In addition to the other comments: If you are on NixOS, you can use nixos-enter [1].
[1] https://wiki.nixos.org/wiki/Change_root
sudo arch-chroot .
This is now my default tool setting up raspberry like computers.
Once I had to rescue a SunOS system and edit /etc/vfstab but I only had ed to my disposal. At that point I had never touched ed before, so that was a bit of a learning exercise, for sure.
In a computer lab I hung out in back in college...
We used rcp to keep passwords in sync. Add the account on the main machine, rcp the password file to the other machine. sudo rcp /etc/password other:/etc/passwd was muscle memory.
One day, someone was getting added to the groups file to be able to work in the server web project. sudo rcp /etc/group other:/etc/passwd
Ooops. Couldn't log in to fix it.
"Is anyone logged into the other machine?" (someone said yes). "Type while 1 sync" ... (ok) ... And we flipped the power switch and brought it up in single user mode (since the password file was invalid). Next, need to establish a minimal /etc/passwd ... emacs /etc/passwd (nope) vi /etc/passwd (nope - invalid terminal 300h not in termcap). "Uhm... cat > /etc/passwd ?" (possible, but a PITA when there is a typo in transcription)
I was a wizard on a lpmud. "I know ed".
And we got a minimal password file restored while reading the hashed values over (no way where we going to have root::0:0:... as the file even for a second) and then rcp'ed the proper /etc/passwd and /etc/group file over to the other machine.
https://www.gnu.org/fun/jokes/ed-msg.txt
Sure, chroot can be useful in a pinch, but it would not be the first thing I'd reach to. If you got the partitions mounted, you can already do quite a lot of things without needing to chroot.
This is usually right. In this particular case the issue was actually in /boot, which might not have been obvious just mounting the main partition.
This is why Linux and Unix are so awesome. It’s really all just files.
Not really, but yes there's a lot of files or things that seem to be files. Try plan9 though.
If you want to blow your mind in what's possible in operating-system isolation read about KeyKOS from 1985: https://css.csail.mit.edu/6.858/2014/readings/keykos.pdf
Such systems are rarely very pragmatic, but do show off a bunch of weird concepts.
Definitely weird, or perhaps I'm just not imaginative enough. I have a hard time figuring out what some KeyKOS constructs mean in familiar terms, particularly whether a given feature is an instance of essential or accidental complexity. I'm more familiar with today's seL4, which has carried on many of the principles of KeyKOS. Learning about the state of the art today is one thing, but learning history is quite another. I rely on simplicity in kernel design to avoid hindering more creative ideas that others have had.
"files" is clearly a misnomer, it should have been "objects".
This is how Linux From Scratch is done, and I've used the same technique to install missing video drivers that the Ubuntu install didn't include for whatever reason.
Haha, learned this technique while installing Void Linux
Ha, that is funny. Void Linux desperately needs a better installer. (IMHO)
What we all know as Docker and containers is really just fancy chroot.
On non Linux systems, it's fancy chroot inside a Linux VM.
Which raises the question of why they need chroot if they have the VM.
Another semi-related but equally odd view: Docker is an operating system and containers are processes.
The difference between Docker and chroot, by the way, is that Docker does a bunch more system calls to ch the root of several other things that are not the filesystem. It also sets up things inside those roots for you by default, so your containers can access the internet, for example.
I found this project to be an interesting POC of how it all actually works.
https://github.com/p8952/bocker
Basically, it runs docker containers using chroot to prove that it's possible.
Docker uses Layered Filesystems (Multiple Filesystems mounted under the same folder on top of one another), something that used to be mostly used when you have a read only Filesystem, like a cdrom, then mount a writeable folder over the same mount point to make that folder writeable.
Bocker does a similar mounting of the layers with chroot to run docker containers.
Chroot has saved my system more times than I care to admit. I didn't really understand how it worked though, interesting.
[dead]
Saved me many times from reaching for the CD to rescue my main. When you install Arch it uses this technique, very lightweight.
[flagged]
What does the use of 'retarded' actually bring to your comment? That's entirely unnecessary.