Until a few years back most of my systems used Linux From Scratch. It's amazing for learning your system and how all its libraries integrate. I created my own boot scripts, my own pacman package manager repository, etc. One of the main reasons I stopped using LFS was the amount of work required for updates.
Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it: harfbuzz, ffmpeg, qt5, poppler and the list goes on.
Chromium takes it to a whole new level. While Linux packages typically have an automake flow (configure, make, make install) Chromium requires ninja, not just ninja but a specific version depending on the Chromium version. Then you need a ton of libraries with specific versions (or custom patches) or use the built-in versions provided by chromium, which may or may not compile at all. Then there is clang, while most software is compiler agnostic, Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile. It seems the chromium build process is mostly built by engineers desperate to reinvent the wheel. While everything can work fine with sane Linux defaults (GCC, automake, system libraries) they convinced themselves their software is so complex and unique it requires a special treatment.
Chromium looks like a build system for a complex integrated project where the engineers have one priority: make the application work as best it can. You will quickly find that when you focus on that then making the build system convenient, especially convenient for distribution packagers with their myriad of own rules and modifications that they want to make on top of your system, is quite a big blocker to the primary goal. You can see similar things with applications like OBS, where they bundle modified libraries to fix problems and then distributions unbundle them and release broken packages, causing no end of grief for users and the developers.
There is not some arbitrary notion of wheel reinvention just for browsers. Calling browsers 'just browsers' is doing their feature set a huge disservice: they are bona fide operating systems or virtual machines, which have to load and run untrusted, unverified data and code within critical system libraries (many of which with kernel-mode access, such as GPU drivers, sound drivers, USB subsystems, and more), and do a huge amount of sandboxing to guard against escalations.
Everything you've listed—from ninja, to Clang, to libraries with custom patches, to Clang releases, including requiring pre-releases sometimes—is optimised for Google's scale, rather than the lone warrior developer. Within Google, a Chromium build uses remote build execution by default, upstream Clang by default (for the latest features and fixes, and Chromium developers regularly contribute to LLVM), and needs to cross-compile easily for at least a dozen different OS-architecture triples (Linux x86-64, Linux ARM, macOS, Windows x86-64, Windows on ARM, iOS, Android x86-64, Android ARM, Android TV, some BSDs, etc).
What you call 'sane Linux defaults' (especially the examples you have listed: GCC, automake, so-called 'system libraries') are highly antagonistic to all of the above. They may make life easy for the so-called 'default case' of desktop Linux (arguable; Chromium was first released on Windows, and secondly since desktop Linux's most common C library, glibc has an unstable ABI, making back-portability impossible), but make it incredibly hard to get out a project of such immense complexity. All of Chromium's tooling and 'complexity' was developed in response to the 'sane' defaults being not so sane after all.
To get a GCC cross-compile for a given set of architecture triples, one needs a complete graph of GCC host-target toolchains. On the other hand Clang is natively a cross-compiler; pass in `--sysroot` and the host compiler will redirect for the target instantly. Automake (and the autotools in general) is slow as molasses compared to ninja. It is based on computing and compilation from the 1970s, and their defaults probe for long-since-dead string comparison functions that are expected to work by default nowadays on most hosted systems. Not to mention it has an extraordinarily complicated configure step[1], and an equally appalling macro-based configure language.
Ninja is purely declarative, and is not even meant to be human-readable, as it is a back-end for a variety of build generator tools (CMake, XMake, build2, Bazel (Blaze internally at Google), Meson, probably more). Google have in fact rewritten Ninja to be a remote-build-capable tool from the outset[2], and they call it Siso. Siso has more than halved my Android build times even compared to Ninja because of its incredible use of RBE.
Frankly I think it should be evident why they have opted for 'rewriting the world' including vendoring and modifying system libraries, as well as including efforts like Skia, ANGLE, and more: directly relying on 'system libraries' is painful. I mentioned GPU drivers. Imagine rewriting the same code over and over to have precisely the same behaviour on D3D, Vulkan, OpenGL, Metal, and every single vendor GPU driver on every single system (and there are a lot more on mobile than on desktop: ARM Mali, Qualcomm Adreno, Samsung Xclipse, Imagination PowerVR, Huawei, MediaTek; this is not even close to exhaustive, just look at the Khronos vendor list).
When comments like yours are made I wonder if there is ever a consideration beyond 'it makes the developer's life hard'. As developers, our ultimate target audience are the users of our software. We should be willing to do anything come hell or high water, even making the dev-ex extraordinarily convoluted, if it means the user experience will be the smoothest and fastest possible. Sure, there is a balance, but I believe it should be strongly skewed towards the user story first.
>glibc has an unstable ABI, making back-portability impossible
Of all the libraries to call out for this, glibc is not the one. It's basically the only library on linux that actually has a strong commitment to providing a stable ABI, with its own symbol versioning system to allow it. It is a bit of a PITA to compile for an older version of that ABI, but it's very possible to be compatible with glibc across a wide range of versions and distributions. It's everything else necessary to interact with the system that's wildly unstable.
I tried Gentoo some 20 years ago or so for fun at home. I guess my naive thinking was that software specifically compiled for my hardware would run noticeably faster and "better".
But I didn't last for very long. I remember starting the install for a bigger app before going to bed, only to find the thing still building when I woke up. And as for the dream of more tailored builds, I never really noticed anything much different - I probably was too uneducated about build configurations, or - more likely - the differences just didn't matter too much.
Eventually, I switched to Ubuntu and never looked back.
So, I'm curious - is anyone using Gentoo as their daily driver, and what is the main incentive for you?
I'm using it as my main OS. I have it installed on 4 machines: a RK3288 SBC, a RK3588 desktop, a x86-64 desktop and a x86-64 laptop. Main incentive for me is customization.
- It makes it very very easy to patch any package before they're installed - just put the patches in /etc/portage/patches/<pkg name>/, and that includes the kernel too.
- I can exclude package features that I don't like. For example, my Firefox doesn't try/offer to turn on the wifi.
- I can also exclude entire packages that I don't like, because the entire dependency chains can be eliminated by disabling USE flags. For example, none of my systems have systemd or dbus. Since last year I don't even have openrc anymore (the other Gentoo service manager). One of them doesn't even have udev. I tweaked it so much, I'm not even sure I can still call it Gentoo.
Daily Gentoo desktop user since 2007. Compilations/ricing is one thing, the variety and flexibility of the portage and USE flag system is another. For me it just makes sense to stay with this proven distro. I use Ubuntu variants on my work PC and they just have all this crap that I don't have to tolerate on my gentoo installs.
The docs were great for learning. And it made it so easy to recompile the kernel because it was right in the initial setup and you just had to walk through it again.
I used it on an off as my daily machine for around 15-20 years I think. It is fast in terms of execution-on-hardware speed, and genuinely puts you in control. For a certain kind of user this is more important than anything.
These days I'm on OSX mostly because of the frequency of business trips meant I needed something small and portable a few years ago, and now the hardware is so much better than what you can get for the PC world. Plus, I am hyper-conscious of my time and can't really afford to be fiddling with new system libraries, kernel configurations and so forth.
Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall, and after enough iterations I'd started to err to Ubuntu at some point. But it was by far the most fun of any Linux I used, save perhaps Slackware which was my first.
I think in a few years' time the next machine I build will be a non-Apple ARM custom portable, hopefully running Gentoo, with a lot of custom machining and hardware features.
> at some point the system state would break and you'd have to reinstall
My advice is to start using btrfs snapshots. Make a backup snapshot, run the updates, and if anything breaks, just "btrfs set-default" to the backup snapshot and everything is working again.
> It is fast in terms of execution-on-hardware speed, and genuinely puts you in control.
After installing Gentoo I felt I should make some kind of change, and I wanted to replace the GNOME foot in the GNOME menu with an icon I found of the Gentoo "g" in a wizard hat.
I found documentation suggesting that the menu's icon was controlled by some setting somewhere, and I changed that setting to point to the wizard-hat icon.
The menu was unaffected. If I viewed the menu bar's properties, the menu was associated with the wizard-hat icon in the properties display, but it continued to show the foot icon in its name.
I had nothing else to go on and gave up. I'm still mad about that.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Well, that did happen to me once. But it happened after I uninstalled glibc. I can't really blame Gentoo for that. I always saw it as positive that Gentoo allowed me to do something that ridiculous.
Compiling Chromium took absolutely forever 5+ years ago. I think a whole system took like 20% of the time that Chromium would take to compile. I can't even imagine with rust extensions added on top now. Gentoo was so awesome though but I had to switch to debian because I would just play with my configuration all the time and break things. One day I hope to return to using it though or setup another machine to use it.
Chromium is a pyramid of components and the higher you go up, the longer the compilation of single file takes, because they depend on so many things. This makes caching difficult. A small change at the bottom can invalidate many files, especially the heavy ones at the top. The build system ninja already does caching, but it only offers limited benefits. Then there is so much activity that if you plan to do a weekly build, you can do it from scratch.
The main problems seem to be that it takes a long time to build all combinations of USE flags and that users complain that volutneers are not working fast enough, causing annoyance of the maintainers. Can you explain how AI would help here?
Firefox is bad to compile yourself, because it's slow and often requires a library version you don't have, such as a new ICU version. Updating ICU also requires you to build packages that depend on it: harfbuzz, ffmpeg, qt5, poppler and the list goes on.
Chromium takes it to a whole new level. While Linux packages typically have an automake flow (configure, make, make install) Chromium requires ninja, not just ninja but a specific version depending on the Chromium version. Then you need a ton of libraries with specific versions (or custom patches) or use the built-in versions provided by chromium, which may or may not compile at all. Then there is clang, while most software is compiler agnostic, Chromium does not compile with gcc. Of course, it needs a specific clang release, sometimes even a pre-release, or again a load of patches to compile. It seems the chromium build process is mostly built by engineers desperate to reinvent the wheel. While everything can work fine with sane Linux defaults (GCC, automake, system libraries) they convinced themselves their software is so complex and unique it requires a special treatment.
Everything you've listed—from ninja, to Clang, to libraries with custom patches, to Clang releases, including requiring pre-releases sometimes—is optimised for Google's scale, rather than the lone warrior developer. Within Google, a Chromium build uses remote build execution by default, upstream Clang by default (for the latest features and fixes, and Chromium developers regularly contribute to LLVM), and needs to cross-compile easily for at least a dozen different OS-architecture triples (Linux x86-64, Linux ARM, macOS, Windows x86-64, Windows on ARM, iOS, Android x86-64, Android ARM, Android TV, some BSDs, etc).
What you call 'sane Linux defaults' (especially the examples you have listed: GCC, automake, so-called 'system libraries') are highly antagonistic to all of the above. They may make life easy for the so-called 'default case' of desktop Linux (arguable; Chromium was first released on Windows, and secondly since desktop Linux's most common C library, glibc has an unstable ABI, making back-portability impossible), but make it incredibly hard to get out a project of such immense complexity. All of Chromium's tooling and 'complexity' was developed in response to the 'sane' defaults being not so sane after all.
To get a GCC cross-compile for a given set of architecture triples, one needs a complete graph of GCC host-target toolchains. On the other hand Clang is natively a cross-compiler; pass in `--sysroot` and the host compiler will redirect for the target instantly. Automake (and the autotools in general) is slow as molasses compared to ninja. It is based on computing and compilation from the 1970s, and their defaults probe for long-since-dead string comparison functions that are expected to work by default nowadays on most hosted systems. Not to mention it has an extraordinarily complicated configure step[1], and an equally appalling macro-based configure language.
Ninja is purely declarative, and is not even meant to be human-readable, as it is a back-end for a variety of build generator tools (CMake, XMake, build2, Bazel (Blaze internally at Google), Meson, probably more). Google have in fact rewritten Ninja to be a remote-build-capable tool from the outset[2], and they call it Siso. Siso has more than halved my Android build times even compared to Ninja because of its incredible use of RBE.
Frankly I think it should be evident why they have opted for 'rewriting the world' including vendoring and modifying system libraries, as well as including efforts like Skia, ANGLE, and more: directly relying on 'system libraries' is painful. I mentioned GPU drivers. Imagine rewriting the same code over and over to have precisely the same behaviour on D3D, Vulkan, OpenGL, Metal, and every single vendor GPU driver on every single system (and there are a lot more on mobile than on desktop: ARM Mali, Qualcomm Adreno, Samsung Xclipse, Imagination PowerVR, Huawei, MediaTek; this is not even close to exhaustive, just look at the Khronos vendor list).
When comments like yours are made I wonder if there is ever a consideration beyond 'it makes the developer's life hard'. As developers, our ultimate target audience are the users of our software. We should be willing to do anything come hell or high water, even making the dev-ex extraordinarily convoluted, if it means the user experience will be the smoothest and fastest possible. Sure, there is a balance, but I believe it should be strongly skewed towards the user story first.
[1]: https://upload.wikimedia.org/wikipedia/commons/8/84/Autoconf...
[2]: https://chromium.googlesource.com/build/+/refs/heads/main/si...
Of all the libraries to call out for this, glibc is not the one. It's basically the only library on linux that actually has a strong commitment to providing a stable ABI, with its own symbol versioning system to allow it. It is a bit of a PITA to compile for an older version of that ABI, but it's very possible to be compatible with glibc across a wide range of versions and distributions. It's everything else necessary to interact with the system that's wildly unstable.
But I didn't last for very long. I remember starting the install for a bigger app before going to bed, only to find the thing still building when I woke up. And as for the dream of more tailored builds, I never really noticed anything much different - I probably was too uneducated about build configurations, or - more likely - the differences just didn't matter too much.
Eventually, I switched to Ubuntu and never looked back.
So, I'm curious - is anyone using Gentoo as their daily driver, and what is the main incentive for you?
- It makes it very very easy to patch any package before they're installed - just put the patches in /etc/portage/patches/<pkg name>/, and that includes the kernel too.
- I can exclude package features that I don't like. For example, my Firefox doesn't try/offer to turn on the wifi.
- I can also exclude entire packages that I don't like, because the entire dependency chains can be eliminated by disabling USE flags. For example, none of my systems have systemd or dbus. Since last year I don't even have openrc anymore (the other Gentoo service manager). One of them doesn't even have udev. I tweaked it so much, I'm not even sure I can still call it Gentoo.
These days I'm on OSX mostly because of the frequency of business trips meant I needed something small and portable a few years ago, and now the hardware is so much better than what you can get for the PC world. Plus, I am hyper-conscious of my time and can't really afford to be fiddling with new system libraries, kernel configurations and so forth.
Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall, and after enough iterations I'd started to err to Ubuntu at some point. But it was by far the most fun of any Linux I used, save perhaps Slackware which was my first.
I think in a few years' time the next machine I build will be a non-Apple ARM custom portable, hopefully running Gentoo, with a lot of custom machining and hardware features.
My advice is to start using btrfs snapshots. Make a backup snapshot, run the updates, and if anything breaks, just "btrfs set-default" to the backup snapshot and everything is working again.
After installing Gentoo I felt I should make some kind of change, and I wanted to replace the GNOME foot in the GNOME menu with an icon I found of the Gentoo "g" in a wizard hat.
I found documentation suggesting that the menu's icon was controlled by some setting somewhere, and I changed that setting to point to the wizard-hat icon.
The menu was unaffected. If I viewed the menu bar's properties, the menu was associated with the wizard-hat icon in the properties display, but it continued to show the foot icon in its name.
I had nothing else to go on and gave up. I'm still mad about that.
> Frankly Gentoo was never polished, at some point the system state would break and you'd have to reinstall
Well, that did happen to me once. But it happened after I uninstalled glibc. I can't really blame Gentoo for that. I always saw it as positive that Gentoo allowed me to do something that ridiculous.
Shared libraries breaking ABI could mean recompiling half your operating system
On the other hand, bundled libraries meant insanely long build times and a sense of impurity/uncleanliness
Unfortunately, it looks like projects like ABI tracker are long dead:
https://github.com/lvc/abi-tracker
To the single guy maintaining it, and no offer of help. Crazy entitlement.
Pretty rich from a compile-from-source distribution