How to Crash Systemd in One Tweet

Blog

September 28, 2016

How to Crash Systemd in One Tweet

The following command, when run as any user, will crash systemd:

NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""

After running this command, PID 1 is hung in the pause system call. You can no longer start and stop daemons. inetd-style services no longer accept connections. You cannot cleanly reboot the system. The system feels generally unstable (e.g. ssh and su hang for 30 seconds since systemd is now integrated with the login system). All of this can be caused by a command that's short enough to fit in a Tweet.

Edit (2016-09-28 21:34): Some people can only reproduce if they wrap the command in a while true loop. Yay non-determinism!

The bug is remarkably banal. The above systemd-notify command sends a zero-length message to the world-accessible UNIX domain socket located at /run/systemd/notify. PID 1 receives the message and fails an assertion that the message length is greater than zero. Despite the banality, the bug is serious, as it allows any local user to trivially perform a denial-of-service attack against a critical system component.

The immediate question raised by this bug is what kind of quality assurance process would allow such a simple bug to exist for over two years (it was introduced in systemd 209). Isn't the empty string an obvious test case? One would hope that PID 1, the most important userspace process, would have better quality assurance than this. Unfortunately, it seems that crashes of PID 1 are not unusual, as a quick glance through the systemd commit log reveals commit messages such as:

Systemd's problems run far deeper than this one bug. Systemd is defective by design. Writing bug-free software is extremely difficult. Even good programmers would inevitably introduce bugs into a project of the scale and complexity of systemd. However, good programmers recognize the difficulty of writing bug-free software and understand the importance of designing software in a way that minimizes the likelihood of bugs or at least reduces their impact. The systemd developers understand none of this, opting to cram an enormous amount of unnecessary complexity into PID 1, which runs as root and is written in a memory-unsafe language.

Some degree of complexity is to be expected, as systemd provides a number of useful and compelling features (although they did not invent them; they were just the first to aggressively market them). Whether or not systemd has made the right trade-off between features and complexity is a matter of debate. What is not debatable is that systemd's complexity does not belong in PID 1. As Rich Felker explained, the only job of PID 1 is to execute the real init system and reap zombies. Furthermore, the real init system, even when running as a non-PID 1 process, should be structured in a modular way such that a failure in one of the riskier components does not bring down the more critical components. For instance, a failure in the daemon management code should not prevent the system from being cleanly rebooted.

In particular, any code that accepts messages from untrustworthy sources like systemd-notify should run in a dedicated process as a unprivileged user. The unprivileged process parses and validates messages before passing them along to the privileged process. This is called privilege separation and has been a best practice in security-aware software for over a decade. Systemd, by contrast, does text parsing on messages from untrusted sources, in C, running as root in PID 1. If you think systemd doesn't need privilege separation because it only parses messages from local users, keep in mind that in the Internet era, local attacks tend to acquire remote vectors. Consider Shellshock, or the presentation at this year's systemd conference which is titled "Talking to systemd from a Web Browser."

Systemd's "we don't make mistakes" attitude towards security can be seen in other places, such as this code from the main() function of PID 1:

/* Disable the umask logic */
if (getpid() == 1)
        umask(0);

Setting a umask of 0 means that, by default, any file created by systemd will be world-readable and -writable. Systemd defines a macro called RUN_WITH_UMASK which is used to temporarily set a more restrictive umask when systemd needs to create a file with different permissions. This is backwards. The default umask should be restrictive, so forgetting to change the umask when creating a file would result in a file that obviously doesn't work. This is called fail-safe design. Instead systemd is fail-open, so forgetting to change the umask (which has already happened twice) creates a file that works but is a potential security vulnerability.

The Linux ecosystem has fallen behind other operating systems in writing secure and robust software. While Microsoft was hardening Windows and Apple was developing iOS, open source software became complacent. However, I see improvement on the horizon. Heartbleed and Shellshock were wake-up calls that have led to increased scrutiny of open source software. Go and Rust are compelling, safe languages for writing the type of systems software that has traditionally been written in C. Systemd is dangerous not only because it is introducing hundreds of thousands of lines of complex C code without any regard to longstanding security practices like privilege separation or fail-safe design, but because it is setting itself up to be irreplaceable. Systemd is far more than an init system: it is becoming a secondary operating system kernel, providing a log server, a device manager, a container manager, a login manager, a DHCP client, a DNS resolver, and an NTP client. These services are largely interdependent and provide non-standard interfaces for other applications to use. This makes any one component of systemd hard to replace, which will prevent more secure alternatives from gaining adoption in the future.

Consider systemd's DNS resolver. DNS is a complicated, security-sensitive protocol. In August 2014, Lennart Poettering declared that "systemd-resolved is now a pretty complete caching DNS and LLMNR stub resolver." In reality, systemd-resolved failed to implement any of the documented best practices to protect against DNS cache poisoning. It was vulnerable to Dan Kaminsky's cache poisoning attack which was fixed in every other DNS server during a massive coordinated response in 2008 (and which had been fixed in djbdns in 1999). Although systemd doesn't force you to use systemd-resolved, it exposes a non-standard interface over DBUS which they encourage applications to use instead of the standard DNS protocol over port 53. If applications follow this recommendation, it will become impossible to replace systemd-resolved with a more secure DNS resolver, unless that DNS resolver opts to emulate systemd's non-standard DBUS API.

It is not too late to stop this. Although almost every Linux distribution now uses systemd for their init system, init was a soft target for systemd because the systems they replaced were so bad. That's not true for the other services which systemd is trying to replace such as network management, DNS, and NTP. Systemd offers very few compelling features over existing implementations, but does carry a large amount of risk. If you're a system administrator, resist the replacement of existing services and hold out for replacements that are more secure. If you're an application developer, do not use systemd's non-standard interfaces. There will be better alternatives in the future that are more secure than what we have now. But adopting them will only be possible if systemd has not destroyed the modularity and standards-compliance that make innovation possible.

Older (View Archive)

Domain Validation Vulnerability in Symantec Certificate Authority

Newer

Systemd is not Magic Security Dust

Comments

Reader me on 2016-09-28 at 20:33:

i just ran it. nothing happened. everything works exactly the same as before

Andrew Ayer on 2016-09-28 at 21:32:

Some people are having to run the command in a loop for the bug to trigger: https://github.com/systemd/systemd/issues/4234#issuecomment-250289253

I'm not sure why; I was able to consistently reproduce with a single call under systemd v215, v229, and v230, on Debian and Ubuntu.

Anonymous on 2016-09-28 at 21:09:

Thank you for providing tangible arguments as to why Systemd is a dangerous option.

Reader Anonymous on 2016-09-29 at 00:13:

Please email them on secalert <@> redhat.com and they will take your issue seriously.

Anonymous on 2016-10-02 at 21:33:

maybe it would have been a good idea to send that email yourself rather then advising someone else to do so. ... and 'yes' I did it now

Anonymous on 2016-10-04 at 00:11:

from secalert@redhat.com

Thank you for this notification. We are aware of this systemd issue and it is being tracked in this flaw:
https://bugzilla.redhat.com/show_bug.cgi?id=1380286

Reader Twirrim on 2016-09-28 at 21:09:

I found I had to use a "while true" loop around that before it would trigger, as it seems it doesn't reliably trigger with every attempt (curious what combination of circumstances trigger it?)

Just out of curiosity, why didn't you contact the RedHat security team to responsibly disclose this vulnerability? (the majority of systemd development work is funded by, and done by, RedHat employees)

Reader Keith Curtis on 2016-09-28 at 22:00:

This article starts with a bug. Note, it's a pretty irrelevant one that doesn't appear to be reproducible on recent builds.

Then it brings up a few other debatable issues, and then it incorrectly generalizes that systemd security is terrible. I say debatable because for example parsing command line parameters isn't something people usually delegate to a separate process.

Unfortunately, it also ignores all the great systemd security features it has added for the average Linux user (such as private tmp, and cgroups), and in fact all the other great features it has.

Reader nextime on 2016-09-29 at 00:44:

parsing command line parameters isn't something people usually delegate to a separate process.

It isn't about parsing command line. It's about parsing text coming from an untrusted socket where anyone can write to, at runtime, in something that runs as PID 1. Not exactly the same thing.

Unfortunately, it also ignores all the great systemd security features it has added for the average Linux user (such as private tmp, and cgroups), and in fact all the other great features it has.

What it's debatable is that systemd add those things. I use private-tmp, namespaces, cgroups and more... and i don't have systemd installed in my system.

And anyway, cgroups are NOT a security feature; they can help to get a better security, but they aren't a security measure.

Reader Antti Laine on 2016-09-29 at 07:22:

Private tmp (being one application of namespaces) and cgroups are features of the Linux kernel, not systemd.

Also, the bug is very reproducible. I just tried it in Ubuntu 16.04 LTS after apt upgrade. Worked very, well^Wbadly.

Anonymous on 2016-09-30 at 08:33:

I couldn't reproduce it on Ubuntu 16.04 LTS 32 bit.

Andrew Ayer on 2016-09-30 at 15:35:

Ubuntu has released a security update that fixes this bug: http://www.ubuntu.com/usn/usn-3094-1/

However, it introduced a new, different DoS vulnerability (file descriptor leak) so expect a second security update soon.

Reader James on 2016-09-29 at 19:58:

Um... I can repeat it on 7 fully patched. I'm not sure what you mean by irrelevant. Now the patch has been submitted. When will a CVE be released. Sticking our heads in the sand is so ... Microsofty .... Let's not follow the win98 crew on that aspect please.

Reader Stefan on 2016-09-28 at 23:12:

Why not report this kind of denial-of-service attack in a responsible disclosure manner?

Reader tim on 2016-09-29 at 13:56:

and you know that it was not responsibly reported how?

Reader Simon Strandman on 2016-10-01 at 05:26:

Where is the bug report then?

Anonymous on 2016-10-02 at 13:51:

He wouldn't have gotten clicks for his blog if he had.

See, nowadays assholes like the OP think they're contributing something but in reality they're just trying to make enough so someone will share their blog.

Hits > Actually doing something worthwhile.

Anonymous on 2016-10-02 at 19:30:

Making the world aware of the sort of thing systemd folks think is okay is the most responsible thing one can do.

Everyone knew systemd was a mess. Everyone knew they were going to try to take over the world. Everyone knew it was bound to have problems like this. They were warned many times already, and chose to go with Lennart's handwaving anyway.

Reader Hendrik Visage on 2016-10-06 at 11:15:

Well, (1) Thanks for showing this error, it is just so...so... so... "expected?"

Yes, "everybody" knows it a bad idea, but then why follow it, 'cause it was solving all the "right" problems, in the wrong way, and everybody was looking at the marketing babble and just wanted the "pains" relieved, instead of understanding their souls their are selling out ;( It sorta started with Upstart, and was unavoidable that something will come in, but then, as is always said in OpenSource: Code talks harder than words...

Wish I had the budget and team to write a "proper" replacements similar to Solaris 10/11's service management framework, but in user mode.

Anonymous on 2016-09-29 at 00:29:

this is the most comprehensive writeup I've yet seen illustrating why systemd is not just a bad idea, but a worse implementation. The very concept is the complete opposite of the past 40 years of UNIX wisdom; that so many (well-respected) systems design folk seem to have completely forgotten their history (and really, just common sense) is depressing indeed.

Reader Scott Francis on 2016-09-29 at 00:30:

(intended to sign that post, doing so here.)

Reader Jussi Sallinen on 2016-09-29 at 07:02:

http://static.usenix.org/event/lisa05/tech/full_papers/adams/adams.pdf This must be what systemd people saw but failed to implement in sane way.

Reader Matthias Koch on 2016-09-29 at 10:43:

Fully agree. I'm just wondering what will this time be the excuse for closing the issue/bug report without even admitting the bug exists, or that it is a bug.

Andrew Ayer on 2016-09-29 at 23:18:

They fixed the bug (although it took them two tries to get it right). But it's not about the bug. The bug is just one example of systemd's terrible programming practices.

Reader rtfa on 2016-10-01 at 17:23:

Do you expect any software to be bug free from day one or just systemd? the sweeping generalisations you make are really stupid

Reader anonnymoose on 2016-10-02 at 05:35:

Do you expect any software to be bug free from day one or just systemd?

This is day 2,337 for systemd. I'm not the OP, but I expect that system-critical system management software gets input sanitization and separations of concerns correct... especially six years after its initial public release.

Anonymous on 2016-10-02 at 13:23:

The Unix philosophy is that you do one thing and to do it well.

That means that you should have a program reaping zombies and do nothing else.

You should have a program that starts the system services taking their dependencies into account. The oldfashioned way is to do them in a specific order, the new way is to calculate the dependencies. You should have a simple program that asks for an IP address on a network interface and does nothing else but to assign that IP address on that interface.

All these are simple programs that can be reasonably comprehended and verified.

Once you integrate all this, the complexity increases, and the amount of work to maintain the code base increases super-linearly. And as agwa says: the consequences of a problem become more troubling. Use the tools that are already there. The "runlevel switching" may want to accept "requests" from userspace. Make that a root-only socket or something.

If you want to make users able to switch runlevels or other "normally root" stuff, delegate that to "sudo" or modern variants of such tools. Or make a daemon that will accept commands from the normal users and passes them on as a privileged one. Again a bug somewhere in this process has the problems contained.

Reader Kyle on 2017-03-09 at 20:14:

"Those who do not understand Unix are condemned to reinvent it, poorly."

                                               -- Henry Spencer

Anonymous on 2016-10-01 at 23:49:

From my previous years in commercial software engineering. "Closed - Functions as intended."

Reader Christopher W. Carpenter on 2016-09-29 at 01:03:

I agree with a large part of this. The only things I disagree about:

1. As a user and hobbyist system admin, systemd has numerous advantages over sysv init(which was the only previous init system to be installed by default on most distributions). So I would not say it has "very few" advantages over other init systems (I haven't used the other more modern init systems enough to have an informed opinion there). I also think having one standard init system between most distributions is very nice, so the fragmentation of the previous non-sysv solutions annoyed me.

2. Systemd, by definition, does not use a "proprietary" DBUS API nor "proprietary" interfaces. See http://www.merriam-webster.com/dictionary/proprietary if you don't believe me. Anyone can legally and practically implement a systemd DNS service clone that communicates via d-bus exactly like the systemd one, if they wish. That being said, I agree that it can still be a problem since it forces others to either imitate systemd's interface or get applications to use something standardized instead/as well.

Andrew Ayer on 2016-09-29 at 01:56:

1. You misread me. I acknowledge that systemd's init system functionality has compelling features, even if terribly implemented. It's the other functionality -- DNS, DHCP, NTP -- where it doesn't offer anything new.

2. Fair enough. I replaced "proprietary" with "non-standard," which is what I meant.

Reader jampola on 2016-09-29 at 02:07:

As a user and hobbyist system admin

You seem to be a better adapted sysadmin than most of the non-hobbyist sysadmin's I know!

Reader James on 2016-09-29 at 20:00:

I am a user and a professional Systems Admin. I might well note that this is the very kind of thing that does nothing for me. And a whole lot to me. I'd like to spend less time working around the things it breaks, please.

Reader Rhy on 2016-09-29 at 05:21:

This article is fantastic. I've been screaming about the rotten state of Linux due to Poettewrong's sysd for months. I've moved everything over to devuan, or left wheezy in place. Oh and I like sysvinit. So there. I REALLY like having log files I can tail too.

Reader ferchunix on 2016-09-29 at 08:59:

Ok Andrew, it works, i tested it in Debian Stable and Ubuntu LTS 16.04, every systemd distro that relies all their init scripts in systemd will crash with that command with an unprivileged user, so my Debian Stable with old legacy init scripts is not vulnerable, new Ubuntu systemdized distros (16.04) will crash systemd, no new processes can be launched, launched processes can't restart/reload, you can't even shutdown or reboot gracefully... It's amazing but what about Responsible Disclosure?

Anonymous on 2017-03-24 at 05:45:

there is a reason slackware is the oldest, still in use, distribution of linux. it is well loved and cared for by its maintainers and does not suffer from "flavor of the week" that became the norm for every other distro that emerged from the late 90's and early 2000's.

== proud slackware user for over 20 years...

Reader freesys59 on 2016-09-29 at 09:52:

Slackware is not using this bullshit of systemd and working fine for many years now !

Reader Andrea Mistrali on 2016-09-29 at 12:17:

Devuan (https://devuan.org/) is Debian 8 systemd-free

Reader Wolf on 2016-09-29 at 10:42:

This does not appear to apply to OpenSUSE 13.2, as the path /run/systemd/notify does not exist. Running the command systemd-notify "" on its own does not appear to trigger the behaviour.

I'm not sure why that path doesn't exist.

Reader James J. on 2016-09-29 at 10:53:

Can you provide a link to the bug report you submitted to the systemd team before writing this blog post?

Reader Juha Autero on 2016-09-29 at 11:45:

People don't seem to understand how assert works. In release builds assert should be no-op. https://github.com/lattera/glibc/blob/master/assert/assert.h#L44 First of all assert is a debug tool, not error handling mechanism. Secondly, exiting with abort signal isn't very good error handling mechanism for systemd.

On the other hand, arguing that this makes systemd defective is hypocritical considering that Go FAQ says

Go doesn't provide assertions. They are undeniably convenient, but our experience has been that programmers use them as a crutch to avoid thinking about proper error handling and reporting. (https://golang.org/doc/faq#assertions)

Andrew Ayer on 2016-09-29 at 23:21:

systemd's assertions are enabled in release builds. However, systemd doesn't exit when an assertion fails. Instead it hangs in the pause system call. This prevents the whole system from crashing, although the system is left in a degraded state.

I would have to say that this bug supports Golang's position on assertions.

Anonymous on 2016-09-29 at 12:23:

:(){ :|: & };:

Anonymous on 2016-10-09 at 13:46:

Handled by age old unix configurations long before systemd came to be...

Reader orlwrite on 2016-09-29 at 12:25:

Nobody I know likes systemd, it's too intrusive in user space and goes way against the core Unix philosophy of doing one thing and doing it well.

The sooner systemd disappears out of the Unix/Linux eco system the better.

Thank you for exposing this bug. We tried it on our up-to-date RedHat 7.2 systems and sure enough they crashed very hard.

Now to get rid of Not-workManager.

Anonymous on 2016-10-12 at 18:10:

Shrike?!

Reader gdfuego on 2016-09-29 at 14:29:

If you haven't read it, check out this issue for another questionable security decision:

https://github.com/systemd/systemd/issues/959

Anonymous on 2016-09-29 at 16:31:

You could think systemd is as dangerous as ISIS

Reader Noryungi on 2016-09-29 at 16:55:

I feel sorry for people using distributions with systemd ''baked in''. During my day job, I am actually one of them, since my place of work is a big user of either Red Hat or Centos.

And then I remember that my personal systems are running either BSD or Slackware. And I feel better about my life.

Reader Andrey Bergman on 2016-09-29 at 18:38:

In some parts of SystemD they seem to use "bad coding practices from textbook":

      } else if (startswith(option, "keyfile-offset=")) {

              if (safe_atou(option+15, &arg_keyfile_offset) < 0) {
                      log_error("keyfile-offset= parse failure, ignoring.");
                      return 0;
              }

Do you see "option+15"? "15" here is a magic number, denoting the length of "keyfile-offset=":

https://cgit.freedesktop.org/systemd/systemd/tree/src/cryptsetup/cryptsetup.c#n106

Reader Damjan on 2016-09-29 at 21:37:

Send patch?

Reader Andrey Bergman on 2016-09-30 at 02:52:

I don't use systemD, so I even can't test the patch.

Reader Stefan on 2016-10-01 at 11:58:

So what's the intention to post this code example on various places (like Golem.de)? You don't use systemd, that's fine! So show us your code - maybe we can learn from you!

Reader Andrey Bergman on 2016-10-01 at 16:27:

Dear Stephan, you are very welcome to learn from me. This code snapshot with my comment teaches you that including magic numbers in the code is considered a bad practice for half a century.

Anonymous on 2016-10-01 at 14:19:

nobody uses systemD, linux basesystem is called systemd you can't test bad coding practices either, but it did not stop you

Anonymous on 2016-10-02 at 06:45:

The linux "basesystem" is called the "kernel" and is normally named vmlinuz. systemd IS NOT the BASESYSTEM of ANY LINUX distro - it is a service manager - which used to handled by initd - which only did one thing - init processes. It IS possible to TEST for magic numbers as well as review code. Good coding principles: 1. DRY - Don't repeat yourself - strings should exist in one place and be declared const. The length of the string if used often should be should be recorded as a singleton value also marked const at initialization. 2. Initialization is Instantiation. Goes without saying. There should not be ANY arithmatic with static numbers when doing text parsing. +15 and +XYZ is the exact same number of instructions in assembly. There is no code savings or clock cycle savings.

Learn to speak intelligently before you post.

Reader Stefan on 2016-10-02 at 08:44:

So feel free to write a bug report or even send a patch (or discuss on systemd-devel).

Otherwise it is damn useless to write comments...

Anonymous on 2016-10-02 at 12:52:

Yeah, it is totally useless if the other party refuses to listen and the only thing he can do is bawling like a baby. That's what you should't do as a developer, go and fix your damn spaghetti code!

Anonymous on 2016-11-06 at 04:37:

Yeah right, reporting bugs to poettering? "closed, not a bug" "closed, works as designed" "closed, i am right and you know nothing" "closed, everything else is crap and systemd does it the right way"

Reader Baylink on 2016-09-29 at 18:48:

I continue not to understand why so many high-profile distributions' Release Configuration Managers drank the Flavor-Aid.

And the bad part is not that they did, but that because they did, packages are no longer required to have initscripts for sane inits, meaning that you have no choice: you either run systemd (read: shoot yourself in the head), or you build initscripts by hand for every package you want to run.

The only thing worse is reading the replies from systemd partisans when it's explained to them why the choices systemd's design makes are bad engineering practice -- am I the only one who thinks the structure of these replies is way too similar to the replies one hears from Trump supporters when it's explained why he is bad engineering practice? :-)

Added this article to my list of "Why systemd is bad" articles; added the supporting commenters to my list of "people never to hire or recommend for responsible software engineering jobs".

Anonymous on 2016-10-01 at 20:48:

Gnome and developer hours available basically.

With Gnome yanking out consolekit support in favor of logind, and logind being part of the system package (not only is everything in the same git, they are all in the same src tree! When the LFS team shifts to eudev because extracting udev from the larger systemd source ball is too much effort you know something is up.) not moving to systemd becomes a hard sell.

Frankly the whole consolekit/logind thing seems overdesigned and aimed squarely at reproducing Microsoft's Active Directory. Is some corporate admin wants that, let him enable it. There is crap all reason for shipping this on every distro unless you think every last of your users are idiots (oh wait, Gnomes already do...).

Anonymous on 2016-10-06 at 08:47:

wouldn’t it be enough to have a simple syntax which can do everything systemd init syntax can?

Something like this?

MOUNTPOINT=/sys/fs/fuse/connections

depend() {
        need localmount
}

start() {
        ebegin "Starting fuse"
        if ! grep -qw fuse /proc/filesystems; then
                modprobe fuse >/dev/null 2>&1 || eerror $? "Error loading fuse module"
        fi
        if grep -qw fusectl /proc/filesystems && \
           ! grep -qw $MOUNTPOINT /proc/mounts; then
                mount -t fusectl none $MOUNTPOINT >/dev/null 2>&1 || \
                        eerror $? "Error mounting control filesystem"
        fi
        eend ${?}
}

stop() {
        ebegin "Stopping fuse"
        if grep -qw $MOUNTPOINT /proc/mounts; then
                umount $MOUNTPOINT >/dev/null 2>&1 || \
                        eerror $? "Error unmounting control filesystem"
        fi
        eend ${?}
}

That’s the syntax of OpenRC. You’ll notice the declarative dependencies along with the simple scripting in start() and stop().

And since it’s a scripting language, the answer to "can it do everything systemd can" is simply: Yes, it can.

A simple standard along with some convenience functions and building on existing tools already provides most of the functionality of systemd.

Anonymous on 2016-10-12 at 18:14:

At $DAY_JOB we have just moved our entire stack to Alpine, which features the wonderful OpenRC :)

Anonymous on 2016-10-02 at 06:48:

It's only because of the declarative syntax of systemd. If we could reverse engineer the systemd syntax with something more stable, we could go back to business as usual. Init scripts are hard to write. Writing configurations that are standard are easy. Systemd just happens to be bad code around a good idea - declarative definitions of services. However, trying to be everything and do everything is causing the bigger problems.

My system hangs because it takes just a little longer for all my items to come online. And that in turns brings 32GB, Core-I7 4720HQ to its knees on a reboot. I always have to restart the display-manager because too many things are starting at once.

Anonymous on 2016-10-02 at 07:19:

Good luck with that, as they seem to pile on new declarations with every release.

There is one guy that is trying though.

http://jdebp.eu./Softwares/nosh/

Anonymous on 2016-10-06 at 08:49:

Comment 31898 should have been a reply to you. Sorry: https://www.agwa.name/blog/post/how_to_crash_systemd_in_one_tweet#comment-31898

Reader funt on 2016-09-30 at 07:54:

Does not work.

It say: "Failed to notify init system"...

Andrew Ayer on 2016-09-30 at 15:34:

That probably means systemd has already crashed. Try using systemctl or rebooting.

Reader David Strauss on 2016-09-30 at 08:02:

I'm a systemd maintainer and have published some of my thoughts in a response: https://www.facebook.com/notes/david-timothy-strauss/how-to-throw-a-tantrum-in-one-blog-post/10108242959884190

This isn't an official response from the project, just one from someone who is very familiar with the issues in question.

Reader David Strauss on 2016-09-30 at 11:07:

At the request of a few readers, I've moved the post to a more sharable/annotation-friendly form on Medium:

https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d

Reader Andrey Bergman on 2016-09-30 at 14:19:

Dear David, you wrote so many words, but the essence of your article is simple "You are wrong! Why? Because!".

Andrew Ayer on 2016-10-01 at 16:16:

With a serving of ad hominem fallacies, strawmen, and dishonest arguments! Is this the best rebuttal systemd can muster?

Reader trivia on 2016-10-01 at 17:30:

Is making crass statements like "defective by design" all you have? if you hate systemd so much, don't use it. By your measure every program that has a bug is "defective by design".

Andrew Ayer on 2016-10-01 at 19:42:

Is making crass statements like "defective by design" all you have?

Nah, it's not like I wrote a blog post explaining my position with evidence and logical arguments and stuff.

if you hate systemd so much, don't use it.

I don't, and I hope others don't also.

By your measure every program that has a bug is "defective by design".

Try harder.

Anonymous on 2016-10-04 at 01:19:

This bug, which I was not able to reproduce on Kali with 215, is hardly related to the other shortcomings for which you despise systemd. The only possibly related one is 'complexity' ... so complex someone was forced to make a < vs <= mistake? plz

Anonymous on 2016-10-04 at 01:48:

Kali appears to be unstable now: unable to login via UI

Reader M on 2017-04-26 at 04:04:

Believe me, should be obvious from this thread, NOT using systemd would absolutely be the first choice for many of us.

Unfortunately, large numbers of us have -- through no choice of our own -- been forced to adopt it basically at gunpoint, as our corporate linuces have betrayed us (yes, we mean you RHEL, Debian, Ubuntu, et al)...

Where we can, we are attempting to move to NON-systemd linuces (Alpine, Devuan, et cetera). Personally, I like many other features of RHEL/CentOS, but systemd may be a deal breaker. (Take note, RHEL. You're actively driving away part of your formerly-loyal installed user base.)

That being observed, no, every program with a bug is not 'defective by design'.

Systemd however, IS -- due to what's been pointed out repeatedly by many individuals, in many venues, upon many occasions. The fundamental underpinnings of the design philosophy is critically flawed, as it not only violates the first premise(1) of UNIX philosophy 'do ONE thing, and do it well', but completely files in the face of it.

An init system (e.g. 'system management loader') does NOT need to take over power management, to control mount points, to handle disk encryption, to manage DNS, to control host naming, or locale, or time management, to add cron-style scheduling, to manage the network interfaces, to control sockets or inetd, or manage login sessions, just to name a few of it's excesses.

Add to that, questionable design choices such as opting for binary logging instead of cleartext, and you're simply adding fuel to the already large pyre a great many of us would gladly toss systemd onto.

Does init.d have it's own flaws. You bet.

Are the intentions of those working on systemd well-meaning? Sure.

Are some of the design objectives good ones? Sure.

But, much more importantly, are the design decisions good? For the most part, certainly no.

Many are fundamentally flawed for many of the same reasons Gnome is a complete disaster. They are overreaching, overcomplicated, with a snowballing morass of poorly written code, a open disdain for modern coding best practices, and design creep that apparently will continue expand and try to take over the world -- though creep doesn't even begin to address the avarice with which systemd claws for control of things that it has no business controling.

Importing flawed, monolithic, Windows-style design concepts into Linux is a poison pill.

The design concepts that should be in use here revolve around KISS -- keep it simple, stupid.

Instead, systemd is an Ode to Zawinksi's Law.

I'm waiting for the day it's announced that systemd has added an email client...

---

(1) Doug McIlroy, The Bell System Technical Journal, 1978

Reader Andrey Bergman on 2016-10-01 at 20:56:

That is why I don't see any reason for discussing article by David Strauss. He, just like most systemd supporters lies from the very beginning.

Anonymous on 2016-10-02 at 02:59:

This is a fair point, but it also isn’t a fundamental design element worthy of calls to abandon systemd. As most, it would justify a call to fork systemd and reverse the umask default.

I liked this bit. Because it is a fundamental design element when the developer team developing your software are completely unwilling to consider any suggestions from the outside. And you know that, because you suggested a fork right away, which of course, never would gain any traction if the fork is only about default policies.

I bet If I were to open a bug proposing to reverse the default behavior to a fail-safe behavior it would just be closed as wontfix with one or more arrogant "systemd is perfect as it is" comments below it. This gives good indications towards the design goals of systemd. It is not meant to be a solid and reliable foundation for Linux systems. That would be boring. It is meant to be an attractive sham-bandwagon to jump onto, selling the feel of flashy and new. Fail-safe behavior gets in the way of that, even if systemd cannot uphold the things it promises for every user.

So yes, it is worth abandoning systemd exactly due to issues like this.

Reader Eugen Rieck on 2016-10-02 at 07:59:

What a brilliant rebuttal - I read as far as "minor security issue".

This exactly validates the critisism of the systemd-mindset: Any non-privileged process being trivially able to de-facto crash a system is a "minor security issue".

Case closed.

Anonymous on 2016-10-06 at 09:03:

I stopped reading the message from David Strauss at

“Ayer’s work shows lots of promise — he’s clearly a talented security and software engineer — but…”

That’s deep in personal attacks and rhetorics.

Anonymous on 2016-10-02 at 06:50:

Did you address why this bug has existed for 2 years? Have you addressed magic numbers and why they appear?

Anonymous on 2016-09-30 at 09:01:

Instead of make it complicated with systemd, make it uncomplicated with https://en.wikipedia.org/wiki/OpenRC

Reader mar77i on 2016-09-30 at 10:36:

Reading the article makes me wonder whether the nsa is actively paying Lennart. As the situation we're now in is so obviously an american trait, something enormous and so reputedly awesome and terrible only depending on who you ask.

Anonymous on 2016-10-02 at 00:00:

I'm an American, I don't code like that. I'm also a sysadmin by trade going on 25 years now. Does that mean I am or am not classified by your 'American traits'?

Reader vilsu33 on 2016-10-04 at 18:52:

Please try not to be offended. He merely is referring to deep state/CIA/NSA -whatever mashup.

Then there are the normal people of USA. The world likes the regular usa citizen as a rule of thumb.

Reader cvvs on 2016-09-30 at 19:23:

Thank you, and the commenters, for providing tangible support to my instinctual distaste of systemd. Maybe it will encourage people to take another look at the BSD's.

Anonymous on 2016-10-01 at 08:43:

Hey folks... try http://smarden.org/runit/ as init-scheme

Anonymous on 2016-10-01 at 12:05:

This does not work. Tested with many systemd versions and distributions.

# Failed to notify init system: Connection refused

This article and most comments are only FUD.

Go and shoot yourself stupid haters

Andrew Ayer on 2016-10-01 at 16:16:

That message means systemd has already crashed, but you're probably too much of a flaming nutjob to understand that.

Anonymous on 2016-10-01 at 14:17:

how to make fool of yourself in one tweet https://medium.com/@davidtstrauss/how-to-throw-a-tantrum-in-one-blog-post-c2ccaa58661d

Andrew Ayer on 2016-10-01 at 16:17:

What systemd supporters lack in logical skills, they make up for by repeating themselves.

Anonymous on 2016-11-06 at 05:09:

Or repeating Poettering's The Correct way of how world is supposed to be.

Reader Matt Shulver on 2016-10-01 at 14:30:

Andrew, thank you for this very lucid article.

I have been philosophically uneasy with systemd ever since it rolled out on Arch and have read many opinions and counter-opinions on the matter. A few months ago I put a Manjaro spin (Fluxbox & openRC) on my laptop and everything worked just fine. I think, in light of the above arguments, I've reached the tipping point to take a deep breath and point my primary Arch system to the openRC alternatives. Thank you.

Reader estan on 2016-10-02 at 00:11:

What most people do not know is that it's possible to purge systemd from Ubuntu 16.06 and everything works well (I actually had to do this because the wiser systemd would be waiting for ethernet network to come up, even with a disconnected cable).

systemd is a bad implementation of what had already been achieved successfuly with SMF https://en.wikipedia.org/wiki/Service_Management_Facility

Anonymous on 2016-10-09 at 13:53:

How long before Pitt and crew gets what they want and purge that option under "too much hassle to maintain"?

Reader Scott Dowdle on 2016-10-02 at 01:07:

Hmm, you mean software has bugs? Why would systemd be any different. Now that the bug has been pointed out, I'm sure a fix will be available RSN... so thanks for helping make systemd better!

Anonymous on 2016-10-06 at 09:09:

Software generally has N-bugs-per-X-lines-of-code, with N and X varying by program type and the care of the development team.

However a program with 20x the lines of code as another program almost always has more bugs. Systemd has about 20x the lines of code as OpenRC.

Anonymous on 2016-10-02 at 01:28:

A nice side effect from all the "bruhaha" of this hack is you can see who works for NSA and who doesn't.

Reader iskander on 2016-10-02 at 07:24:

Totally agree with you Andrew. Nicely summed up.

Recently i kicked Arch Linux to the curb and switched to Void Linux because of systemd.

Regards

Reader mark on 2016-10-02 at 08:38:

This is typical for systemd developers - force it onto everyone, then refuse to fix fatal bugs.

You can see this in countless issue reports at their tracker.

The only sane conclusion is to avoid systemd.

Reader Stefan on 2016-10-02 at 08:48:

So then please give some concrete examples in the systemd bug tracker!

Anonymous on 2016-10-02 at 13:00:

nobody asked for systemd. go and fix it yourself.

Reader idan on 2016-10-02 at 10:12:

the command does not crash centos 7- 1511. checked.

Reader vedeheme on 2016-12-01 at 18:55:

try this on fedora25. I don'y see any crash

Anonymous on 2016-10-02 at 11:49:

'while true ; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify "" ; done' do crash centos 7- 1511. checked.

Reader herauthon on 2016-10-02 at 15:55:

If its needed it will result in a reinstall

I just removed the systemd And it drags the whole desktop environment with it.. into zerobe

will post the results if there is anything to note

be gone... daemon!

Anonymous on 2016-10-02 at 17:31:

As a security analyst for a Fortune 50 company, you can bet I'll be trying this formally. A rogue employee DoSing a system? That's a big problem for us. We throw a lot of money at Red Hat; a hole like this would set off a lot of panic for our IT teams, and some of that will be aimed at RH. Getting an unprivileged shell prompt isn't all that hard, sadly. Parlaying that into a DoS with a short one-liner is a HUGE issue. Strauss' reply isn't reassuring, it's frightening... and will likely make its way into my gap writeup, so the CISO can consider it. Our business is all about managing risk, and it sounds like Red Hat's systemd team is taking, and exposing us to, unnecessary risk.

Anonymous on 2016-10-02 at 17:57:

Sorry I am new to the systemd wars and there is something that I do not understand: If most people like the features that systemd offers but many do not like how they are being implemented, why there is not a fork?

Anonymous on 2016-10-02 at 18:18:

The issues people have with systemd mean forking it wouldn't solve the problem. Systemd flies in the face of Unix philosophy, by having one program try to do it all. Forking it would never fix that, and there are other good alternatives out there such as runit and openrc. There was a fork at one point called uselessd, where the developers tried to remove as much of the unnecessary code in systemd as possible. The problem was that systemd was so large by then that maintaining the fork with only a few people was too much work, so it died. If you don't like systemd, the best option is to just not use it. There are other programs that can do the same thing as systemd, it's just you have to use more than one program (which is the smarter way to do it imo, with systemd you have a single point of failure, which is not good, and with as much code as systemd has, more errors are more likely to occur).

Anonymous on 2016-10-03 at 10:52:

Most people don't care as long as it works.

Of those that care most people both like what it offers an are relatively happy with it's implementation.

No doubt systemd is a major change from sysvinit and some people just don't like change.

Reader AC on 2016-10-03 at 18:00:

People hate stupid and bad change, and for good reason.

Systemd is an abortion. Poettering is a either a short-sighted fool or a malicious tool.

Anonymous on 2016-10-04 at 02:55:

I think the word you meant to use is 'abomination'.

Anonymous on 2016-10-06 at 09:12:

most people just want their programs to work. Systemd was pushed as a dependency of Gnome - which many people need - at least partly due to systemd devs having a say in Gnome development.

Anonymous on 2016-10-09 at 13:59:

Because:

A: there are already multiple alternatives if all you want it dependency based init. But the scope of systemd is creeping ever wider.

B: thanks to the scope creep more and more depends on systemd being present (particularly the stuff coming out of Freedesktop). Gnome for instance wants logind to be present to handle "sessions". These days Logind is responsible for handling sleep and hibernate, with formerly independent powerd/powerkit acting as a compatibility shim. Wayland will depend on Logind and Polkit for managing access to tings like keyboard and GPU.

Reader Justme on 2016-10-02 at 22:13:

The magic number thing reminds me of sony's little gem. . .

int getRandomNumber()
{
    return 4;
}

Reader mary on 2016-10-03 at 03:32:

yeah i just mindlessly opened a terminal and typed that exact command after breakfast while i was still only partially awake. thanks for the warning. i also find it is not good if you highlight all the folders in the / directory then delete them late at night. havent tried this after breakfast

Reader Danielle McLean on 2016-10-12 at 03:25:

The difference is that this command can be performed as any user. To delete everything in /, you need to be root.

Reader Scott Dowdle on 2016-10-03 at 04:19:

Fedora released fixed systemd package for Fedora 24 today. Expect other updates RSN if they haven't already been released. Look, a bug in software that got fixed. Expect billions more in every piece of software ever written.

Reader yo dawg on 2016-10-03 at 16:32:

Systemd is a horrible thing for Linux. All you ubuntu n00bs have caused this

Anonymous on 2016-10-09 at 14:01:

Actually Ubuntu was a late adopter. The 1337 distros Fedora and Arch were the early adopters...

Reader chickenhead on 2016-10-03 at 20:23:

Been saying it for a while...systemd is ready made for a rootkit. And it will be the mother of all rootkits when it happens.

But no matter, the systemd astroturf brigade will convince us to "go to freebsd" if we don't like it, right?

Anonymous on 2016-10-09 at 14:02:

Could have sworn there is already a systemd-dependent bitcoin worm floating around...

Anonymous on 2016-10-05 at 21:08:

nothing happened after running the command in ubuntu 16.04

Reader Willy on 2016-10-06 at 21:44:

You forgot that the most important reason for not using it is that it is totally undebuggable. You cannot strace it, cannot gdb it etc... Just because it's PID1. Can you imagine a system without strace ? It exists with systemd cancering more and more of your system, and it's a real pain each time I end up on such a self-sustaining system which refuses to cooperate with humans.

Anonymous on 2016-10-08 at 03:26:

WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO. WE TOLD YOU SO.

Anonymous on 2016-10-08 at 03:28:

Have fun debugging with those binaries - Oh they are SO much better than scripts. Too bad the millennials destroyed linux with systemd.

Reader Stuart on 2016-10-08 at 15:10:

As well as Devuan other NON Systemd alternatives are:

Manjaro OpenRC (Arch Linux desktop) https://sourceforge.net/projects/manjaro-openrc/

Alpine Linux (also uses openrc) http://alpinelinux.org/

Reader Anonymus on 2016-10-15 at 08:34:

Linux guru talks about the Linux desktop https://www.youtube.com/watch?v=MpDdGOKZ3dg

Reader Strichard Hallman on 2016-10-21 at 02:30:

This made my day. I am stuck with CentOS 6 because I'll never countenance systemd (or pulseaudio). Thank you Ayer!!!

Anonymous on 2016-11-01 at 16:36:

seem this issue has been fixed, run while true; do NOTIFY_SOCKET=/run/systemd/notify systemd-notify ""; done

for 30 minutes on FC24 4.5.5-300.fc24.x86_64 on virtual box with 1.5GB Ram still running fine. Load Average 1.48

Anonymous on 2016-11-01 at 16:49:

Ok, above was too hasty conclusions, ssh is fine, but VM console was unresponsive, also when try to reboot Vm its say

Failed to get properties: Failed to activate service 'org.freedesktop.systemd1': timed out

Failed to open /dev/initctl: No such device or address Failed to talk to init daemon.

daang, issue still not fixed.

Anonymous on 2016-11-03 at 09:59:

RHEL 7.3 is live today but bug still not fixed.

Anonymous on 2016-12-09 at 02:51:

how's this for a stake in the ground: I will never hire anyone who has contributed to systemd. it's only me in my capacity as a software architect with hiring authority, but if you have a commit in the systemd mess, you got a black ball. I will check.

Anonymous on 2017-02-21 at 13:04:

I'm not against systemd in principle (although I don't use distros/versions based on it, as I've had problems with it in the past) - although I do get very disturbed by reports of the ever-increasing scope of its control.

I AM against not having the option whether or not to use it (and I get the red mist when I read that systemd is permanently welded into a particular distro/version, and if I don't like it, I should go elsewhere.)

One thing really puzzles me. We now have devuan - systemd-free fork of debian. Apparently (based on Google searches at any rate) no one has tried producing a systemd-free fork of Red Hat or CentOS 7 (or any other RPM-based distro) (or, if someone has, he/she isn't letting on.) Why not? Given the existence of devuan, I can't believe that it would be an impossible task.

Anonymous on 2017-06-30 at 04:52:

I'm late to the party on this, and doubt anyone will see my comment, but after reading "...Been saying it for a while...systemd is ready made for a rootkit." I felt I had to.

There are already rootkits in the wild. Systemd is a nasty piece of work. I'm back to slackware now and it took eons to get a clean install. My whole system was junked.

I can't go into detail because it would take a dissertation to explain everything and as a linux user of more than 20 years still don't fully understand, but basically it involves the use of systemd, containers - and actually uses portions of selinux against you. You think you su in bash but you're not root. Real root is 'above' root. All formatting say /dev/sda1 does is format the container equivalent of what's called /dev/sda1 so it's a nightmare to break out of. Shit is copied back on reinstall of other systems.

Sorry for incorrect terms (English second language).

Andrew Ayer

Sections

Blog