May 29, 2020

Antoine Beaupré

Upgrading my home server uplink

For more than a few decades now (!), I've been running my own server. First it was just my old Pentium 1 squatting on university networks, but eventually grew into a real server somewhere at the dawn of the millenia. Apart from the university days, the server was mostly hosted over ADSL links, first a handful of megabits, up to the current 25 Mbps down, 6 Mbps up that the Bell Canada network seems to allow to its resellers (currently Teksavvy Internet, or TSI).

Why change?

Obviously, this speed is showing its age, and especially in this age of Pandemia where everyone is on videoconferencing all the time. But it's also inconvenient when I need to upload large files on the network. I also host a variety of services on this network, and I always worry that any idiot can (rather trivially) DoS my server, so I often feel I should pack a little more punch at home (although I have no illusions about my capacity of resisting any sort of DoS attack at home of course).

Also, the idea of having gigabit links at home brings back the idea of the original internet, that everyone on the internet is a "peer". "Client" and "servers" are just a technical distinction and everyone should be able to run a server.

Requirements

So I'm shopping for a replacement. The requirements are:

  1. higher speed than 25/6, preferably 100mbps down, 30mbps up, or more. ideally 1gbps symmetric.

  2. static or near-static IP address: I run a DNS server with its IP in the glue records (although the latter could possibly be relaxed). ideally a /29 or more.

  3. all ports open: I run an SMTP server (incoming and outgoing) along with a webserver and other experiments. ideally, no firewall or policy should be blocking me from hosting stuff, unless there's an attack or security issue, obviously.

  4. clean IP address: the SMTP server needs to have a good reputation, so the IP address should not be in a "residential space" pool.

  5. IPv6 support: TSI offers IPv6 support, but it is buggy (I frequently have to restart the IPv6 interface on the router because the delegated block stops routing, and they haven't been able to figure out the problem). ideally, a /56.

  6. less than 100$/mth, ideally close to the current 60$/mth I pay.

(All amounts in $CAD.)

Contestants

I wrote a similar message asking major ISPs in my city for those services, including business service if necessary:

I have not contacted those providers:

  • Bell Canada: i have sworn, two decades ago, never to do business with that company ever again. They have a near-monopoly on almost all telcos in Canada and I want to give them as little money as possible.

  • Videotron: I know for a fact they do not allow servers on their network, and their IPv6 has been in beta for so long it has become somewhat of a joke now

I might have forgotten some, let me know if you're in the area and have a good recommendation. I'll update this post with findings as they come in.

Keep in mind that I am in a major Canadian city, less than a kilometer from a major telco exchange site, so it's not like I'm in a rural community. This should just work.

TSI

First answer from TSI was "we do not provide 30mbps upload on residential services", even though they seem to have that package on their website. They confirmed that they "don't have a option more than 10 mbps upload."

Oricom

They offer a 100/30 link for 65$ plus 25$ for a static IP.

No IPv6 yet, unlikely to come soon. No services blocked, they have their own PoP within Videotron's datacenters so clients come out from their IP address space.

I can confirm that the IP is fairly static from the office.

Ebox

No response yet.

Beanfield / Openface

Even though they have a really interesting service (50$/mth for unlimited 1gbps), they are not in my building. I did try to contact them over chat, they told me to call, and I left a message

29 May, 2020 06:28PM

hackergotchi for Thomas Goirand

Thomas Goirand

A quick look into Storcli packaging horror

So, Megacli is to be replaced by Storcli, both being proprietary tools for configuring RAID cards from LSI.

So I went to download what’s provided by Lenovo, available here:
https://support.lenovo.com/fr/en/downloads/ds041827

It’s very annoying, because they force users to download a .zip file containing a deb file, instead of providing a Debian repository. Well, ok, though at least there’s a deb file there. Let’s have a look what’s using my favorite tool before installing (ie: let’s run Lintian).
Then it’s a horror story. Not only there’s obvious packaging wrong, like the package provide stuff in /opt, and all is statically linked and provide embedded copies of libm and ncurses, or even the package is marked arch: all instead of arch: amd64 (in fact, the package contains both i386 and amd64 arch files…), but there’s also some really wrong things going on:

E: storcli: arch-independent-package-contains-binary-or-object opt/MegaRAID/storcli/storcli
E: storcli: embedded-library opt/MegaRAID/storcli/storcli: libm
E: storcli: embedded-library opt/MegaRAID/storcli/storcli: ncurses
E: storcli: statically-linked-binary opt/MegaRAID/storcli/storcli
E: storcli: arch-independent-package-contains-binary-or-object opt/MegaRAID/storcli/storcli64
E: storcli: embedded-library opt/MegaRAID/storcli/storcli64: libm
E: storcli: embedded-library … use –no-tag-display-limit to see all (or pipe to a file/program)
E: storcli: statically-linked-binary opt/MegaRAID/storcli/storcli64
E: storcli: changelog-file-missing-in-native-package
E: storcli: control-file-has-bad-permissions postinst 0775 != 0755
E: storcli: control-file-has-bad-owner postinst asif/asif != root/root
E: storcli: control-file-has-bad-permissions preinst 0775 != 0755
E: storcli: control-file-has-bad-owner preinst asif/asif != root/root
E: storcli: no-copyright-file
E: storcli: extended-description-is-empty
W: storcli: essential-no-not-needed
W: storcli: unknown-section storcli
E: storcli: depends-on-essential-package-without-using-version depends: bash
E: storcli: wrong-file-owner-uid-or-gid opt/ 1000/1000
W: storcli: non-standard-dir-perm opt/ 0775 != 0755
E: storcli: wrong-file-owner-uid-or-gid opt/MegaRAID/ 1000/1000
E: storcli: dir-or-file-in-opt opt/MegaRAID/
W: storcli: non-standard-dir-perm opt/MegaRAID/ 0775 != 0755
E: storcli: wrong-file-owner-uid-or-gid opt/MegaRAID/storcli/ 1000/1000
E: storcli: dir-or-file-in-opt opt/MegaRAID/storcli/
W: storcli: non-standard-dir-perm opt/MegaRAID/storcli/ 0775 != 0755
E: storcli: wrong-file-owner-uid-or-gid … use –no-tag-display-limit to see all (or pipe to a file/program)
E: storcli: dir-or-file-in-opt opt/MegaRAID/storcli/storcli
E: storcli: dir-or-file-in-opt … use –no-tag-display-limit to see all (or pipe to a file/program)

Some of the above are grave security problems, like wrong Unix mode for folders, even with the preinst script installed as non-root.
I always wonder why this type of tool needs to be proprietary. They clearly don’t know how to get packaging right, so they’d better just provide the source code, and let us (the Debian community) do the work for them. I don’t think there’s any secret that they are keeping by hiding how to configure the cards, so it’s not in the vendor’s interest to keep everything closed. Or maybe they are just hiding really bad code in there, that they are ashamed to share? In any way, they’d better not provide any package than this pile of dirt (and I’m trying to stay polite here…).

29 May, 2020 10:56AM by Goirand Thomas

hackergotchi for Keith Packard

Keith Packard

picolibc-string-float

Float/String Conversion in Picolibc

Exact conversion between strings and floats seems like a fairly straightforward problem. There are two related problems:

  1. String to Float conversion. In this case, the goal is to construct the floating point number which most closely approximates the number represented by the string.

  2. Float to String conversion. Here, the goal is to generate the shortest string which, when fed back into the String to Float conversion code, exactly reproduces the original value.

When linked together, getting from float to string and back to float is a “round trip”, and an exact pair of algorithms does this for every floating point value.

Solutions for both directions were published in the proceedings of the ACM SIGPLAN 1990 conference on Programming language design and implementation, with the string-to-float version written by William Clinger and the float-to-string version written by Guy Steele and Jon White. These solutions rely on very high precision integer arithmetic to get every case correct, with float-to-string requiring up to 1050 bits for the 64-bit IEEE floating point format.

That's a lot of bits.

Newlib Float/String Conversion

The original newlib code, written in 1998 by David M. Gay, has arbitrary-precision numeric code for these functions to get exact results. However, it has the disadvantages of performing numerous memory allocations, consuming considerable space for the code, and taking a long time for conversions.

The first disadvantage, using malloc during conversion, ended up causing a number of CVEs because the results of malloc were not being checked. That's bad on all platforms, but especially bad for embedded systems where reading and writing through NULL pointers may have unknown effects.

Upstream newlib applied a quick fix to check the allocations and call abort. Again, on platforms with an OS, that at least provides a way to shut down the program and let the operating environment figure out what to do next. On tiny embedded systems, there may not be any way to log an error message or even restart the system.

Ok, so we want to get rid of the calls to abort and have the error reported back through the API call which caused the problem. That's got two issues, one mere technical work, and another mere re-interpretation of specifications.

Let's review the specification issue. The libc APIs involved here are:

Input:

  • scanf
  • strtod
  • atof

Output:

  • printf
  • ecvt, fcvt
  • gcvt

Scanf and printf are both documented to set errno to ENOMEM when they run out of memory, but none of the other functions takes that possibility into account. So we'll make some stuff up and hope it works out:

  • strtod. About the best we can do is report that no conversion was performed.

  • atof. Atof explicitly fails to detect any errors, so all we can do is return zero. Maybe returning NaN would be better?

  • ecvt, fcvt and gcvt. These return a pointer, so they can return NULL on failure.

Now, looking back at the technical challenge. That's a “simple” matter of inserting checks at each allocation, or call which may result in an allocation, and reporting failure back up the call stack, unwinding any intermediate state to avoid leaking memory.

Testing Every Possible Allocation Failure

There are a lot of allocation calls in the newlib code. And the call stack can get pretty deep. A simple visual inspection of the code didn't seem sufficient to me to validate the allocation checking code.

So I instrumented malloc, making it count the number of allocations and fail at a specific one. Now I can count the total number of allocations done over the entire test suite run for each API involved and then run the test suite that many times, failing each allocation in turn and checking to make sure we recover correctly. By that, I mean:

  • No stores through NULL pointers
  • Report failure to the application
  • No memory leaks

There were about 60000 allocations to track, so I ran the test suite that many times, which (with the added malloc tracing enabled) took about 12 hours.

Bits Pushed to the Repository

With the testing complete, I'm reasonably confident that the code is now working, and that these CVEs are more completely squashed. If someone is interested in back-porting the newlib fixes upstream to newlib, that would be awesome. It's not completely trivial as this part of picolibc has diverged a bit due to the elimination of the reent structure.

Picolibc's “Tinystdio” Float/String Conversion

Picolibc contains a complete replacement for stdio which was originally adopted from avr libc. That's a stdio implementation designed to run on 8-bit Atmel processors and focuses on very limited memory use and small code size. It does this while maintaining surprisingly complete support for C99 printf and scanf support.

However, it also does this without any arbitrary precision arithmetic, which means it doesn't get the right answer all of the time. For most embedded systems, this is usually a good trade off -- floating point input and output are likely to be largely used for diagnostics and debugging, so “mostly” correct answers are probably sufficient.

The original avr-libc code only supports 32-bit floats, as that's all the ABI on those processors has. I extended that to 64-, 80- and 128- bit floats to cover double and long double on x86 and RISC-V processors. Then I spent a bunch of time adjusting the code to get it to more accurately support C99 standards.

Tinystdio also had strtod support, but it was missing ecvt, fcvt and gcvt. For those, picolibc was just falling back to the old newlib code, which introduced all of the memory allocation issues we've just read about.

Fixing that so that tinystdio was self-contained and did ecvt, fcvt and gcvt internally required writing those functions in terms of the float-to-string primitives already provided in tinystdio to support printf. gcvt is most easily supported by just calling sprintf.

Once complete, the default picolibc build, using tinystdio, no longer does any memory allocation for float/string conversions.

29 May, 2020 06:16AM

May 28, 2020

hackergotchi for Bits from Debian

Bits from Debian

New Debian Developers and Maintainers (March and April 2020)

The following contributors got their Debian Developer accounts in the last two months:

  • Paride Legovini (paride)
  • Ana Custura (acute)
  • Felix Lechner (lechner)

The following contributors were added as Debian Maintainers in the last two months:

  • Sven Geuer
  • Håvard Flaget Aasen

Congratulations!

28 May, 2020 04:30PM by Jean-Pierre Giraud

Elana Hashman

Presenter mode in LibreOffice Impress without an external display

I typically use LibreOffice Impress for my talks, much to some folks' surprise. Yes, you can make slides look okay with free software! But there's one annoying caveat that has bothered me for ages.

Impress makes it nearly impossible to enter presenter mode with a single display, while also displaying slides. I have never understood this limitation, but it's existed for a minimum of seven years.

I've tried all sorts of workarounds over the years, including a macro that forces LibreOffice into presenter mode, which I never was able to figure out how to reverse once I ran it...

This has previously been an annoyance but never posed a big problem, because when push came to shove I could leave my house and use an external monitor or screen when presenting at meetups. But now, everything's virtual, I'm in lockdown indefinitely, and I don't have another display available at home. And about 8 hours before speaking at a meetup today, I realized I didn't have a way to share my slides while seeing my speaker notes. Uh oh.

So I got this stupid idea.

...why don't I just placate LibreOffice with a FAKE display?

Virtual displays with xrandr

My GPU had this capability innately, it turns out, if I could just whisper the right incantations to unlock its secrets:

ehashman@red-dot:~$ cat /usr/share/X11/xorg.conf.d/20-intel.conf 
Section "Device"
    Identifier "intelgpu0"
    Driver "intel"
    Option "VirtualHeads" "1"
EndSection

After restarting X to allow this newly created config to take effect, I now could see two new virtual displays available for use:

ehashman@red-dot:~$ xrandr
Screen 0: minimum 8 x 8, current 3840 x 1080, maximum 32767 x 32767
eDP1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 310mm x 170mm
   1920x1080     60.01*+  59.93  
   ...
   640x360       59.84    59.32    60.00  
DP1 disconnected (normal left inverted right x axis y axis)
DP2 disconnected (normal left inverted right x axis y axis)
HDMI1 disconnected (normal left inverted right x axis y axis)
HDMI2 disconnected (normal left inverted right x axis y axis)
VIRTUAL1 disconnected (normal left inverted right x axis y axis)
VIRTUAL2 disconnected (normal left inverted right x axis y axis)

Nice. Now, to actually use it:

ehashman@red-dot:~$ xrandr --addmode VIRTUAL1 1920x1080
ehashman@red-dot:~$ xrandr --output VIRTUAL1 --mode 1920x1080 --right-of eDP1

And indeed, after running these commands, I found myself with a virtual display, very happy to black hole all my windows, available to the imaginary right of my laptop screen.

This allowed me to mash that "Present" button in LibreOffice and get my presenter notes on my laptop display, while putting my actual slides in a virtual time-out that I could still screenshare!

Wouldn't it be nice if LibreOffice just fixed the actual bug? 🤷

Well, actually...

I must forgive myself for my stage panic. The talk ended up going great, and the immediate problem was solved. But it turns out this bug has been addressed upstream! It's just... not well-documented.

A couple years ago, there was a forum post on ask.libreoffice.org that featured this exact question, and a solution was provided!

Yes, use Open Expert Configuration via Tools > Options > LibreOffice > Advanced. Search for StartAlways. You should get a node org.openoffice.Office.PresenterScreen with line Presenter. Double-click that line to toggle the boolean value to true.

I tested this out locally and... yeah that works. But it turns out the bug from 2013 had not been updated with this solution until just a few months ago.

There are very limited search results for this configuration property. I wish this was much better documented. But so it goes with free software; here's a hack and a real solution as we all try to improve things :)

28 May, 2020 01:15AM by Elana Hashman

May 27, 2020

hackergotchi for Kees Cook

Kees Cook

security things in Linux v5.5

Previously: v5.4.

I got a bit behind on this blog post series! Let’s get caught up. Here are a bunch of security things I found interesting in the Linux kernel v5.5 release:

restrict perf_event_open() from LSM
Given the recurring flaws in the perf subsystem, there has been a strong desire to be able to entirely disable the interface. While the kernel.perf_event_paranoid sysctl knob has existed for a while, attempts to extend its control to “block all perf_event_open() calls” have failed in the past. Distribution kernels have carried the rejected sysctl patch for many years, but now Joel Fernandes has implemented a solution that was deemed acceptable: instead of extending the sysctl, add LSM hooks so that LSMs (e.g. SELinux, Apparmor, etc) can make these choices as part of their overall system policy.

generic fast full refcount_t
Will Deacon took the recent refcount_t hardening work for both x86 and arm64 and distilled the implementations into a single architecture-agnostic C version. The result was almost as fast as the x86 assembly version, but it covered more cases (e.g. increment-from-zero), and is now available by default for all architectures. (There is no longer any Kconfig associated with refcount_t; the use of the primitive provides full coverage.)

linker script cleanup for exception tables
When Rick Edgecombe presented his work on building Execute-Only memory under a hypervisor, he noted a region of memory that the kernel was attempting to read directly (instead of execute). He rearranged things for his x86-only patch series to work around the issue. Since I’d just been working in this area, I realized the root cause of this problem was the location of the exception table (which is strictly a lookup table and is never executed) and built a fix for the issue and applied it to all architectures, since it turns out the exception tables for almost all architectures are just a data table. Hopefully this will help clear the path for more Execute-Only memory work on all architectures. In the process of this, I also updated the section fill bytes on x86 to be a trap (0xCC, int3), instead of a NOP instruction so functions would need to be targeted more precisely by attacks.

KASLR for 32-bit PowerPC
Joining many other architectures, Jason Yan added kernel text base-address offset randomization (KASLR) to 32-bit PowerPC.

seccomp for RISC-V
After a bit of long road, David Abdurachmanov has added seccomp support to the RISC-V architecture. The series uncovered some more corner cases in the seccomp self tests code, which is always nice since then we get to make it more robust for the future!

seccomp USER_NOTIF continuation
When the seccomp SECCOMP_RET_USER_NOTIF interface was added, it seemed like it would only be used in very limited conditions, so the idea of needing to handle “normal” requests didn’t seem very onerous. However, since then, it has become clear that the overhead of a monitor process needing to perform lots of “normal” open() calls on behalf of the monitored process started to look more and more slow and fragile. To deal with this, it became clear that there needed to be a way for the USER_NOTIF interface to indicate that seccomp should just continue as normal and allow the syscall without any special handling. Christian Brauner implemented SECCOMP_USER_NOTIF_FLAG_CONTINUE to get this done. It comes with a bit of a disclaimer due to the chance that monitors may use it in places where ToCToU is a risk, and for possible conflicts with SECCOMP_RET_TRACE. But overall, this is a net win for container monitoring tools.

EFI_RNG_PROTOCOL for x86
Some EFI systems provide a Random Number Generator interface, which is useful for gaining some entropy in the kernel during very early boot. The arm64 boot stub has been using this for a while now, but Dominik Brodowski has now added support for x86 to do the same. This entropy is useful for kernel subsystems performing very earlier initialization whre random numbers are needed (like randomizing aspects of the SLUB memory allocator).

FORTIFY_SOURCE for MIPS
As has been enabled on many other architectures, Dmitry Korotin got MIPS building with CONFIG_FORTIFY_SOURCE, so compile-time (and some run-time) buffer overflows during calls to the memcpy() and strcpy() families of functions will be detected.

limit copy_{to,from}_user() size to INT_MAX
As done for VFS, vsnprintf(), and strscpy(), I went ahead and limited the size of copy_to_user() and copy_from_user() calls to INT_MAX in order to catch any weird overflows in size calculations.

That’s it for v5.5! Let me know if there’s anything else that I should call out here. Next up: Linux v5.6.

© 2020, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

27 May, 2020 08:04PM by kees

Russ Allbery

Review: Middlegame

Review: Middlegame, by Seanan McGuire

Publisher: Tor
Copyright: May 2019
ISBN: 1-250-19551-9
Format: Kindle
Pages: 528

Roger and Dodger are cuckoo children, alchemical constructs created by other alchemical constructs masquerading as humans. They are halves of the primal force of the universe, the Doctrine of Ethos (which is not what the Doctrine of Ethos is, but that is one of my lesser problems with this book), divided into language and math and kept separate to properly mature. In this case, separate means being adopted by families on opposite coasts of the United States, ignorant of each other's existence and closely monitored by agents Reed controls. None of that prevents Roger and Dodger from becoming each other's invisible friends at the age of seven, effortlessly communicating psychically even though they've never met.

That could have been the start of an enjoyable story that hearkened back to an earlier age of science fiction: the secret science experiments discover that they have more power than their creators expected, form a clandestine alliance, and fight back against the people who are trying to control them. I have fond memories of Escape to Witch Mountain and would have happily read that book.

Unfortunately, that isn't the story McGuire wanted to tell. The story she told involves ripping Roger and Dodger apart, breaking Dodger, and turning Roger into an abusive asshole.

Whooboy, where to start. This book made me very angry, in a way that I would not have been if it didn't contain the bones of a much better novel. Four of them, to be precise: four other books that would have felt less gratuitously cruel and less apparently oblivious to just how bad Roger's behavior is.

There are some things to like. One of them is that the structure of this book is clever. I can't tell you how it's clever because the structure doesn't become clear until more than halfway through and it completely changes the story in a way that would be a massive spoiler. But it's an interesting spin on an old idea, one that gave Roger and Dodger a type of agency in the story that has far-ranging implications. I enjoyed thinking about it.

That leads me to another element I liked: Erin. She makes only fleeting appearances until well into the story, but I thought she competed with Dodger for being the best character of the book. The second of the better novels I saw in the bones of Middlegame was the same story told from Erin's perspective. I found myself guessing at her motives and paying close attention to hints that led to a story with a much different emotional tone. Viewing the ending of the book through her eyes instead of Roger and Dodger's puts it in a different, more complicated, and more thought-provoking light.

Unfortunately, she's not McGuire's protagonist. She instead is one of the monsters of this book, which leads to my first, although not my strongest, complaint. It felt like McGuire was trying too hard to write horror, packing Middlegame with the visuals of horror movies without the underlying structure required to make them effective. I'm not a fan of horror personally, so to some extent I'm grateful that the horrific elements were ineffective, but it makes for some frustratingly bad writing.

For example, one of the longest horror scenes in the book features Erin, and should be a defining moment for the character. Unfortunately, it's so heavy on visuals and so focused on what McGuire wants the reader to be thinking that it doesn't show any of the psychology underlying Erin's decisions. The camera is pointed the wrong way; all the interesting storytelling work, moral complexity, and world-building darkness is happening in the character we don't get to see. And, on top of that, McGuire overuses foreshadowing so much that it robs the scene of suspense and terror. Again, I'm partly grateful, since I don't read books for suspense and terror, but it means the scene does only a fraction of the work it could.

This problem of trying too hard extends to the writing. McGuire has a bit of a tendency in all of her books to overdo the descriptions, but is usually saved by narrative momentum. Unfortunately, that's not true here, and her prose often seems overwrought. She also resorts to this style of description, which never fails to irritate me:

The thought has barely formed when a different shape looms over him, grinning widely enough to show every tooth in its head. They are even, white, and perfect, and yet he somehow can't stop himself from thinking there's something wrong with them, that they're mismatched, that this assortment of teeth was never meant to share a single jaw, a single terrible smile.

This isn't effective. This is telling the reader how they're supposed to feel about the thing you're describing, without doing the work of writing a description that makes them feel that way. (Also, you may see what I mean by overwrought.)

That leads me to my next complaint: the villains.

My problem is not so much with Leigh, who I thought was an adequate monster, if a bit single-note. There's some thought and depth behind her arguments with Reed, a few hints of her own motives that were more convincing for not being fully shown. The descriptions of how dangerous she is were reasonably effective. She's a good villain for this type of dark fantasy story where the world is dangerous and full of terrors (and reminded me of some of the villains from McGuire's October Daye series).

Reed, though, is a storytelling train wreck. The Big Bad of the novel is the least interesting character in it. He is a stuffed tailcoat full of malicious incompetence who is only dangerous because the author proclaims him to be. It only adds insult to injury that he kills off a far more nuanced and creative villain before the novel starts, replacing her ambiguous goals with Snidely Whiplash mustache-twirling. The reader has to suffer through extended scenes focused on him as he brags, monologues, and obsesses over his eventual victory without an ounce of nuance or subtlety.

Worse is the dynamic between him and Leigh, which is only one symptom of the problem with Middlegame that made me the most angry: the degree to this book oozes patriarchy. Every man in this book, including the supposed hero, orders around the women, who are forced in various ways to obey. This is the most obvious between Leigh and Reed, but it's the most toxic, if generally more subtle, between Roger and Dodger.

Dodger is great. I had absolutely no trouble identifying with and rooting for her as a character. The nasty things that McGuire does to her over the course of the book (and wow does that never let up) made me like her more when she tenaciously refuses to give up. Dodger is the math component of the Doctrine of Ethos, and early in the book I thought McGuire handled that well, particularly given how difficult it is to write a preternatural genius. Towards the end of this book, her math sadly turns into a very non-mathematical magic (more on this in a moment), but her character holds all the way through. It felt like she carved her personality out of this story through sheer force of will and clung to it despite the plot. I wanted to rescue her from this novel and put her into a better book, such as the one in which her college friends (who are great; McGuire is very good at female friendships when she writes them) stage an intervention, kick a few people out of her life, and convince her to trust them.

Unfortunately, Dodger is, by authorial fiat, half of a bound pair, and the other half of that pair is Roger, who is the sort of nice guy everyone likes and thinks is sweet and charming until he turns into an emotional trap door right when you need him the most and dumps you into the ocean to drown. And then somehow makes you do all the work of helping him feel better about his betrayal.

The most egregious (and most patriarchal) thing Roger does in this book is late in the book and a fairly substantial spoiler, so I can't rant about that properly. But even before that, Roger keeps doing the the same damn emotional abandonment trick, and the book is heavily invested into justifying it and making excuses for him. Excuses that, I should note, are not made for Dodger; her failings are due to her mistakes and weaknesses, whereas Roger's are natural reactions to outside forces. I got very, very tired of this, and I'm upset by how little awareness the narrative voice showed for how dysfunctional and abusive this relationship is. The solution is always for Dodger to reunite with Roger; it's built into the structure of the story.

I have a weakness for the soul-bound pair, in part from reading a lot of Mercedes Lackey at an impressionable age, but one of the dangerous pitfalls of the concept is that the characters then have to have an almost flawless relationship. If not, it can turn abusive very quickly, since the characters by definition cannot leave each other. It's essentially coercive, so as soon as the relationship shows a dark side, the author needs to be extremely careful. McGuire was not.

There is an attempted partial patch, late in the book, for the patriarchal structure. One of the characters complains about it, and another says that the gender of the language and math pairs is random and went either way in other pairs. Given that both of the pairs that we meet in this story have the same male-dominant gender dynamic, what I took from this is that McGuire realized there was a problem but wasn't able to fix it. (I'm also reminded of David R. Henry's old line that it's never a good sign when the characters start complaining about the plot.)

The structural problems are all the more frustrating because I think there were ways out of them. Roger is supposedly the embodiment of language, not that you'd be able to tell from most scenes in this novel. For reasons that I do not understand, McGuire expressed that as a love of words: lexicography, translation, and synonyms. This makes no sense to me. Those are some of the more structured and rules-based (and hence mathematical) parts of language. If Roger had instead been focused on stories — collecting them, telling them, and understanding why and how they're told — he would have had a clearer contrast with Dodger. More importantly, it would have solved the plot problem that McGuire solved with a nasty bit of patriarchy. So much could have been done with Dodger building a structure of math around Roger's story-based expansion of the possible, and it would have grounded Dodger's mathematics in something more interesting than symbolic magic. To me, it's such an obvious lost opportunity.

I'm still upset about this book. McGuire does a lovely bit of world-building with Asphodel Baker, what little we see of her. I found the hidden alchemical war against her work by L. Frank Baum delightful, and enjoyed every excerpt from the fictional Over the Woodward Wall scattered throughout Middlegame. But a problem with inventing a fictional book to excerpt in a real novel is that the reader may decide that the fictional book sounds a lot better than the book they're reading, and start wishing they could just read that book instead. That was certainly the case for me. I'm sad that Over the Woodward Wall doesn't exist, and am mostly infuriated by Middlegame.

Dodger and Erin deserved to live in a better book.

Should you want to read this anyway (and I do know people who liked it), serious content warning for self-harm.

Rating: 4 out of 10

27 May, 2020 03:27AM

May 26, 2020

Russell Coker

Cruises and Covid19

Problems With Cruises

GQ has an insightful and detailed article about Covid19 and the Diamond Princess [1], I recommend reading it.

FastCompany has a brief article about bookings for cruises in August [2]. There have been many negative comments about this online.

The first thing to note is that the cancellation policies on those cruises are more lenient than usual and the prices are lower. So it’s not unreasonable for someone to put down a deposit on a half price holiday in the hope that Covid19 goes away (as so many prominent people have been saying it will) in the knowledge that they will get it refunded if things don’t work out. Of course if the cruise line goes bankrupt then no-one will get a refund, but I think people are expecting that won’t happen.

The GQ article highlights some serious problems with the way cruise ships operate. They have staff crammed in to small cabins and the working areas allow transmission of disease. These problems can be alleviated, they could allocate more space to staff quarters and have more capable air conditioning systems to put in more fresh air. During the life of a cruise ship significant changes are often made, replacing engines with newer more efficient models, changing the size of various rooms for entertainment, installing new waterslides, and many other changes are routinely made. Changing the staff only areas to have better ventilation and more separate space (maybe capsule-hotel style cabins with fresh air piped in) would not be a difficult change. It would take some money and some dry-dock time which would be a significant expense for cruise companies.

Cruises Are Great

People like social environments, they want to have situations where there are as many people as possible without it becoming impossible to move. Cruise ships are carefully designed for the flow of passengers. Both the layout of the ship and the schedule of events are carefully planned to avoid excessive crowds. In terms of meeting the requirement of having as many people as possible in a small area without being unable to move cruise ships are probably ideal.

Because there is a large number of people in a restricted space there are economies of scale on a cruise ship that aren’t available anywhere else. For example the main items on the menu are made in a production line process, this can only be done when you have hundreds of people sitting down to order at the same time.

The same applies to all forms of entertainment on board, they plan the events based on statistical knowledge of what people want to attend. This makes it more economical to run than land based entertainment where people can decide to go elsewhere. On a ship a certain portion of the passengers will see whatever show is presented each night, regardless of whether it’s singing, dancing, or magic.

One major advantage of cruises is that they are all inclusive. If you are on a regular holiday would you pay to see a singing or dancing show? Probably not, but if it’s included then you might as well do it – and it will be pretty good. This benefit is really appreciated by people taking kids on holidays, if kids do things like refuse to attend a performance that you were going to see or reject food once it’s served then it won’t cost any extra.

People Who Criticise Cruises

For the people who sneer at cruises, do you like going to bars? Do you like going to restaurants? Live music shows? Visiting foreign beaches? A cruise gets you all that and more for a discount price.

If Groupon had a deal that gave you a cheap hotel stay with all meals included, free non-alcoholic drinks at bars, day long entertainment for kids at the kids clubs, and two live performances every evening how many of the people who reject cruises would buy it? A typical cruise is just like a Groupon deal for non-stop entertainment from 8AM to 11PM.

Will Cruises Restart?

The entertainment options that cruises offer are greatly desired by many people. Most cruises are aimed at budget travellers, the price is cheaper than a hotel in a major city. Such cruises greatly depend on economies of scale, if they can’t get the ships filled then they would need to raise prices (thus decreasing demand) to try to make a profit. I think that some older cruise ships will be scrapped in the near future and some of the newer ships will be sold to cruise lines that cater to cheap travel (IE P&O may scrap some ships and some of the older Princess ships may be transferred to them). Overall I predict a decrease in the number of middle-class cruise ships.

For the expensive cruises (where the cheapest cabins cost over $1000US per person per night) I don’t expect any real changes, maybe they will have fewer passengers and higher prices to allow more social distancing or something.

I am certain that cruises will start again, but it’s too early to predict when. Going on a cruise is about as safe as going to a concert or a major sporting event. No-one is predicting that sporting stadiums will be closed forever or live concerts will be cancelled forever, so really no-one should expect that cruises will be cancelled forever. Whether companies that own ships or stadiums go bankrupt in the mean time is yet to be determined.

One thing that’s been happening for years is themed cruises. A group can book out an entire ship or part of a ship for a themed cruise. I expect this to become much more popular when cruises start again as it will make it easier to fill ships. In the past it seems that cruise lines let companies book their ships for events but didn’t take much of an active role in the process. I think that the management of cruise lines will look to aggressively market themed cruises to anyone who might help, for starters they could reach out to every 80s and 90s pop group – those fans are all old enough to be interested in themed cruises and the musicians won’t be asking for too much money.

Conclusion

Humans are social creatures. People want to attend events with many other people. Covid 19 won’t be the last pandemic, and it may not even be eradicated in the near future. The possibility of having a society where no-one leaves home unless they are in a hazmat suit has been explored in science fiction, but I don’t think that’s a plausible scenario for the near future and I don’t think that it’s something that will be caused by Covid 19.

26 May, 2020 01:32PM by etbe

Russ Allbery

Review: The Ten Thousand Doors of January

Review: The Ten Thousand Doors of January, by Alix E. Harrow

Publisher: Redhook
Copyright: September 2019
ISBN: 0-316-42198-7
Format: Kindle
Pages: 373

In 1901, at the age of seven, January found a Door. It was barely more than a frame in a ruined house in a field in Kentucky, but she wrote a story about opening it, and then did.

Once there was a brave and temeraryous (sp?) girl who found a Door. It was a magic Door that's why it has a capital D. She opened the Door.

The Door led to a bluff over the sea and above a city, a place very far from Kentucky, and she almost stayed, but she came back through the Door when her guardian, Mr. Locke, called. The adventure cost her a diary, several lectures, days of being locked in her room, and the remnants of her strained relationship with her father. When she went back, the frame of the Door was burned to the ground.

That was the end of Doors for January for some time, and the continuation of a difficult childhood. She was cared for by her father's employer as a sort of exotic pet, dutifully attempting to obey, grateful for Mr. Locke's protection, and convinced that he was occasionally sneaking her presents through a box in the Pharaoh Room out of some hidden kindness. Her father appeared rarely, said little, and refused to take her with him. Three things helped: the grocery boy who smuggled her stories, an intimidating black woman sent by her father to replace her nurse, and her dog.

Once upon a time there was a good girl who met a bad dog, and they became the very best of friends. She and her dog were inseparable from that day forward.

I will give you a minor spoiler that I would have preferred to have had, since it would have saved me some unwarranted worry and some mental yelling at the author: The above story strains but holds.

January's adventure truly starts the day before her seventeenth birthday, when she finds a book titled The Ten Thousand Doors in the box in the Pharaoh Room.

As you may have guessed from the title, The Ten Thousand Doors of January is a portal fantasy, but it's the sort of portal fantasy that is more concerned with the portal itself than the world on the other side of it. (Hello to all of you out there who, like me, have vivid memories of the Wood between the Worlds.) It's a book about traveling and restlessness and the possibility of escape, about the ability to return home again, and about the sort of people who want to close those doors because the possibility of change created by people moving around freely threatens the world they have carefully constructed.

Structurally, the central part of the book is told by interleaving chapters of January's tale with chapters from The Ten Thousand Doors. That book within a book starts with the framing of a scholarly treatment but quickly becomes a biography of a woman: Adelaide Lee Larson, a half-wild farm girl who met her true love at the threshold of a Door and then spent much of her life looking for him.

I am not a very observant reader for plot details, particularly for books that I'm enjoying. I read books primarily for the emotional beats and the story structure, and often miss rather obvious story clues. (I'm hopeless at guessing the outcomes of mysteries.) Therefore, when I say that there are many things January is unaware of that are obvious to the reader, that's saying a lot. Even more clues were apparent when I skimmed the first chapter again, and a more observant reader would probably have seen them on the first read. Right down to Mr. Locke's name, Harrow is not very subtle about the moral shape of this world.

That can make the early chapters of the book frustrating. January is being emotionally (and later physically) abused by the people who have power in her life, but she's very deeply trapped by false loyalty and lack of external context. Winning free of that is much of the story of the book, and at times it has the unpleasantness of watching someone make excuses for her abuser. At other times it has the unpleasantness of watching someone be abused. But this is the place where I thought the nested story structure worked marvelously. January escapes into the story of The Ten Thousand Doors at the worst moments of her life, and the reader escapes with her. Harrow uses the desire to switch scenes back to the more adventurous and positive story to construct and reinforce the emotional structure of the book. For me, it worked extremely well.

It helps that the ending is glorious. The payoff is worth all the discomfort and tension-building in the first half of the book. Both The Ten Thousand Doors and the surrounding narrative reach deeply satisfying conclusions, ones that are entangled but separate in just the ways that they need to be. January's abilities, actions, and decisions at the end of the book were just the outcome that I needed but didn't entirely guess in advance. I could barely put down the last quarter of this story and loved every moment of the conclusion.

This is the sort of book that can be hard to describe in a review because its merits don't rest on an original twist or easily-summarized idea. The elements here are all elements found in other books: portal fantasy, the importance of story-telling, coming of age, found family, irrepressible and indomitable characters, and the battle of the primal freedom of travel and discovery and belief against the structural forces that keep rulers in place. The merits of this book are in the small details: the way that January's stories are sparse and rare and sometimes breathtaking, the handling of tattoos, the construction of other worlds with a few deft strokes, and the way Harrow embraces the emotional divergence between January's life and Adelaide's to help the reader synchronize the emotional structure of their reading experience with January's.

She writes a door of blood and silver. The door opens just for her.

The Ten Thousand Doors of January is up against a very strong slate for both the Nebula and the Hugo this year, and I suspect it may be edged out by other books, although I wouldn't be unhappy if it won. (It probably has a better shot at the Nebula than the Hugo.) But I will be stunned if Harrow doesn't walk away with the Mythopoeic Award. This seems like exactly the type of book that award was created for.

This is an excellent book, one of the best I've read so far this year. Highly recommended.

Rating: 9 out of 10

26 May, 2020 03:49AM

hackergotchi for Norbert Preining

Norbert Preining

Multi-device and RAID1 with btrfs

I have been using btrfs, a modern modern copy on write filesystem for Linux, since many years now. But only recently I realized how amateurish my usage has been. Over the last day I switched to multiple devices and threw in a RAID1 level at the same time.

For the last years, I have been using btrfs in a completely naive way, simply creating new filesystems, mounting them, moving data over, linking the directories into my home dir, etc etc. It all became a huge mess over time. I have heard of “multi-device support“, but always thought that this is for the big players in the data centers – not realizing that it works trivially on your home system, too. Thanks to an article by Mark McBride I learned how to better use it!

Btrfs has an impressive list of features, and is often compared to (Open)ZFS (btw, they domain openzfs.org has a botched SSL certificate .. umpf, my trust disappears even more) due to the high level of data security. I have been playing around with the idea to use ZFS for quite some time, but first of all it requires compiling extra modules all the time, because ZFS cannot be included in the kernel source. And much more, I realized that ZFS is simply too inflexible with respect to disks of different sizes in a storage pool.

Btrfs on the other hand allows adding and removing devices to the “filesystem” on a running system. I just added a 2TB disk to my rig, and called:

btrfs device add /dev/sdh1 /

and with that alone, my root filesystem grew immediately. At the end I have consolidated data from 4 extra SSDs into this new “filesystem” spanning multiple disks, and got rid of all the links and loops.

For good measure, and since I had enough space left, I also switched to RAID1 for this filesystem. This again, surprisingly, works on a running system!

btrfs balance start -dconvert=raid1 -mconvert=raid1 /

Here, both data and metadata are mirrored on the devices. With 6TB of total disk space, the balancing operation took quite some time, about 6h in my case, but finished without a hiccup.

After all that, the filesystem now looks like this:

$ sudo btrfs fi show /
Label: none  uuid: XXXXXX
	Total devices 5 FS bytes used 2.19TiB
	devid    1 size 899.01GiB used 490.03GiB path /dev/sdb3
	devid    2 size 489.05GiB used 207.00GiB path /dev/sdd1
	devid    3 size 1.82TiB used 1.54TiB path /dev/sde1
	devid    4 size 931.51GiB used 649.00GiB path /dev/sdf1
	devid    5 size 1.82TiB used 1.54TiB path /dev/sdc1

and using btrfs fi usage / I can get detailed information about the device usage and status.

Stumbling blocks

You wouldn’t expect such a deep rebuilding of the intestines of a system to go without a few bumps, and indeed, there are a few:

First of all, update-grub is broken when device names are used. If you have GRUB_DISABLE_LINUX_UUID=true, so that actual device nodes are used in grub.cfg, the generated entries are broken because they list all the devices. This comes from the fact that grub-mkconfig uses grub-probe --target=device / to determine the root device, and this returns in our case:

# grub-probe --target=device /
/dev/sdb3
/dev/sdd1
/dev/sde1
/dev/sdf1
/dev/sdc1

and thus the grub config file contains entries like:

...
menuentry 'Debian GNU/Linux ...' ... {
  ...
  linux	/boot/vmlinuz-5.7.0-rc7 root=/dev/sdb3
/dev/sdd1
/dev/sde1
/dev/sdf1
/dev/sdc1 ro <other options>
}

This is of course an invalid entry, but fortunately grub still boots, but ignores the rest of the command line options.

So I decided to turn back to using UUID for the root entry, which should be better supported. But alas, what happened, I couldn’t even boot anymore. Grub gave me very cryptic messages like cannot find UUID device and dropping you into the grub rescue shell, then having the grub rescue shell being unable to read any filesystem at all (not even FAT or ext2!). The most cryptic one was grub header bytenr is not equal node addr, where even Google gave up on it.

At the end I booted into a rescue image (you always have something like SystemRescueCD on an USB stick next to you during these operations, right?), mounted the filesystem manually, and reinstalled grub, which fixed the problem, and now the grub config file contains only the UUID for root.

I don’t blame btrfs for that, this is more like we are, after sooo many years, we still don’t have a good boot system ðŸ™�

All in all, a very smooth transition, and at least for some time I don’t have to worry about which partition has still some space left.

Thanks btrfs and Open Source!

26 May, 2020 01:01AM by Norbert Preining

May 25, 2020

Elana Hashman

Beef and broccoli

Beef and broccoli is one of my favourite easy weeknight meals. It's savoury, it's satisfying, it's mercifully quick, and so, so delicious.

The recipe here is based on this one but with a few simplifications and modifications. The ingredients are basically the same, but the technique is a little different. Oh, and I include some fermented black beans because they're yummy :3

🥩 and 🥦

Serves 3. Active time: 20-30m. Total time: 30m.

Ingredients:

  • ¾ lb steak, thinly sliced (leave it in the freezer for ~30m to make slicing easier)
  • ¾ lb broccoli florets
  • 3 large cloves garlic, chopped
  • 1 teaspoon fermented black beans, rinsed and chopped (optional)

For the marinade:

  • 1 teaspoon soy sauce
  • 1 teaspoon shaoxing rice wine or dry sherry
  • ½ teaspoon corn starch
  • &frac18 teaspoon ground black pepper

For the sauce:

  • 2 tablespoons oyster sauce
  • 2 teaspoons soy sauce
  • 1 teaspoon shaoxing rice wine or dry sherry
  • ¼ cup chicken broth (I use Better than Bouillon)
  • 2 tablespoons water
  • 1 teaspoon corn starch

For the rice:

  • ¾ cup white medium-grain rice (I use calrose)
  • 1¼ cup water
  • large pinch of salt

Recipe:

  1. Begin by preparing the rice: combine rice, water, and salt in a small saucepan and bring to a boil. Once it reaches a boil, cover and reduce to a simmer for 20 minutes. Then turn off the heat and let it rest.
  2. Mix the marinade in a bowl large enough to hold the meat.
  3. Thinly slice the beef. Place slices in the marinade bowl and toss to evenly coat.
  4. Start heating a wok or similar pan on medium-high to high heat. This will ensure you get a good sear.
  5. Prep the garlic, black beans, and broccoli. If you use frozen broccoli, you can save some time here :)
  6. Combine the ingredients for the sauce and mix well to ensure the corn starch doesn't form any lumps.
  7. By this point the beef should have been marinating for about 10-15m. Add a tablespoon of neutral cooking oil (like canola or soy) to the meat, give it one more toss to ensure it's thoroughly coated, then add a layer of meat to the dry pan. You may need to sear in batches. On high heat, cooking will take 1-2 minutes per side, ensuring each is nicely browned. Once the strips are cooked, remove from the pan and set aside.
  8. While the beef is cooking, steam the broccoli until it's almost (but not quite) cooked through. I typically do this by putting it in a large bowl, adding 2 tbsp water, covering it and microwaving: 3 minutes on high if frozen, 1½ minutes on high if fresh, pausing halfway through to stir.
  9. Once all the beef is cooked and set aside, spray the pan or add oil to coat and add the garlic and black bean (if using). Stir and cook for 30-60 seconds until fragrant.
  10. When the garlic is ready, give the sauce a quick stir and add it to the pan, using it to deglaze. Once there are no more bits stuck to the bottom and the sauce is thick to your liking, add the broccoli and beef to the pan. Mix until thoroughly coated in sauce and heated through. Taste and adjust seasoning (salt, pepper, soy, etc.) if necessary.
  11. Fluff the rice and serve. Enjoy!

But I am a vegetarian???

Seitan can substitute for the beef relatively easily. Finding a substitute for oyster sauce is a little bit trickier, especially since it's the star ingredient. You can buy vegetarian or vegan oyster sauce (they're usually mushroom-based), but I have no idea how good they taste. You can also try making it at home! If you do, let me know how it turns out?

25 May, 2020 11:30PM by Elana Hashman

hackergotchi for Bits from Debian

Bits from Debian

DebConf20 registration is open!

DebConf20 banner

We are happy to announce that registration for DebConf20 is now open. The event will take place from August 23rd to 29th, 2020 at the University of Haifa, in Israel, and will be preceded by DebCamp, from August 16th to 22nd.

Although the Covid-19 situation is still rather fluid, as of now, Israel seems to be on top of the situation. Days with less than 10 new diagnosed infections are becoming common and businesses and schools are slowly reopening. As such, we are hoping that, at least as far as regulations go, we will be able to hold an in-person conference. There is more (and up to date) information at the conference's FAQ. Which means, barring a second wave, that there is reason to hope that the conference can go forward.

For that, we need your help. We need to know, assuming health regulations permit it, how many people intend to attend. This year probably more than ever before, prompt registration is very important to us. If after months of staying at home you feel that rubbing elbows with fellow Debian Developers is precisely the remedy that will salvage 2020, then we ask that you do register as soon as possible.

Sadly, things are still not clear enough for us to make a final commitment to holding an in-person conference, but knowing how many people intend to attend will be a great help in making that decision. The deadline for deciding on postponing, cancelling or changing the format of the conference is June 8th.

To register for DebConf20, please visit our website and log into the registration system and fill out the form. You can always edit or cancel your registration, but please note that the last day to confirm or cancel is July 26th, 2020 23:59:59 UTC. We cannot guarantee availability of accommodation, food and swag for unconfirmed registrations.

We do suggest that attendees begin making travel arrangements as soon as possible, of course. Please bear in mind that most air carriers allow free cancellations and changes.

Any questions about registrations should be addressed to registration@debconf.org.

Bursary for travel, accomodation and meals

In an effort to widen the diversity of DebConf attendees, the Debian Project allocates a part of the financial resources obtained through sponsorships to pay for bursaries (travel, accommodation, and/or meals) for participants who request this support when they register.

As resources are limited, we will examine the requests and decide who will receive the bursaries. They will be destined:

  • Debian funded bursaries are available to active Debian contributors.
  • Debian diversity bursaries are available to newcomers to Debian/DebConf. Especially from under-represented communities.

Giving a talk, organizing an event or helping during DebConf20 is taken into account when deciding upon your bursary, so please mention them in your bursary application.

For more information about bursaries, please visit Applying for a Bursary to DebConf

Attention: deadline to apply for bursaries using the registration form before May 31st, 2019 23:59:59 UTC. This deadline is necessary in order to the organisers to have some time to analyze the requests.

To register for the Conference, either with or without a bursary request, please visit: https://debconf20.debconf.org/register

Participation to DebConf20 is conditional to your respect of our Code of Conduct. We require you to read, understand and abide by this code.

DebConf would not be possible without the generous support of all our sponsors, especially our Platinum Sponsor Lenovo and Gold Sponsors deepin and Matanel Foundation. DebConf20 is still accepting sponsors; if you are interested, or think you know of others who would be willing to help, please get in touch!

25 May, 2020 09:30AM by Tzafrir Cohen, Jean-Pierre Giraud

Russ Allbery

Review: The Last Emperox

Review: The Last Emperox, by John Scalzi

Series: Interdependency #3
Publisher: Tor
Copyright: April 2020
ISBN: 0-7653-8917-7
Format: Kindle
Pages: 318

This is the conclusion of the Interdependency trilogy, which is a single story told in three books. Start with The Collapsing Empire. You don't want to read this series out of order.

All the pieces and players are in place, the causes and timeline of the collapse of the empire she is accidentally ruling are now clear, and Cardenia Wu-Patrick knows who her friends and enemies are. What she doesn't know is what she can do about it. Her enemies, unfettered Cardenia's ethics or desire to save the general population, have the advantage of clearer and more achievable goals. If they survive and, almost as important, remain in power, who cares what happens to everyone else?

As with The Consuming Fire, the politics may feel a bit too on-the-nose for current events, this time for the way that some powerful people are handling (or not handling) the current pandemic. Also as with The Consuming Fire, Scalzi's fast-moving story, likable characters, banter, and occasional humorous descriptions prevent those similarities from feeling heavy or didactic. This is political wish fulfillment to be sure, but it doesn't try to justify itself or linger too much on its improbabilities. It's a good story about entertaining people trying (mostly) to save the world with a combination of science and political maneuvering.

I picked up The Last Emperox as a palate cleanser after reading Gideon the Ninth, and it provided exactly what I was looking for. That gave me an opportunity to think about what Scalzi does in his writing, why his latest novel was one of my first thoughts for a palate cleanser, and why I react to his writing the way that I do.

Scalzi isn't a writer about whom I have strong opinions. In my review of The Collapsing Empire, I compared his writing to the famous description of Asimov as the "default voice" of science fiction, but that's not quite right. He has a distinct and easily-recognizable style, heavy on banter and light-hearted description. But for me his novels are pleasant, reliable entertainment that I forget shortly after reading them. They don't linger or stand out, even though I enjoy them while I'm reading them.

That's my reaction. Others clearly do not have that reaction, fully engage with his books, and remember them vividly. That indicates to me that there's something his writing is doing that leaves substantial room for difference of personal taste and personal reaction to the story, and the sharp contrast between The Last Emperox and Gideon the Ninth helped me put my finger on part of it. I don't feel like Scalzi's books try to tell me how to feel about the story.

There's a moment in The Last Emperox where Cardenia breaks down crying over an incredibly difficult decision that she's made, one that the readers don't find out about until later. In another book, there would be considerably more emotional build-up to that moment, or at least some deep analysis of it later once the decision is revealed. In this book, it's only a handful of paragraphs and then a few pages of processing later, primarily in dialogue, and less focused on the emotions of the characters than on the forward-looking decisions they've made to deal with those emotions. The emotion itself is subtext. Many other authors would try to pull the reader into those moments and make them feel what the characters are feeling. Scalzi just relates them, and leaves the reader free to feel what they choose to feel.

I don't think this is a flaw (or a merit) in Scalzi's writing; it's just a difference, and exactly the difference that made me reach for this book as an emotional break after a book that got its emotions all over the place. Calling Scalzi's writing emotionally relaxing isn't quite right, but it gives me space to choose to be emotionally relaxed if I want to be. I can pick the level of my engagement. If I want to care about these characters and agonize over their decisions, there's enough information here to mull over and use to recreate their emotional states. If I just want to read a story about some interesting people and not care too much about their hopes and dreams, I can choose to do that instead, and the book won't fight me. That approach lets me sidle up on the things that I care about and think about them at my leisure, or leave them be.

This approach makes Scalzi's books less intense than other novels for me. This is where personal preference comes in. I read books in large part to engage emotionally with the characters, and I therefore appreciate books that do a lot of that work for me. Scalzi makes me do the work myself, and the result is not as effective for me, or as memorable.

I think this may be part of what I and others are picking up on when we say that Scalzi's writing is reminiscent of classic SF from decades earlier. It used to be common for SF to not show any emotional vulnerability in the main characters, and to instead focus on the action plot and the heroics and martial virtues. This is not what Scalzi is doing, to be clear; he has a much better grasp of character and dialogue than most classic SF, adds considerable light-hearted humor, and leaves clear clues and hooks for a wide range of human emotions in the story. But one can read Scalzi in that tone if one wants to, since the emotional hooks do not grab hard at the reader and dig in. By comparison, you cannot read Gideon the Ninth without grappling with the emotions of the characters. The book will not let you.

I think this is part of why Scalzi is so consistent for me. If you do not care deeply about Gideon Nav, you will not get along with Gideon the Ninth, and not everyone will. But several main characters in The Last Emperox (Mance and to some extent Cardenia) did little or nothing for me emotionally, and it didn't matter. I liked Kiva and enjoyed watching her strategically smash her way through social conventions, but it was easy to watch her from a distance and not get too engrossed in her life or her thoughts. The plot trundled along satisfyingly, regardless. That lack of emotional involvement precludes, for me, a book becoming the sort of work that I will rave about and try to press into other people's hands, but it also makes it comfortable and gentle and relaxing in a way that a more emotionally fraught book could not be.

This is a long-winded way to say that this was a satisfying conclusion to a space opera trilogy that I enjoyed reading, will recommend mildly to others, and am already forgetting the details of. If you liked the first two books, this is an appropriate and fun conclusion with a few new twists and a satisfying amount of swearing (mostly, although not entirely, from Kiva). There are a few neat (albeit not horribly original) bits of world-building, a nice nod to and subversion of Asimov, a fair bit of political competency wish fulfillment (which I didn't find particularly believable but also didn't mind being unbelievable), and one enjoyable "oh no she didn't" moment. If you like the thing that Scalzi is doing, you will enjoy this book.

Rating: 8 out of 10

25 May, 2020 03:14AM

May 24, 2020

Enrico Zini

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

#3 T^4: Customizing The Shell

The third video (following the announcement, the shell colors) one as well as last week’s shell prompt one, is up in the stil new T^4 series of video lightning talks with tips, tricks, tools, and toys. Today we cover customizing the shell some more.

The slides are here.

This repo at GitHub support the series: use it to open issues for comments, criticism, suggestions, or feedback.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

24 May, 2020 08:51PM

hackergotchi for Holger Levsen

Holger Levsen

20200523-i3statusbar

new i3 statusbar

🔌 96% 🚀 192.168.x.y � 🤹 5+4+1 Qubes (80 avail/5 sys/16 tpl) 💾 77G 🧠 4495M/15596M 🤖 11% 🌡� 50°C 2955� dos y cuarto 🗺  

- and soon, with Qubes 4.1, it will become this colorful too :)

That depends on whether fonts-symbola and/or fonts-noto-color-emoji are being available.

24 May, 2020 01:59PM

François Marier

Printing hard-to-print PDFs on Linux

I recently found a few PDFs which I was unable to print due to those files causing insufficient printer memory errors:

I found a detailed explanation of what might be causing this which pointed the finger at transparent images, a PDF 1.4 feature which apparently requires a more recent version of PostScript than what my printer supports.

Using Okular's Force rasterization option (accessible via the print dialog) does work by essentially rendering everything ahead of time and outputing a big image to be sent to the printer. The quality is not very good however.

Converting a PDF to DjVu

The best solution I found makes use of a different file format: .djvu

Such files are not PDFs, but can still be opened in Evince and Okular, as well as in the dedicated DjVuLibre application.

As an example, I was unable to print page 11 of this paper. Using pdfinfo, I found that it is in PDF 1.5 format and so the transparency effects could be the cause of the out-of-memory printer error.

Here's how I converted it to a high-quality DjVu file I could print without problems using Evince:

pdf2djvu -d 1200 2002.04049.pdf > 2002.04049-1200dpi.djvu

Converting a PDF to PDF 1.3

I also tried the DjVu trick on a different unprintable PDF, but it failed to print, even after lowering the resolution to 600dpi:

pdf2djvu -d 600 dow-faq_v1.1.pdf > dow-faq_v1.1-600dpi.djvu

In this case, I used a different technique and simply converted the PDF to version 1.3 (from version 1.6 according to pdfinfo):

ps2pdf13 -r1200x1200 dow-faq_v1.1.pdf dow-faq_v1.1-1200dpi.pdf

This eliminates the problematic transparency and rasterizes the elements that version 1.3 doesn't support.

24 May, 2020 03:05AM

May 23, 2020

hackergotchi for Rapha&#235;l Hertzog

Raphaël Hertzog

Freexian’s report about Debian Long Term Support, April 2020

A Debian LTS logo Like each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In April, 284.5 work hours have been dispatched among 14 paid contributors. Their reports are available:
  • Abhijith PA did 10.0h (out of 14h assigned), thus carrying over 4h to May.
  • Adrian Bunk did nothing (out of 28.75h assigned), thus is carrying over 28.75h for May.
  • Ben Hutchings did 26h (out of 20h assigned and 8.5h from March), thus carrying over 2.5h to May.
  • Brian May did 10h (out of 10h assigned).
  • Chris Lamb did 18h (out of 18h assigned).
  • Dylan Aïssi did 6h (out of 6h assigned).
  • Emilio Pozuelo Monfort did not report back about their work so we assume they did nothing (out of 28.75h assigned plus 17.25h from March), thus is carrying over 46h for May.
  • Markus Koschany did 11.5h (out of 28.75h assigned and 38.75h from March), thus carrying over 56h to May.
  • Mike Gabriel did 1.5h (out of 8h assigned), thus carrying over 6.5h to May.
  • Ola Lundqvist did 13.5h (out of 12h assigned and 8.5h from March), thus carrying over 7h to May.
  • Roberto C. Sánchez did 28.75h (out of 28.75h assigned).
  • Sylvain Beucler did 28.75h (out of 28.75h assigned).
  • Thorsten Alteholz did 28.75h (out of 28.75h assigned).
  • Utkarsh Gupta did 24h (out of 24h assigned).

Evolution of the situation

In April we dispatched more hours than ever and another was new too, we had our first (virtual) contributors meeting on IRC! Logs and minutes are available and we plan to continue doing IRC meetings every other month.
Sadly one contributor decided to go inactive in April, Hugo Lefeuvre.
Finally, we like to remind you, that the end of Jessie LTS is coming in less than two months!
In case you missed it (or missed to act), please read this post about keeping Debian 8 Jessie alive for longer than 5 years. If you expect to have Debian 8 servers/devices running after June 30th 2020, and would like to have security updates for them, please get in touch with Freexian.

The security tracker currently lists 4 packages with a known CVE and the dla-needed.txt file has 25 packages needing an update.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

23 May, 2020 04:10PM by Raphaël Hertzog

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppSimdJson 0.0.5: Updated Upstream

A new RcppSimdJson release with updated upstream simdjson code just arrived on CRAN. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via some very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. The best-case performance is ‘faster than CPU speed’ as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle use per byte parsed; see the video of the recent talk by Daniel Lemire at QCon (which was also voted best talk).

This release brings updated upstream code (thanks to Brendan Knapp) plus a new example and minimal tweaks. The full NEWS entry follows.

Changes in version 0.0.5 (2020-05-23)

  • Add parseExample from earlier upstream announcement (Dirk).

  • Synced with upstream (Brendan in #12) closing #11).

  • Updated example parseExample to API changes (Brendan).

Courtesy of CRANberries, there is also a diffstat report for this release.

For questions, suggestions, or issues please use the issue tracker at the GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

23 May, 2020 03:09PM

May 22, 2020

hackergotchi for Steve Kemp

Steve Kemp

Updated my linux-security-modules for the Linux kernel

Almost three years ago I wrote my first linux-security-module, inspired by a comment I read on LWN

I did a little more learning/experimentation and actually produced a somewhat useful LSM, which allows you to restrict command-execution via the use of a user-space helper:

  • Whenever a user tries to run a command the LSM-hook receives the request.
  • Then it executes a userspace binary to decide whether to allow that or not (!)

Because the detection is done in userspace writing your own custom rules is both safe and easy. No need to touch the kernel any further!

Yesterday I rebased all the modules so that they work against the latest stable kernel 5.4.22 in #7.

The last time I'd touched them they were built against 5.1, which was itself a big jump forwards from the 4.16.7 version I'd initially used.

Finally I updated the can-exec module to make it gated, which means you can turn it on, but not turn it off without a reboot. That was an obvious omission from the initial implementation #11.

Anyway updated code is available here:

I'd kinda like to write more similar things, but I lack inspiration.

22 May, 2020 05:43AM

hackergotchi for Bits from Debian

Bits from Debian

Debian welcomes the 2020 GSOC interns

GSoC logo

We are very excited to announce that Debian has selected nine interns to work under mentorship on a variety of projects with us during the Google Summer of Code.

Here are the list of the projects, students, and details of the tasks to be performed.


Project: Android SDK Tools in Debian

  • Student(s): Manas Kashyap, Raman Sarda, and Samyak-jn

Deliverables of the project: Make the entire Android toolchain, Android Target Platform Framework, and SDK tools available in the Debian archives.


Project: Packaging and Quality assurance of COVID-19 relevant applications

  • Student: Nilesh

Deliverables of the project: Quality assurance including bug fixing, continuous integration tests and documentation for all Debian Med applications that are known to be helpful to fight COVID-19


Project: BLAS/LAPACK Ecosystem Enhancement

  • Student: Mo Zhou

Deliverables of the project: Better environment, documentation, policy, and lintian checks for BLAS/LAPACK.


Project: Quality Assurance and Continuous integration for applications in life sciences and medicine

  • Student: Pranav Ballaney

Deliverables of the project: Continuous integration tests for all Debian Med applications, QA review, and bug fixes.


Project: Systemd unit translator

  • Student: K Gopal Krishna

Deliverables of the project: A systemd unit to OpenRC init script translator. Updated OpenRC package into Debian Unstable.


Project: Architecture Cross-Grading Support in Debian

  • Student: Kevin Wu

Deliverables of the project: Evaluate, test, and develop tools to evaluate cross-grade checks for system and user configuration.


Project: Upstream/Downstream cooperation in Ruby

  • Student: utkarsh2102

Deliverables of the project: Create guide for rubygems.org on good practices for upstream maintainers, develop a tool that can detect problems and, if possible fix those errors automatically. Establish good documentation, design the tool to be extensible for other languages.


Congratulations and welcome to all the interns!

The Google Summer of Code program is possible in Debian thanks to the efforts of Debian Developers and Debian Contributors that dedicate part of their free time to mentor interns and outreach tasks.

Join us and help extend Debian! You can follow the interns' weekly reports on the debian-outreach mailing-list, chat with us on our IRC channel or reach out to the individual projects' team mailing lists.

22 May, 2020 12:30AM by Donald Norwood

May 20, 2020

hackergotchi for Norbert Preining

Norbert Preining

Plasma 5.19 coming to Debian

The KDE Plasma desktop is soon getting an update to 5.19, and beta versions are out for testing.

In this release, we have prioritized making Plasma more consistent, correcting and unifying designs of widgets and desktop elements; worked on giving you more control over your desktop by adding configuration options to the System Settings; and improved usability, making Plasma and its components easier to use and an overall more pleasurable experience.

There are lots of new features mentioned in the release announcement, I like in particular the much more usable settings application as well as the new info center.




I have been providing builds of KDE related packages since quite some time now, see everything posted under the KDE tag. In the last days I have prepared Debian packages for Plasma 5.18.90 on OBS, for now only targeting Debian/sid and amd64 architecture.

These packages require Qt 5.14, which is only available in the experimental suite, and there is no way to simply update to Qt 5.14 since all Qt related packages need to be recompiled. So as long as Qt 5.14 doesn’t hit unstable, I cannot really run these packages on my main machine, but I tried a clean Debian virtual machine installing only Plasma 5.18.90 and depending packages, plus some more for a pleasant desktop experience. This worked out quite well, the VM runs Plasma 5.18.90.

I don’t have 3D running on the VM, so I cannot really check all the nice new effects, but I am sure on my main system they would work.

Well, bottom line, as soon as we have Qt 5.14 in Debian/unstable, we are also ready for Plasma 5.19!

20 May, 2020 02:33AM by Norbert Preining

May 19, 2020

hackergotchi for Junichi Uekawa

Junichi Uekawa

After much investigation I decided to write a very simple page for getUserMedia.

After much investigation I decided to write a very simple page for getUserMedia. When I am performing music I provide audio in line input with all echo and noise and other concerns resolved. The idea is that I can cast the tab to video conferencing software and conferencing software will hopefully not reduce noise or echo. here. If the video conferencing software is reducing noise or echo from a tab, I will ask, why is it doing so?

19 May, 2020 08:14AM by Junichi Uekawa

May 18, 2020

François Marier

Using Gogo WiFi on Linux

Gogo, the WiFi provider for airlines like Air Canada, is not available to Linux users even though it advertises "access using any Wi-Fi enabled laptop, tablet or smartphone". It is however possible to work-around this restriction by faking your browser user agent.

I tried the User-Agent Switcher for Chrome extension on Chrome and Brave but it didn't work for some reason.

What did work was using Firefox and adding the following prefs in about:config to spoof its user agent to Chrome for Windows:

general.useragent.override=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36
general.useragent.updates.enabled=false
privacy.resistFingerprinting=false

The last two prefs are necessary in order for the hidden general.useragent.override pref to not be ignored.

Opt out of mandatory arbitration

As an aside, the Gogo terms of service automatically enroll you into mandatory arbitration unless you opt out by sending an email to customercare@gogoair.com within 30 days of using their service.

You may want to create an email template for this so that you can fire off a quick email to them as soon as you connect. I will probably write a script for it next time I use this service.

18 May, 2020 09:48PM

How to get a direct WebRTC connections between two computers

WebRTC is a standard real-time communication protocol built directly into modern web browsers. It enables the creation of video conferencing services which do not require participants to download additional software. Many services make use of it and it almost always works out of the box.

The reason it just works is that it uses a protocol called ICE to establish a connection regardless of the network environment. What that means however is that in some cases, your video/audio connection will need to be relayed (using end-to-end encryption) to the other person via third-party TURN server. In addition to adding extra network latency to your call that relay server might overloaded at some point and drop or delay packets coming through.

Here's how to tell whether or not your WebRTC calls are being relayed, and how to ensure you get a direct connection to the other host.

Testing basic WebRTC functionality

Before you place a real call, I suggest using the official test page which will test your camera, microphone and network connectivity.

Note that this test page makes use of a Google TURN server which is locked to particular HTTP referrers and so you'll need to disable privacy features that might interfere with this:

  • Brave: Disable Shields entirely for that page (Simple view) or allow all cookies for that page (Advanced view).

  • Firefox: Ensure that http.network.referer.spoofSource is set to false in about:config, which it is by default.

  • uMatrix: The "Spoof Referer header" option needs to be turned off for that site.

Checking the type of peer connection you have

Once you know that WebRTC is working in your browser, it's time to establish a connection and look at the network configuration that the two peers agreed on.

My favorite service at the moment is Whereby (formerly Appear.in), so I'm going to use that to connect from two different computers:

  • canada is a laptop behind a regular home router without any port forwarding.
  • siberia is a desktop computer in a remote location that is also behind a home router, but in this case its internal IP address (192.168.1.2) is set as the DMZ host.

Chromium

For all Chromium-based browsers, such as Brave, Chrome, Edge, Opera and Vivaldi, the debugging page you'll need to open is called chrome://webrtc-internals.

Look for RTCIceCandidatePair lines and expand them one at a time until you find the one which says:

  • state: succeeded (or state: in-progress)
  • nominated: true
  • writable: true

Then from the name of that pair (N6cxxnrr_OEpeash in the above example) find the two matching RTCIceCandidate lines (one local-candidate and one remote-candidate) and expand them.

In the case of a direct connection, I saw the following on the remote-candidate:

  • ip shows the external IP address of siberia
  • port shows a random number between 1024 and 65535
  • candidateType: srflx

and the following on local-candidate:

  • ip shows the external IP address of canada
  • port shows a random number between 1024 and 65535
  • candidateType: prflx

These candidate types indicate that a STUN server was used to determine the public-facing IP address and port for each computer, but the actual connection between the peers is direct.

On the other hand, for a relayed/proxied connection, I saw the following on the remote-candidate side:

  • ip shows an IP address belonging to the TURN server
  • candidateType: relay

and the same information as before on the local-candidate.

Firefox

If you are using Firefox, the debugging page you want to look at is about:webrtc.

Expand the top entry under "Session Statistics" and look for the line (should be the first one) which says the following in green:

  • ICE State: succeeded
  • Nominated: true
  • Selected: true

then look in the "Local Candidate" and "Remote Candidate" sections to find the candidate type in brackets.

Firewall ports to open to avoid using a relay

In order to get a direct connection to the other WebRTC peer, one of the two computers (in my case, siberia) needs to open all inbound UDP ports since there doesn't appear to be a way to restrict Chromium or Firefox to a smaller port range for incoming WebRTC connections.

This isn't great and so I decided to tighten that up in two ways by:

  • restricting incoming UDP traffic to the IP range of siberia's ISP, and
  • explicitly denying incoming to the UDP ports I know are open on siberia.

To get the IP range, start with the external IP address of the machine (I'll use the IP address of my blog in this example: 66.228.46.55) and pass it to the whois command:

$ whois 66.228.46.55 | grep CIDR
CIDR:           66.228.32.0/19

To get the list of open UDP ports on siberia, I sshed into it and ran nmap:

$ sudo nmap -sU localhost

Starting Nmap 7.60 ( https://nmap.org ) at 2020-03-28 15:55 PDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000015s latency).
Not shown: 994 closed ports
PORT      STATE         SERVICE
631/udp   open|filtered ipp
5060/udp  open|filtered sip
5353/udp  open          zeroconf

Nmap done: 1 IP address (1 host up) scanned in 190.25 seconds

I ended up with the following in my /etc/network/iptables.up.rules (ports below 1024 are denied by the default rule and don't need to be included here):

# Deny all known-open high UDP ports before enabling WebRTC for canada
-A INPUT -p udp --dport 5060 -j DROP
-A INPUT -p udp --dport 5353 -j DROP
-A INPUT -s 66.228.32.0/19 -p udp --dport 1024:65535 -j ACCEPT

18 May, 2020 09:48PM

Displaying client IP address using Apache Server-Side Includes

If you use a Dynamic DNS setup to reach machines which are not behind a stable IP address, you will likely have a need to probe these machines' public IP addresses. One option is to use an insecure service like Oracle's http://checkip.dyndns.com/ which echoes back your client IP, but you can also do this on your own server if you have one.

There are multiple options to do this, like writing a CGI or PHP script, but those are fairly heavyweight if that's all you need mod_cgi or PHP for. Instead, I decided to use Apache's built-in Server-Side Includes.

Apache configuration

Start by turning on the include filter by adding the following in /etc/apache2/conf-available/ssi.conf:

AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

and making that configuration file active:

a2enconf ssi

Then, find the vhost file where you want to enable SSI and add the following options to a Location or Directory section:

<Location /ssi_files>
    Options +IncludesNOEXEC
    SSLRequireSSL
    Header set Content-Security-Policy: "default-src 'none'"
    Header set X-Content-Type-Options: "nosniff"
</Location>

before adding the necessary modules:

a2enmod headers
a2enmod include

and restarting Apache:

apache2ctl configtest && systemctl restart apache2.service

Create an shtml page

With the web server ready to process SSI instructions, the following HTML blurb can be used to display the client IP address:

<!--#echo var="REMOTE_ADDR" -->

or any other built-in variable.

Note that you don't need to write a valid HTML for the variable to be substituted and so the above one-liner is all I use on my server.

Security concerns

The first thing to note is that the configuration section uses the IncludesNOEXEC option in order to disable arbitrary command execution via SSI. In addition, you can also make sure that the cgi module is disabled since that's a dependency of the more dangerous side of SSI:

a2dismod cgi

Of course, if you rely on this IP address to be accurate, for example because you'll be putting it in your DNS, then you should make sure that you only serve this page over HTTPS, which can be enforced via the SSLRequireSSL directive.

I included two other headers in the above vhost config (Content-Security-Policy and X-Content-Type-Options) in order to limit the damage that could be done in case a malicious file was accidentally dropped in that directory.

Finally, I suggest making sure that only the root user has writable access to the directory which has server-side includes enabled:

$ ls -la /var/www/ssi_includes/
total 12
drwxr-xr-x  2 root     root     4096 May 18 15:58 .
drwxr-xr-x 16 root     root     4096 May 18 15:40 ..
-rw-r--r--  1 root     root        0 May 18 15:46 index.html
-rw-r--r--  1 root     root       32 May 18 15:58 whatsmyip.shtml

18 May, 2020 09:48PM

Arturo Borrero González

A better Toolforge: upgrading the Kubernetes cluster

Logos

This post was originally published in the Wikimedia Tech blog, and is authored by Arturo Borrero Gonzalez and Brooke Storm.

One of the most successful and important products provided by the Wikimedia Cloud Services team at the Wikimedia Foundation is Toolforge. Toolforge is a platform that allows users and developers to run and use a variety of applications that help the Wikimedia movement and mission from the technical point of view in general. Toolforge is a hosting service commonly known in the industry as a Platform as a Service (PaaS). Toolforge is powered by two different backend engines, Kubernetes and GridEngine.

This article focuses on how we made a better Toolforge by integrating a newer version of Kubernetes and, along with it, some more modern workflows.

The starting point in this story is 2018. Yes, two years ago! We identified that we could do better with our Kubernetes deployment in Toolforge. We were using a very old version, v1.4. Using an old version of any software has more or less the same consequences everywhere: you lack security improvements and some modern key features.

Once it was clear that we wanted to upgrade our Kubernetes cluster, both the engineering work and the endless chain of challenges started.

It turns out that Kubernetes is a complex and modern technology, which adds some extra abstraction layers to add flexibility and some intelligence to a very old systems engineering need: hosting and running a variety of applications.

Our first challenge was to understand what our use case for a modern Kubernetes was. We were particularly interested in some key features:

  • The increased security and controls required for a public user-facing service, using RBAC, PodSecurityPolicies, quotas, etc.
  • Native multi-tenancy support, using namespaces
  • Advanced web routing, using the Ingress API

Soon enough we faced another Kubernetes native challenge: the documentation. For a newcomer, learning and understanding how to adapt Kubernetes to a given use case can be really challenging. We identified some baffling patterns in the docs. For example, different documentation pages would assume you were using different Kubernetes deployments (Minikube vs kubeadm vs a hosted service). We are running Kubernetes like you would on bare-metal (well, in CloudVPS virtual machines), and some documents directly referred to ours as a corner case.

During late 2018 and early 2019, we started brainstorming and prototyping. We wanted our cluster to be reproducible and easily rebuildable, and in the Technology Department at the Wikimedia Foundation, we rely on Puppet for that. One of the first things to decide was how to deploy and build the cluster while integrating with Puppet. This is not as simple as it seems because Kubernetes itself is a collection of reconciliation loops, just like Puppet is. So we had to decide what to put directly in Kubernetes and what to control and make visible through Puppet. We decided to stick with kubeadm as the deployment method, as it seems to be the more upstream-standardized tool for the task. We had to make some interesting decisions by trial and error, like where to run the required etcd servers, what the kubeadm init file would look like, how to proxy and load-balance the API on our bare-metal deployment, what network overlay to choose, etc. If you take a look at our public notes, you can get a glimpse of the number of decisions we had to make.

Our Kubernetes wasn’t going to be a generic cluster, we needed a Toolforge Kubernetes service. This means we don’t use some of the components, and also, we add some additional pieces and configurations to it. By the second half of 2019, we were working full-speed on the new Kubernetes cluster. We already had an idea of what we wanted and how to do it.

There were a couple of important topics for discussions, for example:

  • Ingress
  • Validating admission controllers
  • Security policies and quotas
  • PKI and user management

We will describe in detail the final state of those pieces in another blog post, but each of the topics required several hours of engineering time, research, tests, and meetings before reaching a point in which we were comfortable with moving forward.

By the end of 2019 and early 2020, we felt like all the pieces were in place, and we started thinking about how to migrate the users, the workloads, from the old cluster to the new one. This migration plan mostly materialized in a Wikitech page which contains concrete information for our users and the community.

The interaction with the community was a key success element. Thanks to our vibrant and involved users, we had several early adopters and beta testers that helped us identify early flaws in our designs. The feedback they provided was very valuable for us. Some folks helped solve technical problems, helped with the migration plan or even helped make some design decisions. Worth noting that some of the changes that were presented to our users were not easy to handle for them, like new quotas and usage limits. Introducing new workflows and deprecating old ones is always a risky operation.

Even though the migration procedure from the old cluster to the new one was fairly simple, there were some rough edges. We helped our users navigate them. A common issue was a webservice not being able to run in the new cluster due to stricter quota limiting the resources for the tool. Another example is the new Ingress layer failing to properly work with some webservices’s particular options.

By March 2020, we no longer had anything running in the old Kubernetes cluster, and the migration was completed. We then started thinking about another step towards making a better Toolforge, which is introducing the toolforge.org domain. There is plenty of information about the change to this new domain in Wikitech News.

The community wanted a better Toolforge, and so do we, and after almost 2 years of work, we have it! All the work that was done represents the commitment of the Wikimedia Foundation to support the technical community and how we really want to pursue technical engagement in general in the Wikimedia movement. In a follow-up post we will present and discuss more in-depth about some technical details of the new Kubernetes cluster, stay tuned!

This post was originally published in the Wikimedia Tech blog, and is authored by Arturo Borrero Gonzalez and Brooke Storm.

18 May, 2020 05:00PM

Russell Coker

A Good Time to Upgrade PCs

PC hardware just keeps getting cheaper and faster. Now that so many people have been working from home the deficiencies of home PCs are becoming apparent. I’ll give Australian prices and URLs in this post, but I think that similar prices will be available everywhere that people read my blog.

From MSY (parts list PDF ) [1] 120G SATA SSDs are under $50 each. 120G is more than enough for a basic workstation, so you are looking at $42 or so for fast quiet storage or $84 or so for the same with RAID-1. Being quiet is a significant luxury feature and it’s also useful if you are going to be in video conferences.

For more serious storage NVMe starts at around $100 per unit, I think that $124 for a 500G Crucial NVMe is the best low end option (paying $95 for a 250G Kingston device doesn’t seem like enough savings to be worth it). So that’s $248 for 500G of very fast RAID-1 storage. There’s a Samsung 2TB NVMe device for $349 which is good if you need more storage, it’s interesting to note that this is significantly cheaper than the Samsung 2TB SSD which costs $455. I wonder if SATA SSD devices will go away in the future, it might end up being SATA for slow/cheap spinning media and M.2 NVMe for solid state storage. The SATA SSD devices are only good for use in older systems that don’t have M.2 sockets on the motherboard.

It seems that most new motherboards have one M.2 socket on the motherboard with NVMe support, and presumably support for booting from NVMe. But dual M.2 sockets is rare and the price difference is significantly greater than the cost of a PCIe M.2 card to support NVMe which is $14. So for NVMe RAID-1 it seems that the best option is a motherboard with a single NVMe socket (starting at $89 for a AM4 socket motherboard – the current standard for AMD CPUs) and a PCIe M.2 card.

One thing to note about NVMe is that different drivers are required. On Linux this means means building a new initrd before the migration (or afterwards when booted from a recovery image) and on Windows probably means a fresh install from special installation media with NVMe drivers.

All the AM4 motherboards seem to have RADEON Vega graphics built in which is capable of 4K resolution at a stated refresh of around 24Hz. The ones that give detail about the interfaces say that they have HDMI 1.4 which means a maximum of 30Hz at 4K resolution if you have the color encoding that suits text (IE for use other than just video). I covered this issue in detail in my blog post about DisplayPort and 4K resolution [2]. So a basic AM4 motherboard won’t give great 4K display support, but it will probably be good for a cheap start.

$89 for motherboard, $124 for 500G NVMe, $344 for a Ryzen 5 3600 CPU (not the cheapest AM4 but in the middle range and good value for money), and $99 for 16G of RAM (DDR4 RAM is cheaper than DDR3 RAM) gives the core of a very decent system for $656 (assuming you have a working system to upgrade and peripherals to go with it).

Currently Kogan has 4K resolution monitors starting at $329 [3]. They probably won’t be the greatest monitors but my experience of a past cheap 4K monitor from Kogan was that it is quite OK. Samsung 4K monitors started at about $400 last time I could check (Kogan currently has no stock of them and doesn’t display the price), I’d pay an extra $70 for Samsung, but the Kogan branded product is probably good enough for most people. So you are looking at under $1000 for a new system with fast CPU, DDR4 RAM, NVMe storage, and a 4K monitor if you already have the case, PSU, keyboard, mouse, etc.

It seems quite likely that the 4K video hardware on a cheap AM4 motherboard won’t be that great for games and it will definitely be lacking for watching TV documentaries. Whether such deficiencies are worth spending money on a PCIe video card (starting at $50 for a low end card but costing significantly more for 3D gaming at 4K resolution) is a matter of opinion. I probably wouldn’t have spent extra for a PCIe video card if I had 4K video on the motherboard. Not only does using built in video save money it means one less fan running (less background noise) and probably less electricity use too.

My Plans

I currently have a workstation with 2*500G SATA SSDs in a RAID-1 array, 16G of RAM, and a i5-2500 CPU (just under 1/4 the speed of the Ryzen 5 3600). If I had hard drives then I would definitely buy a new system right now. But as I have SSDs that work nicely (quiet and fast enough for most things) and almost all machines I personally use have SSDs (so I can’t get a benefit from moving my current SSDs to another system) I would just get CPU, motherboard, and RAM. So the question is whether to spend $532 for more than 4* the CPU performance. At the moment I’ll wait because I’ll probably get a free system with DDR4 RAM in the near future, while it probably won’t be as fast as a Ryzen 5 3600, it should be at least twice as fast as what I currently have.

18 May, 2020 09:09AM by etbe

May 17, 2020

Enrico Zini

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

#2 T^4: Customizing The Shell Prompt

The second video (following the announcement and last week’s shell colors) is up in the stil new T^4 series of video lightning talks with tips, tricks, tools, and toys. Today we cover customizing shell prompts.

The slides are available here. Next week we likely continue on shell customization with aliases.

This repo at GitHub support the series: use it to open issues for comments, criticism, suggestions, or feedback.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 May, 2020 10:27PM

Russ Allbery

krb5-strength 3.2

krb5-strength provides password strength checking for Kerberos KDCs (either MIT or Heimdal), and also provides a password history implementation for Heimdal.

This release adds a check-only mode to the heimdal-history command to interrogate history without modifying it and increases the default hash iterations used when storing old passwords. explicit_bzero is now used, where available, to clear the memory used for passwords after processing. krb5-strength can now optionally be built without CrackLib support at all, if you only want to use the word list, edit distance, or length and character class rules.

It's been a few years since the previous release, so this release also updates all the portability code, overhauls valgrind testing, and now passes tests when built with system CrackLib (by skipping tests for passwords that are rejected by the stronger rules of the embedded CrackLib fork).

You can get the latest release from the krb5-strength distribution page. New packages will be uploaded to Debian unstable shortly (as soon as a Perl transition completes enough to make the package buildable in unstable).

17 May, 2020 05:43PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppArmadillo 0.9.880.1.0

armadillo image

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 719 other packages on CRAN.

Conrad released a new upstream version 9.880.1 of Armadillo on Friday which I packaged and tested as usual (result log here in the usual repo). The R package also sports a new OpenMP detection facility once again motivated by macOS which changed its setup yet again.

Changes in the new release are noted below.

Changes in RcppArmadillo version 0.9.880.1.0 (2020-05-15)

  • Upgraded to Armadillo release 9.880.1 (Roasted Mocha Detox)

    • expanded qr() to optionally use pivoted decomposition

    • updated physical constants to NIST 2018 CODATA values

    • added ARMA_DONT_USE_CXX11_MUTEX confguration option to disable use of std::mutex

  • OpenMP capability is tested explicitly (Kevin Ushey and Dirk in #294, #295, and #296 all fixing #290).

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 May, 2020 03:17PM

hackergotchi for Steve Kemp

Steve Kemp

Some brief sysbox highlights

I started work on sysbox again recently, adding a couple of simple utilities. (The whole project is a small collection of utilities, distributed as a single binary to ease installation.)

Imagine you want to run a command for every line of STDIN, here's a good example:

 $ cat input | sysbox exec-stdin "youtube-dl {}"

Here you see for every (non-empty) line of input read from STDIN the command "youtube-dl" has been executed. "{}" gets expanded to the complete line read. You can also access individual fields, kinda like awk.

(Yes youtube-dl can read a list of URLs from a file, this is an example!)

Another example, run groups for every local user:

$ cat /etc/passwd | sysbox exec-stdin --split=: groups {1}

Here you see we have split the input-lines read from STDIN by the : character, instead of by whitespace, and we've accessed the first field via "{1}". This is certainly easier for scripting than using a bash loop.

On the topic of bash; command-completion for each subcommand, and their arguments, is now present:

$ source <(sysbox bash-completion)

And I've added a text-based UI for selecting files. You can also execute a command, against the selected file:

$ sysbox choose-file -exec "xine {}" /srv/tv

This is what that looks like:

/2020/05/17-choose-file.png

You'll see:

  • A text-box for filtering the list.
  • A list which can be scrolled up/down/etc.
  • A brief bit of help information in the footer.

As well as choosing files, you can also select from lines read via STDIN, and you can filter the command in the same way as before. (i.e. "{}" is the selected item.)

Other commands received updates, so the calculator now allows storing results in variables:

$ sysbox calc
calc> let a = 3
3
calc> a / 9 * 3
1
calc> 1 + 2 * a
7
calc> 1.2 + 3.4
4.600000

17 May, 2020 03:00PM

hackergotchi for Erich Schubert

Erich Schubert

Contact Tracing Apps are Useless

Some people believe that automatic contact tracing apps will help contain the Coronavirus epidemic. They won’t.

Sorry to bring the bad news, but IT and mobile phones and artificial intelligence will not solve every problem.

In my opinion, those that promise to solve these things with artificial intelligence / mobile phones / apps / your-favorite-buzzword are at least overly optimistic and “blinder Aktionismus” (*), if not naive, detachted from reality, or fraudsters that just want to get some funding.

(*) there does not seem to be an English word for this – “doing something just for the sake of doing something, without thinking about whether it makes sense to do so”

Here are the reasons why it will not work:

  1. Signal quality. Forget detecting proximity with Bluetooth Low Energy. Yes, there are attempts to use BLE beacons for indoor positioning. But these use that you can learn “fingerprints” of which beacons are visible at which points, combined with additional information such as movement sensors and history (you do not teleport around in a building). BLE signals and antennas apparently tend to be very prone to orientation differences, signal reflections, and of course you will not have the idealized controlled environment used in such prototypes. The contacts have a single device, and they move – this is not comparable to indoor positioning. I strongly doubt you can tell whether you are “close” to someone, or not.
  2. Close vs. protection. The app cannot detect protection in place. Being close to someone behind a plexiglass window or even a solid wall is very different from being close otherwise. You will get a lot of false contacts this way. That neighbor that you have never seen living in the appartment above will likely be considered a close contact of yours, as you sleep “next” to each other every day…
  3. Low adoption rates. Apparently even in technology affine Singapore, fewer than 20% of people installed the app. That does not even mean they use it regularly. In Austria, the number is apparently below 5%, and people complain that it does not detect contact… But in order for this approach to work, you will need Chinese-style mass surveillance that literally puts you in prison if you do not install the app.
  4. False alerts. Because of these issues, you will get false alerts, until you just do not care anymore.
  5. False sense of security. Honestly: the app does not pretect you at all. All it tries to do is to make the tracing of contacts easier. It will not tell you reliably if you have been infected (as mentioned above, too many false positives, too few users) nor that you are relatively safe (too few contacts included, too slow testing and reporting). It will all be on the quality of “about 10 days ago you may or may not have contact with someone that tested positive, please contact someone to expose more data to tell you that it is actually another false alert”.
  6. Trust. In Germany, the app will be operated by T-Systems and SAP. Not exactly two companies that have a lot of fans… SAP seems to be one of the most hated software around. Neither company is known for caring about privacy much, but they are prototypical for “business first”. Its trust the cat to keep the cream. Yes, I know they want to make it open-source. But likely only the client, and you will still have to trust that the binary in the app stores is actually built from this source code, and not from a modified copy. As long as the name T-Systems and SAP are associated to the app, people will not trust it. Plus, we all know that the app will be bad, given the reputation of these companies at making horrible software systems…
  7. Too late. SAP and T-Systems want to have the app ready in mid June. Seriously, this must be a joke? It will be very buggy in the beginning (because it is SAP!) and it will not be working reliably before end of July. There will not be a substantial user before fall. But given the low infection rates in Germany, nobody will bother to install it anymore, because the perceived benefit is 0 one the infection rates are low.
  8. Infighting. You may remember that there was the discussion before that there should be a pan-european effort. Except that in the end, everybody fought everybody else, countries went into different directions and they all broke up. France wanted a centralized systems, while in Germany people pointed out that the users will not accept this and only a distributed system will have a chance. That failed effort was known as “Pan-European Privacy-Preserving Proximity Tracing (PEPP-PT)” vs. “Decentralized Privacy-Preserving Proximity Tracing (DP-3T)”, and it turned out to have become a big “clusterfuck”. And that is just the tip of the iceberg.

Iceleand, probably the country that handled the Corona crisis best (they issued a travel advisory against Austria, when they were still happily spreading the virus at apres-ski; they massively tested, and got the infections down to almost zero within 6 weeks), has been experimenting with such an app. Iceland as a fairly close community managed to have almost 40% of people install their app. So did it help? No: “The technology is more or less … I wouldn’t say useless […] it wasn’t a game changer for us.”

The contact tracing app is just a huge waste of effort and public money.

And pretty much the same applies to any other attempts to solve this with IT. There is a lot of buzz about solving the Corona crisis with artificial intelligence: bullshit!

That is just naive. Do not speculate about magic power of AI. Get the data, understand the data, and you will see it does not help.

Because its real data. Its dirty. Its late. Its contradicting. Its incomplete. It is all what AI currently can not handle well. This is not image recognition. You have no labels. Many of the attempts in this direction already fail at the trivial 7-day seasonality you observe in the data… For example, the widely known John Hopkins “Has the curve flattened” trend has a stupid, useless indicator based on 5 day averages. And hence you get the weekly up and downs due to weekends. They show pretty “up” and “down” indicators. But these are affected mostly by the day of the week. And nobody cares. Notice that they currently even have big negative infections in their plots?

There is no data on when someone was infected. Because such data simply does not exist. What you have is data when someone tested positive (mostly), when someone reported symptons (sometimes, but some never have symptoms!), and when someone dies (but then you do not know if it was because of Corona, because of other issues that became “just” worse because of Corona, or hit by a car without any relation to Corona). The data that we work with is incredibly delayed, yet we pretend it is “live”.

Stop reading tea leaves. Stop pretending AI can save the world from Corona.

17 May, 2020 12:04PM by Erich Schubert

hackergotchi for Matthew Palmer

Matthew Palmer

Private Key Redaction: UR DOIN IT RONG

Because posting private keys on the Internet is a bad idea, some people like to “redact” their private keys, so that it looks kinda-sorta like a private key, but it isn’t actually giving away anything secret. Unfortunately, due to the way that private keys are represented, it is easy to “redact” a key in such a way that it doesn’t actually redact anything at all. RSA private keys are particularly bad at this, but the problem can (potentially) apply to other keys as well.

I’ll show you a bit of “Inside Baseball” with key formats, and then demonstrate the practical implications. Finally, we’ll go through a practical worked example from an actual not-really-redacted key I recently stumbled across in my travels.

The Private Lives of Private Keys

Here is what a typical private key looks like, when you come across it:

-----BEGIN RSA PRIVATE KEY-----
MGICAQACEQCxjdTmecltJEz2PLMpS4BXAgMBAAECEDKtuwD17gpagnASq1zQTYEC
CQDVTYVsjjF7IQIJANUYZsIjRsR3AgkAkahDUXL0RSECCB78r2SnsJC9AghaOK3F
sKoELg==
-----END RSA PRIVATE KEY-----

Obviously, there’s some hidden meaning in there – computers don’t encrypt things by shouting “BEGIN RSA PRIVATE KEY!”, after all. What is between the BEGIN/END lines above is, in fact, a base64-encoded DER format ASN.1 structure representing a PKCS#1 private key.

In simple terms, it’s a list of numbers – very important numbers. The list of numbers is, in order:

  • A version number (0);
  • The “public modulus”, commonly referred to as “n”;
  • The “public exponent”, or “e” (which is almost always 65,537, for various unimportant reasons);
  • The “private exponent”, or “d”;
  • The two “private primes”, or “p” and “q”;
  • Two exponents, which are known as “dmp1” and “dmq1”; and
  • A coefficient, known as “iqmp”.

Why Is This a Problem?

The thing is, only three of those numbers are actually required in a private key. The rest, whilst useful to allow the RSA encryption and decryption to be more efficient, aren’t necessary. The three absolutely required values are e, p, and q.

Of the other numbers, most of them are at least about the same size as each of p and q. So of the total data in an RSA key, less than a quarter of the data is required. Let me show you with the above “toy” key, by breaking it down piece by piece1:

  • MGI – DER for “this is a sequence”
  • CAQ – version (0)
  • CxjdTmecltJEz2PLMpS4BXn
  • AgMBAAe
  • ECEDKtuwD17gpagnASq1zQTYd
  • ECCQDVTYVsjjF7IQp
  • IJANUYZsIjRsR3q
  • AgkAkahDUXL0RSdmp1
  • ECCB78r2SnsJC9dmq1
  • AghaOK3FsKoELg==iqmp

Remember that in order to reconstruct all of these values, all I need are e, p, and q – and e is pretty much always 65,537. So I could “redact” almost all of this key, and still give all the important, private bits of this key. Let me show you:

-----BEGIN RSA PRIVATE KEY-----
..............................................................EC
CQDVTYVsjjF7IQIJANUYZsIjRsR3....................................
........
-----END RSA PRIVATE KEY-----

Now, I doubt that anyone is going to redact a key precisely like this… but then again, this isn’t a “typical” RSA key. They usually look a lot more like this:

-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAu6Inch7+mWtKn+leB9uCG3MaJIxRyvC/5KTz2fR+h+GOhqj4
SZJobiVB4FrE5FgC7AnlH6qeRi9MI0s6dt5UWZ5oNIeWSaOOeNO+EJDUkSVf67wj
SNGXlSjGAkPZ0nRJiDjhuPvQmdW53hOaBLk5udxPEQbenpXAzbLJ7wH5ouLQ3nQw
HwpwDNQhF6zRO8WoscpDVThOAM+s4PS7EiK8ZR4hu2toon8Ynadlm95V45wR0VlW
zywgbkZCKa1IMrDCscB6CglQ10M3Xzya3iTzDtQxYMVqhDrA7uBYRxA0y1sER+Rb
yhEh03xz3AWemJVLCQuU06r+FABXJuY/QuAVvQIDAQABAoIBAFqwWVhzWqNUlFEO
PoCVvCEAVRZtK+tmyZj9kU87ORz8DCNR8A+/T/JM17ZUqO2lDGSBs9jGYpGRsr8s
USm69BIM2ljpX95fyzDjRu5C0jsFUYNi/7rmctmJR4s4uENcKV5J/++k5oI0Jw4L
c1ntHNWUgjK8m0UTJIlHbQq0bbAoFEcfdZxd3W+SzRG3jND3gifqKxBG04YDwloy
tu+bPV2jEih6p8tykew5OJwtJ3XsSZnqJMwcvDciVbwYNiJ6pUvGq6Z9kumOavm9
XU26m4cWipuK0URWbHWQA7SjbktqEpxsFrn5bYhJ9qXgLUh/I1+WhB2GEf3hQF5A
pDTN4oECgYEA7Kp6lE7ugFBDC09sKAhoQWrVSiFpZG4Z1gsL9z5YmZU/vZf0Su0n
9J2/k5B1GghvSwkTqpDZLXgNz8eIX0WCsS1xpzOuORSNvS1DWuzyATIG2cExuRiB
jYWIJUeCpa5p2PdlZmBrnD/hJ4oNk4oAVpf+HisfDSN7HBpN+TJfcAUCgYEAyvY7
Y4hQfHIdcfF3A9eeCGazIYbwVyfoGu70S/BZb2NoNEPymqsz7NOfwZQkL4O7R3Wl
Rm0vrWT8T5ykEUgT+2ruZVXYSQCKUOl18acbAy0eZ81wGBljZc9VWBrP1rHviVWd
OVDRZNjz6nd6ZMrJvxRa24TvxZbJMmO1cgSW1FkCgYAoWBd1WM9HiGclcnCZknVT
UYbykCeLO0mkN1Xe2/32kH7BLzox26PIC2wxF5seyPlP7Ugw92hOW/zewsD4nLze
v0R0oFa+3EYdTa4BvgqzMXgBfvGfABJ1saG32SzoWYcpuWLLxPwTMsCLIPmXgRr1
qAtl0SwF7Vp7O/C23mNukQKBgB89DOEB7xloWv3Zo27U9f7nB7UmVsGjY8cZdkJl
6O4LB9PbjXCe3ywZWmJqEbO6e83A3sJbNdZjT65VNq9uP50X1T+FmfeKfL99X2jl
RnQTsrVZWmJrLfBSnBkmb0zlMDAcHEnhFYmHFuvEnfL7f1fIoz9cU6c+0RLPY/L7
n9dpAoGAXih17mcmtnV+Ce+lBWzGWw9P4kVDSIxzGxd8gprrGKLa3Q9VuOrLdt58
++UzNUaBN6VYAe4jgxGfZfh+IaSlMouwOjDgE/qzgY8QsjBubzmABR/KWCYiRqkj
qpWCgo1FC1Gn94gh/+dW2Q8+NjYtXWNqQcjRP4AKTBnPktEvdMA=
-----END RSA PRIVATE KEY-----

People typically redact keys by deleting whole lines, and usually replacing them with [...] and the like. But only about 345 of those 1588 characters (excluding the header and footer) are required to construct the entire key. You can redact about 4/5ths of that giant blob of stuff, and your private parts (or at least, those of your key) are still left uncomfortably exposed.

But Wait! There’s More!

Remember how I said that everything in the key other than e, p, and q could be derived from those three numbers? Let’s talk about one of those numbers: n.

This is known as the “public modulus” (because, along with e, it is also present in the public key). It is very easy to calculate: n = p * q. It is also very early in the key (the second number, in fact).

Since n = p * q, it follows that q = n / p. Thus, as long as the key is intact up to p, you can derive q by simple division.

Real World Redaction

At this point, I’d like to introduce an acquaintance of mine: Mr. Johan Finn. He is the proud owner of the GitHub repo johanfinn/scripts. For a while, his repo contained a script that contained a poorly-redacted private key. He since deleted it, by making a new commit, but of course because git never really deletes anything, it’s still available.

Of course, Mr. Finn may delete the repo, or force-push a new history without that commit, so here is the redacted private key, with a bit of the surrounding shell script, for our illustrative pleasure:

#Add private key to .ssh folder
cd /home/johan/.ssh/
echo  "-----BEGIN RSA PRIVATE KEY-----
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
ÄÄÄÄÄÄÄÄÄÄÄÄÄÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::.::
:::::::::::::::::::::::::::.::::::::::::::::::::::::::::::::::::
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLlL
ÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖ
ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ
YYYYYYYYYYYYYYYYYYYYYyYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
-----END RSA PRIVATE KEY-----" >> id_rsa

Now, if you try to reconstruct this key by removing the “obvious” garbage lines (the ones that are all repeated characters, some of which aren’t even valid base64 characters), it still isn’t a key – at least, openssl pkey doesn’t want anything to do with it. The key is very much still in there, though, as we shall soon see.

Using a gem I wrote and a quick bit of Ruby, we can extract a complete private key. The irb session looks something like this:

>> require "derparse"
>> b64 = <<EOF
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
EOF
>> b64 += <<EOF
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
EOF
>> der = b64.unpack("m").first
>> c = DerParse.new(der).first_node.first_child
>> version = c.value
=> 0
>> c = c.next_node
>> n = c.value
=> 80071596234464993385068908004931... # (etc)
>> c = c.next_node
>> e = c.value
=> 65537
>> c = c.next_node
>> d = c.value
=> 58438813486895877116761996105770... # (etc)
>> c = c.next_node
>> p = c.value
=> 29635449580247160226960937109864... # (etc)
>> c = c.next_node
>> q = c.value
=> 27018856595256414771163410576410... # (etc)

What I’ve done, in case you don’t speak Ruby, is take the two “chunks” of plausible-looking base64 data, chuck them together into a variable named b64, unbase64 it into a variable named der, pass that into a new DerParse instance, and then walk the DER value tree until I got all the values I need.

Interestingly, the q value actually traverses the “split” in the two chunks, which means that there’s always the possibility that there are lines missing from the key. However, since p and q are supposed to be prime, we can “sanity check” them to see if corruption is likely to have occurred:

>> require "openssl"
>> OpenSSL::BN.new(p).prime?
=> true
>> OpenSSL::BN.new(q).prime?
=> true

Excellent! The chances of a corrupted file producing valid-but-incorrect prime numbers isn’t huge, so we can be fairly confident that we’ve got the “real” p and q. Now, with the help of another one of my creations we can use e, p, and q to create a fully-operational battle key:

>> require "openssl/pkey/rsa"
>> k = OpenSSL::PKey::RSA.from_factors(p, q, e)
=> #<OpenSSL::PKey::RSA:0x0000559d5903cd38>
>> k.valid?
=> true
>> k.verify(OpenSSL::Digest::SHA256.new, k.sign(OpenSSL::Digest::SHA256.new, "bob"), "bob")
=> true

… and there you have it. One fairly redacted-looking private key brought back to life by maths and far too much free time.

Sorry Mr. Finn, I hope you’re not still using that key on anything Internet-facing.

What About Other Key Types?

EC keys are very different beasts, but they have much the same problems as RSA keys. A typical EC key contains both private and public data, and the public portion is twice the size – so only about 1/3 of the data in the key is private material. It is quite plausible that you can “redact” an EC key and leave all the actually private bits exposed.

What Do We Do About It?

In short: don’t ever try and redact real private keys. For documentation purposes, just put “KEY GOES HERE” in the appropriate spot, or something like that. Store your secrets somewhere that isn’t a public (or even private!) git repo.

Generating a “dummy” private key and sticking it in there isn’t a great idea, for different reasons: people have this odd habit of reusing “demo” keys in real life. There’s no need to encourage that sort of thing.


  1. Technically the pieces aren’t 100% aligned with the underlying DER, because of how base64 works. I felt it was easier to understand if I stuck to chopping up the base64, rather than decoding into DER and then chopping up the DER. 

17 May, 2020 12:00AM by Matt Palmer (mpalmer@hezmatt.org)

May 16, 2020

Sven Hoexter

Quick and Dirty: masquerading / NAT with nftables

Since nftables is now the new default, a short note to myself on how to setup masquerading, like the usual NAT setup you use on a gateway.

nft add table nat
nft add chain nat postrouting { type nat hook postrouting priority 100 \; }
nft add rule nat postrouting ip saddr 192.168.1.0/24 oif wlan0 masquerade

In this case the wlan0 is basically the "WAN" interface, because I use an old netbook as a wired to wireless network adapter.

16 May, 2020 06:44PM

Russ Allbery

DocKnot 3.04

This is a relatively small feature release of my tool for managing software documentation and releases.

I'm slowly moving all of my packages from Travis-CI to GitHub Workflows for automated CI. GitHub Workflows is much easier to configure and control, and I've been a bit worried about the future of Travis-CI since their acquisition. It seems unlikely that GitHub Workflows is going anywhere.

It would be nice to use a fully free software solution for CI, but there doesn't seem to be anything out there that's nearly as easy and straightforward to use, and I have neither the time nor the energy to cobble something together myself. The configuration is fairly straightforward and should be portable to any fully free solution that might materialize in the future.

Anyway, as part of that migration I needed to make some changes to DocKnot to generate status badges from GitHub Workflows instead of Travis-CI. This release includes those changes. There is a backward-incompatible change to make the semantics of the package metadata a bit more straightforward: vcs.travis needs to be changed to vcs.status.travis.

You can get the latest release from the DocKnot distribution page. Debian packages have been uploaded to my personal repository. I plan on uploading DocKnot to Debian proper once I change the metadata format to use YAML instead of relaxed JSON.

16 May, 2020 06:38PM

hackergotchi for Norbert Preining

Norbert Preining

Upgrading AMD Radeon 5700 to 5700 XT via BIOS

Having recently switched from NVIDIA to AMD graphic cards, in particular a RX 5700, I found out that I can get myself a free upgrade to the RX 5700 XT variant without paying one Yen, by simply flashing a compatible 5700 XT BIOS onto the 5700 card. Not that this is something new, a detailed explanation can be found here.

The same article also gives a detailed technical explanation on the difference between the two cards. The 5700 variant has less stream processors (2304 against 2560 in the XT variant), and lower power limits and clock speeds. Other than this they are based on the exact same chip layout (Navi 10), and with the same amount and type of memory—8 GB GDDR6.

Flashing the XT BIOS onto the plain 5700 will not changes the number of stream processors, but power limits and clock speeds are raised to the same level of the 5700 XT, providing approximately a 7% gain without any over-clocking and over-powering, and potentially more by raising voltage etc. Detailed numbers can be found in the linked article above.

The first step in this “free upgrade” is to identify ones own card correctly, best with device id and subsystem id, and then find the correct BIOS. Lots of BIOS dumps are provided in the BIOS database (link already restricting to 5700 XT BIOS). I used CPU-Z (Windows program) to determine this items, see image on the right (click to enlarge). In my case I got 1002 731F - 1462 3811 for the complete device id. The card is a MSI RX 5700 8 GB Mech OC, so I found the following alternative BIOS for MSI RX 5700 XT 8 GB Mech OC. Unfortunately, it seems that MSI is distinguishing 5700 and 5700 XT by their device id, because the XT variant gives 1002 731F - 1462 3810 for the complete device id, meaning that the last digit is 1 off compared to mine (3811 versus 3810). And indeed, trying to flash this video BIOS the normal way (using the Windows version ended in a warning that the subsystem id is different. A bit of search led to a thread in the TechPowerup Fora and this post explaining how to force the flashing in this case.

Disclaimer: The following might brick your graphic card, you are doing this on your own risk!

Necessary software:

I did all the flashing and checking under Windows, but only because I realized too late that there is a fully uptodate flashing program for Linux that exhibits the same functionality. Also, I didn’t know how to get the device id since the current AMD ROCm tools seem not to provide this data. If you are lucky and the device ids for your card are the same for both 5700 and 5700 XT variants, then you can use the graphical client (amdvbflashWin.exe), but if there is a difference, the command line is necessary. After unpacking the AMD Flash program and getting the correct BIOS rom file, the steps taken on Windows are (the very same steps can be taken on Linux):

  • Start a command line shell (cmd or powershell) with Administrator rights (on Linux become root)
  • Save your current BIOS in case you need to restore it with amdvbflash -s 0 oldbios.rom (this can also be done out of the GUI application)
  • Unlock the BIOS rom with amdvbflash -unlockrom 0
  • Force flash the new BIOS with amdvbflash -f -p 0 NEW.rom (where NEW.rom is the 5700 XT BIOS rom file)

This should succeed in both cases. After that shutdown and restart your computer and you should be greeted with a RT 5700 XT card, without twisting a single screw. Starting Windows for the first time gave some flickering, because the driver for the “new” card was installed. On Linux the system auto-detects the card and everything works out of the box. Very smooth.

Finally, a word of warning: Don’t do these kind of things if you are not ready to pay the prize of a bricked GPU card in case something goes wrong! Everything is on your own risk!

Let me close with a before/after image, most of the fields are identical, but the default/gpu clocks both at normal as well as boost levels see a considerable improvement 😉

16 May, 2020 03:06PM by Norbert Preining

Michael Stapelberg

Linux package managers are slow

I measured how long the most popular Linux distribution’s package manager take to install small and large packages (the ack(1p) source code search Perl script and qemu, respectively).

Where required, my measurements include metadata updates such as transferring an up-to-date package list. For me, requiring a metadata update is the more common case, particularly on live systems or within Docker containers.

All measurements were taken on an Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz running Docker 1.13.1 on Linux 4.19, backed by a Samsung 970 Pro NVMe drive boasting many hundreds of MB/s write performance. The machine is located in Zürich and connected to the Internet with a 1 Gigabit fiber connection, so the expected top download speed is ≈115 MB/s.

See Appendix B for details on the measurement method and command outputs.

Measurements

Keep in mind that these are one-time measurements. They should be indicative of actual performance, but your experience may vary.

ack (small Perl program)

distribution package manager data wall-clock time rate
Fedora dnf 107 MB 29s 3.7 MB/s
NixOS Nix 15 MB 14s 1.1 MB/s
Debian apt 15 MB 4s 3.7 MB/s
Arch Linux pacman 6.5 MB 3s 2.1 MB/s
Alpine apk 10 MB 1s 10.0 MB/s

qemu (large C program)

distribution package manager data wall-clock time rate
Fedora dnf 266 MB 1m8s 3.9 MB/s
Arch Linux pacman 124 MB 1m2s 2.0 MB/s
Debian apt 159 MB 51s 3.1 MB/s
NixOS Nix 262 MB 38s 6.8 MB/s
Alpine apk 26 MB 2.4s 10.8 MB/s


The difference between the slowest and fastest package managers is 30x!

How can Alpine’s apk and Arch Linux’s pacman be an order of magnitude faster than the rest? They are doing a lot less than the others, and more efficiently, too.

Pain point: too much metadata

For example, Fedora transfers a lot more data than others because its main package list is 60 MB (compressed!) alone. Compare that with Alpine’s 734 KB APKINDEX.tar.gz.

Of course the extra metadata which Fedora provides helps some use case, otherwise they hopefully would have removed it altogether. The amount of metadata seems excessive for the use case of installing a single package, which I consider the main use-case of an interactive package manager.

I expect any modern Linux distribution to only transfer absolutely required data to complete my task.

Pain point: no concurrency

Because they need to sequence executing arbitrary package maintainer-provided code (hooks and triggers), all tested package managers need to install packages sequentially (one after the other) instead of concurrently (all at the same time).

In my blog post “Can we do without hooks and triggers?”, I outline that hooks and triggers are not strictly necessary to build a working Linux distribution.

Thought experiment: further speed-ups

Strictly speaking, the only required feature of a package manager is to make available the package contents so that the package can be used: a program can be started, a kernel module can be loaded, etc.

By only implementing what’s needed for this feature, and nothing more, a package manager could likely beat apk’s performance. It could, for example:

  • skip archive extraction by mounting file system images (like AppImage or snappy)
  • use compression which is light on CPU, as networks are fast (like apk)
  • skip fsync when it is safe to do so, i.e.:
    • package installations don’t modify system state
    • atomic package installation (e.g. an append-only package store)
    • automatically clean up the package store after crashes

Current landscape

Here’s a table outlining how the various package managers listed on Wikipedia’s list of software package management systems fare:

name scope package file format hooks/triggers
AppImage apps image: ISO9660, SquashFS no
snappy apps image: SquashFS yes: hooks
FlatPak apps archive: OSTree no
0install apps archive: tar.bz2 no
nix, guix distro archive: nar.{bz2,xz} activation script
dpkg distro archive: tar.{gz,xz,bz2} in ar(1) yes
rpm distro archive: cpio.{bz2,lz,xz} scriptlets
pacman distro archive: tar.xz install
slackware distro archive: tar.{gz,xz} yes: doinst.sh
apk distro archive: tar.gz yes: .post-install
Entropy distro archive: tar.bz2 yes
ipkg, opkg distro archive: tar{,.gz} yes

Conclusion

As per the current landscape, there is no distribution-scoped package manager which uses images and leaves out hooks and triggers, not even in smaller Linux distributions.

I think that space is really interesting, as it uses a minimal design to achieve significant real-world speed-ups.

I have explored this idea in much more detail, and am happy to talk more about it in my post “Introducing the distri research linux distribution".

There are a couple of recent developments going into the same direction:

Appendix B: measurement details

ack

You can expand each of these:

Fedora’s dnf takes almost 30 seconds to fetch and unpack 107 MB.

% docker run -t -i fedora /bin/bash
[root@722e6df10258 /]# time dnf install -y ack
Fedora Modular 30 - x86_64            4.4 MB/s | 2.7 MB     00:00
Fedora Modular 30 - x86_64 - Updates  3.7 MB/s | 2.4 MB     00:00
Fedora 30 - x86_64 - Updates           17 MB/s |  19 MB     00:01
Fedora 30 - x86_64                     31 MB/s |  70 MB     00:02
[…]
Install  44 Packages

Total download size: 13 M
Installed size: 42 M
[…]
real	0m29.498s
user	0m22.954s
sys	0m1.085s

NixOS’s Nix takes 14s to fetch and unpack 15 MB.

% docker run -t -i nixos/nix
39e9186422ba:/# time sh -c 'nix-channel --update && nix-env -i perl5.28.2-ack-2.28'
unpacking channels...
created 2 symlinks in user environment
installing 'perl5.28.2-ack-2.28'
these paths will be fetched (14.91 MiB download, 80.83 MiB unpacked):
  /nix/store/57iv2vch31v8plcjrk97lcw1zbwb2n9r-perl-5.28.2
  /nix/store/89gi8cbp8l5sf0m8pgynp2mh1c6pk1gk-attr-2.4.48
  /nix/store/gkrpl3k6s43fkg71n0269yq3p1f0al88-perl5.28.2-ack-2.28-man
  /nix/store/iykxb0bmfjmi7s53kfg6pjbfpd8jmza6-glibc-2.27
  /nix/store/k8lhqzpaaymshchz8ky3z4653h4kln9d-coreutils-8.31
  /nix/store/svgkibi7105pm151prywndsgvmc4qvzs-acl-2.2.53
  /nix/store/x4knf14z1p0ci72gl314i7vza93iy7yc-perl5.28.2-File-Next-1.16
  /nix/store/zfj7ria2kwqzqj9dh91kj9kwsynxdfk0-perl5.28.2-ack-2.28
copying path '/nix/store/gkrpl3k6s43fkg71n0269yq3p1f0al88-perl5.28.2-ack-2.28-man' from 'https://cache.nixos.org'...
copying path '/nix/store/iykxb0bmfjmi7s53kfg6pjbfpd8jmza6-glibc-2.27' from 'https://cache.nixos.org'...
copying path '/nix/store/x4knf14z1p0ci72gl314i7vza93iy7yc-perl5.28.2-File-Next-1.16' from 'https://cache.nixos.org'...
copying path '/nix/store/89gi8cbp8l5sf0m8pgynp2mh1c6pk1gk-attr-2.4.48' from 'https://cache.nixos.org'...
copying path '/nix/store/svgkibi7105pm151prywndsgvmc4qvzs-acl-2.2.53' from 'https://cache.nixos.org'...
copying path '/nix/store/k8lhqzpaaymshchz8ky3z4653h4kln9d-coreutils-8.31' from 'https://cache.nixos.org'...
copying path '/nix/store/57iv2vch31v8plcjrk97lcw1zbwb2n9r-perl-5.28.2' from 'https://cache.nixos.org'...
copying path '/nix/store/zfj7ria2kwqzqj9dh91kj9kwsynxdfk0-perl5.28.2-ack-2.28' from 'https://cache.nixos.org'...
building '/nix/store/q3243sjg91x1m8ipl0sj5gjzpnbgxrqw-user-environment.drv'...
created 56 symlinks in user environment
real	0m 14.02s
user	0m 8.83s
sys	0m 2.69s

Debian’s apt takes almost 10 seconds to fetch and unpack 16 MB.

% docker run -t -i debian:sid
root@b7cc25a927ab:/# time (apt update && apt install -y ack-grep)
Get:1 http://cdn-fastly.deb.debian.org/debian sid InRelease [233 kB]
Get:2 http://cdn-fastly.deb.debian.org/debian sid/main amd64 Packages [8270 kB]
Fetched 8502 kB in 2s (4764 kB/s)
[…]
The following NEW packages will be installed:
  ack ack-grep libfile-next-perl libgdbm-compat4 libgdbm5 libperl5.26 netbase perl perl-modules-5.26
The following packages will be upgraded:
  perl-base
1 upgraded, 9 newly installed, 0 to remove and 60 not upgraded.
Need to get 8238 kB of archives.
After this operation, 42.3 MB of additional disk space will be used.
[…]
real	0m9.096s
user	0m2.616s
sys	0m0.441s

Arch Linux’s pacman takes a little over 3s to fetch and unpack 6.5 MB.

% docker run -t -i archlinux/base
[root@9604e4ae2367 /]# time (pacman -Sy && pacman -S --noconfirm ack)
:: Synchronizing package databases...
 core            132.2 KiB  1033K/s 00:00
 extra          1629.6 KiB  2.95M/s 00:01
 community         4.9 MiB  5.75M/s 00:01
[…]
Total Download Size:   0.07 MiB
Total Installed Size:  0.19 MiB
[…]
real	0m3.354s
user	0m0.224s
sys	0m0.049s

Alpine’s apk takes only about 1 second to fetch and unpack 10 MB.

% docker run -t -i alpine
/ # time apk add ack
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/4) Installing perl-file-next (1.16-r0)
(2/4) Installing libbz2 (1.0.6-r7)
(3/4) Installing perl (5.28.2-r1)
(4/4) Installing ack (3.0.0-r0)
Executing busybox-1.30.1-r2.trigger
OK: 44 MiB in 18 packages
real	0m 0.96s
user	0m 0.25s
sys	0m 0.07s

qemu

You can expand each of these:

Fedora’s dnf takes over a minute to fetch and unpack 266 MB.

% docker run -t -i fedora /bin/bash
[root@722e6df10258 /]# time dnf install -y qemu
Fedora Modular 30 - x86_64            3.1 MB/s | 2.7 MB     00:00
Fedora Modular 30 - x86_64 - Updates  2.7 MB/s | 2.4 MB     00:00
Fedora 30 - x86_64 - Updates           20 MB/s |  19 MB     00:00
Fedora 30 - x86_64                     31 MB/s |  70 MB     00:02
[…]
Install  262 Packages
Upgrade    4 Packages

Total download size: 172 M
[…]
real	1m7.877s
user	0m44.237s
sys	0m3.258s

NixOS’s Nix takes 38s to fetch and unpack 262 MB.

% docker run -t -i nixos/nix
39e9186422ba:/# time sh -c 'nix-channel --update && nix-env -i qemu-4.0.0'
unpacking channels...
created 2 symlinks in user environment
installing 'qemu-4.0.0'
these paths will be fetched (262.18 MiB download, 1364.54 MiB unpacked):
[…]
real	0m 38.49s
user	0m 26.52s
sys	0m 4.43s

Debian’s apt takes 51 seconds to fetch and unpack 159 MB.

% docker run -t -i debian:sid
root@b7cc25a927ab:/# time (apt update && apt install -y qemu-system-x86)
Get:1 http://cdn-fastly.deb.debian.org/debian sid InRelease [149 kB]
Get:2 http://cdn-fastly.deb.debian.org/debian sid/main amd64 Packages [8426 kB]
Fetched 8574 kB in 1s (6716 kB/s)
[…]
Fetched 151 MB in 2s (64.6 MB/s)
[…]
real	0m51.583s
user	0m15.671s
sys	0m3.732s

Arch Linux’s pacman takes 1m2s to fetch and unpack 124 MB.

% docker run -t -i archlinux/base
[root@9604e4ae2367 /]# time (pacman -Sy && pacman -S --noconfirm qemu)
:: Synchronizing package databases...
 core       132.2 KiB   751K/s 00:00
 extra     1629.6 KiB  3.04M/s 00:01
 community    4.9 MiB  6.16M/s 00:01
[…]
Total Download Size:   123.20 MiB
Total Installed Size:  587.84 MiB
[…]
real	1m2.475s
user	0m9.272s
sys	0m2.458s

Alpine’s apk takes only about 2.4 seconds to fetch and unpack 26 MB.

% docker run -t -i alpine
/ # time apk add qemu-system-x86_64
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
[…]
OK: 78 MiB in 95 packages
real	0m 2.43s
user	0m 0.46s
sys	0m 0.09s

16 May, 2020 10:46AM

a new distri linux (fast package management) release

I just released a new version of distri.

The focus of this release lies on:

  • a better developer experience, allowing users to debug any installed package without extra setup steps

  • performance improvements in all areas (starting programs, building distri packages, generating distri images)

  • better tooling for keeping track of upstream versions

See the release notes for more details.

The distri research linux distribution project was started in 2019 to research whether a few architectural changes could enable drastically faster package management.

While the package managers in common Linux distributions (e.g. apt, dnf, …) top out at data rates of only a few MB/s, distri effortlessly saturates 1 Gbit, 10 Gbit and even 40 Gbit connections, resulting in fast installation and update speeds.

16 May, 2020 07:14AM

Lucas Kanashiro

Quarantine times

After quite some time without publishing anything here, I decided to share the latest events. It is a hard time for most of us but with all this time at home, one can also do great things.

I would like to start with the wonderful idea the Debian Brasil community had! Why not create an online Debian related conference to keep people’s minds busy and also share knowledge? After brainstorming, we came up with our online conference called #FiqueEmCasaUseDebian (in English it would be #StayHomeUseDebian). It started on May 3rd and will last until May 30th (yes, one month)! Every weekday, we have one online talk at night and on every Saturday, a Debian packaging workshop. The feedback so far has been awesome and the Brazilian Debian community is reaching out to more people than usual at our regular conferences (as you might have imagined, Brazil is huge and it is hard to bring people at the same place). At the end of the month, we will have the first MiniDebConf online and I hope it will be as successful as our experience here in Brazil.

Another thing that deserves a highlight is the fact that I became an Ubuntu Core Developer this month; yay! After 9 months of working almost daily on the Ubuntu Server, I was able to get my upload rights to the Ubuntu archive. I was tired of asking for sponsorship, and probably my peers were tired of me too.

I could spend more time here complaining about the Brazilian government but I think it isn’t worth it. Let’s try to do something useful instead!

16 May, 2020 03:20AM

May 15, 2020

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

Let’s celebrate Anna!

Today is graduation at Washington University, and just like many other places, the ceremonies are a lot more virtual and surreal that in other years. For Anna today marks the graduation from Biomedical Engineering with a BSc. The McKelvey School of Engineering put a Zoom meeting together yesterday which was nice, and there is something more virtual here. Hopefully a real-life commencenment can take place in a year—the May 30, 2021, date has been set. The university also sent out a little commencement site/video which was cute. But at end of the day online-only still falls short of the real deal as we all know too well by now.

During those years, just about the only thing really I ever tweeted about appears to be soccer related. As it should because ball is life, as we all know. Here is one from 1 1/2 years ago when her Club Team three-peated in their NIRSA division:

And that opens what may be the best venue for mocking Anna: this year, which her a senior and co-captain, the team actually managed to loose a league game (a shocking first in these years) and to drop the final. I presume they anticipated that all we would all talk about around now is The Last Dance and three-peats, and left it at that. Probably wise.

Now just this week, and hence days before graduating with her B.Sc., also marks the first time Anna was addressed as Dr Eddelbuettel. A little prematurely I may say, but not too shabby to be in print already!

But on the topic of gratulations and what comes next, this tweet was very sweet:

As was this, which marked another impressive score:

So big thanks from all of us to WashU for being such a superb environment for Anna for those four years, and especially everybody at the Pappu Lab for giving Anna a home and base to start a research career.

And deepest and most sincere congratulations to Anna before the next adventure starts….

15 May, 2020 07:11PM

Ingo Juergensmann

XMPP: ejabberd Project on the-federation.info

For those interested in alternative social networks there is that website that is called the-federation.info, which collects some statistics of "The Fediverse". The biggest part of the fediverse is Mastodon, but there are other projects (or parts) like Friendica or Matrix that do "federation". One of the oldest projects doing federation is XMPP. You could find some Prosody servers for some time now, because there is a Prosody module "mod_nodeinfo2" that can be used. But for ejabberd there is no such module (yet?) so far, which makes it a little bit difficult to get listed on the-federation.info.

Some days ago I wrote a small script to export the needed values to x-nodeinfo2 that is queried by the-federation.info. It's surely not the best script or solution for that job and is currently limited to ejabberd servers that use a PostgreSQL database as backend, although it should be fairly easy to adapt the script for use with MySQL. Well, at least it does its job. At least as there is no native ejabberd module for this task.

You can find the script on Github: https://github.com/ingoj/ejabberd-nodeinfo2

Enjoy it and register your ejabberd server on the-federation.info! :-)

Kategorie: 

15 May, 2020 03:30PM by ij

hackergotchi for Junichi Uekawa

Junichi Uekawa

Bought Canon TS3300 Printer.

Bought Canon TS3300 Printer. I haven't bought a printer in 20 years. 5 years ago I think I would have used Google Cloud Print which took forever uploading and downloading huge print data to the cloud. 20 years ago I would have used LPR and LPD, admiring the postscript backtraces. Configuration was done using Android app, the printer could be connected via WiFi and then configuration through the app allowed printer to be connected to the home WiFi network, feels modern. It was recognized as a IPPS device by Chrome OS (using CUPS?). No print server necessary. Amazing.

15 May, 2020 12:00PM by Junichi Uekawa

May 14, 2020

Iustin Pop

New internet provider ☺

Note: all this is my personal experience, on my personal machines, and I don’t claim it to be the absolute truth. Also, I don’t directly call out names, although if you live in Switzerland it’s pretty easy to guess who the old provider is from some hints.

For a long time, I wanted to move away from my current (well, now past) provider, for a multitude of reasons. The main being that the company is very classic company, with classic support system, that doesn’t work very well - I had troubles with their billing system that left me out cold without internet for 15 days, but for the recent few years, they were mostly OK, and changing to a different provider would have meant me routing a really long Ethernet cable around the apartment, so I kept postponing it. Yes, self-inflicted pain, I know.

Until the entire work-from-home thing, when the usually stable connection start degrading in a very linear fashion day-by-day (this is a graph that basically reflects download bandwidth):

1+ month download bandwidth test
1+ month download bandwidth test

At first, I didn’t realise this, as even 100Mbps is still fast enough. But once the connection went below (I believe) 50Mbps, it became visible in day to day work. And since I work daily from home… yeah. Not fun.

So I started doing speedtest.net - and oh my, ~20Mbps was a good result, usually 12-14Mbps. On a wired connection. On a connection that officially is supposed to be 600Mbps down. The upload speed was spot on, so I didn’t think it was my router, but:

  • rebooted everything; yes, including the cable modem.
  • turned off firewall. and on. and off. of course, no change.
  • changed the Ethernet port used on my firewall.
  • changed firewall rules.
  • rebooted again.

Nothing helped. Once in a blue moon, speedtest would give me 100Mbps, but like once every two days, then it would be back and showing 8Mbps. Eight! It ended up as even apt update was tediously slow, and kernel downloads took ages…

The official instructions for dealing with bad internet are a joke:

  • turn off everything.
  • change cable modem from bridged mode to router mode - which is not how I want the modem to work, so the test is meaningless; also, how can router mode be faster⁈
  • disconnect everything else than the test computer.
  • download a program from somewhere (Windows/Mac/iOS, with a browser version too), and run it. yay for open platform!

And the best part: “If you are not satisfied with the results, read our internet optimisation guide. If you are still not happy, use our community forums or our social platforms.”

Given that it was a decline over 3-weeks, that I don’t know of any computer component that would degrade this steadily but not throw any other errors, and that my upload speed was all good, I assumed it’s the provider. Maybe I was wrong, but I wanted to do this anyway for a long while, so I went through the “find how to route cable, check if other provider socket is good, order service, etc.” dance, and less than a week later, I had the other connection.

Now, of course, bandwidth works as expected:

1+ month updated bandwidth test
1+ month updated bandwidth test

Both download and upload are fine (the graph above is just download). Latency is also much better, towards many parts of the internet that matter. But what is shocking is the difference in jitter to some external hosts I care about. On the previous provider, a funny thing was that both outgoing and incoming pings had both more jitter and packet loss when done directly (IPv4 to IPv4) than when done over a VPN. This doesn’t make sense, since VPN is just overhead over IPv4, but the graphs show it, and what I think happens is that a VPN flow is “cached” in the provider’s routers, whereas a simple ping packet not. But, the fact that there’s enough jitter for a ping to a not-very-far host doesn’t make me happy.

Examples, outgoing:

Outgoing smokeping to public IPv4
Outgoing smokeping to public IPv4
Outgoing smokeping over VPN
Outgoing smokeping over VPN

And incoming:

Incoming smokeping to public IPv4
Incoming smokeping to public IPv4
Incoming smokeping over VPN
Incoming smokeping over VPN

Both incoming and outgoing show this weirdness - more packet loss and more jitter over VPN. Again, this is not a problem in practice, or not much, but makes me wonder what other shenanigans happen behind the scenes. You can also see clearly when the “work from home” traffic entered the picture and started significantly degrading my connection, even over the magically “better” VPN connection.

Switching to this week’s view shows the (in my opinion) dramatic improvement in consistency of the connection:

Outgoing current week smokeping to public IPv4
Outgoing current week smokeping to public IPv4
Outgoing current week smokeping over VPN
Outgoing current week smokeping over VPN

No more packet loss, no more jitter. You can also see my VPN being temporarily down during provider switchover because my firewall was not quite correct for a moment.

And the last drill down, at high resolution, one day before and one day after switchover. Red is VPN, blue is plain IPv4, yellow is the missing IPv6 connection :)

Incoming old:

Incoming 1-day smokeping, old provider
Incoming 1-day smokeping, old provider

and new:

Incoming 1-day smokeping, new provider
Incoming 1-day smokeping, new provider

Outgoing old:

Outgoing 1-day smokeping, old provider
Outgoing 1-day smokeping, old provider

and new:

Outgoing 1-day smokeping, new provider
Outgoing 1-day smokeping, new provider

This is what I expect, ping-over-VPN should of course be slower than plain ping. Note that incoming and outgoing have slightly different consistency, but that is fine for me :) The endpoints doing the two tests are different, so this is expected. Reading the legend on the graphs for the incoming connection (similar story for outgoing):

  • before/after plain latency: 24.1ms→18.8ms, ~5.3ms gained, ~22% lower.
  • before/after packet loss: ~7.0% → 0.0%→, infinitely better :)
  • before/after latency standard deviation: ~2.1ms → 0.0ms.
  • before/after direct-vs-VPN difference: inconsistent → consistent 0.4ms faster for direct ping.

So… to my previous provider: it can be done better. Or at least, allow people easier ways to submit performance issue problems.

For me, the moral of the story is that I should have switched a couple of years ago, instead of being lazy. And that I’m curious to see how IPv6 traffic will differ, if at all :)

Take care, everyone! And thanks for looking at these many graphs :)

14 May, 2020 09:40PM

hackergotchi for Norbert Preining

Norbert Preining

Switching from NVIDIA to AMD (including tensorflow)

I have been using my Geforce 1060 extensively for deep learning, both with Python and R. But the always painful play with the closed source drivers and kernel updates, paired with the collapse of my computer’s PSU and/or GPU, I decided to finally do the switch to AMD graphic card and open source stack. And you know what, within half a day I had everything, including Tensorflow running. Yeah to Open Source!

Preliminaries

So what is the starting point: I am running Debian/unstable with a AMD Radeon 5700. First of all I purged all NVIDIA related packages, and that are a lot I have to say. Be sure to search for nv and nvidia and get rid of all packages. For safety I did reboot and checked again that no kernel modules related to NVIDIA are loaded.

Firmware

Debian ships the package amd-gpu-firmware but this is not enough for the current kernel and current hardware. Better is to clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git and copy everything from the amdgpu directory to /lib/firmware/amdgpu.

I didn’t do that at first, and then booting the kernel did hang during the switch to AMD framebuffer. If you see this behaviour, your firmwares are too old.

Kernel

The advantage of having open source driver that is in the kernel is that you don’t have to worry about incompatibilities (like every time a new kernel comes out the NVIDIA driver needs patching). For recent AMD GPUs you need a rather new kernel, I have 5.6.0 and 5.7.0-rc5 running. Make sure that you have all the necessary kernel config options turned on if you compile your own kernels. In my case this is

CONFIG_DRM_AMDGPU=m
CONFIG_DRM_AMDGPU_USERPTR=y
CONFIG_DRM_AMD_ACP=y
CONFIG_DRM_AMD_DC=y
CONFIG_DRM_AMD_DC_DCN=y
CONFIG_HSA_AMD=y

When installing the kernel, be sure that the firmware is already updated so that the correct firmware is copied into the initrd.

Support programs and libraries

All the following is more or less an excerpt from the ROCm Installation Guide!

AMD provides a Debian/Ubuntu APT repository for software as well as kernel sources. Put the following into /etc/apt/sources.list.d/rocm.list:

deb [arch=amd64] http://repo.radeon.com/rocm/apt/debian/ xenial main

and also put the public key of the rocm repository into /etc/apt/trusted.d/rocm.asc.

After that apt-get update should work.

I did install rocm-dev-3.3.0, rocm-libs-3.3.0, hipcub-3.3.0, miopen-hip-3.3.0 (and of course the dependencies), but not rocm-dkms which is the kernel module. If you have a sufficiently recent kernel (see above), the source in the kernel itself is newer.

The libraries and programs are installed under /opt/rocm-3.3.0, and to make the libraries available to Tensorflow (see below) and other programs, I added /etc/ld.so.conf.d/rocm.conf with the following content:

/opt/rocm-3.3.0/lib/

and run ldconfig as root.

Last but not least, add a udev rule that is normally installed by rocm-dkms, put the following into /etc/udev/rules.d/70-kfd.rules:

SUBSYSTEM=="kfd", KERNEL=="kfd", TAG+="uaccess", GROUP="video"

This allows users from the video group to access the GPU.


Up to here you should be able to boot into the system and have X running on top of AMD GPU, including OpenGL acceleration and direct rendering:

$ glxinfo
ame of display: :0
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4
...
client glx vendor string: Mesa Project and SGI
client glx version string: 1.4
...

Tensorflow

Thinking about how hard it was to get the correct libraries to get Tensorflow running on GPUs (see here and here), it is a pleasure to see that with open source all this pain is relieved.

There is already work done to make Tensorflow run on ROCm, the tensorflow-rocm project. The provide up to date PyPi packages, so a simple

pip3 install tensorflow-rocm

is enough to get Tensorflow running with Python:

>> import tensorflow as tf
>> tf.add(1, 2).numpy()
2020-05-14 12:07:19.590169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libhip_hcc.so
...
2020-05-14 12:07:19.711478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7444 MB memory) -> physical GPU (device: 0, name: Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT], pci bus id: 0000:03:00.0)
3
>>

Tensorflow for R

Installation is trivial again since there is a tensorflow for R package, just run (as a user that is in the group staff, which normally own /usr/local/lib/R)

$ R
...
> install.packages("tensorflow")
..

Do not call the R function install_tensorflow() since Tensorflow is already installed and functional!

With that done, R can use the AMD GPU for computations:

$ R
...
> library(tensorflow)
> tf$constant("Hellow Tensorflow")
2020-05-14 12:14:24.185609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libhip_hcc.so
...
2020-05-14 12:14:24.277736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7444 MB memory) -> physical GPU (device: 0, name: Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT], pci bus id: 0000:03:00.0)
tf.Tensor(b'Hellow Tensorflow', shape=(), dtype=string)
>

AMD Vulkan

From the Vulkan home page:

Vulkan is a new generation graphics and compute API that provides high-efficiency, cross-platform access to modern GPUs used in a wide variety of devices from PCs and consoles to mobile phones and embedded platforms.

Several games are using the Vulkan API if available and it is said to be more efficient.

There are Vulkan libraries for Radeon shipped in with mesa, in the Debian package mesa-vulkan-drivers, but they look a bit outdated is my guess.

The AMDVLK project provides the latest version, and to my surprise was rather easy to install, again by following the advice in their README. The steps are basically (always follow what is written for Ubuntu):

  • Install the necessary dependencies
  • Install the Repo tool
  • Get the source code
  • Make 64-bit and 32-bit builds
  • Copy driver and JSON files (see below for what I did differently!)

All as described in the linked README. Just to make sure, I removed the JSON files /usr/share/vulkan/icd.d/radeon* shipped by Debians mesa-vulkan-drivers package.

Finally I deviated a bit by not editing the file /usr/share/X11/xorg.conf.d/10-amdgpu.conf, but instead copying to /etc/X11/xorg.conf.d/10-amdgpu.conf and adding there the section:

Section "Device"
        Identifier "AMDgpu"
        Option  "DRI" "3"
EndSection

.

To be honest, I did not follow the Copy driver and JSON files literally, since I don’t want to copy self-made files into system directories under /usr/lib. So what I did is:

  • copy the driver files to /opt/amdvkn/lib, so I have now there /opt/amdvlk/lib/i386-linux-gnu/amdvlk32.so and /opt/amdvlk/lib/x86_64-linux-gnu/amdvlk64.so
  • Adjust the location of the driver file in the two JSON files /etc/vulkan/icd.d/amd_icd32.json and /etc/vulkan/icd.d/amd_icd64.json (which were installed above under Copy driver and JSON files)
  • added a file /etc/ld.so.conf.d/amdvlk.conf containing the two lines:
    /opt/amdvlk/lib/i386-linux-gnu
    /opt/amdvlk/lib/x86_64-linux-gnu

With this in place, I don’t pollute the system directories, and still the new Vulkan driver is available.

But honestly, I don’t really know whether it is used and is working, because I don’t know how to check.


With all that in place, I can run my usual set of Steam games (The Long Dark, Shadow of the Tomb Raider, The Talos Principle, Supraland, …) and I don’t see any visual problem till now. As a bonus, KDE/Plasma is now running much better, since NVIDIA and KDE has traditionally some incompatibilities.

The above might sound like a lot of stuff to do, but considering that most of the parts are not really packaged within Debian, and all this is rather new open source stack, I was surprised that in half a day I got all working smoothly.

Thanks to all the developers who have worked hard to make this all possible.

14 May, 2020 03:55AM by Norbert Preining

May 13, 2020

hackergotchi for Mike Gabriel

Mike Gabriel

Q: Remote Support Framework for the GNU/Linux Desktop?

TL;DR; For those (admins) of you who run GNU/Linux on staff computers: How do you organize your graphical remote support in your company? Get in touch, share your expertise and experiences.

Researching on FLOSS based Linux Desktops

When bringing GNU/Linux desktops to a generic folk of productive office users on a large scale, graphical remote support is a key feature when organizing helpdesk support teams' workflows.

In a research project that I am currently involved in, we investigate the different available remote support technologies (VNC screen mirroring, ScreenCasts, etc.) and the available frameworks that allow one to provide a remote support infrastructure 100% on-premise.

In this research project we intend to find FLOSS solutions for everything required for providing a large scale GNU/Linux desktop to end users, but we likely will have to recommend non-free solutions, if a FLOSS approach is not available for certain demands. Depending on the resulting costs, bringing forth a new software solution instead of dumping big money in subscription contracts for non-free software is seen as a possible alternative.

As a member of the X2Go upstream team and maintainer of several remote desktop related tools and frameworks in Debian, I'd consider myself as sort of in-the-topic. The available (as FLOSS) underlying technologies for plumbing a remote support framework are pretty much clear (x11vnc, recent pipewire-related approaches in Wayland compositors, browser-based screencasting). However, I still lack a good spontaneous answer to the question: "How to efficiently software-side organize a helpdesk scenario for 10.000+ users regarding graphical remote support?".

Framework for Remote Desktop in Webbrowsers

In fact, in the context of my X2Go activities, I am currently planning to put together a Django-based framework for running X2Go sessions in a web browser. The framework that we will come up with (two developers have already been hired for an initial sprint in July 2020) will be designed to be highly pluggable and it will probably be easy to add remote support / screen sharing features further on.

And still, I walk around with the question in mind: Do I miss anything? Is there anything already out there that provides a remote support solution as 100% FLOSS, that has enterprise grade, that up-scales well, that has a modern UI design, etc. Something that I simply haven't come across, yet?

Looking forward to Your Feedback

Please get in touch (OFTC/Freenode IRC, Telegram, Email), if you can fill the gap and feel like sharing your ideas and experiences.

light+love
Mike

13 May, 2020 07:14AM by sunweaver

May 12, 2020

Kunal Mehta

MediaWiki packages for Ubuntu 20.04 Focal available

Packages for the MediaWiki 1.31 LTS release are now available for the new Ubuntu 20.04 LTS "Focal Fossa" release in my PPA. Please let me know if you run into any errors or issues.

In the future these packages will be upgraded to the MediaWiki 1.35 LTS release, whenever that's ready. It's currently delayed because of the pandemic, but I expect that it'll be ready for the next Debian release.

12 May, 2020 08:21PM by legoktm

hackergotchi for Evgeni Golov

Evgeni Golov

Building a Shelly 2.5 USB to TTL adapter cable

When you want to flash your Shelly 2.5 with anything but the original firmware for the first time, you'll need to attach it to your computer. Later flashes can happen over the air (at least with ESPHome or Tasmota), but the first one cannot.

In theory, this is not a problem as the Shelly has a quite exposed and well documented interface:

Shelly 2.5 pinout

However, on closer inspection you'll notice that your normal jumper wires don't fit as the Shelly has a connector with 1.27mm (0.05in) pitch and 1mm diameter holes.

Now, there are various tutorials on the Internet how to build a compatible connector using Ethernet cables and hot glue or with female header socket legs, and you can even buy cables on Amazon for 18€! But 18€ sounded like a lot and the female header socket thing while working was pretty finicky to use, so I decided to build something different.

We'll need 6 female-to-female jumper wires and a 1.27mm pitch male header. Jumper wires I had at home, the header I got is a SL 1X20G 1,27 from reichelt.de for 0.61€. It's a 20 pin one, so we can make 3 adapters out of it if needed. Oh and we'll need some isolation tape.

SL 1X20G 1,27

The first step is to cut the header into 6 pin chunks. Make sure not to cut too close to the 6th pin as the whole thing is rather fragile and you might lose it.

SL 1X20G 1,27 cut into pieces

It now fits very well into the Shelly with the longer side of the pins.

Shelly 2.5 with pin headers attached

Second step is to strip the plastic part of one side of the jumper wires. Those are designed to fit 2.54mm pitch headers and won't work for our use case otherwise.

jumper wire with removed plastic

As the connectors are still too big, even after removing the plastic, the next step is to take some pliers and gently press the connectors until they fit the smaller pins of our header.

Shelly 2.5 with pin headers and a jumper wire attached

Now is the time to put everything together. To avoid short circuiting the pins/connectors, apply some isolation tape while assembling, but not too much as the space is really limited.

Shelly 2.5 with pin headers and a jumper wire attached and taped

And we're done, a wonderful (lol) and working (yay) Shelly 2.5 cable that can be attached to any USB-TTL adapter, like the pictured FTDI clone you get almost everywhere.

Shelly 2.5 with full cable and FTDI attached

Yes, in an ideal world we would have soldered the header to the cable, but I didn't feel like soldering on that limited space. And yes, shrink-wrap might be a good thing too, but again, limited space and with isolation tape you only need one layer between two pins, not two.

12 May, 2020 10:44AM by evgeni

hackergotchi for Daniel Silverstone

Daniel Silverstone

The Lars, Mark, and Daniel Club

Last night, Lars, Mark, and I discussed Jeremy Kun's The communicative value of using Git well post. While a lot of our discussion was spawned by the article, we did go off-piste a little, and I hope that my notes below will enlighten you all as to a bit of how we see revision control these days. It was remarkably pleasant to read an article where the comments section wasn't a cesspool of horror, so if this posting encourages you to go and read the article, don't stop when you reach the bottom -- the comments are good and useful too.


This was a fairly non-contentious article for us though each of us had points we wished to bring up and chat about it turned into a very convivial chat. We saw the main thrust of the article as being about using the metadata of revision control to communicate intent, process, and decision making. We agreed that it must be possible to do so effectively with Mercurial (thus deciding that the mention of it was simply a bit of clickbait / red herring) and indeed Mark figured that he was doing at least some of this kind of thing way back with CVS.

We all discussed how knowing the fundamentals of Git's data model improved our ability to work wih the tool. Lars and I mentioned how jarring it has originally been to come to Git from revision control systems such as Bazaar (bzr) but how over time we came to appreciate Git for what it is. For Mark this was less painful because he came to Git early enough that there was little more than the fundamental data model, without much of the porcelain which now exists.

One point which we all, though Mark in particular, felt was worth considering was that of distinguishing between published and unpublished changes. The article touches on it a little, but one of the benefits of the workflow which Jeremy espouses is that of using the revision control system as an integral part of the review pipeline. This is perhaps done very well with Git based workflows, but can be done with other VCSs.

With respect to the points Jeremy makes regarding making commits which are good for reviewing, we had a general agreement that things were good and sensible, to a point, but that some things were missed out on. For example, I raised that commit messages often need to be more thorough than one-liners, but Jeremy's examples (perhaps through expedience for the article?) were all pretty trite one-liners which perhaps would have been entirely obvious from the commit content. Jeremy makes the point that large changes are hard to review, and Lars poined out that Greg Wilson did research in this area, and at least one article mentions 200 lines as a maximum size of a reviewable segment.

I had a brief warble at this point about how reviewing needs to be able to consider the whole of the intended change (i.e. a diff from base to tip) not just individual commits, which is also missing from Jeremy's article, but that such a review does not need to necessarily be thorough and detailed since the commit-by-commit review remains necessary. I use that high level diff as a way to get a feel for the full shape of the intended change, a look at the end-game if you will, before I read the story of how someone got to it. As an aside at this point, I talked about how Jeremy included a 'style fixes' commit in his example, but I loathe seeing such things and would much rather it was either not in the series because it's unrelated to it; or else the style fixes were folded into the commits they were related to.

We discussed how style rules, as well as commit-bisectability, and other rules which may exist for a codebase, the adherence to which would form part of the checks that a code reviewer may perform, are there to be held to when they help the project, and to be broken when they are in the way of good communication between humans.

In this, Lars talked about how revision control histories provide high level valuable communication between developers. Communication between humans is fraught with error and the rules are not always clear on what will work and what won't, since this depends on the communicators, the context, etc. However whatever communication rules are in place should be followed. We often say that it takes two people to communicate information, but when you're writing commit messages or arranging your commit history, the second party is often nebulous "other" and so the code reviewer fulfils that role to concretise it for the purpose of communication.

At this point, I wondered a little about what value there might be (if any) in maintaining the metachanges (pull request info, mailing list discussions, etc) for historical context of decision making. Mark suggested that this is useful for design decisions etc but not for the style/correctness discussions which often form a large section of review feedback. Maybe some of the metachange tracking is done automatically by the review causing the augmentation of the changes (e.g. by comments, or inclusion of design documentation changes) to explain why changes are made.

We discussed how the "rebase always vs. rebase never" feeling flip-flopped in us for many years until, as an example, what finally won Lars over was that he wants the history of the project to tell the story, in the git commits, of how the software has changed and evolved in an intentional manner. Lars said that he doesn't care about the meanderings, but rather about a clear story which can be followed and understood.

I described this as the switch from "the revision history is about what I did to achieve the goal" to being more "the revision history is how I would hope someone else would have done this". Mark further refined that to "The revision history of a project tells the story of how the project, as a whole, chose to perform its sequence of evolution."

We discussed how project history must necessarily then contain issue tracking, mailing list discussions, wikis, etc. There are exist free software projects where part of their history is forever lost because, for example, the project moved from Sourceforge to Github, but made no effort (or was unable) to migrate issues or somesuch. Linkages between changes and the issues they relate to can easily be broken, though at least with mailing lists you can often rewrite URLs if you have something consistent like a Message-Id.

We talked about how cover notes, pull request messages, etc. can thus also be lost to some extent. Is this an argument to always use merges whose message bodies contain those details, rather than always fast-forwarding? Or is it a reason to encapsulate all those discussions into git objects which can be forever associated with the changes in the DAG?

We then diverted into discussion of CI, testing every commit, and the benefits and limitations of automated testing vs. manual testing; though I think that's a little too off-piste for even this summary. We also talked about how commit message audiences include software perhaps, with the recent movement toward conventional commits and how, with respect to commit messages for machine readability, it can become very complex/tricky to craft good commit messages once there are multiple disparate audiences. For projects the size of the Linux kernel this kind of thing would be nearly impossible, but for smaller projects, perhaps there's value.

Finally, we all agreed that we liked the quote at the end of the article, and so I'd like to close out by repeating it for you all...

Hal Abelson famously said:

Programs must be written for people to read, and only incidentally for machines to execute.

Jeremy agrees, as do we, and extends that to the metacommit information as well.

12 May, 2020 07:00AM by Daniel Silverstone

Jacob Adams

Roman Finger Counting

I recently wrote a final paper on the history of written numerals. In the process, I discovered this fascinating tidbit that didn’t really fit in my paper, but I wanted to put it somewhere. So I’m writing about it here.

If I were to ask you to count as high as you could on your fingers you’d probably get up to 10 before running out of fingers. You can’t count any higher than the number of fingers you have, right? The Romans could! They used a place-value system, combined with various gestures to count all the way up to 9,999 on two hands.

The System

Finger Counting (Note that in this diagram 60 is, in fact, wrong, and this picture swaps the hundreds and the thousands.)

We’ll start with the units. The last three fingers of the left hand, middle, ring, and pinkie, are used to form them.

Zero is formed with an open hand, the opposite of the finger counting we’re used to.

One is formed by bending the middle joint of the pinkie, two by including the ring finger and three by including the middle finger, all at the middle joint. You’ll want to keep all these bends fairly loose, as otherwise these numbers can get quite uncomfortable. For four, you extend your pinkie again, for five, also raise your ring finger, and for six, you raise your middle finger as well, but then lower your ring finger.

For seven you bend your pinkie at the bottom joint, for eight adding your ring finger, and for nine, including your middle finger. This mirrors what you did for one, two and three, but bending the finger at the bottom joint now instead.

This leaves your thumb and index finger for the tens. For ten, touch the nail of your index finger to the inside of your top thumb joint. For twenty, put your thumb between your index and middle fingers. For thirty, touch the nails of your thumb and index fingers. For forty, bend your index finger slightly towards your palm and place your thumb between the middle and top knuckle of your index finger. For fifty, place your thumb against your palm. For sixty, leave your thumb where it is and wrap your index finger around it (the diagram above is wrong). For seventy, move your thumb so that the nail touches between the middle and top knuckle of your index finger. For eighty, flip your thumb so that the bottom of it now touches the spot between the middle and top knuckle of your index finger. For ninety, touch the nail of your index finger to your bottom thumb joint.

The hundreds and thousands use the same positions on the right hand, with the units being the thousands and the tens being the hundreds. One account, from which the picture above comes, swaps these two, but the first account we have uses this ordering.

Combining all these symbols, you can count all the way to 9,999 yourself on just two hands. Try it!

History

The Venerable Bede

The first written record of this system comes from the Venerable Bede, an English Benedictine monk who died in 735.

He wrote De computo vel loquela digitorum, “On Calculating and Speaking with the Fingers,” as the introduction to a larger work on chronology, De temporum ratione. (The primary calculation done by monks at the time was calculating the date of Easter, the first Sunday after the first full moon of spring).

He also includes numbers from 10,000 to 1,000,000, but its unknown if these were inventions of the author and were likely rarely used regardless. They require moving your hands to various positions on your body, as illustrated below, from Jacob Leupold’s Theatrum Arilhmetico-Geometricum, published in 1727:

Finger Counting with Large Numbers

The Romans

If Bede was the first to write it, how do we know that it came from Roman times? It’s referenced in many Roman writings, including this bit from the Roman satirist Juvenal who died in 130:

Felix nimirum qui tot per saecula mortem distulit atque suos iam dextera computat annos.

Happy is he who so many times over the years has cheated death And now reckons his age on the right hand.

Because of course the right hand is where one counts hundreds!

There’s also this Roman riddle:

Nunc mihi iam credas fieri quod posse negatur: octo tenes manibus, si me monstrante magistro sublatis septem reliqui tibi sex remanebunt.

Now you shall believe what you would deny could be done: In your hands you hold eight, as my teacher once taught; Take away seven, and six still remain.

If you form eight with this system and then remove the symbol for seven, you get the symbol for six!

Sources

My source for this blog post is Paul Broneer’s 1969 English translation of Karl Menninger’s Zahlwort und Ziffer (Number Words and Number Symbols).

12 May, 2020 12:00AM

May 11, 2020

Tim Retout

hackergotchi for Markus Koschany

Markus Koschany

My Free Software Activities in April 2020

Welcome to gambaru.de. Here is my monthly report (+ the first week in May) that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games

Playonlinux
  • Scott Talbert did a fantastic job by porting playonlinux, a user-friendly frontend for Wine, to Python 3 (#937302). I tested his patch and uploaded the package today. More testing and feedback is welcome. Scott’s work basically prevented the removal of one of the most popular packages in the games section. I believe this will also give interested people more time to package the Java successor of playonlinux called Phoenicis.
  • Reiner Herrmann ported ardentryst, an action role playing game, to Python 3 to fix a release critical Py2 removal bug (#936148). He also packaged the latest release of xaos, a real-time interactive fractal zoomer, and improved various packaging details. I reviewed both of them and sponsored the upload for him.
  • I packaged new upstream releases of minetest, lordsawar, gtkatlantic and cutemaze.
  • I also sponsored a new simutrans update for Jörg Frings-Fürst.

Debian Java

Misc

  • I packaged new versions of wabt and binaryen, required to build Webassembly code from source.

Debian LTS

This was my 50. month as a paid contributor and I have been paid to work 11,5 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • I completed the security update of Tomcat 8 in Stretch released as DSA-4673-1 and Tomcat 8 in Jessie soon to be released as DLA-2209-1.
  • I am currently assigned more hours and my plan is to invest the time in a project to improve our knowledge about embedded code copies and their current security impact which I want to discuss with the security team. The rest will be spent on Stretch security updates which will become the new LTS release soon.

ELTS

Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 7 „Wheezy“. This was my 23. month and I have been paid to work 2 hours on ELTS.

  • I prepared the fix for CVE-2019-18218 in php5 released as ELA-227-1.
  • I checked jetty for unfixed vulnerabilities and discovered that the version in Wheezy was not affected by CVE-2019-17632. No further action was required.
  • It turned out that the apache2 package in Wheezy was not affected by vulnerable embedded expat code because it depends on already fixed system packages.

Thanks for reading and see you next time.

11 May, 2020 05:53PM by apo

hackergotchi for Gunnar Wolf

Gunnar Wolf

Certified printer fumes

After losing a fair bit of hair due to quality and reliability issues with our home laser multifunctional (Brother DCP1600-series, which we bought after checking it was meant to work on Linux… And it does, but with a very buggy, proprietary driver — Besides being the printer itself of quite low quality), we decided it was time to survey the market again, and get a color inkjet printer. I was not very much an enthusiast of the idea, until I found all of the major manufacturers now offer refillable ink tanks instead of the darn expensive cartridges of past decades. Lets see how it goes!

Of course, with over 20 years of training, I did my homework. I was about to buy an Epson printer, but decided for an HP Ink Tank 410 Wireless printer. The day it arrived, not wanting to fuss around too much to get to see the results, I connected it to my computer using the USB cable. Everything ran smoothly and happily! No driver hunting needed, print quality is superb… I hope, years from now, we stay with this impression.

Next day, I tried to print over WiFi. Of course, it requires configuration. And, of course, configuration strongly wants you to do it from a Windows or MacOS machine — which I don’t have. OK, fall back to Android — For which an app download is required (and does not thrill me, but what can I say. Oh — and the app needs location services to even run. Why‽ Maybe because it interacts with the wireless network in WiFi Direct, non-authenticated way?)

Anyway, things seem to work. But they don’t — My computers can identify and connect with the printer from CUPS, but nothing ever comes out. Printer paused, they say. Entering the printer’s web interface is somewhat ambiguous — Following the old HP practices, I tried http://192.168.1.75:9100/ (no point in hiding my internal IP), and got a partial webpage sometimes (and nothing at all othertimes). Seeing the printer got detected over ipps://, my immediate reaction was to try pointing the browser to port 631. Seems to work! Got some odd messages… But it seems I’ll soon debug the issue away. I am not a familiar meddler in the dark lands of cups, our faithful print server, but I had to remember my toolkit..

# cupsenable HP_Ink_Tank_Wireless_410_series_C37468_ --release

Sucess in enabling, but self-auto-disabled right away… lpstat -t was not more generous, reporting only it was still paused.

… Some hours later (mix in attending kids and whatnot), I finally remember to try cupsctl --debug-logging, and magically, /var/log/cups/error_log turns from being quiet to being quite chatty. And, of course, my first print job starts being processed:

D [10/May/2020:23:07:20 -0500] Report: jobs-active=1
(...)
D [10/May/2020:23:07:25 -0500] [Job 174] Start rendering...
(...)
D [10/May/2020:23:07:25 -0500] [Job 174] STATE: -connecting-to-device
(...)

Everything looks fine and dandy so far! But, hey, given nothing came out of the printer… keep reading one more second of logs (a couple dozen lines)

D [10/May/2020:23:07:26 -0500] [Job 174] Connection is encrypted.
D [10/May/2020:23:07:26 -0500] [Job 174] Credentials are expired (Credentials have expired.)
D [10/May/2020:23:07:26 -0500] [Job 174] Printer credentials: HPC37468 / Thu, 01 Jan 1970 00:00:00 GMT / 28A59EF511A480A34798B6712DEEAE74
D [10/May/2020:23:07:26 -0500] [Job 174] No stored credentials.
D [10/May/2020:23:07:26 -0500] [Job 174] update_reasons(attr=0(), s=\"-cups-pki-invalid,cups-pki-changed,cups-pki-expired,cups-pki-unknown\")
D [10/May/2020:23:07:26 -0500] [Job 174] STATE: -cups-pki-expired
(...)
D [10/May/2020:23:08:00 -0500] [Job 174] envp[16]="CUPS_ENCRYPTION=IfRequested"
(...)
D [10/May/2020:23:08:00 -0500] [Job 174] envp[27]="PRINTER_STATE_REASONS=cups-pki-expired"

My first stabs were attempts to get CUPS not to care about expired certificates, but it seems to have been hidden or removed from its usual place. Anyway, I was already frustrated.

WTF‽ Well, yes, turns out that from the Web interface, I paid some attention to this the first time around, but let it pass (speaks wonders about my security practices!):

Way, way, way too expired cert

So, the self-signed certificate the printer issued at itself expired 116 years before even being issued. (is this maybe a Y2k38 bug? Sounds like it!) Interesting thing, my CUPS log mentions the printer credentials to expire at the beginning of the Unix Epoch (01 Jan 1970 00:00:00 GMT).

OK, lets clickety-click away on the Web interface… Didn’t take me long to get to Network ⇒ Advanced settings ⇒ Certificates:

Can manage certs!

However, clicking on Configure leads me to the not very reassuring…

Way, way, way too expired cert

I don’t remember what I did for the next couple of minutes. Kept fuming… Until I parsed again the output of lpstat -t, and found that:

# lpstat -t
(...)
device for HP_Ink_Tank_Wireless_410_series_C37468_: ipps://HPF43909C37468.local:443/ipp/print
(...)

Hmmmm… CUPS is connecting using good ol’ port 443, as if it were a Web thingy… What if I do the same?

Now we are talking!

Click on “New self-signed certificate�, click on Next, a couple of reloads… And a very nice color print came out of the printer, yay!

Now, it still baffles me (of course I checked!): The self-signed certificate is now said to come from Issuer : CN=HPC37468, L=Vancouver, ST=Washington, C=US, O=HP,OU=HP-IPG, alright… not that it matters (I can import a more meaningful one if I really feel like it), but, why is it Issued On: 2019-06-14 and set to Expires On: 2029-06-11?

Anyway, print quality is quite nice. I hope to keep the printer long enough to rant at the certificate being expired in the future!

Comments

Jeff Epler (Adafruit) 2020-05-11 20:39:17 -0500

“why is it Issued On: 2019-06-14 and set to Expires On: 2029-06-11?� → Because it’s 3650 days

Gunnar Wolf 2020-05-11 20:39:17 -0500

Nice catch! Thanks for doing the head-scratching for me 😉

11 May, 2020 07:00AM by Gunnar Wolf

May 10, 2020

Enrico Zini

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, April 2020

I was assigned 20 hours of work by Freexian's Debian LTS initiative, and carried over 8.5 hours from March. I worked 26 hours this month, so I will carry over 2.5 hours to May.

I sent a (belated) request for testing an update of the linux package to 3.16.82. I then prepared and, after review, released Linux 3.16.83, including a large number of security fixes. I rebased the linux package onto that and will soon send out a request for testing. I also spent some time working on a still-embargoed security issue.

I did not spend signficant time on any other LTS activities this month, and unfortunately missed the contributor meeting.

10 May, 2020 05:27PM

Russell Coker

IT Asset Management

In my last full-time position I managed the asset tracking database for my employer. It was one of those things that “someone” needed to do, and it seemed that only way that “someone” wouldn’t equate to “no-one” was for me to do it – which was ok. We used Snipe IT [1] to track the assets. I don’t have enough experience with asset tracking to say that Snipe is better or worse than average, but it basically did the job. Asset serial numbers are stored, you can have asset types that allow you to just add one more of the particular item, purchase dates are stored which makes warranty tracking easier, and every asset is associated with a person or listed as available. While I can’t say that Snipe IT is better than other products I can say that it will do the job reasonably well.

One problem that I didn’t discover until way too late was the fact that the finance people weren’t tracking serial numbers and that some assets in the database had the same asset IDs as the finance department and some had different ones. The best advice I can give to anyone who gets involved with asset tracking is to immediately chat to finance about how they track things, you need to know if the same asset IDs are used and if serial numbers are tracked by finance. I was pleased to discover that my colleagues were all honourable people as there was no apparent evaporation of valuable assets even though there was little ability to discover who might have been the last person to use some of the assets.

One problem that I’ve seen at many places is treating small items like keyboards and mice as “assets”. I think that anything that is worth less than 1 hour’s pay at the minimum wage (the price of a typical PC keyboard or mouse) isn’t worth tracking, treat it as a disposable item. If you hire a programmer who requests an unusually expensive keyboard or mouse (as some do) it still won’t be a lot of money when compared to their salary. Some of the older keyboards and mice that companies have are nasty, months of people eating lunch over them leaves them greasy and sticky. I think that the best thing to do with the keyboards and mice is to give them away when people leave and when new people join the company buy new hardware for them. If a company can’t spend $25 on a new keyboard and mouse for each new employee then they either have a massive problem of staff turnover or a lack of priority on morale.

10 May, 2020 01:42PM by etbe

hackergotchi for Norbert Preining

Norbert Preining

Updating Dovecot for Debian

A tweet of a friend pointed me at the removal of dovecot from Debian/testing, which surprised me a bit. Investigating the situation it seems that Dovecot in Debian is lagging a bit behind in releases, and hasn’t seen responses to some RC bugs. This sounds critical to me as dovecot is a core part of many mail setups, so I prepared updated packages.

Based on the latest released version of Dovecot, 2.3.10, I have made a package starting from the current Debian packaging and adjusted to the newer upstream. The package builds on Debian Buster (10), Testing, and Unstable on i386 and x64 archs. The packages are available on OBS, as usual:

For Unstable:

deb https://download.opensuse.org/repositories/home:/npreining:/debian-dovecot/Debian_Unstable/ ./

For Testing:

deb https://download.opensuse.org/repositories/home:/npreining:/debian-dovecot/Debian_Testing/ ./

For Debian 10 Buster:

deb https://download.opensuse.org/repositories/home:/npreining:/debian-dovecot/Debian_10/ ./

To make these repositories work, don’t forget that you need to import my OBS gpg key: obs-npreining.asc, best to download it and put the file into /etc/apt/trusted.gpg.d/obs-npreining.asc.

These packages are provided without any warranty. Enjoy.

10 May, 2020 12:54PM by Norbert Preining

NOKUBI Takatsugu

Virtual Background using webcam

I made a webpage to produce virtual background with webcam.

https://knok.github.io/virtbg/

Sorcecode:
https://github.com/knok/knok.github.io/tree/master/virtbg

Some online meeting software (Zoom, Microsoft Teams) supports virtual background, but I want to use other software like Jitsi (or google meet), so I made it.

To make this, I referred the article “Open Source Virtual Background”. 
The following figure is the diagram.

It depends on docker, GPU, v4l2loopback (only works on Linux), so I want to make more generic solution. To make as a webpage, and using OBS
Studio with plugins (obs-v4l2sink, OBS-VirtualCam or OBS (macOS) Virtual Camera) you can use the solution on more platforms.

To make as a single webpage, I can reduce overhead using inter-process commuication using http via docker.

This is an example animation:

y4069-ju7wv.gif

Using jisti snapshot:

image.png

Unfortunately, BodyPix releases only pretraind models, no training data.

I need more improvements:

  • Accept any background images
  • Suppot choose camera device
  • Useful UI

10 May, 2020 10:50AM by knok