worrdev

[worrdev]

It’s been awhile y’all

It's been a hot minute since I've documented some of my work, so I guess in keeping with making a main blog post, I'll make a devblog post today too.

cfn-mode / flycheck-cfn

https://gitlab.com/worr/cfn-mode/

I've been an emacs user for some time, and at my current job, I've been hurting for good support for cloudformation templates in my editor. I wrote this mode and flychecker to at least add some basic syntax highlighting and linter support. I'm currently in the process of getting them added to MELPA.

imdb-api

I made a bunch of changes fairly recently to imdb-api, most notably adding front-end support, migrating to Gitlab and migrating to ky after node-requests was deprecated. Normally I'd link patches, but there are too many since my last update. Here's the changelog: https://gitlab.com/worr/node-imdb-api/-/blob/master/CHANGELOG.md

fluent-bit

At work, we discovered an issue where our fluent-bits were sticky to the same instance of fluentd if we turned on keepalive and used a load-balancer.

To mitigate this, I ended up adding a new option to fluent-bit that will recycle keepalive connections after sending a number of messages, to cycle between backend instances periodically.

https://github.com/fluent/fluent-bit/commit/44190c2a1c4b939dc9ecb2908148d38c82a40831

https://github.com/fluent/fluent-bit-docs/commit/8d43b502123e366a1722a0051918ce7d78a8506b

fluentd-s3-plugin

Also at work, we found a case where the fluend plugin for s3 would spend forever trying to upload logs. By default, the naming scheme for the log chunks would be something like <time_slice>_<idx>. The time slice is the time when the log was uploaded, and the idx value is a monotonically increasing integer.

The problem, is that if you have mutliple threads uploading (or multiple fluentd's, or both), they have to check the presence of the filename to ensure that the formulated filename doesn't exist. Additionally, it doesn't track the last-used index, so when doing this check, fluentd will start at 1, check, increment, check again, increment again, etc. This obviously doesn't scale very well when you are outputing a ton of logs.

We fixed this my changing our file format to include a uuid and disabling the behavior to check for collisions.

However, since the defaults are dangerous, I've submitted this PR to try and make things less dangerous for new users (not accepted yet at the time of this writing).

https://github.com/fluent/fluent-plugin-s3/pull/355/files

This works by tracking the last used index in an atomic that's shared between threads. As outlined in the PR, it doesn't solve the whole problem, but it does make the defaults considerably safer.

logging-operator

Perhaps you've noticed a theme with my recent, work-driven contributions. :)

logging-operator is a kubernetes operator to automate adminitration of logging infrastructure in a kubernetes cluster. I've been contributing a bit to it lately, since we adopted it fairly early, and have needed to add a few features to make it work for us.

This first diff adds support not just for the configurable I added to fluent-bit that I mentioned earlier, but exposes all net parameters as configurables.

https://github.com/banzaicloud/logging-operator/commit/3c9e3938590209716918bc7cc197b43b09bb4361

There was a string conversion bug in how the operator would report on prometheus configuration.

https://github.com/banzaicloud/logging-operator/commit/86503b52150cf0dcf62d4b636eb247d0807101e7

We needed to configure where in s3 fluentd was uploading these logs

https://github.com/banzaicloud/logging-operator/commit/29fccfc2b8cee6c38c88fb34cf73a112eeb534de

We also needed way more support for configuring certain scheduling attributes in fluentd and fluentbit

https://github.com/banzaicloud/logging-operator/commit/45dffe5ebb38a3dbba4ecb217235f45c13f7856e

https://github.com/banzaicloud/logging-operator/commit/961fd788bb90f8f46d188a731aac0a916b30f933

https://github.com/banzaicloud/logging-operator/commit/0ec91f72831e1e63bd560224450454b33084553d

I also had to expose a number of these features in their helm charts

https://github.com/banzaicloud/logging-operator/commit/efc74711c5336063a6da72bf39239c57c81c7dff

https://github.com/banzaicloud/logging-operator/commit/f581da2e9daadae9b786362f69d379f8151ad918

https://github.com/banzaicloud/logging-operator/commit/4e74e36dfe7d63212b19401fe645a198734da1fd

wsh

Someone reached out to me privately to report several Solaris 11 compatibility bugs with wsh, my multi-host ssh tool.

Use -std=c11 flag with SunStudio: https://github.com/worr/wsh/commit/b11d2668ef6b85913d1901cfbfe6eb612be69bdc

Don't use __attribute__ with SunStudio, since none of the ones I used were supported: https://github.com/worr/wsh/commit/25ed3fc6fa36a1202e33c8fb36893d03cd5bce8c

Don't unconditionally compile in memset_s (Found because Solaris actually has a libc with this function): https://github.com/worr/wsh/commit/3876745a5cc4bce80d5e9fff0ab70b2dc429287f

This also led to a protobuf PR for better SunStudio support, which it looks like I need to follow up on.

https://github.com/protocolbuffers/protobuf/pull/7049/files

python-fido2

Last post, I mentioned I was working on getting my yubikey to work on OpenBSD. Part of that included adding support in ykman, which also required changes in python-fido2.

First, I added general OpenBSD support

https://github.com/Yubico/python-fido2/pull/82/files

This impl is arguably a bit brittle, since I essentially had to build the device probing for it in scratch from python, using the primitives from libusbhid to probe every uhid(4) device to see if it was a yubikey.

However, some time later, fido(4) was rolled into OpenBSD meaning that this code could be greatly simplified. I think someone reached out to me about this directly? I don't really remember, since it was awhile ago.

https://github.com/Yubico/python-fido2/pull/87/files

What a year

That's basically been the last year or so for me. Honestly, it's been a weird one, and I haven't been able to really do as much OSS as I've wanted to. A lot of it has been through work, which while nice, doesn't touch the types of projects that I want to be doing.

I am working on a gemini server on OpenBSD, which has been feeling quite rewarding, and I have other projects kicking around in my head that I'm going to be following up on.

[worrdev]

usb, wsh, sceptre, more

I haven't updated this in a hot minute, but since I've gotten back into the swing of working on stuff, I thought I'd share some commits from the past weeks or so.

First, the big theme that I'm working towards has been getting U2F working on OpenBSD. No real progress on that so far, since I need to touch several projects for that to get traction. That said, I have contributed small bits to other related projects.

In that vein:

libusb

In testing some of my U2F-related code, I found a number of issues with libusb on OpenBSD

The biggest was the fields in the initialization of the OpenBSD backend incorrectly corresponding to how they were declared in the struct definition. This would cause an immediate segfault when libusb would try and get the time on an OpenBSD host, since clock_gettime was mapped to a NULL pointer. The fix was to swith to a designated initializer (tbh I never knew what it was called until today) to ensure that we don't make this same mistake again.

That revealed another bug, where obsd_clock_gettime would emit a debug statement, but then the libusb_dbg function would call into clock_gettime...causing infinite recursion. Fixed here.

Then like formatting goofs

ykman

As you can imagine, one of the first stops in my U2F trek has been just talking to my yubikey with the ykman tool. This tool doesn't officially support OpenBSD...at all as it depends on python-fido (which is one of the libraries I'm writing an OpenBSD backend for) which depends on pyu2f (OpenBSD backend also in progress).

Using it for testing revealed not only the libusb bugs I found above, but also a new issue that was introduced recently where I'd get a use-after-free just enumerating my Yubikeys.

This was caused by this commit. To understand why this commit did this though, we need to talk about pyusb.

pyusb is a python wrapper around not just libusb1, but also libusb0 and some other library called openusb. You can choose whichever one you want, or even switch between them at runtime. I have libusb1, so that's the backend ykman was choosing to use.

When you call libusb1.get_backend(), you get back a _LibUSB object. However the important thing about this object, is that they have it call libusb_exit in its finalizer (it's worth noting that _finalize_object is this libraries way of calling code in __del__).

This is where the bug lies, since libusb_exit tears down important structures, like a bunch of mutexes. If you still had a device object that relied on these mutexes, that would be a use-after-free.

ykman would create a backend every time one would be asked for](https://github.com/Yubico/yubikey-manager/commit/1e8b16ca644c382f912051f144253e2e7bf83bc5), and then persist descriptor objects created from that backend. Since now, the descriptors would outlive the backend, when those descriptors tried to acquire a mutex to talk to a device, you'd get a use-after-free.

My fix to this was not to create a global backend like existed before, but simply to add a reference to the backed that generated the descriptor.

wsh

wsh is a multissh tool that I wrote back at LinkedIn. Apparently, some of my old coworkers still use this since I got 3 new fresh PRs for it the other day.

https://github.com/worr/wsh/pull/108

The most significant, I apparently don't handle partial authentication correctly. The weird thing is that the version of libssh that travis has doesn't seem to have the SSH_AUTH_PARTIAL symbol.

https://github.com/worr/wsh/pull/107

Allows the user to specify which MACs they'll allow on the command-line.

https://github.com/worr/wsh/pull/106

Here, I failed to account for the hard rlimit of nofile to be set to infinity, so on systems where this is so, we fail. Hopefully this ends up setting the soft rlimit to infinity or some other high value if that's permitted.

sceptre

At work we use AWS heavily, and define our infrastructure through Cloudformation templates. To deploy them, we use a tool called sceptre. I had been tasked with building a tool to bundle static analyzers into our pipeline so that we could verify correctness of our cfn before we deployed it anywhere.

Through sceptre, we make use of templating pretty heavily, and so we don't have the luxury of just piping our cfn through a static analyzer. My approach was just to call sceptre generate and feed that compiled output through to the analyzers.

The big problem I encountered was this bug: https://github.com/Sceptre/sceptre/issues/658

Sceptre would fail to generate any template that used the YAML tag notation Amazon has for calling Cloudformation functions.

While my implementation went through changes over time, we ended up settling on implementing a custom yaml loader that would handle the tags currently published by Amazon: https://github.com/Sceptre/sceptre/commit/ed8279330c6927fe97c054cecfc83b71ef409fda#diff-985c5ac1f9746c911b11118d6c80b109R312

Firefox

I use keepassxc for my password storage, and it has a firefox plugin! Neat!

However it requires native messaging to communicate to the browser.

I noticed that it would never successfully connect on OpenBSD. On other browsers, it worked fine, but OpenBSD always failed. I looked at the browser console and saw this: cannot read contents of null pointer ctypes.char.ptr.ptr(ctypes.UInt64("0x0")).

Whoops! Firefox, when trying to launch a process reads the environ global to fetch the current environment (admittedly I'm not sure if environ is actually global or if it's thread-local. Doesn't matter for my purposes though). On OpenBSD, it's possible for environ to point to NULL. Firefox didn't check this before dereferencing, and naturally would fail to read the environment.

https://hg.mozilla.org/mozilla-central/rev/6b88d6c20e5795c1830cf542960f00b03595f578

[worrdev]

imdb-api releases and more

It's been a few weeks since I last updated this blog. But also like, I haven't been super busy.

imdb-api

I've shipped a few new versions of imdb-api. Mostly I'm dealing with some garbage of the downstream I'm calling changing their interface and requiring payment.

I've also added some very necessary features, like searching and timeouts. Not sure how I didn't include these in the first round but you know, live and learn.

mypy-rt

I've started a project that takes python function type annotations and enforces them at runtime, by dynamically injecting bytecode into the compiled function.

This is largely useful if you maintain a library, so that you can enforce types at your API boundary without writing extra code beyond your type annotations.

Code here

tsserver

I added a tiny commit that fixes tsserver on OpenBSD

[worrdev]

Angband on OpenBSD

This weekend, I was lucky enough to go to the Roguelike Celebration in SF! I've honestly only really played nethack, and only in passing; I preferred working on the codebase to playing it!

I got to see a talk from some of the Angband developers, which definitely piqued my interest in the game. At the conference, I popped open my new OpenBSD laptop to install it and play.

The version in ports was a bit old (3.3.2), so I set out to update it to 4.0.5. Building the game had changed a bit, and introduced some issues with detecting ncurses on OpenBSD. After some poking, I found out it was trying to use ncurses5-config to grab the cflags and ldflags of ncurses - which isn't present on OpenBSD! I quickly added an additional check using AC_CHECK_LIB, in the case we failed to find find ncurses5-config.

Once that was done, I found another problem: angband -mgcu would hang on startup. I found out that mvwaddnwstr was guarded by _XOPEN_SOURCE_EXTENDED. By not defining that macro, I'd get implicit decl warnings from the compiler, and it'd only be runnable because that symbol would be found in the linked libncurses.

So why does this work when I define _XOPEN_SOURCE_EXTENDED? mvwaddnwstr (and many other functions) are implemented both as function-like macros and as functions. Normally the function implementations aren't used - you have to define NCURSES_NOMACROS. It looks like (I've seen one other report of this as well) that the mvwaddnwstr function has a bug that can sometimes cause it to loop infinitely. The macro version works just fine, so once I unguard it by defining the _XOPEN_SOURCE_EXTENDED macro, everything works fine.

Submitted upstream and to ports.

https://github.com/angband/angband/pull/463

https://marc.info/?l=openbsd-ports&m=147434362321356&w=2

[worrdev]

imdb-api and wsh

This weekend, I wanted to spend some serious time working on projects. While I wanted to focus on privdrop support for pkgsrc, I got sidetracked by a few smaller OSS projects that I created.

wsh

wsh is a tool that I wrote in college that manages performing commands on a large number of machines relatively quickly. It ended up getting used at LnkedIn, which is pretty cool.

Big features I worked on:

Passing ssh options on the command line

Figure out how many threads to run automatically

Killing idle threads

I also did some PR management

Sudo without invoking a shell

imdb-api

node-imdb-api is one of my smallest, oldest projects, but is surprisingly used. I don't like admitting that I had a javascript phase, but uh, it happens. :) Luckily, for my sanity I wrote it in typescript.

I mostly did a rewrite this weekend, which keeps most of the same API, but uses an entirely different backend, and adds a lot more type safety. I won't link all the commits here, but I will link the big ones.

omdb switch for movies

use promises internally

type safety for remote calls

youshouldnt.date

This weekend I also launched my bathroom reading site, youshouldnt.date. It simply aggregates relationship stories from reddit, so I can peruse completed stories rather than having to wade through a bunch of relationship questions. It's very rough, and needs styling / more features. I'll do a proper announcement once I'm happy with it. Code is all here.

[worrdev]

{py}gradle

My company just released a Gradle plugin so that you can build Python projects with Gradle! imo this is mostly useful if you have many existing projects at your company that also build with Gradle, and you want to standardize on a single build system for all projects.

As a hackday project, I wanted to add an optional linter check using mypy. After writing all the code to make this work, which was pretty easy despite not know Java nor Groovy, I wrote an integration test that would actually run mypy when your build.gradle specified that you were using python 3.5 and when you opted into the check.

To my surprise, my tests failed immediately – but not because of my changes. Turns out the in the initial release, pygradle didn't actually support the documented method for setting the python version didn't actually work. In fact, the whole details block where you could specify your python version, python path or venv didn't work.

I talked with one of the maintainers, Ethan Hall, and he and I worked to gether to come up with a solution. I was super out of my depth working on this, but I did learn quite a bit about Groovy and Gradle on the way.

https://github.com/linkedin/pygradle/pull/36

[worrdev]

pkgsrc + uwsgi

Just like my last post, I get to start this one off by saying that it's been a long time since I've done any work on FOSS :) I've been busy traveling and seeing a bunch of old friends - both good problems to have.

That said, I feel obligated to post a little bit about what I've been doing lately.

uwsgi

I've been using uwsgi quite a bit lately, and I've found a few minor issues.

https://github.com/unbit/uwsgi/pull/1229/commits/5b20f8cead01e06fbdf5a950c500f9de17efc441

I added support for statsd and socket metrics to be emitted over a unix socket.

https://github.com/unbit/uwsgi/pull/1336/commits/f6d96c88d2b0022548163feb914d8dbd1b6473b4 https://github.com/unbit/uwsgi/pull/1331/commits/90d9749ae629ad8eeb935088edabb23e8237e173

After working with the codebase a little bit, I got a bit curious and ran clang's static analyzer on it. I fixed two of the more major issues. Luckily, the maintainers started doing the same afterwards, and fixed more of the problems that had cropped up in the results.

pkgsrc / pkgin / libnbcompat

I started using pkgsrc and pkgin on my work laptop + desktop as a replacement for homebrew and installing ancient RHEL packages. I found some features I'd like to add, but before I was able to do that, I found two issues...

https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=51424

libnbcompat did not properly guard the definition of their strdup replacement, so that if you had a project that tested for strdup presence, the libnbcompat declaration would shadow the system one. Worse, if strdup is a macro (like on Fedora amd64), then compilation blows up.

https://github.com/NetBSDfr/pkgin/pull/72/commits/c111a49f248d9f972c34bdc1e89f44c57c59a651

In pkgin, I found a small misstep that made compilation fail if strsep was defined as a macro. In this case, since there was no check for the function definition, HAVE_STRSEP would be undefined, causing libnbcompat's version to get picked up. Like in the last case, if strsep was a macro, compilation would blow up.

[worrdev]

Calibre Wizard

It's been awhile since I've actually worked on any FOSS, which is a damn shame. Things at work have been in a bit of an upheaval for me as I'm about to switch teams, and on top of that, I recently moved to a new apartment in SF.

I finally subscribed to the excellent Model View Culture quarterly, and wanted to put the first issue of the year on my Kindle, so I started setting up Calibre on my linux desktop.

The Kindle integration in Calibre is a bit interesting - rather than treating the Kindle as a USB mass storage device, they opt for emailing content to the email address associated with your Kindle. To make this setup easy, they have a wizard to help you set this up the first time you start the program.

Well, when I was setting up Calibre, I made the mistake of entering the wrong port for my mail server. While waiting for the test request to time out, I noticed that the entire wizard UI was blocked on sending the email.

I started digging, and found that Calibre has a job framework for starting up new threads that don't block the UI. Unfortunately, this is heavily tied to the main Calibre process, which isn't running at the time of the wizard invocation. Welp.

I bitched about it to my new, infinitely-patient roommate, and while talking to her, remembered Qt signals/slots. The Qt framework, which Calibre makes heavy use of, has a convenient mechanism for inter-object communication: signals and slots.

Signals are more or less what they sound like (although aren't the same as POSIX signals). They define messages that you can send to a receiving slot. Slots receive and act on signals. As long as a slot accepts the correct type of signal, you can hook them up any which way. They're super convenient. My particular usecase was having the email testing thread update the GUI thread when the test had completed.

All-in-all, I came up with this super simple patch that I've submitted for review: https://github.com/worr/calibre/commit/24bf2685b527170fae1de04ca677a42ff224baf1

However, the maintainer scrapped this, and re-worked that dialog entirely obviating my patch.

[worrdev]

Upcoming Unicode blog?

I've been feeling pretty burnt out on tech projects atm, which is why my project blog has been, well, quiet. I've been taking it easy and trying to enjoy the free time I have around trying to get my shit together to move and staying on top of my job.

That said, I have recently started putting together code samples and such for a new blog post about one of my new favorite topics - Unicode! I'm not quite ready to show off the code samples/testcases just yet, but I am at least getting that together.

To kind of sum up my goal with this blog, I'm going to putting together a blogpost about dealing with Unicode in C, python2/3, rustlang and golang. I'm going to be writing the bulk of the code in C, using that to write about a lot of the gotchas when dealing with Unicode, and then use the other languages as examples of how things have changed, as well as an indication as to where the Unicode-handling landscape is now. I'd like to touch on problems like dealing with string length, character classes, character composition, and normalization? I'm worried it might get a bit long, so I might split it into two parts.

If you have suggestions, questions or requests for anything specific to talk about, let me know via whichever way is easiest for you to get in touch with me.

[worrdev]

bsdinstall(8) troubles

rust xattr

I made a blog post a few days ago on my main blog about extended attributes. This reminded me that my rust patch is still in limbo, so I pinged the author, and he was nice enough to take a look and provide fantastic feedback.

https://github.com/Stebalien/xattr/pull/3/

bsdinstall

After working on fixing up my patches, I was in the mood to try and figure out why loading echo.ko was causing kernel panics. I started setting up a new FreeBSD instance in bhyve(8) to figure this one out.

Installing FreeBSD proved to be a problem.

This is the first time that I’m using the new bsdinstall over sysinstall, and I had a few problems actually make it to a successfully installed FreeBSD system.

The major issue that I encountered is that some components would either drop me back to the beginning of the installer, or drop me back to the beginning of that component. For example, partedit errors would drop me back to partedit, and zfsboot errors would restart the installation.

What made going through the installation over and over again extra frustrating, was that netconfig_ipv4 would error if you tried to configure an interface with a running dhclient. I submitted a report and simple patch to kill any dhclients if you’re configuring your installation interface to use dhcp.

Another bug that I found while proceeding through the install was that modifying a ZFS partition and just setting the mountpoint would cause partedit to segfault. Bug report and patch.

While I was looking through all this, I happened to find a potential issue that would hang the installer. When writing the bootcode to disk, partedit reads the file in a loop without checking for read(2) errors. If read(2) fails once for whatever reason, this will hang this installer, as that loop will never terminate. There was also an unchecked malloc(3) that took the result of an unchecked lseek(2) as well. I replaced all of that with an mmap(2)..

[worrdev]

CFEngine on bad variable expansions - part 3

This title is misleading, since I did more than this. But whatever, it's part of a series, I guess.

First, the things that have nothing to do with bad variable exapnsions, in increasing relevancy.

twitter-trollers

I had a shower thought, and decided to create a quicky Twitter bot that would watch a stream for polls, and vote on random results. I do this manually for fun anyway, so I thought it'd be trivial to automate.

Part way through the project, I discovered that Twitter still hasn't exposed a poll API. I'm not wholly surprised, since Twitter has only become more developer-hostile ever since the whole API key revocation scandal years ago.

Code is here.

cookiecutter-pypackage

I discovered a cute little project that can generate language-agnostic project templates. I hate doing scaffolding, so I latched onto this immediately. I forked the author's pypackage template, and stripped out half of the shit I didn't care about, and amended it for my particular python requirements (mypy, only 3.5+).

I'm thinking of putting together a basic C template for some of my simple C projects as well.

My python template is here.

CFEngine PR update

I made fixups to my silent-returnszero patch.

CFEngine testall --verbose

I finally discovered the testall script in CFEngine's tests/acceptance folder. I'll need to amend HACKING.md to include some more detailed information about testall, since it's easy to miss. It's also a pain to develop tests without it.

PR is here.

And finally...

CFEngine now fails promises when variables don't fully expand

It looks like I've almost finished this patch. To be honest, I'm not happy with the implementation just yet. I decided to tackle part of this problem by amending the Rval struct to include a field indicating whether or not that Rval has been fully expanded. That doesn't seem like the best place to put that information, but as an experiment it was the easiest.

After doing that, I was able to run almost the entire test suite without error. This is huge progress, especially since this safeguard will prevent all kinds of CFEngine policy errors.

Commit is here.

Lastly...perl 6

Perl 6 finally came out. Somehow my name got on the list of contributors even though the code I wrote was almost certainly excised a few VMs ago. Neat though.

[worrdev]

CFEngine on bad variable expansions - part 2

Last time

Today, I started taking a stab at actually implementing the logic to start bailing on bad variable expansions (this is not the finished commit message).

I started right off the bat, by taking all of the extant calls to ExpandScalar and starting to guard their return values, since literally no invocation of ExpandScalar checks the success of variable expansion. However, starting off by guarding these expansions didn't actually work - largely because even on unexpanded variables, ExpandScalar was returning success.

It shouldn't surprise anyone that a function whose return value was never checked returned bogus results in some cases.

After fixing that bug (and running all the tests), the guards suddenly work, and I got this delightful message from my test-case:

error: Could not expand variable '$(bar)'.

My super simple test-case:

bundle agent main { files: "/foo" copy_from => cp("$(bar)"); } body copy_from cp(source) { source => "$(source)"; }

This was already a massive success. I continued guarding, running tests, and I got the commit linked above, with no test failures. I haven't finished just yet, but it's a great start.

One problem I will have soon though, will be guarding variable expansions in rvalues in variable assignments. Currently, ExpandPrivateRval doesn't seem to have any mechanism for reporting error or referencing the calling promise, so it makes it difficult to report error on bad vars there.

Also this weekend, I made a quick PR for the work I've done on replacing select(2) with poll(2) in CFEngine. I added a poll(2) implementation for Windows from LibreSSL, and have been replacing select(2) calls where I can.

[worrdev]

CFEngine on bad variable expansions

A massive problem that I've had at work, is that we sometimes lay down templates, or run commands, or name files after unexpanded CFEngine variables. Currently, CFEngine doesn't have any built-in defense for handling this, and we've built our own series of hacks to deal with this - in most cases.

Today I started work on a patch to CFEngine that will detect this probablem and fail to keep the promise if this happens. After some discussion on the mailing list, we decided that this feature needs to be opt-in for now, and possibly add per-promise options to toggle it. While I'm normally against such dials, this drastic of a change in behavior would break too many installations to turn it on all at once.

Since I had a relatively busy evening, I only implemented the global option, and haven't added any support for failing in this case yet. I'll hopefully get to that later this week.

https://github.com/worr/core/commit/2bbef753f8645e4c23595dc6e2691d2aa1e3e17a

[worrdev]

CFEngine’s returnzero() and FreeBSD’s pkg(8)

I had a very quiet and uneventful break from work the last couple of days, and I spent it doing pretty much nothing technical. It was actually surprisingly relaxing - but I'm back on the horse now.

I spent the last hour or so doing some veryh simple work. On CFEngine, I updated my PR to silence the returnszero function. returnszero is a function that executes a command, and returns a boolean indicating the success of that command (where 0 == success). It's unforutnately noisy, in that the command is run using the parent's stdout and stderr, polluting the output of cf-agent. While my patch had been inflight for awhile, we determined that it would be a better idea to expose this output on higher verbosity levels (verbose and info). Honestly, that makes more sense than my original proposal, so I quickly updated my PR.

I think I submitted one of the dumbest changes I've ever proposed... pkg(8) sometimes prefixes print statements with the name of the jail that pkg is currently operating on. I find this invaluable, so much so that I finally submitted a PR with my changes to prepend the jailname to more output. I've been running this for a few weeks, and it's made my life far easier, since I typically run pkg on all of my jails.

[worrdev]

cf-lololol and Rust’s mkstemp

The other night, I dreamt about a kind of an interesting idea. It'd be super convenient to build a tool that used configuration management to continuously produce container images. In this case, organizations that maintain config management repos for non-containerized systems can use their existing codebase and knowledge to produce new images, and to continuously generate those base images as policy changes over time.

This started as a kind of joke project, until my roommate mentioned that this wasn't actually a bad idea.

I started working on a PoC yesterday, using CFEngine and Docker. I didn't get super far, since we had company over that evening, but I did make a surprising amount of progress. Since I need to call into CFEngine's libpromises, I ended up setting up a lot of the same scaffolding that I've used in my rust kernel modules, which I really need to look at automating. Once I finish my PoC, I'll turn it into something that's actually...good. :)

I also was somehow at a loss as to the best way to create tempfiles in Rust. There didn't seem to be anything in the stdlib to do so, and I couldn't find anything in Cargo. This, obviously, led me to quickly write up some quickie bindings to mkstemp(3) and then publish it on crates.io. If there's a better way to do this rather than writing my own crate, let me know.

Both projects, while in their infancy, are great examples of how trivial interop with C is in Rust. It's been an amazing language to work with so far.

Edit: Turns out I was too drunk to find this. I looked through the source, and this appears to be a pure-rust implementation of mkstemp and mkdtemp, with some extra features. That's pretty great.

[worrdev]

Spinning up a new puppet master

Today was a pretty quiet day. I recently started using a new VPS service called Vultr, as it's one of the few that meet the my requirements:

OpenBSD support

Native IPv6

The cost for what I'm getting is much better than my old provider, ARP Networks, and it also allows me to drop VPSes in different parts of the world. I know these are terribly silly things to get excited about if you live in Linux VPS land, but it's tough finding such a feature-rich provider that supports OpenBSD.

A day or two ago, I spun up a new mail server, to replace my old baremetal mail server sitting on CSH. Today, I took all of the manual steps that I needed to perform (setting up login.conf, cron jobs, certs, etc.) and dropped them into puppet. Most of my mail setup was there already, but this last push made mail deployment fully automated.

Once that was done, I also dropped my puppet master configs into puppet. The VPS I was spinning up today is going to be my replacement puppet master, so I figured I would take the manual steps out of setting that up as well.

Once I got OpenBSD installed on the new VPS, bootstrapping was a simple puppet agent --test --server <old puppetmaster> away.

[worrdev]

New rcstrings

rcstring

After I got the rcstrings crate submitted to crates.io the other day, I decided to show it off to /r/rust. It's amazing how newcomers are encouraged to get involved early, ask questions and are encouraged to show off their first crates to the rust community! The community has been immensely helpful, and pointed out a number of problems with my code.

To make my CString struct easy to use, I used to do some translation to *mut i8. This threw away lifetimes, which meant that my struct could possibly outlive the data it was pointing to. Oops!

To solve this, I used a neat feature in Rust called PhantomData. It's a zero-size struct that allows me to use generic types or lifetimes in my struct declaration, without actually using them in a useful field. In my case, I really only needed lifetimes, so I ended up with this. Now, unless an API consumer uses my unsafe functions, I can guarantee that my struct won't outlive the data it's pointing to.

echo.ko

I've been working on converting one of the FreeBSD kernel module dev tutorials to Rust to try out ideas for my token ring driver. I've made quite a bit of progress, in that I have build scaffolding that will actually get the module loaded into the kernel, after building it with Cargo.

The major breakthrough for me was when I found out about custom target definitions. This allows you to define the set of options for a custom compiler target in a JSON file. This is super useful to me, since the FreeBSD kernel doesn't handle dynamic relocations...at all. Before, I was handling this by using cargo rustc -- -C relocation-model=static, however this only affects the last artifact produced by the build. The different with custom build targets is that they contain a set of definitions that is used for every artifact produced against that triple.

I can now just specify cargo --target x86_64-kernel-freebsd and get the build definitions from here.