Generating a “Vanity” PGP Key ID Signature

Here’s a quick bash script I used to generated a “vanity” PGP key with the last two bytes (four characters) set to FFFF.

#!/usr/bin/env bash

while :
do
gpg --debug-quick-random -q --batch --gen-key << EOF
Key-Type: RSA
Key-Length: 2048
Name-Email: user@domain
Name-Real: Real Name
Passphrase: yourverylongpassphrasegoeshere
EOF

if gpg -q --list-keys | head -4 | tail -c 5 | grep FFFF
then
        echo Break
        exit 1
else
        gpg2 --batch -q --yes --delete-secret-and-public-key `gpg -q --list-keys
| head -4 | tail -n 1`
fi

done

I also added no-secmem-warning to ~/.gnupg/options to suppress the insecure memory warnings. When I set it to a 1024-bit key, it only took about 3 hours, while 2048-bit took 20 hours across.

It goes without saying, my use of insecure randomness is a terrible idea for those facing a serious threat model. Also, you’re basically picking a number at random out of 65,535 hoping for the right combination – but I’m just having fun with it.

Passing by Reference: C’s Garbage Collection

The C programming language has no built-in garbage-collection mechanism – and it very likely never will. This can (and does) lead to memory leaks by even the best programmers. It is also an imputes for the Rust language. However, depending on your use-case, it is still possible to structure your code to use the stack as a sort of zero-cost “garbage collector”.

Lets jump directly into the code!

This is how many applications instantiate and utilize a structure or arbitrary object.

struct resource *instance;
instance = malloc(sizeof(struct resource));
get_resource(instance); ... free(instance);

While this is a perfectly fine snippet of code, it requires the program to explicitly free(3) instance when it is no longer needed or risk a memory leak. There is also a slight performance loss from the malloc(3) and free(3).

Therefore, lately I have been using another method.

struct resource instance;
get_resource(&instance);

Rather than allocating memory, this uses the stack. When the variable is “destroyed” immediately after falling out of scope without the need for a free(3).

The downside, of course, is losing the ability to pass the pointer elsewhere after the initial allocating function closes. But, this can be overcome by creating the variable in the parent function to all those that need it.

Thoughts?

SHA1 on FreeBSD Snippet

I needed some code that produces SHA1 digests for a project I am working on. I hunted through the FreeBSD’s sha1(1) code and produced this minimal snippet. Hopefully this helps someone else in the future.

Compile and run as follows:

$ cc shatest.c -o shatest -lmd
$ ./shatest
10d0b55e0ce96e1ad711adaac266c9200cbc27e4
$ printf "bismillah" | sha1
10d0b55e0ce96e1ad711adaac266c9200cbc27e4

Thanks to FreeBSD for maintaining such clean code!

Including optimized-out kernel symbols in dtrace on FreeBSD

Warning: This is a hack that involves modifying the build scripts. tldr; modify /usr/src/sys/conf/kern.pre.mk to change all references of -O2 to -O0.

Have you ever had dtrace(1) on FreeBSD fail to list a probe that should exist in the kernel? This is because Clang will optimize-out some functions. The result is ctfconvert(1) will not generate debugging symbols that dtrace(1) uses to identify probes. I have a quick solution to getting those probes visible to dtrace(1).

In my case, I was trying to instrument on ieee80211_ioctl_get80211, whose sister function ieee80211_ioctl_set80211 has a dtrace(1) probe in the generic FreeBSD 11 and 12 kernels. Both functions are located in /usr/src/sys/net80211/ieee80211_ioctl.c.

My first attempt was to add to /etc/make.conf as follows and recompile the kernel.

CFLAGS+=-O0 and -fno-inline-functions

This failed to produce the dtrace(1) probe. Several other attempts failed and I was getting inconsistent compilation results (Is it me or is ieee80211_ioctl.c compiled with different flags if NO_CLEAN=1 is set?). When I manually compiled the object file by copying the compilation line for the object file and adding -O0 -fno-inline-functions, nm(1) on both the object file and kernel demonstrated that the symbol was present. I installed the kernel, rebooted and it was listed as a dtrace probe. Great!

But as I continued to debug my WiFi driver (oh yeah, I’m very slowly extending rtwn(4)), I found myself rebuilding the kernel several times and frequently rebooting. Why not do this across the entire kernel?

After hacking around, my solution was to modify the build scripts. My solution was to edit /usr/src/sys/conf/kern.pre.mk and modify all optimization level 2 to optimization level 0. The following is my diff(1) on FreeBSD 12.0-CURRENT.

diff --git a/sys/conf/kern.pre.mk b/sys/conf/kern.pre.mk
index c1bbf0d30bf..9a99f1065aa 100644
--- a/sys/conf/kern.pre.mk
+++ b/sys/conf/kern.pre.mk
@@ -57,14 +57,14 @@ CTFFLAGS+=  -g
.if ${MACHINE_CPUARCH} == "powerpc"
_MINUS_O=      -O      # gcc miscompiles some code at -O2
.else
-_MINUS_O=      -O2
+_MINUS_O=      -O0
.endif
.endif
.if ${MACHINE_CPUARCH} == "amd64"
.if ${COMPILER_TYPE} == "clang"
-COPTFLAGS?=-O2 -pipe
+COPTFLAGS?=-O0 -pipe
.else
-COPTFLAGS?=-O2 -frename-registers -pipe
+COPTFLAGS?=-O0 -frename-registers -pipe
.endif
.else
COPTFLAGS?=${_MINUS_O} -pipe

My dtrace -l | wc -l went from 71432 probes to 91420 probes.

A few thoughts:

  • This seems like a hack rather than a long-term solution. Either the problem is with the hard-coded optimization flags, or the inability to overwrite them in all places in make.conf.
  • Removing optimizations is only something I would do in a non-production kernel, so its as if I have to choose between optimizations for a production kernel or having dtrace probes. But dtrace explicitly markets itself as not impactful on production.
  • Using the dtrace pony as your featured image on WordPress does not render properly and must be rotated and modified. Blame Bryan Cantrill.

If you have a better solution, please let me know and I will update the article, but this works for me!

Linux maintains bugs: The real reason ifconfig on Linux is deprecated

In my third installment of FreeBSD vs Linux, I will discuss underlying reasons for why Linux moved away from ifconfig(8) to ip(8).

In the past, when people said, “Linux is a kernel, not an operating system”, I knew that was true but I always thought it was a rather pedantic criticism. Of course no one runs just the Linux kernel, you run a distribution of Linux. But after reviewing userland code, I understand the significant drawbacks to developing “just a kernel” in isolation from the rest of the system.

Lets say a userland program wants to request an object from the kernel. The kernel structure might be something like this:

struct foo {
     size_t size;
     char name[20];
     int val;
};

On POSIX systems, a typical way to communicate with the kernel is to open a file descriptor to the appropriate system and send an ioctl(1) with a pointer to where the kernel should store the responding data. FreeBSD might perform this task as follows:

struct foo x;
ioctl(fd, CMD_REQUEST_FOO, &x);

Linux should do the same and to be fair it typically does. This manifests as software source that requires the Linux kernels headers. But because userland tools are maintained independent of the kernel, and sometimes are even explicitly written to be cross-platform, they typically maintain their own copy of data structures and macros independent of the Linux source tree.

So far so good. This might even produce the exact same binary output. But what happens if the kernel structure or behavior changes? This could be due to a bug fix, an added feature or an optimization – either way, the structure may change.

On FreeBSD this is not a problem. They update the kernel and userland tools in tandem. In fact, because both the kernel and userland application are in the same source tree they can even share the same header files. For 3rd party userland applications, FreeBSD provides highly stable libraries that do all the kernel-interactions, such as lib80211(3) – its worth noting that OpenBSD and NetBSD do not have these libraries because the kernel interface itself is highly stable anyways. FreeBSD even provides a COMPAT layer in the rare cases that an older binary fails to run on modern versions of FreeBSD.

Conversely on Linux, because the kernel and the rest of the operating system are not developed in tandem, this means updating or fixing a kernel struct would almost guarantee to break a downstream application. The only to prevent this would be to conduct regular massively coordinated updates to system utilities when the kernel changes, and properly version applications for specific kernel releases. Quite a herculean endeavor. This also explains why systemtap, one of Linux’s many answers to dtrace(1), does not work on Ubuntu.

Also, Linux can never have an equivalent of a lib80211(3) because there is no single standard library set. Even for the standard C library set, Linux has Glibc, uClibC, Dietlibc, Bionic and Musl. Rather than guessing the underlying C library implementation or falling into “dependency hell“, applications default to the most low-level implementation or their requested functionality. Some tools, such as ifconfig(8), resort to just reading from the /proc filesystem.

Linux’s solution to this problem was to create a policy of never breaking userland applications. This means userland interfaces to the Linux kernel never change under any circumstances, even if they malfunction and have known bugs. That is worth reiterating. Linux maintains known bugs – and actively refuses to fix them. In fact, if you attempt to fix them, Linus will curse at you, as manifest by this email.

And this leads back to the topic. Have you ever wondered why nearly every distribution deprecated ifconfig(8), a standard networking tool dating back to classic Unix? When Linux first implemented multiple IPv4 addresses on the same physical interface, it did so by cloning the interface in software and assigning each clone a unique IPv4 address. For example, eth0 could be cloned with eth0:1, eth0:2, etc. From a programmatic perspective, eth0 still only had one IPv4 address. As time passed and developers updated the kernel, it allowed users to assign multiple IPv4 addresses directly to the same interface., bypassing the need for cloning.

But Linux’s API has not changed. It still only returns a single legacy IPv4 address per interface. An interface could have multiple IPv4 addresses but ifconfig(8) will still only report a single address. In other words, as it currently stands ifconfig(8) lies to you. I do not fully understand they did not just update ifconfig(8) – random IRC rumors say there was a failed attempt due to ifconfig(8)’s convoluted code-base. But for whatever reason, this led to the completely new tool ip(8).

By contrast, FreeBSD just updates their ifconfig(8) in tandem with any kernel updates and there were no problems. Simple.

This also explains why Linux has multiple tools for seemingly highly correlated network tasks. Rather than working together to create a consolidate tool, Linux has iw(8), iwconfig(8) and brctl(8), etc, whereas FreeBSD just has different drivers for its ifconfig(8) implementation. For the record, I think ip(8)’s syntax is cleaner than ifconfig(8)’s syntax, as the latter is a victim of IPv4 legacy syntax. If both tools worked just fine, it might be worth having ifconfig(8) for legacy scripts during a transitionary period, but making ip(8) the future. That would be perfectly fine, but it would be ideal if both tools just worked, rather than needing to abandon the tool because it is broken.

Written with love a laptop running OpenBSD 6.3.

Thoughts?

My Backup Solution Leveraging OpenZFS, rsync, WOL and crontab

The other day I managed to destroy my hard drive’s partition table as I attempted to fix a grub(2) issue. To make matters worse, my backup of important files was old, and while attempting to make a boot-disk to re-install Linux, I selected the wrong disk and over-wrote the backup! Clearly I needed a more robust backup solution.

My requirements for the solution were as follows:

  • Runs during idle time, in my case 5:00 am and 5:00 pm.
  • Does not require me to leave my computer on 24/7. I try to reduce my power consumption.
  • Does not require non-standard (read: non-open source) software.

My setup is as follows:

  • Linux Mint desktop. Hate on me if you want, I love the Cinnamon desktop.
  • FreeBSD server, which serves as the ZFS backup sink (and a lot more).
  • A small Linux (soon-to-be NetBSD) Raspberry Pi 1 that functions as my IPv6 router, VPN endpoint and jump host when I need to tunnel into my home network from the outside.

Lets begin!

At 4:58 AM, the Raspberry Pi’s crontab(1) runs a script to ping my desktop. If the ping is successful, it notes that the computer is online by writing 0 to /var/log/pc-status. Otherwise, it notes that the computer is off-line, writes 1 to /var/log/pc-status and sends a Wake-On-LAN (WOL) frame to my computer to power on the machine. The script is as follows:

#!/bin/sh
ping6 -w 0.5 -c 1 -q PC_IP6_ADDRESS 2> /dev/null > /dev/null
if [ $? -ne 0 ]; then
     # It was off, so write 1
     sudo wakeonlan PC_MAC_ADDRESS 2> /dev/null > /dev/null
     sudo sh -c 'echo 1 > /var/log/pc-status'
else
     # Its already on, so write 0
     sudo sh -c 'echo 0 > /var/log/pc-status'
fi

I put this in the crontab(1) as follows: 58 4 * * * /home/farhan/bin/wake-script.sh.

Next, at 5:00 AM, the Linux desktop performs an rsync(1) to the FreeBSD backup machine to synchronize all files. Since this is rsync(1) and not scp, it does not waste time on files that are already up to date. Upon completion, the Linux desktop queries the Raspberry Pi’s power-status record to determine if it was just on or off. If the value is on, Linux will do nothing. If the value is off, it will suspend the machine. This is done as follows in another script:

rsync -6 -q -a --exclude "VirtualBox VMs" --include ".*" /home/farhan/* farhan@FREEBSD_SERVER:/usr/local/home/farhan/pc_home/
ssh farhan@RASPBERRY_PI cat /var/log/pc-status | grep 1 -q
if [ $? -ne 0 ]; then
     sudo pm-suspend 2> /dev/null > /dev/null # The redirects are unnecessary.
fi

My storage device is 4 TB and mostly unused, so I do not even bother with the --delete option. Yes, my ~/Downloads directory will likely grow quite large in the next few weeks, but that is not a problem. However, I excluded the Virtual Machines’ directory because even just powering on a VM results in changing the VDI disk image. And similarly to the wake-script, this is in crontab(1) as follows: 0 5 * * * /home/farhan/bin/sleep-script.sh.

Finally, on the FreeBSD side I run daily OpenZFS snapshots and daily prunes of snapshots that are older than 14 days. My crontab(1) is as follows:

@daily /sbin/zfs snapshot -r Data@daily-`date "+\%Y-\%m-\%d"`
@daily /sbin/zfs list -t snapshot -o name | /usr/bin/grep tank/home/farhan/pcbackup@daily- | /usr/bin/sort -r | /usr/bin/tail -n +14 | /usr/bin/xargs -n 1 /sbin/zfs destroy -r

And that’s it! A low-powered, open-source backup solution that relies 100% on Unix tools.

Notes: The only way to make this solution more elegant would be if the Linux desktop ran OpenZFS and used zfs(8) send command. But given that Linux Mint does not support ZFS out of the box, I am concerned what might happen if the OpenZFS module fails to load and I am stuck with a non-functional machine. Also, notice the explicit use of IPv6, not legacy IP.

fsync(2) on FreeBSD vs Linux

Even with our modern technology, hard-disk operations tend to be the slowest and most painful part of any modern system. As such, modern operations implement buffering mechanism. In this schema, when an application calls write(2), rather than immediately performing physical disk operations, the operating stores data in a kernel buffer. When the buffer exceeds a certain amount or the when an application falls the fsync(2) system call, the kernel begins writing to the disk.

This scheme is significantly faster, perhaps most demonstrably by the massive performance differential between the GNU vs BSD yes(1), as initially noted here. Note: FreeBSD’s yes(2) has now reached parity with GNU.

So far so good. But what happens when a disk write operation fails? This could be due to a hardware or network failure, but ultimately it is not the fault of the operating system. However, the operating must properly handle the failure.

On Linux, when an application’s fsync(2) call fails, the kernel returns a disk error. However, it then clears the buffer and properly sets the buffer as “dirty” (EIO flag). When the application issues another fsync(2) and the disk succeeds, the kernel clears the error bit, and reports a successful write to the application. As such the previously failed data never hit the disk and, if discarded by the application, the data was lost.

On FreeBSD, when an application’s fsync(2) call fails, the kernel also returns an error. Similar to Linux, it also reports the error to the application. But unlike with Linux, it maintains the “dirty” bit, thus not re-writing over the kernel buffer, until the page buffer is cleared, even if the successive fsync(2) is successful. This way, the page data is not lost.

This is another example of the superiority of FreeBSD over Linux. FreeBSD can better survive a disk failure, while Linux’s implementation is fundamentally broken. In the past I have experienced Linux’s ext4 fail into read-only mode to prevent disk corruption. While that might be a fall-back mechanism, it is not a long-term solution. Instead, userland applications have to keep track of whether the kernel was successful or not. Depending on your perspective, this is a stack violation.

Additionally, any long-term solution to change the behavior of the operating system would mean all user-land applications would potentially break. Linus Torvalds has notoriously stated:

Breaking user programs simply isn't acceptable

In fact, he’s repeated this policy in more colorful language here. So you’re stuck with bad behavior.

Now consider if you want to build an operating system that will run for potentially a hundred years and produce zero errors or catch errors and properly perform exception handling. Go with FreeBSD.

Its worth noting that Illumos (Solaris) properly implements fsync(2), whereas OpenBSD and NetBSD also failed on this issue and I fully anticipate them to fix the problem.

Tracing ifconfig commands from userspace to device driver

I am currently working on expanding FreeBSD’s rtwn(4) wireless device driver. I have the basics down, such as initialization, powering on and off, loading the firmware, etc, and am now trying to fill in specific ifconfig(8) methods. This requires having an in depth knowledge of how ifconfig(8) commands pass are ultimately delivered to the driver. I could not find concise documentation that outlines each stage of the process. So I wrote one! 🙂

This article is specific to FreeBSD 12.0-CURRENT, but it should apply to any future version and other operating systems that utilizes net80211(4), such as OpenBSD, NetBSD, DragonFlyBSD and illumos. I hope it serves to help the FreeBSD community continue to develop WiFi and other device drivers.  This is not an exhaustive guide as there is far too many code, but it should provide you with the basic order of operations.

In this example, I will walk through changing the channel on your WiFi card and placing it in monitor mode as follows:

# ifconfig wlan0 channel 6 wlanmode monitor

High Level Summary

FreeBSD’s ifconfig(8) utilizes the lib80211(3) userspace library which functions as an API to populate kernel data structures and issue ioctl(2) syscall. The kernel receives the ioctl(2) syscall in a new thread, interprets the structure and routes the command to the appropriate stack. In our case this is net80211(4). The kernel then creates a new queued task and terminates the thread. Later on, a different kernel thread receives the queued task and runs the associated net80211(4) handler which immediately delivers execution to the device driver.

To summarize again:

Lets begin!

Userspace: ifconfig(8) + lib80211(3) library

Starting: ifconfig(8) executable

Startnig early in ifconfig(8), it opens a SOCK_DGRAM socket in /usr/src/sbin/ifconfig/ifconfig.c as follows:

s = socket(AF_LOCAL, SOCK_DGRAM, 0)

This socket functions as the interface for userspace to kernel communication. Rather than tracing from the if-else maze in main()1, I grepped for the string “channel” and found it in ieee80211_cmd[] defined at the end of /usr/src/sbin/ifconfig/ifieee80211.c. This table enumerates all ieee80211 ifconfig(8) commands. The “channel” command is defined as follows:

DEF_CMD_ARG("channel", set80211channel)

Note the second argument. I looked up DEF_CMD_ARG and found that it was a pre-processor macro that defines what function is run when the user sends ifconfig(8) a command. A quick grep search shows set80211channel is defined in /usr/src/sbin/ifconfig/ifieee80211.c. The parameters are fairly easy to identify: val is the new channel number (1 through 14) and s is the socket we opened earlier. This executes ifconfig(8)‘s set80211 function whose sole purpose is to cleanly transfer execution into the lib80211(3) library.

Userspace: lib80211(3) library

lib80211(3) is an 802.11 wireless network management library to formally communicate with the kernel. Its worth noting that neither OpenBSD nor NetBSD have this library and instead opt to communicate directly to the kernel.

As mentioned, ifconfig(8)‘s set80211 function calls lib80211_set80211, located in /usr/src/lib/lib80211/lib80211_ioctl.c. The lib80211_set80211 function populates an ieee80211req data structure, used for user-to-kernel ieee80211 communication. In the below example, this is the ireq variable, which contains the WiFi interface name and intended channel. The library then calls the ioctl(2), as follows:

ioctl(s, SIOCS80211, &ireq)

This runs the syscall to formally enter kernel-space execution. In essence, ifconfig(8) is nothing more than a fancy ioctl(2) controller. You could write your own interface configuration tool that directly calls the ioctl(2) syscall and get the same result. Now on to the kernel!

The Kernel: Kernel Command Routing to net80211(4)

There are two brief explanations before we proceed.

First, at a high-level the BSD kernel operates like an IP router in that it routes execution through the kernel, populating relevant data values along the way, until the execution reaches its destination handling functions. The following explanation shows how the kernel will identify the syscall type, determine that it is for an interface card, determine the type of interface card and finally queue a task for future execution.

Second, the BSD kernel utilizes a common pattern of using template methods that call a series of function pointers. The exact function pointers are conditionally populated, allowing the code to maintain a consistent structure while the exact implementation may differ. It works very well but can make tracing execution paths difficult if you are just reading the code straight through. When I had trouble, I typically used illumos’s OpenGrok or dtrace(1) .

Brief Dtrace Detour

Solaris’s dtrace(1) is a dynamic tracing tool imported to FreeBSD that is used to monitor a kernel or process in real time. It is useful in understanding what the operating system is doing and saves you the trouble of using printf(3)-style debugging. I used dtrace(1) in writing this guide identify what the kernel was executing, function arguments, and the stack trace at any given moment.

For example, if I wanted to monitor the ifioctl function, I might run this:

# dtrace -n '
> fbt:kernel:ifioctl:entry {
> self->cmd = args[1];
> stack(10);
> }
> fbt:kernel:ifioctl:return {
> printf("ifioctl(cmd=%x) = %x", self->cmd, arg1);
> exit(0);
> } '

This dtrace(1) one-line command sets up handlers for ifioctl‘s entry and return probes. On entry, dtrace(1) records the value of the 2nd argument cmd, and displays the last 10 elements of the stack. On return, it displays the function argument and return value. I used variations of this basic command template throughout my research, especially when I was confused in tracing the code or could not identify a function’s arguments.

Syscall Interception

The first non-assembly function is the amd64-specific syscall handler amd64_syscall that receives a new thread structure and identifies the type as a syscall. In our case it is for an ioctl(2) so amd64_syscall calls sys_ioctl located in /usr/src/sys/kern/sys_generic.c.

On FreeBSD sys_ioctl performs input validation and formats the data it receives. It then calls kern_ioctl which determines what type of file descriptor the ioctl(2) is working with, what the capabilities for the socket are and assigns the function pointer fo_ioctl accordingly. (NetBSD and OpenBSD do not have kern_ioctl. For them sys_ioctl directly calls fo_ioctl.) Our file descriptor corresponds to an interface, so FreeBSD assigns fo_ioctl as a function pointer to ifioctl, which handles interface-layer ioctl(2) calls. This function is located in /usr/src/sys/net/if.c.

Network IOCTL

The function ifioctl is responsible for all sorts of interfaces: Ethernet, WiFi, epair(4), etc. ifioctl starts with a switch-condition based on the cmd argument. This checks if the command can be handled by net80211(4) without needing to jump into the driver, such as creating a clone interface or updating the MTU. A quick dtrace(2) probe reveals that the cmd argument is SIOCS80211, which fails to meet any switch-conditions, so execution jumps to the bottom. The function continues and calls ifp->if_ioctl, which in the case of WiFi is a function pointer to ieee80211_ioctl, located in /usr/src/sys/net80211/ieee80211_ioctl.c.

WiFi IOCTL

ieee80211_ioctl contains another switch-case. With cmd set to SIOCS80211, execution matches the associated case and calls ieee80211_ioctl_set80211, located in /usr/src/sys/net80211/ieee80211_ioctl.c.

ieee80211_ioctl_set80211 has yet another switch-case with a few dozen conditions2. The ireq->i_type was set to IEEE80211_IOC_CHANNEL by lib80211(3) so it will match the associated case and execute ieee80211_ioctl_setchannel. The gist of this function is to determine if the input channel is valid or if the kernel needs to set any other values. It concludes by calling setcurchan, which does two things. First, it determines the validity of the channel and if any additional values must be set. Second, it runs ieee80211_runtask, that makes the final thread-level call to taskqueue_enqueue.

The Kernel: Task Execution

taskqueue_enqueue is not an ieee80211(9) function, but its worth a brief review. In a nutshell, the taskqueue(9) framework allows you to defer code execution into the future. For example, if you want to delay execution for 3 seconds, running the kernel equivalent of sleep(3) would cause the entire CPU core to halt for 3 seconds. This is unacceptable. Instead, taskqueue(9) allows you specify a function that the kernel will execute at a later time.

In our channel change example, the scheduled function is the net80211(4) function update_channel, located in /usr/src/sys/net80211/ieee80211_proto.c. When taskqueue(9) reaches our enqueued task, it will first initiate the update_channel handler to receive the task and immediately hand over execution to the driver code pointed to by ic_set_channel.

To summarize, up to this point the kernel has routed the command to the network stack, which routed to the WiFi-specific stack, where it was scheduled as a task for future execution. When taskqueue(9) reaches the task, it immediately jumps to the driver-specific code. At last, we entered the driver!

The Driver

From here on, the code is driver-specific and I will not get into the implementation details, as each device has its own unique channel changing process. I am currently working on rtwn(9), which is located in /usr/src/sys/dev/rtwn. NetBSD and OpenBSD separate USB and PCI drivers, so the same driver is located in /usr/src/sys/dev/usb/if_urtwn.c and /usr/src/sys/dev/pci/if_rtwn.c, respectively.

Operating Systems need a standard way to communicate with device drivers. Typically, the driver provides a structure containing a series of function pointers to driver-specific code and the kernel uses this as an entry-point into the driver code. In the case of WiFi, this structure is ieee80211com, located in /usr/src/sys/net80211/ieee80211_var.h. By convention, all BSD-derived systems use the variable name ic to handle ieee80211(9) methods.

In our case, we are changing the channel, so the operating system will call ic->ic_set_channel, which is a pointer to the driver’s channel changing function. For rtwn(9), this is rtwn_set_channel, which itself is a function pointer to r92c_set_chanr92e_set_chan or r12a_set_chan, depending on which specific device you are using.

The specifics of rtwn(9) are outside of the scope of this article, but it is worth discussing how the driver communicates to the hardware.

The softc structure is a struct that maintains the device’s run-time variables, states, and method implementations. By convention, each driver’s softc instance is called sc. You might wonder why you need yet another method function pointer when ieee80211com provides that. This is because ieee80211com‘s methods point to command handlers, not necessarily to device routines. A device drivers may have their own internal methods that are not part of ieee80211com. Also, the softc structure can handle minor variations between device versions. rtwn(9)‘s softc struct is called rtwn_softc and located in /usr/src/sys/dev/rtwn/if_rtwnvar.h.

How does a driver send data to the driver? rtwn(9) uses the rtwn_write_[1|2|4] and rtwn_read_[1|2|4] methods to actually send or receive a byte, word or double-word3. rtwn_read_1 is a pointer to the sc_read_1 method.

The driver assigns the sc_read class of functions at initialization to either the rtwn_usb_read_* and rtwn_usb_write_* methods or rtwn_pci_read_* and rtwn_pci_write_*. The aforementioned class of functions are abstractions to the PCI and USB buses. In the case of PCI, these function calls will eventually call bus_space_read_* and bus_space_write_*, which are part of the PCI subsystem. In the case of USB, the driver will call usbd_do_request_flags, which is part of the USB subsystem. A well-written driver should abstract these bus-specific layers and provide you with clean read and write methods for various data sizes. As an aside, FreeBSD is long overdue for an SDIO stack and this is a major impediment for the Raspberry Pi, Chromebooks and other embedded devices. But I digress…

As an example, the driver uses the following line to enable hardware interrupts.

rtwn_write_4(sc, R92C_HIMR, R92C_INT_ENABLE);

This will write the value R92C_INT_ENABLE to the R92C_HIMR device register.

The End

To summarize this long journey, the ifconfig(8) opens a socket and passes it to the lib80211(3) library. lib80211(3) sends a userspace-to-kernel command structure to the kernel with an ioctl(2) syscall. The syscall triggers the kernel to run a new kernel thread. From here, the kernel determines that theioctl(2) command corresponds to a network card, specifies the type as a WiFi card, then identifies the exact command type. The ieee80211(9) tells taskqueue to create a new task to change the WiFi channel, then terminates. Later on, the taskqueue(9) runs the ieee80211(9) task handler that transfers execution to the driver. The driver communicates to the hardware using the PCI or USB buses to change the WiFi channel.

In conclusion, in my opinion, FreeBSD is technically superior to Linux, but lacks in several critical areas, among which is hardware support. I hope this article serves the FreeBSD community to continue to produce high-quality, faster device drivers.

Thank you


Notes

  1. Linux has a point when they argue that the classic ifconfig(8) is antiquated. Its syntax is inconsistent and this is reflected in the spaghetti-code of if-then conditions.
  2. Note: on my FreeBSD 11.1-RELEASE kernel this function was optimized out, so dtrace(1) probes failed. You should be able to add CFLAGS= -O0 -fno-inline to your /etc/make.conf, but that did not seem disable the optimization for me. Your mileage may vary.
  3. Lets use rtwn_read_1 for now, but the concepts apply to the others.

[This article was also published in the January/February 2018 edition of the FreeBSD Journal]

DNS over TLS: A Brief Analysis

The following is a quick write-up I presented to my senior leadership regarding DNS over TLS. It was rooted in the mistake presumption that Google was going to “enforce DNS-over-TLS”. In short. Interestingly, this system is currently in use by Android, but I do not believe this will ever attain mainstream adaptation.

High Level Summary

DNS over TLS is a 2016 protocol that allows clients to resolve a hostname over the HTTPS (TLS) protocol. The client will issue a GET request specifying the hostname and request type and the server will respond with the requested data in JSON. All requests are over TCP/853.

The implementation of DNS over TLS is in the user-space resolution libraries and should not may even be unnoticed by user-land application. From a security perspective, this has a noticeable but easily manageable security impact around TLS security and allowing traffic over port 853.

Except for Android, currently no major operating system natively supports DNS over TLS. I do not foresee this protocol gaining mass implementation nor do I see Google’s public DNS servers mandating it for all clients.

Relation to DNSSEC

DNSSEC and DNS over TLS are parallel features of the DNS protocol. DNSSEC is a DNS protocol extension that provides integrity, but fails to provide confidentiality. As such, a Man-in-the-Middle (MitM) attacker could identify potential endpoints or targets. The DNS over TLS protocol provides both integrity and confidentiality, independent of DNSSEC. Additionally, the DNS over TLS client and server does not communicate over the DNS protocol.

Security Implications

There are three (3) broad security implications from implementing DNS over TLS. These implications are specific to TLS, not DNS over TLS.

FIPS 140-2 Encryption Module

The FIPS 140-2 publication is a recommended standard for encrypted modules. As such, any encryption on government IT systems are subject to this standard. In the case of DNS-over-TLS, this must be through an approved encryption module, most typically OpenSSL.

TLS is not an encryption cipher. TLS is a protocol that provides three aspects of protection: Authentication via a certificate or user certificate, an encryption cipher and hashing mechanism.

Protocol Version

Any implementation of TLS over DNS would have to ensure that the TLS version is free from publicly known or feasible attacks. The current version of TLS is revision 1.2, with TLS 1.3 in draft format. All versions of TLS below 1.1 and all versions of Secure Socket Layer (SSL) are vulnerable to various attacks, namely POODLE and ORACLE.

Certificate Chain

Proper TLS implementations typically utilize a certificate signed by a trusted certificate authorities. In the case of DNS-over-TLS, this requires additional dependence on the certificate authorities for every resolution. This may not be a problem for an end-user who trusts commonly trusted root-level certificate authorities. However, root-level certificates are often subject to distrust or influence from hostile state actors and high-secure environments should not blindly trust the decision of Microsoft or Redhat.

Encryption Cipher and Hashing Mechanism

TLS is a mechanism to facilitate encryption over a network and the hashing algorithm provides data verification. Both encryption ciphers and hashing mechanisms are in slow flux and should be closely followed. For example, in 2015 numerous agencies and security researchers reported that they could compromise RC4 cipher. Google Security researchers reported that they can perform a collision attack against the SHA1 hashing algorithm. Both algorithms were widely implemented in the industry.

Open Connections

A TLS connection initialization is computationally expensive. Therefore, the RFC suggests that the client maintain an indefinite open TCP connection over port 853. This may require an additional firewall rule to the DNS server.

Implementations

The DNS over TLS protocol was formalized in 2016. Due to its relatively young age, currently there are currently very few implementations.

As the RFC documentation specifies, DNS over TLS should be implemented at the host resolution library level, particularly the getaddrinfo(3) and gethostbyname(3) functions. As such, the operating system only needs to maintain library ABI compatibility, but the application does not need implement anything. Currently, only the Android operating system has implemented DNS over TLS while some Linux user-land tools can perform DNS over TLS resolutions.

Google Resolution

Google has currently implements DNS over TLS on 8.8.8.8, 8.8.4.4, 2001:4860:4860::8888 and 2001:4860:4860::8844. Google also offers a web-interface which submits a JSON GET request.

For example, to URL https://dns.google.com/resolve?name=farhan.codes would resolve the hostname farhan.codes. The formatted response is as follows:

Standard Implementation Method

There are several implementations of DNS over TLS encapsulated in simplified Docker containers. In summary, the containers utilize a standard web server to handle the HTTP layer and communicates to the DNS server over the DNS protocol. This is a standard method of isolating the HTTP layer from the application layer.

Future Speculation

I do not believe that the DNS over TLS protocol will attain mass implementation, nor that Google will mandate it for use of their DNS servers. There are three (3) primary reasons why:

  1. Architecture: Historically, short-term add-ons to a protocol are superseded by permanent change to the protocol or a parallel revision. If the goal is confidentiality, this can be achieved via an extension to the protocol.
  2. Performance: Per Google research, DNS is a bottleneck when a URL has multiple external sources. However, I suspect the current DNS resolution is still significantly faster. A UDP-based connection requires only a UDP socket with a simple sendto(2) call, whereas DNS over TLS requires multiple layers of conversion across potentially multiple machines. Specifically, from TLS to HTTP across the internet, converted to the DNS server and back across the same route.
  3. Standardization: There does not appear to be a TLS over DNS standardization. Most implementations utilize HTTP, but this is not specified in the RFC. Additionally, the JSON format differs between the implementations.

Documentation

This paper is based on multiple sources. The primary sources are cited below.

That time I Reverse-Engineered the Motorola CBEP Protocol

This is the tale of how I reverse-engineered a Motorola CPS radio protocol to make it work on Linux. While this may have been of questionable legality and thus lost interest in the project, I learned a lot on how to reverse engineer. I’m writing this entry more than a year after I initially did this, so I may be a little rusty on the details, but this is the gist of it.

My father worked in radio communications so when he passed I inherited his old EOL Motorola XTS 3000. I got an FCC ham radio license and wanted to utilize this device in service of my fledgling new radio hobby. Turns out, this device was the Rolls Royce of radios in its day. It can operate within the ham bands, can do encryption, digital communication, P.25, D-Star, trunking, and pumps out a very clear signal. In short, this was a heavy-duty mission-radio.


Motorola XTS3000 Radio

So I purchased a new $30 lithium ion battery, a programming cable, I ran into a few unfortunate roadblocks. First two…

  1. This device cannot be front-face programmed, which is a fancy way of saying you cannot just set in an arbitrary frequency on the fly. Kinda sucks if you want to change to another random frequency.
  2. This could only be done by a proprietary Motorola CPS (Computer Programming Software) – but this was trivially easy to download.

…All of that was trivial compared to the next bomb-shell…

You could only program this device using a hardware serial port running on native 32-bit Windows. This means no Windows 7/8/10, no Virtual Machines, no Linux, no USB-to-Serial port.

Radio Reference users lamented that they were forced to maintain an old Windows XP laptop with a serial port for programming their device. I personally went out and purchased a $75 computer off Craigslist. Damn!

Up to this point, here are my thoughts: A serial RS-232 port is a “dumb” port compared to modern USB or PCI devices. In fact, serial does not even have a single standard, its whatever the device programmers decide. RS-232 is as close to raw data over a wire as you can get. So why doesn’t this work in a VM? And, why can’t I just capture this data, replay it and have the device function as normal?

Chipping Away at the Secret

I questioned the assumptions and put in an FTDI cross-over cable I made. One end went into the Windows machine, the other end went into My Linux machine, a final serial to radio cable connected to the device. This way my computer was essentially doing a Man-in-the-Middle (MitM) attack, but for serial.


FTDI Cross-over-Cable

I whipped up some C code to relay messages between the Windows machine and device. When I initialized the connection, the radio beeped! And then nothing happened…the software timed out and complained that the connection to the radio failed. I captured the initialization bytes 0x01,0x02,0x01,0x40,0xF7 and replaying them clearly made the radio do something, but immediately stopped afterwards.


Serial Capture Cable

I tried this process several times, but it failed. Annoyed, I looked into purchasing an RS-232 snooping cable: a cable with two regular connections that transfer data as normal and two tap ports, one that taps pins 2 and another that taps pin 3. For whatever reason, well-built cables cost upwards of $100 online and proper hardware snooping devices cost $500, way above my budget. So I decided it was much cheaper to build my own damn cable. I have the program I whipped up to read the bits from the transfer.

And it worked!

I saw a flurry of bits fly across the terminal. In the mix, I noticed a subtle pattern: A pause in the transfer, 7 bytes sent back, echoed back, another pause, and then another flurry of bits. For example, I saw:

0xF5,0x11,0x20,0x00,0x00,0x00,0xD9.

Later on I saw:

0xF5,0x11,0x20,0x00,0x00,0x20,0xB9

and then

0xF5,0x11,0x20,0x00,0x00,0x40,0x99

If you didn’t catch it, the last two bytes are increasing by 0x20 (32) while the last bit ends in 9. (Spoiler, the repeating 9 was coincidental). I interpreted this as a value increasing by 32, and the last byte being a checksum. This was actually a lucky half-guess, because I had no way to know that.

Again I tried to replay these same bits, I ran into the same failure.

I briefly attempted to run the program in IdaPro, GNU Debugger for Windows and Immunity Debugger, but this approach failed and I am still not certain why. For example, I found a string that was clearly only utilized in one place in the binary and set an IdaPro breakpoint when the binary accessed that memory address. But for whatever reason, it did not break. Moreover, I learned the hard-way that the Win32 GUI API controls the application flow and is far from linear, so I could not just break after, say, 30,000 GUI instructions and expect to reliably step into the desired code.

I also briefly tried to use some RS-232 snooping tools, but every tool I found relied on a modern .NET framework not available on Windows XP. Moving on…

Win32 API Monitor

A Windows XP zealot on IRC informed me of a Win32 API monitoring application that would monitor DLL files and produce a list of when they were executed and their arguments. That might be useful.

I spent some time reading up on how Windows communicates over Serial: It uses CreateFileA() to open the COM1 port, typically listed as "\\.\COM1" and then use ReadFileA() call, similar to Unix’s read(2) syscall. I expected to see this in the API capture.

Nope! Instead, I saw that the binary used CreateFileA() against Commsb9. Next, I saw ReadFileA(), sending over 0xF5,0x01,0x02,0x01,0x40 but not the trailing 0xF7. Win32 API even told me driver was communicating to it using USB IOCTLs — not only is this device serial, USB was barely invented when this program was created. What’s going on here?

Reading the code, I identified that these Read/Write commands were taking place in VcomSB96.dll, and filtering by that DLL file, I saw that it was loading Commsb96.sys and Commsbep.sys. In my experience sys files are always driver files.

The Driver

Looks like we are working with a driver. With my limited background in writing a Linux USB driver, Microsoft’s excellent documentation and websites like this, I had an idea of what I needed to hunt for. The C code would look like this:

NTSTATUS DriverEntry( IN PDRIVER_OBJECT pDriverObject,
IN PUNICODE_STRING pRegistryPath )
{
NTSTATUS ntStatus = 0;
UNICODE_STRING deviceNameUnicodeString, deviceSymLinkUnicodeString;
...
pDriverObject->DriverUnload = OnUnload;
pDriverObject->MajorFunction[IRP_MJ_CREATE] = Function_IRP_MJ_CREATE;
pDriverObject->MajorFunction[IRP_MJ_CLOSE] = Function_IRP_MJ_CLOSE;
pDriverObject->MajorFunction[IRP_MJ_DEVICE_CONTROL] = Function_IRP_DEVICE_CONTROL;

...
}

This snippet is assigning the driver methods to the pDriverObject struct. I opened up the Commsb96.sys in IdaPro, expecting to hunt through, starting from the obligatorily exported DriverEntry symbol and trace where the DeviceIoControl initializes. To my surprise I saw this:


IdaPro with Full Symbols

Wow, that was easy. Turns out, a now defunct company called Vireo Software, Inc produced this driver in 1997 and failed to strip out the kernel symbols, making it much easier to reverse. Looks like they used C++ and compiled with optimization. That produced assembly that is a bit difficult to follow, but which I eventually traced back to where the DeviceIoControl messages landed in the kernel, and from there tracked down the USB message that Win32 API Monitor detected.

I finally traced code that read 5 bytes form the stack, and ran them through a loop, calculated a single byte, then returned that byte. I wish I could claim to have written the code below, but it was actually done by another IRC contact. He had a (probably bootleg) professional version of IdaPro that produced the following C code.

unsigned char sbCRC(const unsigned char *msgbuf, const int len) {
     const unsigned char table[8] = {}; // REDACTED FROM THE BLOG TO AVOID LEGAL TROUBLE!

     unsigned char a, b, n = 0;
     int i = 0;

     while (i < len) {
          n = (unsigned char)*(msgbuf + i) ^ n;
          a = ((unsigned char)((signed char)n >> 1) >> 1) ^ n;
          b = a;
          a = ((signed char)a << 1) & 0xF0;
          b = (signed char)b >> 1;
          if (b & 0x80)
               b = ~b;
          n = (a + (b & 0x0F)) ^ table[n & 0x07];
          i++;
     }

     return n;
}

I tested this function against known values such as 0xF5,0x11,0x20,0x00,0x00,0x00 that I cited before, ran it through the function and it resulted in 0xD9. Boom! Checksum reversed!

The Missing Link – RS-232 Flow Control

Coming of age in the 2000s, I learned about effectively full-duplex buses such as USB or PCI. In modern buses, you can effectively asynchronously send data but the RS-232 often requires manual flow-control by the CTS, DSR, RTS and DTR pins.

Providentially, around this time I found the amazing tool 232Analyzer, which was the only tool of its sort that did not require a modern .NET framework. Had I found it earlier, this would have saved me a lot of time! But I learned so much along this process.


232Analyzer Capture

Putting it all Together

With that, I modified my python code to emulate the flow control, calculate the checksum and sent the resulting bytes over. And this time….it worked! I could replay messages from original CPS and the radio responded with the same flutter of meaningless data.

Asking around on forums, IRC and reading up on the Motorola hardware, I learned that these sorts of devices do not request or set specific values from the radio as an API abstraction layer might do. Instead, you request memory locations and interpreting those locations according to a pre-known memory map. I deduced that the 0xF5,0x11 bytes meant “read”, the 0x20 is some sort of divider (or maybe more?), the next 3 bytes are memory locations, and the final byte is a checksum.

Armed with this hypothesis, I found the memory location of the device serial number and my code could read serial. I tested this on my second radio and it resulted it correctly retrieved the serial number.

To find other values, I recorded reading the radio memory, made a single change, then read the radio memory again and compared the difference to isolate values. I was able to find frequency values, channel names, signal strength values, etc! With time, I could mapped out the entire address space! I even found the write-prefix, but was too scared to test it in fears of bricking my radio.

Anti-Climactic Ending

Somewhere along this, I wondered “wait…is this legal?” I contacted the EFF. They were extremely eager to get back to me and after a long conversation the lawyer suspected that because the CPS has a copyright notice and I did not…um…come into acquisition of it through a financial transaction (sure, that phrasing works), it was likely illegal to distribute the reverse-engineered code.

And mapping out the memory got really tedious and annoying. And I started watching The Walking Dead.

And right about here my journey ended.

::cough::

But I learned so much!

  1. How Windows device drivers work
  2. Windows API calls (turns out, they don’t suck)
  3. How to reverse engineer code with IdaPro
  4. How RS-232 traffic flow works
  5. A buncha tools!

Thoughts? Should I have kept going?