DNS over TLS: A Brief Analysis

The following is a quick write-up I presented to my senior leadership regarding DNS over TLS. It was rooted in the mistake presumption that Google was going to “enforce DNS-over-TLS”. In short. Interestingly, this system is currently in use by Android, but I do not believe this will ever attain mainstream adaptation.

High Level Summary

DNS over TLS is a 2016 protocol that allows clients to resolve a hostname over the HTTPS (TLS) protocol. The client will issue a GET request specifying the hostname and request type and the server will respond with the requested data in JSON. All requests are over TCP/853.

The implementation of DNS over TLS is in the user-space resolution libraries and should not may even be unnoticed by user-land application. From a security perspective, this has a noticeable but easily manageable security impact around TLS security and allowing traffic over port 853.

Except for Android, currently no major operating system natively supports DNS over TLS. I do not foresee this protocol gaining mass implementation nor do I see Google’s public DNS servers mandating it for all clients.

Relation to DNSSEC

DNSSEC and DNS over TLS are parallel features of the DNS protocol. DNSSEC is a DNS protocol extension that provides integrity, but fails to provide confidentiality. As such, a Man-in-the-Middle (MitM) attacker could identify potential endpoints or targets. The DNS over TLS protocol provides both integrity and confidentiality, independent of DNSSEC. Additionally, the DNS over TLS client and server does not communicate over the DNS protocol.

Security Implications

There are three (3) broad security implications from implementing DNS over TLS. These implications are specific to TLS, not DNS over TLS.

FIPS 140-2 Encryption Module

The FIPS 140-2 publication is a recommended standard for encrypted modules. As such, any encryption on government IT systems are subject to this standard. In the case of DNS-over-TLS, this must be through an approved encryption module, most typically OpenSSL.

TLS is not an encryption cipher. TLS is a protocol that provides three aspects of protection: Authentication via a certificate or user certificate, an encryption cipher and hashing mechanism.

Protocol Version

Any implementation of TLS over DNS would have to ensure that the TLS version is free from publicly known or feasible attacks. The current version of TLS is revision 1.2, with TLS 1.3 in draft format. All versions of TLS below 1.1 and all versions of Secure Socket Layer (SSL) are vulnerable to various attacks, namely POODLE and ORACLE.

Certificate Chain

Proper TLS implementations typically utilize a certificate signed by a trusted certificate authorities. In the case of DNS-over-TLS, this requires additional dependence on the certificate authorities for every resolution. This may not be a problem for an end-user who trusts commonly trusted root-level certificate authorities. However, root-level certificates are often subject to distrust or influence from hostile state actors and high-secure environments should not blindly trust the decision of Microsoft or Redhat.

Encryption Cipher and Hashing Mechanism

TLS is a mechanism to facilitate encryption over a network and the hashing algorithm provides data verification. Both encryption ciphers and hashing mechanisms are in slow flux and should be closely followed. For example, in 2015 numerous agencies and security researchers reported that they could compromise RC4 cipher. Google Security researchers reported that they can perform a collision attack against the SHA1 hashing algorithm. Both algorithms were widely implemented in the industry.

Open Connections

A TLS connection initialization is computationally expensive. Therefore, the RFC suggests that the client maintain an indefinite open TCP connection over port 853. This may require an additional firewall rule to the DNS server.

Implementations

The DNS over TLS protocol was formalized in 2016. Due to its relatively young age, currently there are currently very few implementations.

As the RFC documentation specifies, DNS over TLS should be implemented at the host resolution library level, particularly the getaddrinfo(3) and gethostbyname(3) functions. As such, the operating system only needs to maintain library ABI compatibility, but the application does not need implement anything. Currently, only the Android operating system has implemented DNS over TLS while some Linux user-land tools can perform DNS over TLS resolutions.

Google Resolution

Google has currently implements DNS over TLS on 8.8.8.8, 8.8.4.4, 2001:4860:4860::8888 and 2001:4860:4860::8844. Google also offers a web-interface which submits a JSON GET request.

For example, to URL https://dns.google.com/resolve?name=farhan.codes would resolve the hostname farhan.codes. The formatted response is as follows:

Standard Implementation Method

There are several implementations of DNS over TLS encapsulated in simplified Docker containers. In summary, the containers utilize a standard web server to handle the HTTP layer and communicates to the DNS server over the DNS protocol. This is a standard method of isolating the HTTP layer from the application layer.

Future Speculation

I do not believe that the DNS over TLS protocol will attain mass implementation, nor that Google will mandate it for use of their DNS servers. There are three (3) primary reasons why:

  1. Architecture: Historically, short-term add-ons to a protocol are superseded by permanent change to the protocol or a parallel revision. If the goal is confidentiality, this can be achieved via an extension to the protocol.
  2. Performance: Per Google research, DNS is a bottleneck when a URL has multiple external sources. However, I suspect the current DNS resolution is still significantly faster. A UDP-based connection requires only a UDP socket with a simple sendto(2) call, whereas DNS over TLS requires multiple layers of conversion across potentially multiple machines. Specifically, from TLS to HTTP across the internet, converted to the DNS server and back across the same route.
  3. Standardization: There does not appear to be a TLS over DNS standardization. Most implementations utilize HTTP, but this is not specified in the RFC. Additionally, the JSON format differs between the implementations.

Documentation

This paper is based on multiple sources. The primary sources are cited below.

That time I Reverse-Engineered the Motorola CBEP Protocol

This is the tale of how I reverse-engineered a Motorola CPS radio protocol to make it work on Linux. While this may have been of questionable legality and thus lost interest in the project, I learned a lot on how to reverse engineer. I’m writing this entry more than a year after I initially did this, so I may be a little rusty on the details, but this is the gist of it.

My father worked in radio communications so when he passed I inherited his old EOL Motorola XTS 3000. I got an FCC ham radio license and wanted to utilize this device in service of my fledgling new radio hobby. Turns out, this device was the Rolls Royce of radios in its day. It can operate within the ham bands, can do encryption, digital communication, P.25, D-Star, trunking, and pumps out a very clear signal. In short, this was a heavy-duty mission-radio.


Motorola XTS3000 Radio

So I purchased a new $30 lithium ion battery, a programming cable, I ran into a few unfortunate roadblocks. First two…

  1. This device cannot be front-face programmed, which is a fancy way of saying you cannot just set in an arbitrary frequency on the fly. Kinda sucks if you want to change to another random frequency.
  2. This could only be done by a proprietary Motorola CPS (Computer Programming Software) – but this was trivially easy to download.

…All of that was trivial compared to the next bomb-shell…

You could only program this device using a hardware serial port running on native 32-bit Windows. This means no Windows 7/8/10, no Virtual Machines, no Linux, no USB-to-Serial port.

Radio Reference users lamented that they were forced to maintain an old Windows XP laptop with a serial port for programming their device. I personally went out and purchased a $75 computer off Craigslist. Damn!

Up to this point, here are my thoughts: A serial RS-232 port is a “dumb” port compared to modern USB or PCI devices. In fact, serial does not even have a single standard, its whatever the device programmers decide. RS-232 is as close to raw data over a wire as you can get. So why doesn’t this work in a VM? And, why can’t I just capture this data, replay it and have the device function as normal?

Chipping Away at the Secret

I questioned the assumptions and put in an FTDI cross-over cable I made. One end went into the Windows machine, the other end went into My Linux machine, a final serial to radio cable connected to the device. This way my computer was essentially doing a Man-in-the-Middle (MitM) attack, but for serial.


FTDI Cross-over-Cable

I whipped up some C code to relay messages between the Windows machine and device. When I initialized the connection, the radio beeped! And then nothing happened…the software timed out and complained that the connection to the radio failed. I captured the initialization bytes 0x01,0x02,0x01,0x40,0xF7 and replaying them clearly made the radio do something, but immediately stopped afterwards.


Serial Capture Cable

I tried this process several times, but it failed. Annoyed, I looked into purchasing an RS-232 snooping cable: a cable with two regular connections that transfer data as normal and two tap ports, one that taps pins 2 and another that taps pin 3. For whatever reason, well-built cables cost upwards of $100 online and proper hardware snooping devices cost $500, way above my budget. So I decided it was much cheaper to build my own damn cable. I have the program I whipped up to read the bits from the transfer.

And it worked!

I saw a flurry of bits fly across the terminal. In the mix, I noticed a subtle pattern: A pause in the transfer, 7 bytes sent back, echoed back, another pause, and then another flurry of bits. For example, I saw:

0xF5,0x11,0x20,0x00,0x00,0x00,0xD9.

Later on I saw:

0xF5,0x11,0x20,0x00,0x00,0x20,0xB9

and then

0xF5,0x11,0x20,0x00,0x00,0x40,0x99

If you didn’t catch it, the last two bytes are increasing by 0x20 (32) while the last bit ends in 9. (Spoiler, the repeating 9 was coincidental). I interpreted this as a value increasing by 32, and the last byte being a checksum. This was actually a lucky half-guess, because I had no way to know that.

Again I tried to replay these same bits, I ran into the same failure.

I briefly attempted to run the program in IdaPro, GNU Debugger for Windows and Immunity Debugger, but this approach failed and I am still not certain why. For example, I found a string that was clearly only utilized in one place in the binary and set an IdaPro breakpoint when the binary accessed that memory address. But for whatever reason, it did not break. Moreover, I learned the hard-way that the Win32 GUI API controls the application flow and is far from linear, so I could not just break after, say, 30,000 GUI instructions and expect to reliably step into the desired code.

I also briefly tried to use some RS-232 snooping tools, but every tool I found relied on a modern .NET framework not available on Windows XP. Moving on…

Win32 API Monitor

A Windows XP zealot on IRC informed me of a Win32 API monitoring application that would monitor DLL files and produce a list of when they were executed and their arguments. That might be useful.

I spent some time reading up on how Windows communicates over Serial: It uses CreateFileA() to open the COM1 port, typically listed as "\\.\COM1" and then use ReadFileA() call, similar to Unix’s read(2) syscall. I expected to see this in the API capture.

Nope! Instead, I saw that the binary used CreateFileA() against Commsb9. Next, I saw ReadFileA(), sending over 0xF5,0x01,0x02,0x01,0x40 but not the trailing 0xF7. Win32 API even told me driver was communicating to it using USB IOCTLs — not only is this device serial, USB was barely invented when this program was created. What’s going on here?

Reading the code, I identified that these Read/Write commands were taking place in VcomSB96.dll, and filtering by that DLL file, I saw that it was loading Commsb96.sys and Commsbep.sys. In my experience sys files are always driver files.

The Driver

Looks like we are working with a driver. With my limited background in writing a Linux USB driver, Microsoft’s excellent documentation and websites like this, I had an idea of what I needed to hunt for. The C code would look like this:

NTSTATUS DriverEntry( IN PDRIVER_OBJECT pDriverObject,
IN PUNICODE_STRING pRegistryPath )
{
NTSTATUS ntStatus = 0;
UNICODE_STRING deviceNameUnicodeString, deviceSymLinkUnicodeString;
...
pDriverObject->DriverUnload = OnUnload;
pDriverObject->MajorFunction[IRP_MJ_CREATE] = Function_IRP_MJ_CREATE;
pDriverObject->MajorFunction[IRP_MJ_CLOSE] = Function_IRP_MJ_CLOSE;
pDriverObject->MajorFunction[IRP_MJ_DEVICE_CONTROL] = Function_IRP_DEVICE_CONTROL;

...
}

This snippet is assigning the driver methods to the pDriverObject struct. I opened up the Commsb96.sys in IdaPro, expecting to hunt through, starting from the obligatorily exported DriverEntry symbol and trace where the DeviceIoControl initializes. To my surprise I saw this:


IdaPro with Full Symbols

Wow, that was easy. Turns out, a now defunct company called Vireo Software, Inc produced this driver in 1997 and failed to strip out the kernel symbols, making it much easier to reverse. Looks like they used C++ and compiled with optimization. That produced assembly that is a bit difficult to follow, but which I eventually traced back to where the DeviceIoControl messages landed in the kernel, and from there tracked down the USB message that Win32 API Monitor detected.

I finally traced code that read 5 bytes form the stack, and ran them through a loop, calculated a single byte, then returned that byte. I wish I could claim to have written the code below, but it was actually done by another IRC contact. He had a (probably bootleg) professional version of IdaPro that produced the following C code.

unsigned char sbCRC(const unsigned char *msgbuf, const int len) {
     const unsigned char table[8] = {}; // REDACTED FROM THE BLOG TO AVOID LEGAL TROUBLE!

     unsigned char a, b, n = 0;
     int i = 0;

     while (i < len) {
          n = (unsigned char)*(msgbuf + i) ^ n;
          a = ((unsigned char)((signed char)n >> 1) >> 1) ^ n;
          b = a;
          a = ((signed char)a << 1) & 0xF0;
          b = (signed char)b >> 1;
          if (b & 0x80)
               b = ~b;
          n = (a + (b & 0x0F)) ^ table[n & 0x07];
          i++;
     }

     return n;
}

I tested this function against known values such as 0xF5,0x11,0x20,0x00,0x00,0x00 that I cited before, ran it through the function and it resulted in 0xD9. Boom! Checksum reversed!

The Missing Link – RS-232 Flow Control

Coming of age in the 2000s, I learned about effectively full-duplex buses such as USB or PCI. In modern buses, you can effectively asynchronously send data but the RS-232 often requires manual flow-control by the CTS, DSR, RTS and DTR pins.

Providentially, around this time I found the amazing tool 232Analyzer, which was the only tool of its sort that did not require a modern .NET framework. Had I found it earlier, this would have saved me a lot of time! But I learned so much along this process.


232Analyzer Capture

Putting it all Together

With that, I modified my python code to emulate the flow control, calculate the checksum and sent the resulting bytes over. And this time….it worked! I could replay messages from original CPS and the radio responded with the same flutter of meaningless data.

Asking around on forums, IRC and reading up on the Motorola hardware, I learned that these sorts of devices do not request or set specific values from the radio as an API abstraction layer might do. Instead, you request memory locations and interpreting those locations according to a pre-known memory map. I deduced that the 0xF5,0x11 bytes meant “read”, the 0x20 is some sort of divider (or maybe more?), the next 3 bytes are memory locations, and the final byte is a checksum.

Armed with this hypothesis, I found the memory location of the device serial number and my code could read serial. I tested this on my second radio and it resulted it correctly retrieved the serial number.

To find other values, I recorded reading the radio memory, made a single change, then read the radio memory again and compared the difference to isolate values. I was able to find frequency values, channel names, signal strength values, etc! With time, I could mapped out the entire address space! I even found the write-prefix, but was too scared to test it in fears of bricking my radio.

Anti-Climactic Ending

Somewhere along this, I wondered “wait…is this legal?” I contacted the EFF. They were extremely eager to get back to me and after a long conversation the lawyer suspected that because the CPS has a copyright notice and I did not…um…come into acquisition of it through a financial transaction (sure, that phrasing works), it was likely illegal to distribute the reverse-engineered code.

And mapping out the memory got really tedious and annoying. And I started watching The Walking Dead.

And right about here my journey ended.

::cough::

But I learned so much!

  1. How Windows device drivers work
  2. Windows API calls (turns out, they don’t suck)
  3. How to reverse engineer code with IdaPro
  4. How RS-232 traffic flow works
  5. A buncha tools!

Thoughts? Should I have kept going?

Migrating from FreeNAS to FreeBSD

I love FreeNAS. Its awesome, well built, well-supported. But as my needs increased, I wanted to use my FreeNAS box for more than the basics. In particular, I was moving towards a single host to run as a:

  1. Family NAS server
  2. Development server
  3. IRC client
  4. VM server
  5. Web server
  6. Email Server
  7. Git Server
  8. Home Firewall
  9. Home IPv6 gateway
  10. IPv6 VPN and Jump box

FreeNAS could easily do all of this. But I found myself using the device for everything but a NAS server. Also, as my experience on FreeBSD reaching proficient-status, I wanted to jump in the deep end and manually configure a production system from scratch. So I thanked FreeNAS for their contribution, yanked out the USB disks and installed FreeBSD 11.1 on a separate USB disk.

During installation, I was careful not to touch the /dev/ada devices, as that would destroy my precious files. Instead, I installed to the second USB disk, /dev/da1, while the installation medium was /dev/da0. This was obviously a problem, because at reboot the USB disk would become /dev/da0 and the kernel would panic upon not finding a /dev/da1. So I dropped to the terminal and mounted zroot/ROOT/default volume,  which is the / directory, to /tmp/root as follows.

zfs set mountpoint=/tmp/root zroot/ROOT/default
zfs mount zroot/ROOT/default

Then I edited /tmp/root/etc/fstab and changed /dev/da1p2 to /dev/da0p2, umounted, reset the machine and FreeBSD booted without a glitch.

As mentioned, I plan on using this system fairly heavily going forward so the 8 GB USB disk would definitely not be sufficient. FreeBSD has an amazing feature where it isolates the base system from any user-installed applications or configurations. Rather than using symlink magic, my strategy was to store all application data on my two 4TB NAS disks.

First things first, I imported the pool as follows:

zpool import -f tank

The -f flag was necessary because for whatever reason ZFS thought tank was currently utilized. A quick zfs list revealed that FreeNAS had been mounting my disks to /tank. Unfortunately, the /tank directory is not utilized by default by FreeBSD. Therefore, I renamed each ZFS volume to a new /usr/local as follows. First, I created a zfs volume for tank/usr/share as follows.

zfs create tank/usr/local

Then I renamed the old paths to map to my new intended directory structure, as follows

zfs rename tank/old/path tank/usr/local/new/path
zfs set mountpoint=/usr/local/new/path tank/usr/local/new/path

This took a bit of time, but after completing these for all partitions, I ran:

zfs mount -a

With that, all ZFS shares were mounted as /usr/local subdirectories. All of my data was successfully migrated over without a single bit of data loss!

From here, I needed to re-create the jails. FreeNAS’s excellent jail web-based GUI allows you to create jails with their own independent network stack. This feature is called VIMAGE and is useful to isolate network services from the host FreeBSD system. VIMAGE is pre-compiled into the FreeNAS kernel. It is on by default on FreeBSD 12.0, but not 11.x and must be compiled in. To do this, you need to download and uncompress the src distribution, edit /usr/src/sys/amd64/conf/GENERIC and add in the following line:

options VIMAGE

Next, compile the kernel and install it as follows.

make -j 5 buildkernel
make installkernel

The -j 5 is because this machine is an i3 with 4 cores – feel free to adjust this depending on the number of cores you have.

With a successful reboot, I was now ready to migrate the jails over. I did so by moving the zfs jails volume to /usr/local/jail, such that my IRC client jail was /usr/local/jail/irc. Now the complicated part: Configuring the jails!

Since a jail using VIMAGE has a completely separate network stack, by default it renders a jail unable to communicate outside of itself. The way to allow communication you have to create an epair(4) pair and pass one side to the jail, as follows:

ifconfig epair create
ifconfig epair0a vnet JAILNAME

In this configuration epair0a would belong to the jail while epair0b would belong to the base FreeBSD host, such that they could communicate. But how to setup connectivity? I had a lot of options to have the jails connect outside, including:

  • Being on the same subnet (192.168.1.0/24)
  • Being on a separate VLAN from the rest of the network (might be the long-term plan)
  • Have a single VLAN, have legacy IPv4 addresses identifiably different for ease, but have a single IPv6 network. I opted for this for now. Its simple and works.

This means creating an if_bridge(4) and attaching the network interface card, in my case an em(4) card and epairXb. Any frame to the bridge is relayed to the relevant epair(4). (Note, this not a route). I set my jail IP range as 192.168.100.0/24, just for organizational purposes. I also set the ISPs IP subnet to be 192.168.0.0/16, otherwise it would drop packets from 192.168.100.0/24. I am using TunnelBroker for my IPv6 traffic, as Verizon Fios does not offer IPv6. (As an side, this may be a good thing, since ISPs typically blocks ports, whereas TunnelBroker is completely unfiltered.) With that, Boom, network connectivity!

But…I wanted something repeatable per reboot, in the event of a power failure or loss. This meant I needed to go a little further. And here’s the complicated part. It took me about 4 hours to properly configure /etc/jail.conf:

/* Template */
host.hostname = "${name}.my.domain.prefix";

$ip4_route      = "192.168.100.1";
$ip6_route      = "IPV6PREFIX::1";

vnet;
vnet.interface = "epair${if}b";

persist;
allow.mount;
mount.devfs;
allow.sysvipc;

exec.prestart =  "ifconfig epair${if} create up";
exec.prestart += "ifconfig epair${if}a up";
exec.prestart += "ifconfig bridge0 addm epair${if}a up";

#exec.start += "/sbin/ifconfig epair${if}b up";
exec.start += "/sbin/ifconfig epair${if}b inet  ${ip4_addr}/24 up";
exec.start += "/sbin/ifconfig epair${if}b inet6 ${ip6_addr} prefixlen 64 up";

exec.start += "/sbin/route -4 add default ${ip4_route}";
exec.start += "/sbin/route -6 add default ${ip6_route}";

exec.start += "/sbin/ifconfig epair${if}b down";
exec.start += "/sbin/ifconfig epair${if}b up";

exec.start += "/bin/sh /etc/rc";

exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig bridge0 deletem epair${if}a";
exec.poststop = "ifconfig epair${if}a destroy";

irc {
        path = /usr/local/jail/irc;
	$if = "0";
	$ip4_addr 	= "192.168.100.2";
	$ip6_addr 	= "IPV6PREFIX::2";
}

www {
        path = /usr/local/jail/www;
	$if = "1";
	$ip4_addr 	= "192.168.100.3";
	$ip6_addr 	= "IPV6PREFIX::3";
}

In short, upon initialization, this creates a new epair(4) as specified by $if, attaches it to the jail, assigns the relevant IPv4/IPv6 information, and starts the init scripts. Shutdown is a mere detachment from the bridge and destruction of the epair(4). I also needed to assign the legacy IPv4 address to my em(4) interface.

Finally, I added the following sysctl(8) settings to /etc/sysctl.conf:

net.inet.ip.forwarding: 1
net.inet6.ip6.forwarding: 1

I did a lot of testing, reboot, restarting the jail, etc, and every time it worked. From the jails’ perspective, they didn’t even “know” they were migrated from one system to another. I wish I had tested if a FreeNAS plugin survived the migration, but I never used FreeNAS plugins anyways (what is this Plex I keep hearing about?).

Going forward, I plan:

  • Place the jails on a properly separate VLAN to segment the network
  • Consider use pfSense running in bhyve(8) to function as the Jail’s firewall of choice
  • Look into vale(4) to replace if_bridge(4). But I can’t find any documentation on it!
  • Figure out why TunnelBroker is failing on FreeBSD, but works just fine on my Linux Raspberry Pi – likely the fault of the ISP router.

My only regret: not installing HardenedBSD with LibreSSL.

Thoughts?

FreeBSD kernel Makefile variables SRCTOP and SYSDIR

I am currently writing a FreeBSD device driver and find myself lugging around the entire src. As you can imagine, this is quite large, especially if you are using any sort of version tracking system. So following the example here, I extracted out:

/usr/src/sys/modules/rtwn/
/usr/src/sys/dev/rtwn/

into

/home/user/src/rtwn/sys/modules/rtwn/
/home/user/src/rtwn/sys/dev/rtwn/

However, when I ran make(1) in the /home/user/src/rtwn/sys/modules/rtwn, I received an error saying:

make: don't know how to make r92c_attach.c. Stop

This error message is extremely non-descriptive of the actual issue. After reviewing the aforementioned functioning Makefiles, I identified that the SRCTOP and SYSDIR were not set correctly.

SRCTOP is the equivalent of /usr/src. If your src directory differs from /usr/src, such as $HOME/src/freebsd12src, you would set SYSDIR to $HOME/src/freebsd12src/.

SYSDIR is similar. Ordinarily it would be /usr/src/sys, but now it might be $HOME/src/freebsd12src/sys/.

This can be resolved two ways:

  1. Command-line over-ride. I am doing this:
    make VARIABLE="something"
    For me, that would be:
    make SRCTOP=$HOME/src/freebsd12src/ SYSDIR=$HOME/src/freebsd12/sys/ -C sys/modules/rtwn load.
  2. Permanent method: Edit the Makefile in question, in my case sys/modules/rtwn/Makefile.
    SRCTOP="/home/user/src/freebsd12src/"
    SYSDIR="/home/user/src/freebsd12src/sys"

And of course, you have to have at least one correct src directory in order to compile a kernel object. This is pretty simple, but it confused me for a while. Hope this helps! Keep writing that BSD code!

Linux kernel code vs FreeBSD kernel code

Linux driver code contains some serious garbage. I heard this refrain, but I did not realize how bad it was until I looked at it myself. Here is just one example.

Device drivers typically read static memory, typically known as EEPROM or ROM, from the chip to identify version, hard-coded information, device capabilities, etc. These values are used throughout execution of the driver. The reading process is among the first things when the device is attached and powered on.

In the case of FreeBSD, after the kernel reads the ROM, it uses a struct pointer with all the variables pre-populated, and points it at the ROM blob data stored in memory. For example:

struct r88e_rom {
	uint8_t		reserved1[16];
	uint8_t		cck_tx_pwr[R88E_GROUP_2G];
	uint8_t		ht40_tx_pwr[R88E_GROUP_2G - 1];
	uint8_t		tx_pwr_diff;
	uint8_t		reserved2[156];
	uint8_t		channel_plan;
	uint8_t		crystalcap;
#define R88E_ROM_CRYSTALCAP_DEF		0x20

	uint8_t		thermal_meter;
	uint8_t		reserved3[6];
	uint8_t		rf_board_opt;
	uint8_t		rf_feature_opt;
	uint8_t		rf_bt_opt;
	uint8_t		version;
	uint8_t		customer_id;
	uint8_t		reserved4[3];
	uint8_t		rf_ant_opt;
	uint8_t		reserved5[6];
	uint16_t	vid;
	uint16_t	pid;
	uint8_t		usb_opt;
	uint8_t		reserved6[2];
	uint8_t		macaddr[IEEE80211_ADDR_LEN];
	uint8_t		reserved7[2];
	uint8_t		string[33];	/* "realtek 802.11n NIC" */
	uint8_t		reserved8[256];
} __packed;

_Static_assert(sizeof(struct r88e_rom) == R88E_EFUSE_MAP_LEN,
    "R88E_EFUSE_MAP_LEN must be equal to sizeof(struct r88e_rom)!");

Notice the assertion at the bottom, which ensures that the ROM struct’s size equals a pre-defined length. The code will fail to compile if this assertion is not valid. Later, the kernel will instantiate a struct pointer and point it to the ROM, stored in the variable buf, as follows:

struct r88e_rom *rom = (struct r88e_rom *)buf;

Now, rom->channel_plan is set to the correct value. Simple.

Unfortunately, this is not how the same code is written on Linux. As mentioned, the Linux driver also begins by reading the ROM blob and storing it in a value called hwinfo. But rather than creating an equivalent struct pointer, the Linux code uses offset values of the ROM on an as-needed basis. For example, the driver reads the channel_plan as follows:

rtlefuse->eeprom_version = *(u16 *)&hwinfo[params[7]];

In this example, params[7] comes from a list of ROM offsets values set in the previous calling function. (That alone made tracing difficult.) The rtlefuse->eeprom_version is now the same as FreeBSD’s rom->version. This manual process repeats for every variable in the ROM.

While that may be just annoying and require a negligible bit more CPU power, this is not be a problem if it was done all in one place. But instead, the driver reads from the hwinfo blob on a seemingly as-needed during execution. And because these as-needed instances are during normal execution, the driver reads-in the same static value from hwinfo every a simple WiFi function occurs, such as changing the channel.

Okay, but even that might not be too difficult…right? Here’s the real kicker.

Sometimes, the driver works by using incrementing offsets from the ROM blob. For example, consider at read_power_value_fromprom (in drivers/net/wireless/realtek/rtlwifi/hw.c). It initializes eeaddr as a u32 (uint32_t), then assigns it with the offset value EEPROM_TX_PWR_INX. So far so good. But then, rather than using new offsets for every successive value, it increments the eeaddr value in multiple doubly-nested for-loops. Here is a simplified version of the code:

for (rfpath = 0 ; rfpath < MAX_RF_PATH ; rfpath++) {
		/*2.4G default value*/
		for (group = 0 ; group < MAX_CHNL_GROUP_24G; group++) { pwrinfo24g->index_cck_base[rfpath][group] =
			  hwinfo[eeaddr++];
			if (pwrinfo24g->index_cck_base[rfpath][group] == 0xFF)
				pwrinfo24g->index_cck_base[rfpath][group] =
				  0x2D;
		}
}

Notice the line hwinfo[eeaddr++]! Merely reading in that variable changes the offset. Its the Heisenberg Uncertainty Principle equivalent of code. This is a cleaned-up version of the 188-line function. The actual function has 6 nested for-loops, some with if-statements, each incrementing the eeaddr parameter as they go along.

Why would anyone do it this way? You are needlessly using up the CPU, making the code difficult to follow, repeatedly reading in static values and making any minor modifications and re-ordering or re-structuring will essentially break the entire function.

And perhaps the worst offender is when 20 functions deep you are not even working with hwinfo anymore. You are working to a pointer to hwinfo that has been incremented God-knows where, with their own offsets that are near impossible to track down.

In my efforts to port this driver to FreeBSD, I literally resorted to printing out the entire ROM, manually finding the memory, and backing into the equivalent offset. Other bizarre code: I have seen if-conditions that are impossible to reach, misplaced code that should go in the previous function, code that does bits of a tasks, while another function does the entire task – so repeat code, unnecessarily repeated code, etc.

How does this make it into the Linux Kernel?

To be fair, this does not appear to be the fault of Larry Finger, who maintains this driver. This is the fault of Realtek, for vomiting this terrible driver in the first place, providing absolutely zero documentation and refusing to respond to any contact attempts.

I hope my FreeBSD port is cleaner and more performant!

Switched from Ubuntu-based to Fedora

tl;dr: Fedora’s debugging packages work, Ubuntu’s are out of date.

Linux = Linux = Linux, whether Arch or Slackware or Ubuntu or OpenSUSE or Linux from scratch as I once did (before there were instructions!). Unless and until the kernel forks and someone decides to modify the syscall table, they all use the same basic syscalls, they typically share the same basic libraries and core utilities, etc. They’re all the same.

Why did I use Ubuntu-based distributions? (Note: Not Debian) Because Ubuntu came pre-configured with all the things I did not care to learn or manually configure: ACPI, firmware, X11, a pretty WM theme, etc. I did not particularly care whether I was running Mint, Elementary, Ubuntu MATE or basic Ubuntu (except Unity…nah). As long as it did not do strange things like remove /sbin/ifconfig or have a radically different file structure than I was used to. I felt at home with knowing where the standard file paths were, and knew how to administer my machine. Their package repository was pretty solid. It had almost everything I wanted – and what little was not on it was typically available in Debian-package format. The broader Linux community effectively standardized on this package format. This is crucial. Debian’s apt and FreeBSD’s pkg are in my muscle-memory at this point.

Literally one thing pushed me over: Ubuntu’s SystemTap was broken. Utterly broken!

I got into OS-level programming, specifically, porting a Linux WiFi driver to FreeBSD. I wanted to use SystemTap, Linux’s answer to DTrace, to help understand what is going on during live execution. But SystemTap does not work on Ubuntu – at least currently.

But wait, I thought Linux = Linux = Linux and programs from 20 years ago will still work. Why does SystemTap fail?

SystemTap works by producing C code for a kernel module, compiling it and loading it into memory. Sometime ago, the kernel team changed the get_user_pages() kernel API call. This meant that any code compiled against the old function definition failed. I encountered this in the professional space when the VMWare kernel modules failed to build and I hacked it until it worked. (They think I’m a wizard now). I was on Kernel 4.10 but the version of systemtap Ubuntu used was nearly 2 years old. This meant no one from the Ubuntu team was using it.

I submitted a bug report and installed Fedora 26.

SystemTap was developed by Red Hat and was trivial to get working under Fedora. And while not every single package is available (Bitcoin, Steam thus far), there is enough that moving over was trivial. Also, they come in Cinnamon, which I prefer, with a pretty theme. And it provides a clean terminal out-of-the-box. Which I need. (I would rather use stock XFCE if their terminal was clean than fully-loaded CentOS with an ugly terminal)

dnf took a little getting used to, but a hop-over from apt. So whatever on that front….

I would be willing to try OpenSUSE again, but the latest time I did, they got rid of /sbin/ifconfig for /sbin/ip, which is unacceptable. Silly, perhaps…Does it come in Cinammon? What does it offer? Are the packages as clean and up to date? I may never know, unless another business-need arises. I do not care to run any of these “hardware” distributions, like Arch. I paid my Linux dues around kernel 2.2 on Slackware and its time to move on from that.

Thoughts?

But look, if you’re 99% of the Linux world, any specific distribution is trivial. Pick one and go with it. Unless you’re doing very specific tasks like me, it really does not matter what you use. So stop Distro Hoping!

Custom Kernel Modules for Chromebook

Note: I wrote this about a year and a half ago, but I refer to it all the time. Hopefully the instructions have not changed too much! Enjoy!

I recently purchased a Chromebook. It’s great, it symbolizes the direction the PC market should head – inexpensive, low-powered ARM processor, defense in depth resistance to malware and simple for non-technical users. And with crouton, it functions quite cleanly as a Debian-based workstation.

With its simplicity and low price, there are certain key features that are lacking in the stripped down Linux kernel that can make it frustrating for a power-user. Unfortunately, Chromium addons have not or cannot satisfy some tasks that require kernel-level functionality. Even in crouton, you may find your ability limited to the user-space. Those looking for casual additions, recompiling the kernel may seem like daunting over-kill. Instead, compiling and inserting a single module may serve as an apt alternative. In this guide, I will explain how to compile a custom kernel module to add additional functionality to your Chromebook and how to circumvent the built-in security mechanisms that prevent you from adding into the kernel-space.. This guide is specifically written for an ARM-based CPU using kernel 3.10.18 for the CIFS (SMB) module, but can be trivially ported to any other architecture, kernel and module.

Compiling the Kernel Module

As mentioned, Chromium OS is a stripped down version of Linux. Therefore, you should be able to compile and dynamically link kernel modules from the stock kernel into Chromium.

Per Google’s documentation, you must compile the kernel and modules on an x86_64 CPU, even if you will be compiling an ARM or 32-bit x86 module. This is possible thanks to GNU C Compiler’s cross-platform capability. The documentation also specifies using Ubuntu, but it worked just fine on my Debian 8 workstation.

If you have not already done so, install git, subversion and perform the basic configurations:

sudo apt-get install git subversion
git config --global user.email “name@domain.tld”
git config --global user.name "Your Name"

Google manages its various git repositories with wrapper depot_tools, a custom git wrapper. You can clone the associated git repository and set your PATH environmental variable to include the wrapper scripts as follows.

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

export PATH=`pwd`/depot_tools:"$PATH"

Next, make a directory where your Chromium OS build will reside, download the Chromium source, and synchronize it to the latest updates. This take around 30 minutes to complete.

mkdir chromiumos && cd chromiumos
repo init -u https://chromium.googlesource.com/chromiumos/manifest.git
repo sync

Once completed, you will need to download the cross-platform SDK environment, build the dependencies and enter a chroot(1) environment. This will take another 30 minutes.

cros_sdk

Now that you are inside the chroot(1) environment, you need to specify the hardware configuration for your Chromebook device, either x86-generic, amd64-generic or arm-generic. You can determine your architecture by running uname -m on your Chromebook. For my ARM-based CPU, I did the following:

export BOARD=arm-generic

Now you must prepare the core packages associated with your board.

./setup_board --board=${BOARD}
./build_packages --board=${BOARD}

Change directory to ~/trunk/src/third_party/kernel/ and then to whichever subdirectory is associated with your kernel (ie, v3.10 for 3.10.18). You can determine your kernel version by running uname -r on your Chromebook.

Next, we will need to tell the kernel which hardware platform you are on and start with the base configuration of the kernel. A list the options of base configurations by running find ./chromeos/config. In my case I am using NVIDIA’s Tegra motherboard, which is ./chromeos/config/armel/chromeos-tegra.flavour.config, so I use chromeos-tegra as follows:

./chromeos/scripts/prepareconfig chromeos-tegra

If you are compiling for a non x86_64 CPU, set the architecture and compiler settings as follows:

export ARCH=arm
export CROSS_COMPILE=armv7a-cros-linux-gnueabi-

This next portion is the same as compiling any other kernel module. Configure the kernel by running make menuconfig

Select whichever controls you would like to install and save. Once completed you will have a .config file that corresponds to your hardware. Since we are only compiling the kernel modules, you can either run make modules to compile all kernel modules, or make fs/cifs/cifs.ko to build only a specific module. I prefer the former because your module may require other dependencies in other modules, such as with crypto/md4.ko for cifs. You can verify that the file was built for the right architecture by running file fs/cifs/cifs.ko. Great! On to inserting the module!

ChromiumOS’s Security Mechanisms

ChromeOS is the official signed release of ChromiumOS, which is what you run in developer mode. Even in developer mode, Google implemented multiple defensive mechanisms to slow down a would-be attacker from gaining access the underlying system. To protect the kernel, Google utilized the Linux Security Module (LSM), which validates files from the root partition against a list of cryptographic hash values stored in the kernel, thereby preventing an attacker from loading a malicious kernel modules. In effect, the only way to insert a kernel module is to have it stored on the root partition. But by default, the root partition is set to read-only, so you cannot simply move a file to the root partition and load it.

Therefore, we must disable the root partition verification running the following script.

sudo /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partitions 4

Now, reboot the machine and from ChromiumOS remount the root partition to be read-writeable, as follows:

sudo mount -o remount,rw /

From here, you should be able to simply insert the kernel module with insmod. Now, you can install
Enjoy!

Draw this shape without picking up your pen

For many years, while in a meeting or in a moment of free time, I have tried to draw this shape without picking up my pen or drawing over the same two points twice.

shape

At best I would get 1 line away, but never completed the shape.

I wanted to know if it was even possible. So I wrote some python code to try every possible combination.

But, the code is below.

#!/usr/bin/env python3

import copy
import sys

lines = {
        1:[2,3],
        2:[1,3,4,6,7],
        3:[1,2,5,6,7],
        4:[2,5,6,7],
        5:[3,7],
        6:[2,3,4,7,8],
        7:[2,3,5,6,8],
        8:[6,7]
    }

def check(cstate):
    for offset in lines:
        if sorted(lines[offset]) != sorted(cstate[offset]):
            return
    print("Solution!")
    sys.exit()

def iteration(clocation, cstate):

    if len(cstate) == 8:
        check(cstate)

    for ilocation in lines[clocation]:
        nstate = copy.deepcopy(cstate)
        y = nstate.get(clocation, [])
        x = nstate.get(ilocation, [])

        if ilocation in y:
            continue

        y = y + [ilocation]
        x = x + [clocation]

        nstate[clocation] = y
        nstate[ilocation] = x
        iteration(ilocation, nstate)

iteration(1, {})
iteration(2, {})

The lines list is an abstraction of the possible points in the shape and where they can connect to. Point 1 is the top point, 2 and 3 are the top corners of the square, 4 and 5 are the far left and right points of the triangle, etc.

Starting at points 1 and 2. Point 1 is functionally the same as points 4, 5 and 8, while point 2 is the same as 2, 3, 6 and 7. No need for unnecessary iterations. Give its current location, the code recursively builds lines to all possible connection points. If no points are available, it just returns.

It breaks when all possible links are met, as seen by the check function. This is done by checking if every point is touched at least one, and then iterating through all points to see if that point is connected to every possible other line.

Turns out it is not possible.

Sucks.

FreeBSD and Linux Remote Dual Booting

The following is a quick and dirty guide on how to setup remote dual booting for FreeBSD (12.0-CURRENT) and Linux (Ubuntu 16.04). Granted, this method is slightly a hack, but it works and suits my needs.

Why remote dual-booting? I am currently developing a FreeBSD kernel module for a PCIe card. The device is supported on Linux and I am using the Linux implementation as documentation. As such, I find myself frequently rebooting into Linux to look printk() outputs, or booting into FreeBSD to test kernel code. This device is located at my house, and I typically work on it during my downtime at work.

Why not use Grub? I would have preferred Grub! But for whatever reason, Grub failed to install on FreeBSD. I do not know why, but even a very minimalistic attempt gave a non-descriptive error message.

efibootmgr? Any change I made with efibootmgr failed to survive a reboot. This is apparently a known problem. Also, this tool only exists on Linux, as FreeBSD does not seem to have an efibootmgr equivalent.

Ugh, so what do I do???

The solution I came up with was to manually swap EFI files on the EFI partition no an as-needed basis.

First, I went into the BIOS and disabled legacy BIOS booting, enabled EFI booting, and disabled secure booting.

Then, I installed Ubuntu. I had to manually create the partition tables, since by default the installer would consume the entire disk. However, this does not automatically create the EFI partition. So, you must manually create one. I set mine to 200MBs as the first partition. After installation, I booted up, mounted the /dev/sda1. I found that ubuntu had created /EFI/ubuntu/grubx64.efi and other related files. Great!

Next, I installed FreeBSD and while manually setting up the partition tables, FreeBSD auto-created an EFI partition. One already exists, so I safely deleted it, and proceeded with the rest of the install. Right before rebooting, I mounted /dev/ada0p1 (sda1 on Linux) as /boot.local/ and /dev/da0p1 as /boot.installer/. I then copied /boot.installer/EFI/BOOT/BOOTX64.EFI too /boot.local/EFI/BOOT/EFIBOOT/BOOTX64.EFI (I think I had to re-create EFI/BOOT, I’m forgetting off-hand). Then I rebooted.

When I rebooted the machine, Ubuntu still came up. This is because Ubuntu edits the EFI boot order and places ubuntu as the first partition. Ordinarily you should be able to use efibootmgr here to boot into FreeBSD and use the non-existent FreeBSD equivalent to boot back, but with the lack of that option, I mounted the EFI partition (/dev/sda1) as /boot/efi, and when I wanted boot into FreeBSD, I renamed /boot/efi/EFI/ubuntu/grubx64.efi to ubuntu.efi and then copied /boot/efi/EFI/BOOT/BOOTX64.EFI to /boot/efi/EFI/ubuntu/grubx64.efi. When I rebooted, FreeBSD came back up! Then on the FreeBSD side, I mounted /dev/sda1 to /boot/efi and did copied /boot/efi/EFI/ubuntu/ubuntu.efi to /boot/efi/EFI/ubuntu/grubx64.efi.

And that’s it! I can now remotely boot back and forth between the two systems.

Ugly? Yes. But it does the job.

Linux could fix this problem by debugging their efibootmgr utility and FreeBSD could fix this by having an efibootmgr equivalent at all.

Thoughts?

My python3 Programming Environment

UPDATE: I have since started using a very good vimrc. I recommend it over mine listed below. My only modification is that I removed all line numbers, eww.

I ssh into a FreeBSD jail with everything setup.

The Jail runs on code.mydomainname.com, which has an internet-routable IPv6 address – and IPv4 behind a NAT, (boo!)

I have a virtualenv already built-out. (more about my pip list later)

The set my ~/.bashrc to execute source enter-env.sh (even though I run ksh)

My REPL is ptpython, which just requires touch ~/.ptpython/config.py.

I use gitlab, since they offer free repositories, and then periodically manually backup my code at other locations. If there are automatic ways of doing this, I would be interested.

My project’s gitlab wiki has copy-paste instructions to install all necessary packages, both on FreeBSD and Debian (well….Ubuntu) and subsequent python3 packages that you install with pip.

My default browser is vim, and I set ~/.vimrc to: set ts=4 / set expandtab. I used to set syn on, but that does not seem necessary anymore.

My project requires a PostgreSQL database, so I included the very simple instructions on installation and configuration in the gitlab wiki.

Finally, though I typically code off of a FreeBSD Jail, everything is configured to run on Debian. The main reason it works on Debian is because my personal computer (before my Chromebook took over) is was Mint, but I intend to run this code on a FreeBSD server, primarily for ZFS. I used to code on a Raspberry Pi, but it was too slow.

It takes me about 5 minutes to rebuild this environment, in the event that it goes down (which it never does).

Thoughts?