TunnelFS

October Update: Learning for TunnelFS

Posted on by Savagedlight

It has been a while since I announced I would work on an exciting new project, TunnelFS, and I think it’s time for a status update.

In that post, I mentioned I had a lot to learn before I could get anywhere with this project. I had an idea of how much it was, but it turned out to be more. I’ve been working on learning C — I can read C without much trouble, except the really complex cases, and I can actually write a little on my own. For example, I’ve been going through the entire FreeBSD SCTP implementation in an effort to document its sysctl MIBs (there’s still outstanding work on this, but I’ll cycle back to it at some point). I’ve been reading some books on the subject of C, and it makes sense to me. It’ll take a while to really learn the grammar, but the concepts are nothing new. It feels a bit like learning a new language which is similar to your own, but with slightly different grammar and vocabulary.

This week I’ve been trying to figure out FUSE. It’s difficult. Documentation seems to be virtually non-existent, the code that I have found isn’t documented, and what little documentation I have found is on some sourceforge page, or specific to the kernel-side implementation of the API with little or no concern of how one would use that API. I’ve been trying to figure out how to compile some example code using Clang, but have given up and will be using GCC for now. I don’t have to solve every problem all at once. The important thing is getting a grasp of the API. I hope to implement fuse-tunnelfs in C to make it easier to port it as a native file system, keeping the options open for future potential features such as booting from TunnelFS.

I’ve done some thinking on how I want TunnelFS to be implemented, logistically speaking. I’ve come up with two implementation alternatives, but might land on a middle ground if possible. The only difference between the two implementation options is how it handles filesystem I/O on the host side.

TunnelFS-Implementation1The first option has the filesystem I/O be handled by a separate process, which may run as a different user than root if desired, communicating with bhyve over UNIX socket or some other means which I have yet to investigate. This makes the data have to be copied or streamed three times: Between fuse-tunnelfs and the TunnelFS virtio driver (guest side), between TunnelFS VirtIO driver guest side and bhyve side (backend), and between that backend and the separate process. It also has a smaller chance of breaking Bhyve, as it leaves less code running in Bhyve’s context. It could also be slightly more portable, as the filesystem I/O code isn’t directly tied to Bhyve in any way.

TunnelFS-Implementation2The second option has the filesystem I/O implemented into Bhyve itself. This leaves a bigger footprint in Bhyve, meaning there’s a higher chance of breaking Bhyve. But it also means there’s one less data-copying step, as it won’t have to communicate with a separate process. It is possibly less portable than the first implementation, but it could remain easily portable if done right (i.e. not implementing the I/O logic directly in the VirtIO backend code). Filesystem I/O would run in the bhyve context, meaning all operations would be executed as root.

I think this is a performance-vs-security/reliability decision. I’d err on the side of security/reliability, but this is all quite far off into the future, so we’ll see what I find out. If anyone have any input on this please let me know; your knowledge is appreciated.

I think that sums up this month, and the status of TunnelFS pretty well. Hopefully, I’ll have skeleton code by the end of the year, but we’ll see. :)

A farewell to the old, and a hello to the new

Posted on by Savagedlight

It’s late August, which means it’s currently late summer, and soon fall, where I live. It’s time to decide on a winter project. Previously, I’ve had coding sprees on various things related to the MMORPG Anarchy Online. Some examples, in no particular order: AOItems (an items database), Towerwar Tracker (mobile) (shows Player versus Player land control fights), PlanetMap Viewer, and more. Although I have to admit some of those stretched into year-around and even multi-year projects. :) In idle moments, I’ve taken stabs at improving things in the FreeBSD land as best as I can, by writing some guides and giving a helping hand on various social media, forums and IRC.

Anarchy Online has been the main focus point of many of my IT-related hobby projects. And much of my home infrastructure, powered by FreeBSD, is the way it is now because it had to be that way to support and run those projects. I started playing the game in December 2004, and stopped playing it early 2009. I have played it a week here or there when large patches have hit, but for the most part, just logged in to check if my AO-related tools were still working, or in some other way related to improving those tools. It’s now August 2015, and it’s six years since I stopped playing that game. I’ve been making tools for the game for more than ten years, and it has been fun. It has been an incredible learning experience. I’ve done many crazy things, such as:

  • Making a multi-process map compiler using PHP and syncing state on disk (4 processes was about 50% faster than one)
  • Porting said compiler to C#.Net, heavily optimized with threads. About 40x faster than single-process PHP version :)
  • Creating a LARGE toolchain (20 libraries/applications) for extracting item info from the game client and making sense out of it, storing it in a DB and displaying it on a website using PHP. Toolchain parses through 14 years of patch history in about 15 minutes. (similar tools were said to be taking weeks to do the same). This toolchain also verifies integrity, and automatically creates reports on human-induced mistakes in the data, for easy and detailed report submission to the games developers.
  • For the planetmap viewer, using a hooking library to run my own code in the context of the game client, effectively using the game client as an API.
  • Much more :)

If it wasn’t for this game, its awesome community, and all the related projects I’ve worked on, I wouldn’t be where I am today. I wouldn’t have known C#.NET as well as I do, if at all. There are many friends I would never have made otherwise, all over the world. If you ever read this, you should know who you are. :)

All that being said, 10 years is a long time. It’s a third of my life on this planet. It was fun, it was quite the experience. It really was. But now it’s time to change focus. I will maintain the existing projects for the foreseeable future, until a suitable successor can be found. No new features are likely to arrive. All the tools except those related to AOItems.com are open source, so anyone who’s up to the task can fork and improve them. I’ll keep AOItems updated for the foreseeable future. If I ever stop maintaining it, I will make the tools available to the community, so that others can pick up the torch where I left it. People can do amazing things if you let them. :)

To the whole Anarchy Online community, who know me as “Demoder”: Thank you for being truly awesome, and inspiring me to do ‘crazy’ things. I’ll try not to be a stranger!

The New

I plan on learning to play the (musical) keyboard. Exercise more. Be more out-going. Maybe quit a bad habit or two. All of those things that people write on their blogs, and sometimes follow up on. And often don’t. But there’s more.

I was introduced to the world of FreeBSD and Linux back in 2001 or so. The story is long and for another blog post, but the point is this: FreeBSD has been with me in varying degrees for nearly 15 years, or half my life, and I’m now at a point in life where I feel like contributing more than I have in the past. I started earlier this year, submitting a PR and a patch for adding LibXo support to iscsictl(8), and then proceeding with a thorough technical review of the book “FreeBSD Mastery: ZFS” by Michael W. Lucas and Allan Jude. And I feel these are the kinds of things I want to do with the large chunk of my spare time which is labeled “geeky things”. Make contributions to something, hopefully making a positive change for other people in the process.

I’ve been using jails for a long time. For the Linux folks out there, think containers. For the Solaris people out there, think zones. I love how easy it is to manage large amounts of data with ZFS, and how trouble-free it is to share this data with the jail environments, with (nearly) no overhead.

The past couple of years, some awesome people have been implementing a native hypervisor to FreeBSD called “Bhyve”. I love it. I really do. It lets me do things that jails wouldn’t. It will eventually let me retire my VMWare ESXi server. There are a few things which are annoying with it though, but most of those are being worked on by very skilled people. The one feature I miss the most,  is an easy and cheap way to leverage ZFS with no performance penalty. Think of jails with nullfs, or having a ZFS dataset delegated to a jail. That’s super awesome.

My primary use case would be to set up a virtualized file server, leveraging ZFS on the host without ZFS on ZFS or similar overhead, and avoiding the growing complexity of things like NFS configurations. As such, my new winter project is FreeBSD-related. It involves Bhyve (hypervisor), file systems, host/vm communication, and simplifying administration. It involves learning C properly, virtio, FUSE, and FreeBSD kernel internals. Its name is Tunnel File System (https://tunnelfs.io/). The goal is to share files between host and guest like you do between host and jails using nullfs. It aims to be simple to configure/use by the system administrator. It aims to be predictable. It aims to make your life simpler. You can read about it in more detail on the project’s site.

I have a pretty good idea (or so i think right now!) on how to implement it. I know what I need to learn. I know I have a LOT to learn. But that’s okay. I LOVE learning. Almost anything fun in life involves learning something new. So I’ll get started on that. And I’ll probably find out that I can’t do these things the way I wanted, but that’s okay too. It’s a learning experience, and I will get to the finish line eventually. :)

PS: If you haven’t read FreeBSD Mastery: ZFS, and use or plan to use ZFS, you should go read it. Even if you’re not using FreeBSD. If you don’t own it, you should buy it. It’s awesome, and it will look good on your book shelf.