A changelog-ish journal of what's going on.
Sorry for the delayed update. My schedule has been packed lately. Due to a few straggling shipments, hypervisors #4 and #5 will be going online tomorrow instead of today. I'm also catching up invoicing and emails today.
Ugh... sorry guys. I know it's been a long wait. I've been migrating the frontend I had in-progress to a more turn-key solution (Svelte + Vercel). IT WILL BE LIVE TODAY. If you have shipped a drive, you will be able to pull up photos that I took before loading it in the hypervisor. I am also emailing you the photos. (Mainly for those who decided to drop-ship drives -- and therefore have never seen the drives or their serials.)
I'm also building an "equity page". It will show how many payments have been made for your drive(s) and an option to pay them off.
We ordered more drives. I will be reaching out to the next batch of rent-to-own users this week. If you are able to pay for your drive outright, I can prioritize your order. Please let me know if this is the case for you.
We have successfully registered AS399197 with ARIN. We are currently working on securing IP transit and obtaining additional IPv4 addresses. Ideally, this will lower our bandwidth costs.
Some users have mentioned seeing ~4.00 GB of daily bandwidth consumption
for their idle machines. This was caused by local broadcast traffic. The
reports should be more accurate now. Give it a shot by running
I hope everyone had a festive new year's celebration! Couple of items on the agenda:
zz is a quick n' dirty tool to check your instance's data usage / drive temperature stats. Demo video here: zz/demo.mp4.
Now that the year has rolled over, we will be tallying data usage according to our pricing page. I'm about to render this on the website, but for now, this tool accomplishes the same thing.
Take a look at the source code to see which API endpoints are being used.
Feel free to use the endpoints yourself. The backend looks at your source
IP to determine the instance. All of the data exposed is read-only.
In order to modify your machine with
zz, you will need to
set your API key.
Attaching .iso files, triggering hard-reboots, and opening a VNC are all on the roadmap.
# Download compressed binary $ wget https://zfs.rent/zz.gz $ gzip -d zz.gz $ chmod +x zz $ ./zz # Run from source $ git clone https://github.com/radious-subsystems/zfs.rent $ cd zfs.rent/zz $ ./build.sh $ ./zz.js
We are in the process of registering for an ASN and acquiring a set of IPv4 and IPv6 addresses for long-term use.
Hypervisors #4 and #5 will be deployed on January 11th. If you are in this round, you have already been notified. Hypervisor #6 will likely be deployed on January 18th.
A couple of users have been asking for IPv6 support. It will be rolled out this weekend. Your DHCP client will pick up the router advertisements. Disable IPv6 ahead of time if you are not interested.
First of all, I want to wish everyone a relaxing winter break.
I am taking a couple weeks off my dayjob to help enroll new users. Response time via email should be less than 24 hours. Feel free to email directly or make post on our GitHub discussion board.
In the spirit of transparency and longevity, we are open-sourcing our software, database schemas, and runbooks. Safely migrating everything to the public repo will probably take a week or so.
It has been about two weeks since our last public update. In the mean time, we have on-boarded several users and gathered feedback. From here on out, there will be a weekly update posted on Fridays. In addition, minor status updates may be posted on our Twitter.
As we mentioned in our philosophy.txt document, we strive for an API-first system. Everything exposed on our dashboards/graphs can be called directly over JSON over HTTP APIs.
The current API is being fleshed out. But please check out these two endpoints for hypervisor #1:
Similar endpoints exist for user data, (e.g. your own drive temperature stats and bandwidth usage.)
In order to simplify our on-boarding process, we have decided to use email-based auth codes for the time being. If you do not have account, simply enter your email in the login dashboard and one will be created for you. A corresponding API key will be created for you automatically.
(Hover for captions.)
Here are a few roadblocks (and solutions!) we have encountered over the past two weeks.
Shortly after launch, we assembled a 2x 8TB system in a small 1U chassis to benchmark disk speeds and log temperatures within the datacenter. Unfortunately, cooling was an issue. We did not install any fans directed at the drives. (Note: "m2_drive" refers to "machine-2", not the M.2 device standard.) The cyclical spikes you are seeing correspond to the datacenter's ambient temperature day/night cycle. Next trip, I plan to install some ambient temperature sensors in the rack.
These issues have been mitigated with our 4U chassis. In each chassis, we install five pressure-optimized fans in a push-pull configuration. There are three intake fans that push air directly into the drive cages. Each drive is separated by roughly 0.25 inches and the air flows through them, like fins. At the rear of the chassis, two exhaust fans accelerate the airflow.
With the new fans and larger chassis, the drive temperatures are in a reasonable range!
These plots are regenerated every 5 minutes. The backend software will be open-sourced generically at: radious-subsystems/metrics. (Note: the repo hasn't been made public yet. But it will be soon!)
It was bad timing to launch a hardware-centric business in the midst of an ongoing silicon shortage.
In other news, IBM/RedHat have recently dropped their 2029 LTS timeline for CentOS 8.... which is a real bummer. Additionally, their upgrade to CentOS 8.3 broke new OpenZFS 0.8.5 installs. In the interim, we installed ZFS 2.0.0 on several systems and has been working smoothly. The ZFS team has a fairly solid track record of cautious updates and I trust 2.0.0 not to break anything.
Luckily, the OpenZFS devs are quick. Within two weeks, ZFS 0.8.6 now installs on CentOS 8.3 without any additional effort.
I anticipate that by the time Jan. 2022 rolls around, Rocky Linux will provide a viable transition path from CentOS 8.
This is a royal pain-in-the-ass because of Linux raw sockets. (https://superuser.com/a/1457487)
DHCP requests and responses skip over iptables and ebtables rules. I plan on doing a write-up of our solution in the docs later on. We use DHCP to map static IPs to each VM's mac address. It was important to guarantee that outside DHCP queries would not be able to leak in/out of our systems.
Eh... I thought this ER diagram generator made the DB schema look slick, so I'd might as well post it. All of our base entities are tracked in the database. It records their power usage, bandwidth consumption, temperature, etc.
I'm going to port over a project that I've been working on to plot lightweight .png graphs of the systems. (example 1, example 2) Look ma! No JS!
(Axes labels will be added to the zfs.rent graphs of course.)