#Jidoteki Meta

[unscramblegk]

Meta OVA release v13 (1.13)

At the end of November 2019, we quietly released v1.13 of our On-Prem Meta appliance.

This release contains a few security fixes, minor bug fixes, and a major OS upgrade from TinyCore 7.x to 10.x.

Some notable changes below:

Improved token setup

One feature which we quickly implemented in the Jidoteki Admin API was the ability to configure the initial API token on first-use. Unfortunately this has some important security implications for those deploying an OVA in a hostile network (or on a public network).

To fix this, we've enabled the ability to optionally enable our "first-run" feature (disabled by default). This generates a random passphrase as an API token when the OVA is first launched. The random passphrase can only be seen from the console (ex: by an infrastructure admin), or programmatically by an app running on the OVA appliance.

This initial passphrase must then be used to reset the API token and complete the "first-run" process.

Broken GitHub integration

Our GitHub integration was broken since the day we launched it. Although it worked on our test appliance because we weren't testing it correctly. Oops. This is fixed, and the Meta OVA can now receive GitHub push webhook events and automatically pull the latest code changes.

New options in certain API calls

We're often adding new input/output parameters to API calls to provide more functionality for our customers without breaking existing functionality. A favourite of mine is the builddate, which is central to pretty much every API call. Certain API endpoints required you to parse the reply to determine the builddate, so for those endpoints we now return that value directly at the top level of the JSON response, to facilitate chaining API calls which depend on the builddate.

We've also added the ability to suppress the log output when polling for a build's status. The log output could easily reach a few hundred KB, and polling that much data every 5 seconds was not efficient. Now it can be disabled and the status output reduced to just a few KB instead.

Finally, for our customers building multiple flavours of the same appliance (ex: small, medium, large), it is now possible to only build the small OVA, or any combination of flavours through the ova_files parameter on the POST /builds endpoint. This is good for testing when you only need to test one OVA (since technically, the only difference between various flavours are the disk sizes). For production builds, the ova_files parameter can simply be omitted.

The future

Ths year we've begun working on Meta OVA v20 (v2.x) which is so far quite a significant improvement.

Our focus has always been to help our customers deploy Enterprise on-prem virtual appliances, so with that in mind, our next OVA will contain many more enterprise features which can also be integrated into customer appliances.

We've also recently launched a new set of Open Source tools for working with the Meta OVA, so make sure to read our next blog post for details on that.

As usual, feel free to contact us and tell us your story. We'll be happy to help you get up and running with On-Prem Meta so you can start building and deploying your own on-prem appliances.

[unscramblegk]

Six months of unreleased features

Today, we finally released an update to Jidoteki Meta - v1.9.0, which contains a handful of security updates, and only one new major feature: backups.

The security updates are the typical ones you would find in any Linux system: OpenSSL, Nginx, Kernel, etc.

The backups feature makes it possible to quickly retrieve the most important Jidoteki Meta data (logs and database), and restore them on another appliance to get up and running quickly.

Of course, this release does contain a multitude of useful bug fixes and minor improvements, but nothing major like our previous releases.

During the last six months, we had our heads down pumping out feature after feature, adding new whiz-bang functionality, only to drop them a couple weeks upon completion. Why? Because focus.

One of our biggest concerns is adding features nobody will use. Since we're also daily users of Jidometa, we quickly realized that our "new features" were not actually very useful. They definitely had a cool factor, but if we aren't using those features daily, then they're not very important.

Who got the boot?

Some of the features we built include: lsyncd+rsync+ssh replication, auto-update from a remote server, user and group management, S3 object data storage, kexec fast reboots, and server-side HTML page rendering.

Yes, we've essentially been busy spinning our wheels, but the outcome is that our Jidometa OVA remains a 60Mb download, and does its job extremely well, with no overhead or confusion.

What's next?

We do plan to rehash some of those unreleased features into better and more lean implementations, but we'll wait for the right time for that (i.e: when we see a real need for it).

At the moment, we want to focus on a Jidometa appliance which makes our customers (and our own) lives simpler, while helping us work, develop, and build faster. We've listened to feedback and already have a list of important features to implement, but I promise we won't waste our time, this time around.

One of the main things to look forward to, is the ability to manage a cluster of Jidometa appliances from one central location. This would allow builds to be distributed across developer workstations, while allowing everyone to share their builds with the rest of the team. We want to provide the ability to manage the .tcz extensions uploaded to the appliance, so updating something like nodejs can be done without our intervention. Finally, we're looking at integrating SyncThing as part of every OVA (dev option enabled), to allow developers to quickly test and iterate on their app directly in an existing appliance, without going through the full build/update/reboot/test loop (note: we've been doing this for over a year inside our dev Jidometa appliance, and it's been a huge time saver).

Stay tuned for future updates.

[unscramblegk]

Meta OVA release v12

Today we're releasing our On-Prem Meta OVA release v12.

This release contains a few bug fixes, and a ton of awesome new features.

Update Oct. 1, 2018: We shipped a new minor update with a ton of performance optimizations.

Extensions management

A feature added to our TODO list back in 2016, it is now possible to manage the TinyCore .tcz extensions directly through the Meta UI or API. Not only can you view all the existing extensions on disk, but it's also possible to upload new .tcz extensions and start building with them. It is not yet possible to "automatically" download extensions from the official TinyCore repos, but that will be added eventually.

OVA signing

VirtualBox and VMware provide the ability to validate an imported OVA's "signature". This feature is not well known, but it's similar to signed Android or iOS apps. Essentially, it allows the person importing the OVA to confirm who built the OVA, and to ensure it wasn't tampered with in transit. When uploading RSA certificates to the Meta OVA, they will automatically be used to sign all future OVAs.

While working on this feature, we discovered an issue with the way our .vmdk disks were created, so we fixed that as well, which led to the following two additional features:

Auto-generating VMDK disks

With our new approach to creating .vmdk disks, it is now possible to create them on-the-fly for every build. This provides our customers with more flexibility for their OVA configurations, since the disks don't need to be created in advance, by us, anymore. We allow arbitrary sizes in 1GiB increments and the value is specified in the Ansible builder config file.

Exporting to QCOW2, RAW, VHD

Since we were knee-deep in the OVA disk world, we decided to also provide the ability to export the OVA's OS disk to .qcow2, .raw, and .vhd formats.

This is a build-time option and results in a disk1..tar.gz which can be downloaded and easily deployed to various hypervisors and/or cloud providers (such as Xen, AWS, GCP, Azure).. and that leads to the last new feature of this Meta release:

Deploying an AMI on Amazon EC2

We opted not to implement an automated AWS -> AMI export feature for various reasons. Instead, "Cloud documentation" is accessible through from support section, with instructions on how to deploy the RAW OVA disk image to Amazon's EC2 service.

We plan on adding instructions for other cloud providers in our next release.

Of course, this means our On-Prem Meta appliance is also officially available on Amazon Web Services.

Performance optimizations

We weren't satisfied with the v1.12.0 release, so we worked hard to ship v1.12.1 as quickly as possible, with a ton of performance improvements.

To list some of the changes we've made:

trimmed hundreds of DB queries down to one

replaced gzip with pigz for multi-core parallel compression

disabled compression for update packages, at the expense of slightly larger files

lazy-loading images only when entering a specific help docs section

changed the way we create disk1 (NBD vs Loopback)

move certain files from memory to disk to reduce memory usage during a build

With these changes, our own Meta appliance builds are now 2x faster (down from 4 to 2 minutes).

One more thing

We've moved away from updating a wiki with our release information, and built dedicated Release pages and an RSS feed to centralize everything. We'll be rolling that out to all our customers shortly.

As usual, feel free to contact us and tell us your story.

[unscramblegk]

Meta OVA release v11

We recently released v11 of our On-Prem Meta OVA. This release contains a handful of security updates, some great new features, and a huge memory optimization.

Memory

Previously, one build would require nearly 3x the amount of RAM for the size of the build (i.e: a 350MB build would require 1GB). We've made some changes which allow us to lower that requirement between 33% and 50% less RAM for each new build. This means it's now possible to do 2x more concurrent builds, or provision a 2x smaller OVA.

Features

The biggest new feature is the official integration of GitHub webhooks. The Meta OVA can now directly accept a GitHub push event, and can automatically clone/pull the repository changes prior to making a build. This removes the need to manually push Git changes to the appliance, and opens the door for automatically launching builds when a webhook is received.

We've also added a few new API endpoints and parameters, with no breaking changes.

Security

The usual suspects, such as Nginx. OpenSSL, and Qemu, have received security updates.

Of course, we've incorporated many bug fixes implemented over the last few months, particularly in regards to race conditions with certain asynchronous tasks.

We're already preparing for our v12 release, which will include a few long-awaited and often requested features. Stay tuned for more updates.

[unscramblegk]

Inside Jidometa: No more caching

In [part 2 of Inside Jidometa], we discussed our implementation of caching for JSON responses.

The initial goal was to solve the “slow page load” issues, but after a few months of running this in production, we eventually found ourselves battling more and more odd race conditions.

In retrospect, caching was a bad idea

I knew this from the start, but we were quite confident in our ability to implement a proper cache invalidation scheme to remove any possible race conditions.

The reality is that as our app grew, new features were added, and those features introduced more places where race conditions would surface.

Finding the root cause

I tasked myself with the goal of finding the exact root cause of our slow page loads. This required the disabling of caching, and then optimizing of SQL queries.

Our SQL backend was performing simple and multiple SQL queries per page load (~30 queries per load). This was extremely inefficient and un-necessary.

I identified every page which had such issues, and managed to reduce and combine queries down to 2 or 3 queries per page, thus significantly decreasing page load time.

Some numbers

Without caching, loading a page (or API call) with ~700 builds would previously take nearly 30 seconds.

After my optimizations, this was reduced to 300 milliseconds, or 0.3 seconds. Those changes were sufficient for us to completely remove our JSON caching implementation, and also fixed other random issues/race conditions as a side-effect.

The lesson here is: caching sucks. Don’t do it unless.. just don’t do it. It’s pretty much always a hack, and will eventually somehow someway end up biting you in the rear. Even if you’re a seasoned expert in race-condition handling and cache invalidation, other things which you didn’t even know exist will surface and cause headaches and random failures -- because of caching. The typical approach to solving caching issues is to add more caching (see Intel CPUs), or to perform more ugly hacks to work around the caching issues.

I’m a fan of using no caching, and then moving on with my happy life ;)

Moving forward

We released Jidoteki Meta v1.8 at the beginning of June, which included these changes and a few other extremely useful features.

We’re continuously working to improve Jidometa in order to provide the best, fastest, and most reliable way of consistently building virtual appliances designed to run on-premises.

As usual, feel free to contact us if you’re in the process of or thinking of providing your app to enterprise customers. We’ll be more than happy to help.

[unscramblegk]

Bricked virtual appliances

We recently released v1.7.0 of Jidoteki Meta, and with it came a little unexpected surprise: it bricked a few customer appliances.

In this post, I’ll explain how not to brick a customer appliance, as well as a few techniques to recover when it happens.

Oops!

Our build process is entirely automated - duh, that’s why it’s called Jidoteki (automatically, in Japanese) - but our release process isn’t. There’s a series of roughly 40 steps on a long checklist prior to making a Jidometa release. Most tasks are automated and only take a few seconds to complete/validate, but others are manual and require a bit more time.

In our most recent release, we changed something which shouldn’t have been changed (the default /etc/fstab file), which prevented the /data disk from being mounted - but only on older deployments.

Problems compounded

To make matters worse, our fstab blunder had the cascading effect of not starting Openssh, Nginx, or the Jidoteki API services, which made remote administration impossible (thus impossible to update the appliance).

A missing validation

What we didn’t realize was that our release process didn’t include a step to validate our builds/updates against, you guessed it, older deployments! We did test the updates against slightly older appliances, but not against the oldest ones - the ones which some of our great customers were still running.

No worries though, we’ve added that to our process/checklist and can guarantee it won’t happen again.

Recovery

Luckily (and stupidly), our initial appliances shipped with a default root password (known only to us). We were able to provide instructions to login via the Linux console, start the necessary services, and then access the UI to upload an update package containing the fstab fix.

In our latest appliances, which are slightly more secure, there is no default root password anymore. In fact, logins are completely disabled, even by SSH. We provide the ability to change the admin password via the console GUI, but that only provides access to the files in /data (customer files), not root.

Recovery without password

We’ve customized the boot menu to provide just enough time to modify the boot command. Simply removing the ,/boot/rootfs entry will load the default TinyCore installation, which includes a root user with no password! omg! That’s a good thing. It means it’s still possible to fix a bricked appliance.

I know, it seems quite insecure at first glance, but the reality about a Linux virtual appliance is that, anyone with access to the host machine can get root. There’s no way to prevent that.

Other techniques include mounting the disk(s) in another appliance, or booting from a recovery ISO/CD.

In the end, there’s so many ways to obtain root access and to recover from such issues, that there’s no real point in preventing your customers from having it. Security through obscurity is a wasted effort.

Moving forward

To avoid bricking customer appliances, we’ve decoupled essential services from the boot process. They will start no matter what, and always provide remote administration capabilities.

Secondly, the design of Jidoteki appliances makes it very easy to either obtain root access (from the console, obviously not over the network), that we don’t need to worry about the consequences of a “bricked” appliance. The customer data is never touched by the updates, and they’re free to obtain access and perform a manual recovery procedure.

We’re working on an automated process which actually validates (integration test?) an appliance once it’s built, against a set of criteria (ex: does X service start, are disks mounted correctly, etc). We’ve already written the test suite and have been using it for a while on newly built appliances (not updated ones). Our last step is to integrate it to Jidometa and automatically run it against the builds/updates when they complete.

Finally, wouldn’t it be nice for an appliance to self-heal? Yes, it would, and we’re working on just that. The idea isn’t a new one (I implemented something similar in 2009 while working on a custom Linux OS), but essentially rather than overwrite the existing OS during an update, we could rename the file and have a second “recovery” boot option which boots from a working version when the primary one fails.

Contact us

As usual, if you’re planning on providing your On-Prem or Installer-based application to enterprise customers, contact us so we can discuss the details of your setup.

[unscramblegk]

Jidoteki: Updated pricing, easier to get started

We’ve recently updated our pricing to make it even easier to get started with Jidoteki.

Having celebrated 5 years of being in business a few days ago, we decided to make our service more accessible to companies, by providing a new pricing structure.

Previous Setup plan

Previously, we offered only one “Setup” plan to get started with Jidoteki. It included lots of custom work, consulting, and onboarding in order to get a custom appliance up and running. It’s great for companies with lots of custom requirements for their software to work on-premises.

One recent example involved us converting a Java-based installer application to an on-prem virtual appliance with custom tools and scripts.

Splitting the Setup plans

We’ve renamed the original Setup plan to “Pro Setup” plan, which includes a fully customized virtual appliance covering all your needs for the perfect on-prem appliance. It includes customizations at every level to ensure you can easily transition from Installer/SaaS -> to On-Premises.

Today, we’re also announcing the “Business Setup” plan, which includes a slightly more generic virtual appliance. We still provide the same open source tools, scripts, and features of our rock-solid enterprise virtual appliance, along with its great management features. The only difference is we decreased the amount of custom work to finalize the OVA. Of course, we’ll still customize a few things to ensure your app actually works as intended.

Benefits of the Business Setup

The main benefit of the new setup plan is that we can have the appliance ready in just a few hours. Typically with lots of customization, the time to completion can take between one week and up to one month, or more.

... ready in just a few hours

To be eligible for our Business Setup plan, we require your stack to conform to the typical software stacks such as LAMP, LEMP, MEAN, etc. With those types of stacks, we can get you up and running real quickly, and you can start providing your on-prem OVA to enterprise customers with very little delay.

Another great advantage is that it’s always possible to upgrade to the “Pro Setup” in the future, if the need arises.

Moreover, Jidoteki Meta will work just as well with either Setup plan, so you’re still free to maintain full control of your builds, on your own premises.

Get your feet wet

I want to extend an invitation to companies to try our service. We can pretty much guarantee you’ll love the end product and will want to continue working with us in the long-term. It’s a win-win situation, as you’ll also gain the ability to serve enterprise customers (which translates to $$$).

Contact us and we’ll be more than happy to discuss your situation and requirements.

[unscramblegk]

Inside Jidometa: Loose coupling of customer data

[Read part 3 of Inside Jidometa]

One of the major differences between Jidoteki appliances, and most other setups, is that we isolate customer data and ensure it’s not tightly coupled to the system.

Looking at the graph above, we can see the customer data is stored on disk 2, while the OS/apps/kernel are stored on disk 1.

Issues with tight coupling

Adding disks to a virtual appliance is quite trivial. Moving them to another appliance is just as simple. We know this, so we made a plan to support it.

In a typical non-Jidoteki appliance, databases, log files, application settings and data are stored on the same disk. This tight coupling introduces problems which are almost impossible to workaround once deployed:

Failed OS updates could accidentally wipe all customer data

It’s impossible to expand the disk once it’s full

It’s difficult to move customer data to a newly deployed appliance

Customers may not have full control of their data

The disk must be sufficiently large from the start

Our disk setup

Jidoteki appliances use a minimum of two disks. The first disk is quite small (8GB sparse disk) and used exclusively for storing the system and custom settings (ex: network settings). The small OS disk means importing an OVA for the first time is very quick. The second disk is sized based on our customers’ requirements, but typically range between 10GB and 2TB (sparse disk), depending on the application. In any case, the customer can opt to not import the second disk, or simply delete it and re-create it with whatever size they want.

The best part about disk 2 is we deploy it using LVM, which allows anyone to attach a third, fourth.. eighth disk (max 8 disks) of virtually any size. If a deployed appliance uses 2TB and needs to increase to 42TB, an additional disk can be added and the appliance will automatically grow the LVM disk.

Other benefits to loose coupling

By dedicating the entire disk to the customer data, we can provide full permissions to the appliance admin (the customer) to login via SSH and modify any file(s) on that disk. The customer controls their data and remains free to alter it how they want (at their own risk, of course). We do this in our own on-premises virtual appliance, Jidoteki Meta (Jidometa), and guide our customers in doing it this way as well.

Another benefit is having the ability to detach the disk from the appliance, deploy a new appliance on a different (physical?) server, and then re-attach the disk. It’s not as fancy as live migration, but moving disks is effortless compared to transferring data with rsync or some other wild migration scheme.

Of course, if it’s that easy to move data, that means it’s just as easy to backup the data, or create a fault-tolerant setup for disaster recovery scenarios.

Finally, as mentioned in a previous blog post, dedicating the second disk to customer data means it can also be replaced by network storage, such as NFS, AoE, iSCSI, or NBD. I can’t imagine the logistics (and coding nightmare) of having to support network storage when the application was designed for storing on the same disk. With Jidoteki, we simply mount the network storage to /data, instead of mounting the local disk 2 to /data. No software changes required. Easy.

Moving forward

As always, we’re happy to discuss your requirements if you plan on providing an on-prem appliance to enterprise customers. We’re familiar with enterprise requirements, and try our best to provide the most rock-solid on-premises virtual appliances out there. In a few days we’ll be celebrating 5 years of doing this stuff. It really is our specialty, and we’d love to help more companies go on-prem, the right way.

Feel free to contact us, or signup for our new monthly newsletter to stay up-to-date with what we’re doing.

[unscramblegk]

Ansible tweaks to get 500% faster runs

In this post, I’ll explain how we significantly decreased the time it takes to create an appliance build in Jidoteki Meta (Jidometa). We managed to bring the build times down from 6 minutes to a little less than 60 seconds.

Profiling

Before trying to optimize Ansible runs, it’s best to start by profiling the tasks to see what’s taking the most amount of time. The Jidoteki Meta virtual appliance is quite small (~60MB), so we decided to profile a customer build instead, using this simple plugin (example output):

Various approaches

A simple search for “faster ansible” or “speeding up ansible” returns many results, which all provide rehashed versions of SSH/remote optimizations. Unfortunately they are useless for us since we use ansible in local mode. Fortunately I did find one gem hidden at the bottom of an Async post, which is the basis for our huge speed gains (ironically, it has nothing to do with async).

Why not async?

The Ansible async module is quite complex, and requires another module (async_status), does polling in the background, and is a big ugly hack that’s more suitable for SSH/remote ansible runs. It wasn’t designed for us, so we didn’t include it in our optimization schemes.

Copying files

There is a known “bug” (feature) in Ansible since the early days: the copy module is terribly slow. Their recommended approach is to use the synchronize module which uses rsync behind the scenes, but that is also just a big hack.

Our 6 slowest tasks involve looping through a long array of with_items (~120 entries) and copying files from source -> destination. The gem was in this line:

','.join(dependencies)

If we could concatenate our list to a space-separated string, and then transform our task into a shell task (instead of copy), we could make one single call to the cp command, containing all 120 files. The processed command looks something like this:

cp file1 file2 file3 fileN file120 /dest/dir/

Without the need for looping or using with_items.

Previously, those 6 tasks went from taking a combined 180 seconds, to a total of only 3 seconds.

Wow!

Too simple

Of course, that solution was just too simple and probably should have been there from the start, so we made up for it by optimizing something a bit more complex ;)

At the end of our Ansible run, we generate a rootfs which is really just a CPIO archive containing the files/directory structure from the Ansible output. Since our Ansible repos are versioned in Git, every single Ansible run using the same git commit ref should produce the exact same rootfs (see our Reproducible Builds post). Of course, some minor differences exist in regards to file/directory modification times, and randomly generated passwords/keys, but everything else remains the same.

What I realized, was that if we can find a previous build that used the exact same git commit ref as the current build, then we can re-use the entire rootfs instead of re-generating it from scratch. Well, the Ansible role for generating the rootfs, which previously took 120 seconds, was reduced to only 8 seconds.

The sum of its parts

With all these optimizations in build times, we effectively managed to obtain a 500% speed increase for the final builds created with Jidometa.

Going back to our own Jidometa appliance build, which previously "only” took 38 seconds, now takes exactly 5 seconds.

I’m laughing as I type this.

Dogfooding

I may sound like a broken record, but something really magical occurs when you use your own product on a daily basis. We discover and implement solutions which solve our own problems, which happen to also solve our customers problems.

Our focus is on creating the most solid enterprise-ready on-prem virtual appliances. We work exclusively with customers who want to distribute professional, complex applications to the enterprise, in a self-contained file, without having to figure out all the tough parts.

Feel free to contact us, and we’ll be happy to discuss your requirements.

[unscramblegk]

Automatically updating a cluster of offline appliances

In this post, we’ll describe a new feature in Jidoteki Meta, which makes it possible to generate special update packages aimed at updating a cluster of appliances on an internal network.

A solid foundation

This feature builds on top of a recent feature, Full update packages, which can be applied to ANY appliance regardless of their version.

Some of our customers deploy multiple types of appliances to their enterprise customers. In those situations, where you have a cluster of node (slave) appliances and one or more server (master) appliances on your internal network, it can be quite bandwidth intensive to perform online updates. If the node appliances don’t have internet access, manual updates are required, which is troublesome when each appliance is on a different version.

Distributing updates

To solve this problem, we can now create a special update package for the server appliance which includes the Full update package of the node appliances.

This makes it possible for the server appliance to hold and distribute updates to each node appliance, acting as an update server, allowing the node appliances to automatically update themselves to the latest version without requiring internet access or manual intervention.

Creating the updates

In Jidoteki Meta, the option is labeled as the “Node source”, which is the build ID of any previously built appliance. When building a server update package, and a node_source is selected, that build’s Full update package is added to the build’s OVA in /mnt/sda1/boot/ , and it’s added to the build’s update packages.

Complexity reduction

From the outside, it seems like a complex feature, but in fact it’s very simple and extremely useful in situations where an admin has more than a handful of appliances to update.

With Jidoteki, we build our solutions to solve problems which not only our customers have, but also problems their enterprise customers have.

If you’re currently distributing your software on-premises, or planning on doing that sometime soon, please feel free to contact us, as we’re focused exclusively on making the absolute best virtual appliances and automated offline update process. We’ll be more than happy to discuss your situation and see how we can help.

[unscramblegk]

New flexible options for updating a virtual appliance

Don’t worry, we haven’t changed our update process since the last post (part 4), since we’re certain it’s the absolute best  approach. On the other hand, we’ve added two new features to provide even more flexibility.

Diff update packages

Our previous update packages only had the ability to apply a binary diff to existing files. The advantage of a binary diff is the update package’s file size can be dramatically reduced, since it only contains a delta between two versions.

The problem with binary diffs is apparent when an appliance is several versions behind the latest version. Updating to the latest version would require one to apply each binary diff update, sequentially, and in order. This lead us to creating bundle updates.

Bundle update packages

A bundle update package wraps multiple binary diffs into one package. That bundle detects the version of the appliance, and then successively applies each update up to the latest version. This is great as it simplifies the task of applying multiple updates to an outdated system.

The problem with bundles is they can occasionally grow quite large, sometimes even surpassing the size of a full appliance. In that case, a different strategy is needed.

Full update packages

Our latest creation: a full update package contains the exact same files found on the boot disk of an appliance. The update process atomically overwrites each file, rather than applying a binary diff. This makes it possible to update ANY appliance with just one update package - regardless of what version it was on. And yes, it’s backwards compatible with every appliance built with atomic updates (~2015).

Of course, we use encryption and cryptographic signatures to ensure an update package can’t be applied to the wrong system (ex: company A’s update packages won’t work on company B’s appliance).

Since the update package doesn’t contain the actual VM disks, its filesize is guaranteed to always be smaller than the full appliance. It’s a great solution that fits perfectly between Diff and Bundle update packages.

Jidoteki Meta

All of this can be managed directly by the Jidoteki Meta on-prem virtual appliance. We’ve since added the ability to generate one or more types of update packages, all at once. This provides the freedom to test different update approaches, and provide various packages to customers based on their needs. The added flexibility helps ensure customers can always update their appliance down the line.

As always, we’re available to help with the creation and updating of on-prem virtual appliances. If you’re currently shipping your application as an Installer, or as a SaaS, and you want to provide the absolute best experience to your enterprise customers, then feel free to contact us, we’ll be more than happy to help.

[unscramblegk]

Frequently Asked Questions about our service

Over the years, we’ve be able to compile a list of the most frequently asked questions regarding our service. This blog post has recently been summarized on our website, so I’ll provide more details here.

Business questions

Q: Why did you get rid of the Jidoteki SaaS? A: The Jidoteki SaaS we were offering was shutdown sometime in 2015, when we decided to focus on providing a Managed Service for building virtual appliances. The first and most obvious reason, was that we followed demand. Most of our customers didn’t want to, or were unable to do all the “provisioning” work on their own - that is, the step of actually configuring the appliance with Puppet/Chef/Ansible. One of the issues of hosting a SaaS for Jidoteki was that virtual appliance builds required a large amount of memory, disk space, bandwidth, and CPU. The cost was high for us, and would end up being higher for our customers. Downloading large files from our servers in Tokyo introduced latency issues as well. Finally, being a small bootstrapped company meant we needed to avoid supporting servers 24/7 - aka being on-call.

Q: Your services are not exactly cheap. Why? A: The primary reason to use Jidoteki as a managed service is to offload the most difficult part of going on-prem: creating and updating the appliance, to a team of experts. In 5 years of running this business, we’ve accumulated a plethora of knowledge which we can transfer directly to our customers, in the form of solutions to the most difficult edge-cases, and with an unbelievably fast turnaround time-to-ship. With everything we’ve built so far, depending on the complexity of a customer’s setup, we can typically have a beta virtual appliance ready in less than one week, containing all the features of a professional updateable and secure on-prem virtual appliance. It takes (and took) time to do this work, and so we charge for that.

Q: What’s included in the Setup plan? A: Other than what’s already listed on the site, the Setup plan is just a consulting contract allowing us to build your first virtual appliance (OVA) from scratch, with all the features required for you to ship to production. It also includes the Ansible scripts to build the custom rootfs (see Technical Questions below). The scripts are openly licensed to you, so you’re free to modify and re-use them as you see fit. Since the Setup plan is only for the initial OVA, you will be left to manage the updates of that OVA on your own, along with the creation of new OVAs. We can provide suggestions on how to create updates and new OVAs on your own, but it’s also a lot of work, which is why we offer the Support and Meta plans.

Q: Are the Support and Meta plans really necessary? A: A full project is typically completed in one month - from the day we start work, to the day you’re shipping to your customers. During that time, we constantly iterate on the appliance until it’s built exactly as needed. The Support and Meta plans are completely optional, and only really needed once the appliance has been released. The Support plan includes access to our team for help with updating the appliance (creating Update Packages), as well as updating dependencies such as OpenSSL, handling weird edge-cases, and adding new features such as NFS support, etc. We’ll do all that work for you under our Support plan. The Meta plan is a self-serve DIY solution, also known as Jidoteki v3. If you’re comfortable with updating the appliance dependencies, and want to host everything internally, the Jidoteki Meta appliance is the best option. It runs entirely offline, and can easily be integrated into a Continuous Integration system such as Jenkins, to automate the creation of your Update Packages and OVAs. This is a somewhat repackaged version of the original Jidoteki SaaS, although it’s quite different behind the scenes (we got rid of Ruby and NodeJS).

Technical questions

Q: What is the difference between a rootfs and an OVA? A: The rootfs is a single cpio archive built from scratch, which contains custom init scripts, custom config files, boot scripts, and other things which we build specifically for your virtual appliance. The rootfs also contains all of your app’s software dependencies (ex: nodejs and the node_modules). The rootfs is the only thing that’s really “custom” about your OVA, other than the OVF metadata file - but that’s just a detail. The OVA is just a standard tar archive which contains the OVF file, and 2 vmdk disk images. The first disk image is the bootable disk, which contains among other things: the rootfs, the unmodified base OS (TinyCore Linux), and a vanilla Linux kernel. To update an OVA, you can boot it and simply replace the existing rootfs, but that process is manual and error-prone.

Q: How is Jidoteki better than InstallAnywhere, Bitnami, Replicated, Gravitational? A: InstallAnywhere is garbage. Garbage in, garbage out. If you enjoy employing (and paying) a full-time engineer just to build your Installer or OVAs, then that software is the way to go, but you’ll deeply regret it in the short and long-term. Bitnami is focused more on creating Open Source stacks - which is admirable since we love Open Source - but their process is not aimed at building secure, hardened, on-prem, small-footprint, easy-to-update appliances the way ours are built. Case in point: their base OS is Ubuntu. Replicated and Gravitational are not really in the same category as us, since they chose to redefine the term “on-premises” to mean “on Amazon’s or Google’s cloud”. They do seem to provide some nice features for management of appliances, and we respect them for that, but their solutions include a hidden "feature”: vendor lock-in. Choosing to work with those closed-source SaaS platforms locks you into their technology, and their vendor platforms, and their pricing models. Finally, they both focus on Docker at the base of the system. If you’re not using Docker and don’t care for it, then you will be forced to use it with them. What we provide - which is a much more sane solution - is a 100% open source platform for your appliance, without Docker, and with a very tiny OS to boot. Every single tool, script and software included in your appliances is open sourced. This allows you to validate that it does what it’s supposed to do (no telemetry or analytics phoning home, no timebombs or illegal data logging). It also provides complete freedom to modify your solution as you see fit, as opposed to being locked to our way of doing things. That in itself is the most valuable tool for any business. Our Jidoteki Meta appliance does contain some closed-source software, but none of it is obfuscated (we distribute the sources), and as our customer, you’re more than welcome to look at the code and suggest improvements/bug fixes if you want. Again this benefit is a long-term one, where even if you don’t “want” to modify our code, if we happen to stop working on Jidoteki Meta, you “can” continue using it.

Q: Why do you dislike Docker and systemd? A: Garbage in, garbage out.

Q: How do you handle updates when there’s no internet? A: I don’t want to say it’s our secret sauce, because we’ve blogged about it here, here, here, and here. Essentially, by running the OS entirely in memory, it allows us to atomically update the system’s OS, kernel, and custom rootfs, without affecting the running system. A reboot is required to “activate” the changes, but once that’s done, they are up and running with a fully updated system. To handle offline updates, we package the update files into an encrypted tar package, and distribute that. The package can be used to fully update the system to the latest version, without requiring internet access. If an update happens to fail, it is automatically reverted and “nothing changes” to the running system.

Q: How do you handle database migrations? A: Since the OS is only “updated” on reboot, all DB migrations should run on boot. We use shmig to handle the DB migrations after a reboot, and it has worked flawlessly for us since the start. It’s your job to ensure the DB migration scripts don’t fail when run. Unfortunately that’s not our area of expertise, but the following answer might help.

Q: Can you revert a failed migration? A: Of course we can! By default, all our appliances ship with a second hard disk for storing the database and other persistent data. We ensure to include LVM, to allow you to perform a database snapshot right before running the database migration. If the migration fails, you can simply revert the snapshot and go back to a working state. In any case, if a migration fails and leaves the DB in a bad state where your application won’t start, the included (open source) Jidoteki Admin tools will allow your customers to either upload a new Update Package, to debug the appliance, and even login via SSH to perform manual system administration.

Q: Can you do hardware builds? A: Yes! We also have a set of Ansible scripts to build a system for the Beagle Bone Black and Raspberry Pi family. The final system is almost identical to a virtual appliance, the only difference is rather than having a custom rootfs layered on top of the unmodified OS, we’re forced to merge them all into one rootfs (because the ARM bootloader can’t chainload them). Of course, the binaries are all compiled for the armv7 architecture instead of x86 or x86_64.

Q: What’s the typical size of a final OVA and Update Package? A: Our latest Jidoteki Meta appliance is 57MB, and Update Packages can vary wildly, but typically between 100KB and 5MB. The Update packages contain binary diffs of the previous OVA files, which allows them to be much smaller than a full OVA. Of course, if you’re updating a Java application and constantly making changes to the file/directory structure, adding new dependencies, etc, then Update Packages can grow quite large - to multiple hundred MBs or more. There’s no rule of thumb for the OVA size, but our default appliance with all the base dependencies, kernel, etc end up at roughly 35MB. Everything else will depend on your application size, and its dependencies.

Misc questions

Q: What does Jidoteki mean? A: In Japanese it’s written 自動的（じどうてき）which means “automatically”. Our plan has always been to automate the build of virtual appliances, through scripting, continuous integration, etc, so we thought this name was apt even though it sounds a bit weird. We’re also based in Japan.

Q: Is Jidoteki Meta going to be open sourced? A: Yes, we plan to fully open source it eventually. For the moment we’re releasing parts of it at a time, but we haven’t written full documentation or tests for all the tools yet. It would be irresponsible of us to open source something with no tests and no documentation ;) We would like to fund development of Jidoteki Meta by selling it first, as a “partially” closed source solution, and within the next year or two once the product is mature, we would be happy to release it publicly for the community to enjoy and use/maintain collectively.

Q: What other benefit(s) do you provide? A: The greatest benefit is that you can focus exclusively on your software, while we (experts) handle your on-prem builds and setup. This has an important impact on your engineering team, since you don’t need to dedicate any resources to something that’s not core to your business. This translates to huge cost savings. Moreover, with your newfound ability to quickly iterate on your appliance and updates, you can now quickly provide updates for your customers, new versions, and new features - which translates to increased revenues. In the end, the cost of working with us will have an important positive financial impact on your business, and as long as that’s true, it’s win-win for both of us.

I hope this long and detailed FAQ answers questions which many of you may already have regarding building on-premise virtual appliances. If you’re interested in getting started with Jidoteki as a managed service, please feel free to contact us, and we’ll be happy to discuss your requirements and see how we can help. Thanks for reading!

[unscramblegk]

Inside Jidometa: Caching JSON responses

[Read part 1 of Inside Jidometa]

We recently implemented a feature which greatly improves the browser experience in Jidoteki Meta (Jidometa).

In the past...

The Jidoteki Meta UI is a simple JavaScript layer on top of a PicoLisp REST API. The API returns JSON responses for every successful request, but occasionally we would see rather slow navigation between sections of the UI.

Certain responses tend to take more time as the size of data increases. There is no way around this, since many tables and entries need to be queried on every request, or so we thought..

The present...

To “solve” this problem, we decided caching responses for common GET requests would be a good approach.

Our idea was to only cache the most queried items (the list of all builds, and the details for each build). By caching this data, it makes it possible to quickly navigate between different builds and the full list of builds, without making any calls to the API.

Caching gotchas

Before implementing any caching solution, it’s important to fully understand the gotchas that accompany it. In most cases, the most important requirement is to never serve stale data. To invalidate the cache requires us to ensure any cached elements which are affected by a change, will immediately be evicted from the cache.

Another gotcha, is avoiding the temptation to add yet another layer of complexity to the stack. We always aim to do things as simply as possible with the least amount of software and overhead.

We needed the ability to easily disable caching or wipe the cache if needed, and we needed to ensure the browser’s own caching features didn’t interfere with ours.

Caching as simply as possible

Since our Jidometa virtual appliance (as well as our customer appliances) run entirely in memory, it was easy for us to assign a temporary directory for storing cached responses.

We store responses in /tmp/jidometa/cache/ - which is not actually “on disk”, but rather an in-memory filesystem (tmpfs). Those responses are regenerated on every GET request to the API. Fortunately there’s no overhead for that.

Each cached response is a simple .json file which is absolutely identical to what would have been served by the API. The files are named according to the build ID they represent, so there’s only ever one cached response per build.

It’s hard to get any simpler than that.

Serving the cached responses

We use Nginx as a frontend for static files and the API. Below are examples of the two Nginx directives we added for serving the above cached files:

location /cache/ {  add_header Cache-Control no-cache;  alias /tmp/jidometa/cache/;  try_files $uri @nocache; }

location @nocache {  rewrite    /cache/builds/(.*)-details.json /builds/$1/details break;  rewrite    /cache/builds.json /builds break;  proxy_pass http://jidometa; }

As an example, a request will first try to serve the /cache/builds.json from /tmp/jidometa/cache/builds.json - if the file doesn’t exist, it tries the @nocache directive. That directive will then send the request to /builds on the API, which will return a fresh response (and generate a new cached builds.json).

We also send the Cache-Control: no-cache HTTP header to ensure browsers don’t cache the cached response... but here’s the thing, browsers WILL cache the response regardless, except the no-cache directive will force the browser to use the Last-Modified or ETag headers in future requests, to revalidate whether the cached entry was changed or not. Well, assuming browsers handle those headers according to the RFC specs ;)

If the cached file hasn’t changed, the browser won’t fetch the entire file again.

Invalidating the cache

Builds are linked to their parent build (if any), as well as their child builds (if any).

When a certain build is modified (ex: the build’s status is changed from unreleased -> released), its cached response needs to be invalidated. We decided the simplest and easiest way to invalidate a cached entry is to delete the build’s cached response, and all the other responses linked to it (its parent and children).

Here’s the code we use for invalidation (PicoLisp):

(de remove-cached-file (Filename)  (call 'rm "-f" (pack "/tmp/jidometa/cache/" Filename)) )

(de remove-cached-build (Build)  (remove-cached-file (pack "builds/" Build "-details.json")) )

(de remove-all-cached-builds (Builds)  (mapcar '((N)            (remove-cached-build (; N 1)) )            Builds ) )

A call to (remove-all-cached-builds) with the list of Builds as the only argument, will handle removing each cached response individually. What’s nice is we can also use (remove-cached-file “builds.json”ˆ) to remove the entire builds list from the cache.

Of course, we wrapped all those calls in a simple (invalidate-cache) function which can be called anywhere in our code; anywhere that updates an existing build, or inserts a new one.

If a build has one parent and three child builds, then a total of six cached responses will be removed from /tmp/jidometa/cache/, including the builds.json (list of all builds). This is quite an aggressive cache invalidation strategy, but it ensures there are never any stale responses, and a new cached response will be regenerated as soon as a request is made for the “missing” file. Easy.

Issues with caching

We have yet to encounter any issues with cached responses, but if they do occur, it’s quite simple to identify exactly which response is problematic, as well as delete it manually from the cache directory. It’s also simple to completely wipe the cache from the system and start over from scratch. This happens after reboot anyways.

The main issue with a fresh cache is the first page load will be slightly slower than the others. It might take a few clicks to “warm” the cache with fresh responses, but considering the use-case for Jidometa (on-prem virtual appliance), we can live with that. Had there been a requirement to support hundreds of concurrent users, we would have considered an in-memory cache with persistence, such as Redis... maybe.

In conclusion

I would go into further detail, but really there’s nothing more to it. Serving cached JSON responses the way we do is somewhat of a “poor man’s solution”, but the lack of overhead and the simplicity of the implementation are a net win for everyone.

Stay tuned for part 3, where we’ll provide more details regarding the Jidometa internals.

[unscramblegk]

Announcing Jidoteki Meta: on-prem appliance for building on-prem appliances

We created our own on-premises virtual appliance - called Jidoteki Meta - designed to build on-premises virtual appliances.

Here’s an overview of the current features:

Console boot GUI

Before you can access an on-premises appliance, you’ll need a way to configure the network settings. This basic console boot GUI provides a menu for managing those settings prior to accessing the web UIs.

Admin UI and API

A simple open source Admin UI and API can be used to monitor the appliance’s health, to manage network and security settings, and to apply system updates. The REST API allows you to automate tasks using Jenkins or other tools of your choice.

Jidometa UI and API

The Jidometa UI and API are not (yet?) open sourced. They use the same core as the Jidoteki Admin API, but provide additional API endpoints and a separate UI for creating and managing builds, and other settings. Of course, it’s also possible to automate the entire build process thanks to the REST API with tools such as Jenkins.

Integrations with Slack

We’ve only scratched the surface of what’s possible with Jidometa (Jidoteki Meta). We’ve recently included a Slack integration which allows the build status to be pushed to your Slack channel. When a new build is created, when the status changes, when the download files are ready, you’ll know about them right away, and can download them directly from the Slack channel UI.

More to come

We’ve got quite a few more features planned for the near future. We’re really happy that our customers can now maintain full control of their entire on-prem virtual appliance builds, in their own local environment. Of course, we’re glad to continue providing support for one-off builds and custom requests.

Why did we do this?

One of the biggest issues our customers faced with the Jidoteki service, was the turnaround time for creating/downloading/testing updates and new OVAs. Our servers hosted in Tokyo introduced additional delays for downloading files which were occasionally between 500MB and 1GB. Being a fully-managed service meant we also had to be available to create the builds on our customer’s behalf.

In the early days, we provided Jidoteki as a SaaS for our customers to use on their own. The transition to fully-managed directed us to use the SaaS on their behalf, but the SaaS service still meant our customers relied on our infrastructure to be available 24/7.

Going on-prem

Making Jidoteki available as a on-prem appliance was a logical and obvious decision on our end. It provides the benefit that our customers can create and manage their builds independently from us (i.e: without relying on our availability). It also completely eliminates file download latency, since the appliance runs in their own local private network, behind their firewall. For us, it means we can focus exclusively on Jidoteki and supporting customers, rather than managing a complex and resource-intensive production infrastructure.

The biggest benefit though, is that we’re dogfooding our own product. By having our own on-prem appliance, our customers directly benefit from having the latest updates and improvements, since our customer appliances, and our Jidometa appliance share the same base OS and packages. If there’s an issue, we’re the first to know and test it on Jidometa, and we can quickly move those changes upstream to our customer’s appliances.

Get started

We’ve been doing this Jidoteki thing for over 4 years (plus my time working on GitHub Enterprise). I’m confident to say that we’re experts at this, and our constant iterations on Jidoteki have greatly helped us progress and provide better and better solutions for going on-prem.

If you’re interested in providing secure, offline, on-premises and easily updateable virtual appliances to your enterprise customers, please contact us to discuss your situation. We’ve converted desktop software, bash scripts, SaaS, and even installers over to the virtual appliance format.00

Our setup fee starts at USD $995 (depends on your requirements), and we’ll be more than happy to discuss your needs and requirements.