sumanthvepa

[sumanthvepa]

Live Blogging my Open Source Contributions

I thought I’ll try something new this December 2020. I’ve decided to live blog some of my open source development project. Today, and tomorrow I’m building a really simple command line tool to convert CSV files to arbitrarily complex JSON files. The idea is that this forms the basis of an open source ETL toolset aimed at #devops engineers eventually. 

[sumanthvepa]

Python: Set Literals

Discovered the syntax for set literals in python today. Should have been obvious. But it didn’t occur to me that sets were a native type in python.

aset = {'a', 2, 'b'} # You can perform common set operations such as union and interection s0 = {'a', 'b'} s1 = {'c', 'd'} s2 = {'b', 'c'} s = s0 | s1 # Set union produces {'a', 'b', 'c', 'd'} s = s0 & s2 # Set intersection produces {'b'}

Look at the python manual in the section on built-in types for more information.

[sumanthvepa]

MacOS: Automounting NFS shares

If you come from Linux, the way MacOSes’ does network mounts may seem baroqe to you. So here’s how you mount an NFS network mount on a Mac.

First edit /etc/auto_master to add the line shown at the bottom of the file.

# # Automounter master map # +auto_master # Use directory service /net -hosts -nobrowse,hidefromfinder,nosuid /home auto_home -nobrowse,hidefromfinder /Network/Servers -fstab /- -static /- auto_nfs -nobrowse,nosuid

Then create the file /etc/auto_nfs referenced in the auto_master file. Make sure it as permissions of 0644. For some weird reason MacOS no longer allows you to mount to /Volumes. Not sure why. So choose another destination. For example I use /media.

/media/localmountpoint -fstype=nfs,noowners,nolockd,noresvport,hard,bg,intr,rw,tcp,nfc nfs://yournfsserver.example.com/remotedir

Now get the automounter to activate the mounts as follows:

$ sudo automount -cv

And you should see the remote mounts. Make sure of course that the remote server allows you to mount its remote drives. On Linux, you have to edit /etc/exports, followed by an exportfs -a command.

[sumanthvepa]

GNOME 3: Location of Standard folders

You can customize the location of standard folders on GNOME 3 by editing the file ~/.config/user-dirs.dirs.

XDG_DESKTOP_DIR="$HOME/Desktop" XDG_DOWNLOAD_DIR="$HOME/Downloads" XDG_TEMPLATES_DIR="$HOME/" XDG_PUBLICSHARE_DIR="$HOME/" XDG_DOCUMENTS_DIR="$HOME/Documents" XDG_MUSIC_DIR="$HOME/Music" XDG_PICTURES_DIR="$HOME/Pictures" XDG_VIDEOS_DIR="$HOME/Videos"

You'll have to log out and log back in to your account for this to take effect. This primarily applies to the vanilla GNOME3 setup on CentOS or Fedora.

This configuration file is used by the xdg-user-dirs-update program to determine the location of the standard user dirs.

[sumanthvepa]

MacOS: Enabling X11 forwarding when SSHing into a Mac

Late model MacOSes tend to come with sshd configured to not allow X11 forwarding when one is attempting to log into a Mac. To fix this do the following:

Edit /etc/ssh/sshd_config and change/replace the following lines:

X11Fowarding yes XauthLocation /opt/X11/bin/xauth X11UseLocalhost no

The meaning of those magic incantations above is as follows:

X11Forwarding yes: Enable X11 forwarding. This allows X11 data to be tunnelled over incomming SSH connections.

XauthLocation /opt/X11/bin/xauth: This specifies the location of the xauth binary. xauth, according to it's man page is "used to extract authorization records from one machine and merge them in on another (as is the case when using remote logins or granting access to other users.)"

X11UseLocalhost no: By default this configuration option is set to yes, which means that the ssh server only accepts X11 forwarding requests from the localhost, not remote connections (i.e. it listens on the loopback interface for X11 forwarding requests) To cause it to listen for X11 forwarding reqeusts on all interface, and hence support remote X11 servers, change this field's value to no.

Then stop and restart your ssh daemon to get the daemon to pick up the configuration changes. You can do so as follows:

sudo launchctl stop openssh.sshd # Not strictly required sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist sudo launchctl load /System/Library/LaunchDaemons/ssh.plist sudo launchctl start openssh.sshd # Not strictly required

Unfortunately, every time Apple updates your OS, you'll need to redo this work.

[sumanthvepa]

Python: Iterating with an index

This tip come courtesy of pylint. If you want to iterate over a list (or any other iterable and you need an index value to use during the iteration, use the python builtin enumerate

Code below:

alist = ['a', 'b', 'c'] for n, item in enumerate(alist): print str(n) + ':' + str(item)

[sumanthvepa]

Pylint Errata

Some notes on runing pylint under VisualStudio Code. Normally pylint looks for a pylintrc file in package path of the file being linted. But with the Python extension in VS Code this doesn’t happen. You have to place your pylintrc file in the root folder of your workspace. And if it isn’t the same as the root package of your python package tree linting won’t use your preferences.

[sumanthvepa]

A New Web Project Part 1: Choosing an OS for Web Development

Since I was starting a new project from scratch. I have the luxury of choosing any development environment I like. Since I’m working on a personal project, I’m less concerned about stability and more interested in exploring some of the newer technologies and OSes/Distributions out there. However, I don’t want to get too cutting edge. I still want the site to ship and be maintainable. I’d also like some of the learning applicable to what I do at work. 

Windows or Unix

Basically the choices boil down to the following:

Windows

Unix/Unix-like: (Linux, BSD, Mac OS X)

Windows web development is a whole other ball of wax, so I’ll address those in a future set of posts. 

In the Unix world, one can essentially choose between Linux, BSD, and Mac OS X. The latter (Mac OS X) is not a very realistic option if you want to use cheap hosting for your website, as I do. There are not many cheap hosting options.

Any of the BSD variants would be fine. They are very stable, and I’ve had a good experience with them at Yahoo!, but BSD.

FreeBSD

has good support for Python and Django. AWS does provide FreeBSD images. However, BSD is a little bit outside my core comfort zone (I’m primarily a Linux guy.) For this project I’ll stick a little closer to my knitting. In a future project I’ll explore BSD.

So the the choice boils down to Linux.

Supports Python, Django and Mezzanine very well

I’m very familiar with the OS and its quirks

Cheap hosting is available

Finally lots of expertise on the web with this stuff (so a Google search is all that’s needed to solve most problems)

In addition I’d like to host my website on one of the relatively cheap AWS Linux instances. So I’ll have to pick AWS.

Which Linux Distribution?

There are many good Linux distributions, but I explored only those I had at least some passing familiarity with.

CentOS

Ubuntu

Fedora

For production work, I would normally choose CentOS. It is stable and reliable (and free!) 

Ubuntu would also make a great distribution. I’m a little less familiar with Ubuntu. I’ve only recently started using it as a Desktop environment. So I’ll keep learning the environment there, while I explore something else for this project. I imagine, most of what I do would seamlessly work on Ubuntu.

However, as I’m looking to explore the edges a little, I’m going to go with Fedora. It’s a distribution that I’m very familiar with, as I’ve used it in some form or the other for16 years, even before it’s present branding.  It’s a little too unstable for my taste, for production work. But for a personal project, it’s great. It exposes me to some the things that I’m likely to encounter in CentOS down the line.

The project provides cloud images  and is the distribution is well supported by AWS and other cloud hosting providers. 

Of course Python/Django and Mezzanine are available (either as yum packages or pip install-able packages.

So Fedora it is for this project!

[sumanthvepa]

A New Web Project

My personal blog at vepa.in is in need of a revamp. I’ve wanted to move to python/django  powered website that uses Mezzanine as it’s content management system.  So this week, I plan on blogging about that.

[sumanthvepa]

Just cleaned up my partitions, I used to have a DATA partition further than a home and system partition. How is your disk organized?

[sumanthvepa]

Customizing Workspaces in the Unity Launcher

Since I switched to Ubuntu this week, I’ve been playing around with customizing my desktop experience.

Unity is a variant of the GNOME desktop environment. It seems like they’ve replaced the GNOME Shell with a different one. Some applications like GNOME Terminal may have been patched as well. 

One immediate consequence is that customizing stuff on the Unity launcher requires the use of different techniques.  Here are the steps to Unity workspaces joy:

1. Turn on Workspaces

First off, by default, Unity does not enable Workspaces: a critical productivity feature for a pixel hog like me.  Enabling workspaces is relatively straight forward. Just go to Settings -> Appearance -> Behavior (tab) and click Enable workspaces. All illustrated very beautifully in this StackOverfllow post: How do I turn on workspaces?

2. Setup the number & layout of Workspaces

Next up you may want to organize your workspaces in a manner more convenient to you than the default 4x4 layout you get. To do this after you’ve setup your desktop requires you install the unity-tweak-tool. All explained in this equally informative StackOverflow post: How can I reduce or increase the number of workspaces in Unity?

Have fun.

[sumanthvepa]

Setting the title of a Unix (or Unix-like) Terminal window

I spend a lot of time on Unix like termnals, on my Mac as well as on Linux. Occasionally, I find the title on the terminal window annoying and would like to change it. Here’s a little bash script that can set your terminal window’s title. I’m not the author of the original control character sequence, but I’ve forgotten where on the Internet I found the original snippet. So, with thanks to the anonymous author. Here's the code. Add this to the .bashrc file of your home director.

function styleCode() { if (typeof disableStyleCode != 'undefined') { return; } var a = false; $('code').each(function() { if (!$(this).hasClass('prettyprint')) { $(this).addClass('prettyprint'); a = true; } }); if (a) { prettyPrint(); } } $(function() {styleCode();});

# Command to set the terminal title. set_term_title(){ echo -en "\033]0;$1\a" }

I've tested the script to see that it works with GNOME terminal, Xterm, and the Mac Terminal app. Your mileage on other terminal emulators may vary.

[sumanthvepa]

Back to social media

I've been off social media for a few months. Somewhat unintentionally. The pressure of work has kept me off. But now I'm back, and perhaps I can talk about some of the things I've been working on.  For today though the topic the revamp of my personal website http://vepa.in/. I've disucssed the issues involved in detail there.

[sumanthvepa]

Designing a configuration file format with YAML

This afternoon, I'm exploring how provisioning options may be specified in the configuration file of my Virtual Machine creation service zaphod. The objectives for a config file format, would be:

Simplicity: Humans should easily understand what the config file is saying, without necessarily having to look at the user manual.

Ease of Handling the Parsed data structure in a programming language. The data structure returned by a YAML parser must be easy to work with within code of zaphod.

Obviously the first consideration is paramount. The programmer working on zaphod itself (me!) can be given lesser consideration. :-) 

So let's get started with a YAML parser/dumper written in python. It's trivially simple. It just slurps in YAML files and prints out the results of parsing them.

The little python script shown above simply takes a YAML file parses it, and prints it's contents as a python data structure.

Next step: we need to create a YAML file specifying provisioning, and use this script to parse the configuration file to understand what config file formats work best.

Shown below is the first cut for a YAML specification for provisioning:

The file is quite simple and hopefully, intuitive. The problem is with the parsed output:

As you can see from the output above, all the host and vm path strings are munged into a single vector of dictionaries. Each dictionary having only one value.

A much better system would be a vector with each dictionary having two values. Let's see how that can be done...

That's much better. The scp field is a sequence of dictionaries. Each dictionary, contains a specification of host path and vm path for the provisioning script. (It could also specify the username to be used when creating files on the VM or when running those files on the vm.) Let's see what the the parsed python data structure looks like:

Perfect! We have a winner.  Now onto more complex things...

[sumanthvepa]

Upgrading to Fedora 20

Exploring an upgrade to Fedora 20 over the course of this month. My previous experience with upgrading to a Fedora release has been that the release quality is very spotty and you can't assume that because a previous release was good that this one is going to be better. Reviews are unfortunately not hugely useful as the way the reviewer uses Fedora may differ greatly from my use of it: I use Fedora as my primary development environment for web development and systems software development.

However, at least one review seems to be positive, so I'm going to check it out today on a virtual machine.

I'm going with the DVD install which is a pretty big download for India. (I use the torrent, to cope with the frequent disruptions to connectivity here.)