dublindonut

[dublindonut]

Meandering is a word I like to think of as synonymous with procrastination. When I meander from subject to subject in an attempt to avoid actual work (as I am doing this very morning) I typically come up with ideas for blog posts. This idea however came up while listening to an awesome podcast by the two greatest podcasters of all time, Life of Alexander. If you're not familiar with these guys go listen to Life of Caesar (recently moved to Life of Augustus) to get a sense of their amazing work.

Now back to Meandering. The word meander gets its meaning from the Meander river in Turkey which is well known for its winding, random path. Back in Alexander's day, the Meander connected the city Priene to the Mediterranean Sea, but after much silting over a couple thousand years it has diverted elsewhere. Priene is important to Alexander because, as often stated in the Life of Alexander podcast, it was well known for its temple of Athena which burnt down the night of Alexander's birth. Feeling very bad for having distracted the Gods with his birth (Alexander thought of himself as a demigod and as being descended from Zeus and (actually) Achilles), Alexander offered to repair the temple, but the locals refused (kind of them considering how broke Alexander's campaign was).

What some people don't know about meander, is that the term also refers to a formation in a river called... you guessed it, a meander. There's a shocking amount of complicated mathematics, geology involving sine waves and angular momentum, but the easiest way to understand just what a meander is, is to look at the picture below.

The wikipedia also has a picture of what they call meander scars, along the Rio Negro river in Argentina.

Pretty random, pretty cool stuff.

[dublindonut]

Flux (kinda)

Horay I finally took the time to get flux working (kinda)!

There's a lot of improvements to make but here is the start (hint: open your console and see all the useless repetitive console.log stuff I left behind that shouldn't be there). End goal for this little page is to use an expression tree to calculate whatever expression is entered to the input. Right now all it does is multiply your value by 2. You can see my full implementation on my github, and it may also be handy to take a look at this blog post on writing es6 flux stores that I found really handy.

As we all know, flux consists of actions, constants, a dispatcher, and a store. Here's a quick list of things that happen to get data processed...

bunch of unnecessary logging

doubles the value and places it in the store instance's data object

emits a change event

my component's onChange function calls the setstate method

when it was mounted, my component passed it's onchange method as a callback to the store's addChangeListener method

when the change event from 4.3 is emitted, the addChangeListener's callback is executed

my components state is updated and reflected in the input field

This still has some problems because once my state has been updated I can't update the input field again, I can only click "calculate" again to double the value. Work in progress as always....

[dublindonut]

SQL Reversion Script

When people new to sql think of databases they tend to think of a table of numbers that ends up in a graph of some sort. What they don't think of is the configuration and user settings that are also stored in that database. So lets say you have a column in the database that should be a series of comma separated values, but was for whatever reason is actually tab delimited. You want to update each column, but you don't want to run the risk of accidentally changing everybody, so here we can actually make a query that generates a series of queries that set the column back to the original value. Check out the example/gist below. You can simply copy paste your results into a new sql file, then run that sql file if anything goes wrong! Keep in mind that the triple quotes are for the case that your value to be changed is a string. If the value is an integer you won't be needing them.

[dublindonut]

Metaprogramming

Metaprogramming is an interesting topic, I'm not totally sure what it is, but the way I understand it is making a program to generate code. I've been calling that scripting for quite some time for no particular reason. I think that technically metaprogramming would involve allowing commands to be input, but I need to do more research (I have done none). I recently ran across a problem in which I needed to generate the same method for each numeric data type in C#.NET, including nullables. Instead of writing this out each time, or copy-pasting plus edit, I decided to script out my solution in chrome's javascript console (mostly because I want to learn javascript better).  My basic steps are 1. write the different data types, 2. go through the nullable ones to write out nullable functions, 3. similar for non-nullables. Something to note... the .includes() is specific to es6, though it should be implemented in most browsers.

[dublindonut]

Grocery Store Labeling Needs to Change

As a supposed adult I now do things like shop for groceries. One thing I've noticed and ranted about with my roommate is that the labeling on items needs to change immediately. Nothing is ever near it's label. In some aisles the labels are above the item then in the next they'll be below the item and four feet to the left. Or the labels are old, especially for fruits like apples that come in different varieties. Then on top of all this you find six labels with item names written more cryptically than 14 year old girl's text to her crush. All vowels removed, words truncated, 6 pt yellow font on a white background, and why is the unit price in per 100? This is just a loaf of bread. So that's an exaggeration, but all I need is a big sticker stamped on my bread bag that says $2.99, or maybe the $/lb on my apples. Grocery stores around the US, time to get your shit together and start labeling individual items correctly. This is why I buy everything I can online.

[dublindonut]

Timsort

So quicksort is pretty cool. Everybody knows "how" it works, meaning we've all seen a visualization of quicksort picking a pivot, going through and finding values less than or greater than and moving them around the pivot. Unfortunately very few people correctly implement the algorithm correctly on their first try, or if they did, they've never implemented it since. Less common that quicksort is the man behind the curtain (edit: I made this post when I was pretty exhausted, is "man behind the curtain" even a phrase? Does it make sense here?)...Timsort. Timsort is based on Tim Peter's list sort for python described here. It even runs O(n lg n) in the worst case, and often faster on real world data! As an adaptive sorting routine, Timsort uses a binary sort when it recurses deeply enough and combines sets of data that were ordered to begin with. (More edits... Tim sort is also a stable sorting algorithm, meaning that if you want to compare array vs array like with Strings, you can compare each piece of that array. This means objects with equal keys will not be sorted arbitrarily.) Aside from the coolness of having beaten quicksort Timsort has its downsides, for example it does also use n/2 extra space where qiucksort does sorting in-place. In most applications I would have, the extra space would be no problem, however for big data analysis n/2 extra space could be GBs or TBs.

find the original here

[dublindonut]

Sorry I’ve Neglected You Dublin Donut

I’ve been busy, got another blog at imbadatcleverurls.blogspot.com. Here’s a cross post.

XOR Linked Lists:

Today was fairly slow at work so when Walter messaged me on gchat I responded. Having just finished App Academy Walter is very up to date on the most efficient data structures, and had apparently spent the morning researching xor (exclusive or) linked lists. I am currently taking an evening class in data structures so when he messaged me to try and teach me about them I was interested to hear more. In fact, my professor had mentioned them, but not demonstrated an implementation. 

To understand xor linked lists we need to understand the problem they solve. Suppose you're super memory constricted and want a linked list you can traverse both forward and backward, but with only one extra reference rather than the prev and next references typical of doubly linked lists. That's where xor linked lists come in; one reference at each node containing information for both next and previous. As indicated in the name, the crucial part of this is the xor operator (^ in javascript) which returns the higher of two values if they are different, and zero if they are the same. So when xoring two 16 bit ints, 00000000000000001^00000000000000010 = 00000000000000011. This means 1 xor 2 = 3, and as Walter told me to do, you can check this in the javascript console of your web browser with 1^2. Walt also informed me that because xor is lossless, 1^2 =3, 1^3 =2, 3^1 = 2, 3^2 =1 ... and so on for all permutations of the three numbers. (Before I really get into it here's some notation I just invented, this won't hold true elsewhere, but in the following, {} is a variable resulting from the xor operation and anything in [] is an object storing our node data and the previously mentioned variable.) Now think for a moment that this property of xor holds true for everything, not just numbers in bitwise format. Think if we had [null], [1], [2], [3], [4], [null] as our example list, and {[null] xor [2]} stored in node [1], {[1] xor [3]} stored in node 2, and so on, where the objects being xor'd are actually addresses in memory! Then given the information [null], [1], {[null] xor [2]} we can take {[null] xor [2]} xor [null] to get the memory address that [2] is stored in. We can now drop [null] and store [1], [2] in memory. In [2] we'll also find the value for {[1] xor [3]}, and by taking {[1] xor [3]} xor [1] we will find the address for the value after [2] is [3]. Continuing this pattern we could traverse the whole tree. In all we would need to store four nodes of information, our first two and last two, then use the above pattern to traverse the list forward or backward. I did some googling and there are a few problems with this technique. First it's pretty risky in Java, where you cannot access memory directly and are liable to have your unreferenced objects get gc'd. Also, it apparently really messes with debuggers and makes it hard or impossible to step through code. Finally, memory is cheap, this is a data structure I could see used in some kind of controller with little memory, but not a modern desktop or laptop. I know this was a long post and probably difficult to read, but thanks for reading anyways. Future posts will be much less daunting, and there will probably be days where I post something like "I'm hungover and learned that naps are great". A million thanks to Walter for having sparked this post, check out his github.

[dublindonut]

2048.1: A Variation on 2048

I spent the last couple of days building a slight variation to the game 2048 (If you haven’t played it, you need to). I call it... 2048.1! Very majestic and creative, I know. But wait, what does the “.1″ mean? Instead of having the starting tiles as 2 or 4, I have all of the starting tiles begin at 1. While this somewhat reduces the complexity of the game, it does make reaching the 2048 tile much harder.

When I started building this thing, I spent a good amount of time playing the original to learn the mechanics of the game. How does the board populate? How do the tiles slide? How do the tiles merge? etc... But after I got around to understanding the rules of the game, I wanted to understand how to beat the game.

Full disclosure, I’m seriously horrible at 2048. I don’t think I’ve ever gotten to the 1024 tile and only on a few occasions have I reached the 512 tile. Not impressive, I know. Fortunately, a bunch of people I know are super duper good! And, I learned a lot about the strategy needed to reach a high score. The key? Keep high value tiles on the corners. From that, I developed a simple heat map / gradient of where tiles should be located on any particular move. Here are some visuals:

(Just a brief explanation, the darker the blue, the better the tile position)

With this in mind, I realized I could actually develop an AI capable of playing the game. My first thought was to use the Minimax algorithm because well.. it’s the only AI algorithm I know. Unfortunately, I sorta struggled to implement this in the beginning and stumbled upon these slides from CS294 at The University of Washington (thanks, btw) which paved the way for my AI.

It described the Minimax algorithm in simple terms:

Base case: return the heuristic for the current board state

Maximizing player: return the maximum value for each child state

Minimizing player: return the minimum value for each child state

It showed me alpha-beta pruning which gave performance gains

It showed me an even better algorithm called Expectiminimax. This is very similar to the Minimax algorithm except it has an extra case -- instead of always having a minimizing player, it accounts for random cases, perfect for my random tiles that show up.

So, with the Expectiminimax, I replaced my Minimizing Player condition and replaced it with the random condition, and BOOM. My AI beat 2048.1!!!

Here’s a link to the GitHub repo: https://github.com/waltertan12/2048.1

- Walter

[dublindonut]

BACK AGAIN

Once again, I’ll start the post with ‘LONG TIME NO POST’ Sorry for the extra long hiatus but I do have some blog worthy updates.

First things first, I quit my engineering job in Aiken, SC back in July! Big news, yeah? Even more exciting, I moved to San Francisco to attend a web development bootcamp called App Academy. I’m 4 weeks into the 12 week program and I gotta say, it is pretty intense. While the workload is less than what I had for my senior ChemE capstone class, it’s going to be continual grind for about three months. Definitely a challenge and I can feel myself burning out... But hey, I’m learning a ton. Way, way more than I was when I was still working.

Feel free to check out what I’ve been learning / what I’ve been building by visiting my GitHub. I’ll try to post here a bit more on my daily projects so stay tuned!

- Walter

[dublindonut]

The Lebron Effect

Not to be outdone by my co-blogger, Walter, I’ve come back from the dead with one graph. Year vs Cavaliers game attendance. Tell me if you can point to where Lebron joined/left/joined!

See the R script at my github

https://github.com/michaelAlvarino/Analyses/blob/master/lebron%20effect.R

[dublindonut]

Fun With Data

Long time no post, yeah?

Well, this should be a pretty good one.

I’ve taken a brief break from Ruby on Rails to do a little bit of work with Python. To be honest, I was a getting slightly bored / frustrated with web development so I thought I’d try my hand at “data-mining” and “data science.” So, with the CrossFit Regionals coming, I thought it’d be interesting to analyze what it takes to be a top CrossFit competitor. Specifically, I wanted to know what factors would affect placement in the CrossFit Open.

The first thing I needed to do was gather data.

I used a couple Python libraries to help me do this: Requests and Beautiful Soup. Requests downloads the html of a page while Beautiful Soup allows for simple searching / manipulation of the text. With some regular expressions and some looping, I managed to scrape all the relevant data off of an athlete page and store the competitor info in an SQLite database.

The next step was analysis.

I immediately made a couple graphs with PR vs overall rank. They’re pretty cool actually!

I also wanted to see how age might affect performance in the open so I made a chart detailing the best score of each age group. Graphically, it’s easy to see the teenagers and 30+ aged athletes struggle. Even more interesting is the peak of performance at around age 25 (I HAVE ONE MORE YEAR TO BE SUPER PHYSICALLY FIT!).

The next step was to actually develop the model to predict performance. I got stuck here for a little bit. I had a brief conversation with Mike on how to perform multiple regression but my mind couldn’t process it all (Note: Mike explained it really well. I simply couldn’t grasp it right away...)

Fortunately, it all sort of came together today and I made a simple model based of multiple linear regression, specifically sum of least squares over twelve variables. I don’t have a web app built to showcase the model but the Excel spreadsheet I have incorporates it. You can find the Excel spreadsheet and the Python scripts on my GitHub account: http://github.com/waltertan12/crossfit-games

I think the next step for me is to use some better techniques, like a random forest, to build a better model. I found the Scikit-Learn libary which seems to have some great machine learning tools so I’ll give that a go at soon. Oh, and I’ll most likely build a simple web app to display the results.

- Walter

[dublindonut]

Fitness and Netflix

People know that they should eat better, exercise more, and in general, live a healthy lifestyle. Unfortunately, despite the vocalizations of wanting to be healthier, it doesn’t really happen.

How do we change that?

Sir, you are supposed to lift the weight, not sit on it.

My gym ran a program trying to get the willing but unmotivated motivated. I’m not quite sure what they titled the thing but maybe Heart Rate METS Tracking Challenge for Prizes? (I did not sign up)

The premise:

Make participants wear a heart rate monitor at the gym

Track the total METS (metabolic equivalents) of each user in a certain time period,

Participant with most METS wins the challenge and gets a prize.

I thought the concept was great and I wanted to try and expand it.

So, here’s the new idea

Set a goal of how much exercise you want per week

We will recommend whatever the government says: 90 min a week of moderate intensity exercise

Turn this goal into a METS target

Link FitBit or heart rate monitor to web app

Track weekly METS

If you reach the weekly target, you get access to movies the following week

Repeat steps 3-5

A simple way to implement this with a friend would be to give your friend your Netflix account and password and make them track your workouts. If you reach your quota of working out, he tells you the password. If you don’t reach the quota, you don’t get the password :(

I’m still thinking of ways to scale this upward. Does anyone have suggestions on how to do this? Also, do you think this would motivate you to work out more? Or would you just start to hate your “friend” and spend $10 on another Netflix account?

[dublindonut]

Sitting Disease

It's been awhile but I've finally made another page for Dublin Donut. This time the focus is on sitting disease.

I first came across the notion of sitting disease when a co-worker forwarded me an article from the Washington Post nearly a year ago. Honestly, I didn't take it seriously, as the drawing of the man in the article made me laugh, but some good came out of it. Instead of sitting all day, I would stand up every couple hours, proclaim that I was "fighting sitting disease," and proceed to work standing up for an hour. It was all a big joke in my mind.

He looks kind of weird right?

However, the biggest joke was on me for ironically getting into a good habit of standing up every now and then. Here's the train of thought that turned me:

Daily Routine

Time I spend sleeping per day: 8 hours

Time I spend working out: 1 hour

Time I spend sitting at work: 10 hours

Time I spend in the car, sitting: 1 hour

Time I spend sitting when I get home from work: 4 hours

These numbers are not exact but I paints a frightening picture. As someone that considers themselves fairly active, I am actually not active at all. In fact, I spend an overwhelming majority of every day of my life sedentary.

So, to encourage myself to stand a bit more while working, I made a simple standing/sitting tracker. Check it out here or here! Hopefully it will encourage you to stand up and "fight sitting disease..."

- Walter

[dublindonut]

IoT: Temperature Logging

Here’s the latest with my Raspberry Pi:

I bought a few digital temperature sensors a couple weeks ago and finally got around to installing them this weekend. I had grand plans for a temperature logger hosted on a web app. However, not everything goes as planned. Initially, I wired the temperature sensors backwards and immediately destroyed the device. Fortunately, I bought two in case something like this happened, though I almost did the same with my backup sensor.

After some online tutorial-ing I got the temperature sensor to register with my Pi. Then, I wrote a couple scripts in Ruby and Python to read the temperature and log the info into an sqlite3 database.

But again, not everything goes as planned — I got really lazy and didn’t want to make a web app for the data. So, instead of actually using my database as intended, I looked up the Plot.ly API streamed my temperature data. Take a look!

Check out the source code on GitHub: github.com/waltertan12/temp-stream

Walter

[dublindonut]

Another Look At The Stages Of Learning

I often read random blog posts that make it to the top of reddit or hackernews, and maybe I haven’t read enough, but I get the sense that most people judge the stages of learning software development in terms of confidence vs competence. Take a look at the graph below:

Though this graph is obviously a parody, the Dunning-Kruger effect was hypothesized in 1999 by Cornell professor David Dunning and University of Illinois professor Justin Kruger, then tested and shown to be true in their paper “Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments”. The test subjects were Cornell undergraduates, meaning they were most likely educated, white, intelligent, and not a truly random sample, but the point comes across nonetheless. For their contributions, Dunning and Kruger were awarded the satirical Ig Nobel Prize in 2000.

Walter and I have walked this path very accurately, and lately we have been hanging in the hammock of staleness. Though it is very accurate, the graph above just doesn’t feel right. I want to propose a re-labeled, discontinuous, poorly drawn, and poorly named version of this graph that applies specifically to software development. I call it the “Software Development Learning Cycle”.

Why am I here:

During this stage you’ve forgotten how the random clicking at 2 am brought you to an html tutorial. You bookmark it anyways.

Cool:

You read a tutorial and go “oh that’s how those web pages show up? Is that how images are put in? Is that how you style it?”.

But how do you:

Here you wonder how pages become interactive with dropdown menus, so you look it up and discover javascript. The inviting language is simple and has plenty of code samples.

Cheat Codes:

Here you work your way through a tutorial, I call it cheat codes because tutorials teach you the structures around a pre-defined, simple goal. Wow that’s a beautiful todo app you have there.

But how do you:

Now you’re asking real questions, how do I make something new and innovative? How the hell do I persist data? What about users and logins? Security? Now I need to know a database language? Maybe there’s an easier way…

NOTE: At this point (for unknown reasons) restart entirely with a new language. This is the “Cycle” part of SDLC. Repeat until you’re sick of repeating.

Why did I come here:

Here you may remember that book you wanted to read, that you have a non-software related job, or your final exam is actually more important than the tutorial.

Google your log files:

This is the complicated part, “Google your log files” comes in many forms. In reality it’s just a day during the hammocky low point in the middle where you’re not too tired or distracted to learn a bit more. Then there might come a day when you actually have something with log files, like an nginx server, and it’s returning errors, and you google those errors and find a solution. It’s a revolution, but a Franklin-kite-lightning one as we will soon see.

Disappointment:

The next day you find there are more errors, and you can’t find any good resources. Since the error is unintelligible you just look the other way for a while and let the kite burn. Where did you put that book again? Is it midterms already?

Find it in the documentation:

Known to many as RTFM (read the f’ing manual), you return one day, possibly after a long winter vacation, to Franklin’s sizzling kite and see that one part of the error message makes sense. You google that and come to some Rails documentation saying something about an environmental variable and Rails version differences. So you do some more research, create the missing environment variable, and the error changes to something else, or maybe your website works? Who knows?

What we have here is the difference between learning the mechanics of a language, or cycling through everything before the “NOTE” part of the post, and learning how to think critically, or solve unique problems specific to your application by investigating. I can now see that learning the basics of a programming language can be relatively smooth, and continuous, but solving problems and critical thinking is a weird, jerky, angular, discontinuous experience with high peaks and low valleys. In my opinion the issue is that results are so binary, either the server works and serves pages or you type in a url and nothing happens. There’s probably not many cases of half an email address getting put in your database, either it’s all being inserted properly or it isn’t (let’s ignore regex errors). As a result, you have either a work in progress or a wonderful, working site.

I hope you found my post encouraging, it was meant to remind people that learning is done at your own pace, and though your startup may move up and to the right, we don’t always follow it perfectly. After “Find it in the documentation” the process probably repeats, but I don’t know yet. I haven’t been there.

[dublindonut]

Heat Rises?

I live in the south. It’s pretty warm, even in the winter. Check out the 5 day forecast:

In fact, it’s so warm, one of my co-workers started talking about how he still hasn’t turned on his heat.

Tall co-worker: Dude, we haven’t turned on our heat this past weekend. It’s been really nice. Bearded co-worker: Tell me about it. I haven’t turned on my heat at all yet. Tall co-worker: Really? Don’t you live on the second floor? Bearded co-worker: Yeah. Tall co-worker: I lived on the third floor of that apartment complex and still needed heat. Bearded co-worker: Ahhh that’s weird. Tall co-worker: I assumed the heat would rise from the first and second floor which would make it a better option than the second floor.

That got me thinking. Does the insulation from the third floor keep the second floor warmer? Is it that measurable of a difference? So, I decided develop a model and see. Turns out it’s better to be on the second floor in the winter.

While I was making this, I realized the second floor would be the better option, at least in the model. The top floor doesn’t take heat from both the first and second floor, just the floor directly below it. Plus, it has more heat sinks. Less heat sources and more heat sinks equals slightly colder apartment.

Here's a link to the Excel sheet.

Technical stuff / assumptions after the jump.

I’ve got two models going: one for the second floor with no heat on, one for the top floor with no heat on. I’ll start off with my assumptions

General Assumptions

Assume outside temperature moves sinusoidally from 30-60 F

12 AM is the coldest; 12 PM is the warmest

Thermal conductivities and heat capacities listed in the Excel sheet

Uniform heat distribution in all apartments

Wall and roofing are the same material; Floors are composed of different material

All apartment dimensions: 25’ x 30’ x 10’ (LxWxH)

Second Floor Model

Assume first and third floor have heat and are at a constant 68 F

Third Floor Model

Assume the second floor has heat and is constant at 68 F

Some additional notes. I really, really simplified this 3D heat transfer problem by arranging it as a sum of 1D transfers.

Energy Accumulation = Wall Loss + Wall Loss + Wall Loss + Wall Loss + Floor Loss + Ceiling Loss

I’m really not sure if this is an acceptable solution but it my solutions seemed to pass my sanity check. The things I really want to fix mainly depend on wall material. In my model, walls pass a lot of heat. Currently, I say walls are made solely of drywall and fiberglass insulation but I’m not sure if that’s true. If you've got any suggestions for the sheet, just let me know! I'd love to incorporate it.

[dublindonut]

Memory

For the past week and a half, I've noticed a plastic bag full of chobanis placed clumsily on the door shelf of the work refrigerator. I've never seen chobani in there before, other than mine, and thought, "Cool, someone else eats chobani besides me."

Well, day after day I've looked at this pile of greek yogurt and found it quite suspicious that the number of yogurts never diminished. It was like this: Monday, 5 yogurts; Tuesday, 5 yogurts; Wednesday, 5 yogurts; etc. Who brings yogurt to work and doesn't eat them?

Today, curiosity got the better of me, and I decided to look into the bag, just to see the flavors. The first one that caught my eye was passion fruit. I remembered buying the exact same flavor yogurt a couple weeks ago but never eating it. I also remembering finding it odd I never ate it that flavor and guessed I lost that yogurt, whatever that meant.

Then, it hit me.

These were my yogurts. I put them in a plastic grocery bag, I brought them to work, and I put them in the fridge the Friday before a long weekend...

Anyway, it really is a strange feeling when you lose and find something all in the same moment.

- Walter