#planetpython

[ramrachum]

GridRoyale - A life simulation for exploring social dynamics

GridRoyale - A life simulation for exploring social dynamics

Another day, another project :)

This is a project that I wanted to do for years. I finally had the opportunity to do it. Check out the GridRoyale readme on GitHub for more details and a live demo.

GridRoyale is a life simulation. It's a tool for machine learning researchers to explore social dynamics.

It's similar to Game of Life or GridWorld, except I added game mechanics to encourage the players to behave socially. These game mechanics are similar to those in the battle royale genre of computer games, which is why it's called GridRoyale.

The game mechanics, Python framework and visualization are pretty good-- The core algorithm sucks, and I'm waiting for someone better than me to come and write a new one. If that's you, please open a pull request.

[releaseteam]

via Twitter https://twitter.com/releaseteam

[ch3n2k]

via Twitter https://twitter.com/ch3n2k

[opexxx]

From Twitter: Graham Dumpleton: Launching applications in Docker containers. http://bit.ly/1zmyC1i— Planet Python (@planetpython) December 16, 2014

↬@planetpython

[ramrachum]

Live-coding a music synthesizer

Live-coding a music synthesizer

After so much work and waiting, the video of my EuroPython talk is finally released!

This is a fun live-coding session using NumPy and SoundDevice. The goal of this talk is to make the computer produce realistic-sounding instrument sounds, using nothing but math.

The talk starts with creating a simple sound using a sine wave. We gradually make it sound more like a real instrument, learning a little bit about music theory on the way. We add features one-by-one until by the end of the talk, we hear our synthesizer play a piece of classical music.

[ramrachum]

Improving Python exception chaining with raise-from

Improving Python exception chaining with raise-from

This is going to be a story about an esoteric feature of Python 3, and how I spent the last few months reviving it and bringing it into the limelight.

Back in the yee-haw days of 2003, Raymond Hettinger wrote an email to the python-dev mailing list, sharing an idea for improving the way that Python handles exceptions that are caught and replaced with other exceptions. The goal was to avoid losing information about the first exception while reporting the second one. Showing the full information to the user would make debugging easier, and if you've followed my work before, you know there's nothing I love better than that.

That idea was polished and refined by many discussions on python-dev. A year later, Python core developer Ka-Ping Yee wrote up a comprehensive PEP that was then known as PEP 344, later to be renamed to PEP 3134. That idea was detailed there, with all the loose ends, potential problems and solutions. Guido accepted the PEP, and it was implemented for the infamous Python 3.0, to be used... By no one. For a long time.

If there's one thing I don't miss, it's waiting 10 years for the Python ecosystem to adopt Python 3. But finally, it happened. Almost all the packages on PyPI support Python 3 now, and getting a job writing Python 3 code is no longer a luxury. Only a few days ago, NumPy finally dropped Python 2 support. We live in good times.

When a modern Python developer catches an exception and raises a new one to replace it, they can enjoy seeing the complete information for both tracebacks. This is very helpful for debugging, and is a win for everybody.

Except... For one thing.

Two cases of exception chaining

There was one interesting detail of PEP 3134 that was forgotten: It has to do with the question, "What does it mean when one exception is replaced with another? Why would someone make that switcheroo?"

These can be roughly divided into two cases, and PEP 3134 provided a solution for each case.

The first case is this:

"An exception was raised, we were handling it, and something went wrong in the process of handling it."

The second case is this:

"An exception was raised, and we decided to replace it with a different exception that will make more sense to whoever called this code. Maybe the new exception will make more sense because we're giving a more helpful error message. Or maybe we're using an exception class that's more relevant to the problem domain, and whoever's calling our code could wrap the call with an except clause that's tailored for this failure mode."

That second case is quite a mouthful, isn't it? It didn't help that the first case was defined as the default. The second case ended up falling by the wayside. Most Python developers haven't learned how to tell Python that the second case is what's happening in their code, and to listen when Python is telling them that it's happening in code that they're currently debugging. This resulted in a Catch 22 situation, not that different from the one that slowed down Python 3 adoption in the first place.

Before I tell you what I did to break that Catch 22, I'll bring you into the fold and show you how to make that feature work in your project.

Exception causes, or `raise new from old`

I'm going to show you both sides of this feature: How to tell Python that you're catching an exception to replace it with a friendlier one, and how to understand when Python is telling you that this is what's happening in code that you're debugging.

For the first part, here's a good example from MyPy's codebase:

try: self.connection, _ = self.sock.accept() except socket.timeout as e: raise IPCException('The socket timed out') from e

See the from e bit at the end? That's the bit that tells Python: The IPCException that we're raising is just a friendlier version of the socket.timeout that we just caught.

When we run that code and reach that exception, the traceback is going to look like this:

Traceback (most recent call last): File "foo.py", line 19, in self.connection, _ = self.sock.accept() File "foo.py", line 7, in accept raise socket.timeout socket.timeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File "foo.py", line 21, in raise IPCException('The socket timed out') from e IPCException: The socket timed out

See that message in the middle, about the exception above being the direct cause of the exception below? That's the important bit. That's how you know you have a case of a friendly wrapping of an exception.

If you were dealing with the first case, i.e. an exception handler that has an error in it, the message between the two tracebacks would be:

During handling of the above exception, another exception occurred:

That's it. Now you can tell the two cases apart.

What I did to push this feature

I found that almost no one knows about this feature, which is sad, because I think it's a useful piece of information when debugging. I decided I'll do my part to push the Python community to use this syntax.

I wrote a little script that uses Python's ast module to analyze a codebase and find all instances where this syntax isn't used and should be. The heuristic was simple: If you're doing a raise inside an except then in 99.9% of cases you're wrapping an exception.

I took the output from that script and used it to open PRs to a slew of open-source Python packages. Some of the projects I fixed are: Setuptools, SciPy, Matplotlib, Pandas, PyTest, IPython, MyPy, Pygments and Sphinx. Check out my GitHub history for the full list.

I then added a rule to PyLint, now known as W0707: raise-missing-from. After the PyLint team makes the next release, and the thousands of projects around the world that use PyLint upgrade to that release, they will all get an error when they fail to use raise from in places they should.

Hopefully, in a few years' time, this feature of Python will become more ingrained in the Python community.

What you can do to help

Do you maintain a Python project that already dropped Python 2 support? Install the latest version of PyLint from GitHub. You can do this in a virtualenv if you'd like to keep your system Python clean. Run this to install:

pip install git+https://github.com/PyCQA/pylint

Then, run this line on your repo:

pylint your_project_path | grep W0707

You'll get a list of lines showing where you should add raise from in your code. If you're not getting any output, your code is good!

[ramrachum]

Symlinks and hardlinks, move over, make room for reflinks!

Symlinks and hardlinks, move over, make room for reflinks!

If you've been around Linux for a while, you know about symlinks and hardlinks. You've used them and you know the differences between how each of them behaves. Besides being a useful filesystem tool, they're also a favorite interview question, used to gauge a candidate's familiarity with filesystems.

What you might not know is that there's also a thing called reflink. Right now it's supported only on a handful of filesystems, such as Apple's APFS, used on all Apple devices, XFS which is used on lots of Linux file-sharing servers, Btrfs, OCFS2, and Microsoft's ReFS.

If a symlink is a shortcut to another file, and a hardlink is a first-class pointer to the same inode as another file, what's a reflink, and when is it useful?

A reflink is a tool for doing copy-on-write on the filesystem.

If you've heard the term copy-on-write before, I'm willing to bet that it was in the context of the Linux fork call. Let's talk a bit about that.

Copy-on-write when forking a process

When you fork a process in Linux, the new process has a new copy of the old process's memory. This is essential, because if the new process shared the old process's memory, either process could crash if the other process was making an unexpected change to their shared memory. Therefore, Linux needs to make a copy.

However, Linux is smart, and it knows better than to just make a naive copy. Making a naive copy could be a waste of memory, especially if your process has several gigabytes of memory allocated, and you're forking lots of processes for small tasks. If Linux were to make naive copies, you could find yourself with an out-of-memory crash very quickly.

When you fork a process, Linux uses copy-on-write to create the new process's memory. This means that it holds off on making actual copies of the existing memory pages until the last possible moment; which means, the moment when the two processes start having different ideas on what the content of these memory pages should be. In other words, as soon as one of these processes start writing to these memory pages, Linux makes a copy of it, assigning the original page to the original process, and the new copy to the newly-forked process.

This is a huge boon, because most of the time, the new process will either only be reading the memory, or not even that. So many copy actions are avoided thanks to this technique. The beauty part is that these shenanigans are completely transparent to the process, and to the developer who's writing the logic that this process performs. The new process behaves as if it has its own copy of the parent's memory pages, and the floor is being paved ahead of it as it walks forward, so to speak. It'll never even know that copy-on-write was performed.

Now we're ready to talk about reflinks.

Reflinks are copy-on-write for the filesystem

If you read the section above, you already know 90% of what you need to know to understand and use reflinks.

A reflink is a copy of a file, except that copy isn't really created on the hard-drive until the last possible moment. Like the forking version, this logic is invisible. You could do a reflink of a 10 gigabyte file, and the new "copy" would be created immediately, because the 10 gigabytes wouldn't really be duplicated. They'll only be duplicated once you start modifying one of the copies.

All the while, you could treat the reflink as if it was a completely legitimate copy of your original file.

How do you create reflinks?

On Linux, run the following:

$ cp --reflink old_file new_file

On Mac, there's a different flag for some reason:

$ cp -c old_file new_file

If you're creating reflinks programmatically, you could also use dedicated libraries such as this one for Python.

When are reflinks useful?

Here's an example of where I've used reflinks for a client of mine years back. They had a tool for developers that takes their entire codebase and copies it into a Docker container to run tests on it. (Don't ask.)

That recursive copying took a while, and the developers couldn't change their code in the meanwhile, or checkout any other branches, because then an inconsistent version of their code would be copied into the container. That was pretty annoying for me personally, because I was twiddling my thumbs whenever I started the test process.

I figured, why not use reflinks?

I wrote some Python code that creates reflinks to the code in a temporary folder, and then does a real copy from that temporary folder to the Docker container. The big advantage here is that as soon as the reflinks were created, I could modify the original code as much as I wanted, without affecting the tests.

Fortunately, all the developers were using Macs in that company, so I knew I didn't have to worry about filesystem support.

How can reflinks go wrong?

You might be thinking, "What happens if I create a reflink of a huge file, that's bigger than the amount of space I have available on the harddrive?"

I've never tried this, but here's what I heard: The reflink will be created, but then you'll get an error as soon as one of the copies will be changed, and an actual copy will need to be created. I haven't tested this, but this is something you should take into account if you're relying on reflinks in your business logic.

[ramrachum]

PySnooper: Never use print for debugging again

PySnooper: Never use print for debugging again

I just released a new open-source project!

https://github.com/cool-RR/PySnooper/.

[ramrachum]

Developers: When using Frecency, please consider the context

Developers: When using Frecency, please consider the context

Nowadays there are many applications that use the wonderful concept of Frecency.

“Frecency” is a combination of “frequency” and “recency”. For example, say that I have a dear friend named Bob Jones to whom I write emails every day. I may have many Bobs in my address book, but when I start a new message in GMail and start typing “Bob”, GMail is smart enough to offer me Bob Jones as the first choice, because that’s the Bob to whom I’ve written the most often to recently.

This is a really awesome feature! It saves a lot of time sifting between non-popular choices. I <3 Frecency.

Many applications use this concept of frecency. When you start typing anything into Chrome’s Omnibar, it uses frecency to guess which website you want to visit. I use a launcher for Windows called Launchy. When you press Start-Space, you can start typing the name of the program that you want to launch, and Launchy will usually guess that program after 1-3 keystrokes. That’s a really fast and convenient feature, powered by frecency.

But… There’s room for improvement

There’s one thing about most frecency-using programs that I find annoying. Let me explain it with an example.

I’ve already mentioned my good friend Bob Jones to whom I write emails every day. I also have a business partner called John Schiller to whom I write frequently, though less than to Bob. Sometimes I mean to write an email to John, so I start typing jo in the address field, but then instead of getting John Schiller as the first suggestion, I get Bob Jones.

GMail is offering me Bob Jones because I write to Bob Jones more frequently than to John Schiller, and jo happens to be the first 2 letters in Bob Jones’ name.

But, if GMail had a better algorithm, than it would realize that I never start an email to Bob Jones by typing his last name. Ever. Therefore if I’m writing jo, the odds are slim that I’m writing an email to Bob.

How to fix it?

The way to fix this problem is to always check how certain selections are made. If I always start an email to Bob Jones by typing Bob, don’t offer that selection in a high position when I start typing Jones. Every time I make a selection, save the original piece of text that I typed in to make that selection, and then consult that data next time the program is trying to guess my selection.

I have a similar problem with Launchy. Sometimes I want to launch Rosetta Stone, so I start typing ro, but then Launch offers to launch Chrome, because I launch Chrome much more often than Rosetta Stone, and “Chrome” has “ro” in it. Again, a mistake. I never type ro when I’m launching Chrome. Never. So the smart thing for Launchy to do is to notice that so it could guess better at what I mean.

I hope that developers writing frecency algorithms would listen to this advice so we could all have smarter programs!

[ramrachum]

Why making a cool project is a good idea for an aspiring software developer

Why making a cool project is a good idea for an aspiring software developer

As a software developer, I get to know a lot of other software developers. Many of the software developers I know are either running a startup, working as an early employee for a startup, doing contract jobs for a startup, or dreaming of one day starting a startup. My point is that software developers are very interested in startups, either starting them or working for them.

My advice to anyone who’s interested in startups: Before you ever start a startup, try to make at least one cool and useful project. It doesn’t have to make money; in fact, it’s better if your project doesn’t even try to make any kind of profit.

Why is making a cool project important?

When you’re doing a startup, you essentially have to do two things at once:

Create some kind of product or solution that answers your users’ needs; this is usually the software ingredient of your startup; and

Create a business around that product that brings in money for your company.

Both of these things are difficult. Really difficult. It’s easy to mess either of these up. And you have to do them both at the same time. Get one right but mess the other one up, and you still failed.

Here’s a metaphor: Starting a startup is like juggling while riding a unicycle. Riding a unicycle is hard. Juggling is hard. Doing them both at the same time? Really freaking hard.

If your goal in life was to juggle while riding a unicycle, would you get on the unicycle while holding a few juggling clubs and try to juggle while riding the unicycle? Not if you have any common sense. You would first try to ride the unicycle. You’d take your time until you’re sufficiently skilled with that. Ditto for juggling. Then you’d carefully try to do both of them at once.

The same applies to startups. Making a successful product is hard enough; it’s all too common for companies to invest a lot of resources in making an ambitious product, and when it’s launched it’s just “meh”. 

Building a business on top of a product is hard as well. There are many places where you can screw it up, both by making technical mistakes or strategic ones.

As you can guess, the chance of you messing either the software side or the business side is very big.

Start a project, don’t try to make money.

That’s the essence of my advice. Start a project which is free of the burden of profit. Sure, you won’t get rich off of it, and you’re gonna have to support yourself financially while you work on it, but it’s your best chance at learning how to make a successful product– It definitely has a much better ROI than going to college.

Not being burdened with having to make a profit will make it easier for you to build a successful product. You’d never have to think about limiting or exploiting your users, which is something that many software businesses have to do. Think about those crippled “starter” versions of Windows that sell for cheap, or DRM, or annoying ads on websites, or desktop software that installs annoying toolbars and spyware. These are all attempts to make money by limiting or exploiting users. Software needs to act like a butler, which is difficult enough; imagine how hard it is to have a good butler who also tries to upsell you some crap as he serves you.

As a person who’s making a project that doesn’t aim to make money, you’ll be free from all of those dangerous practices.

Making a useful product is still hard, mind you, and you still have a good chance of failing; but at least when you crash from the unicycle and try to pick yourself up, you won’t have juggling clubs falling on your head.

[ramrachum]

Mike Driscoll interviewed me on his blog

Mike Driscoll interviewed me on his blog

I recently had the pleasure of being interviewed by Mike Driscoll on his blog, The Mouse vs. The Python.

Mike is well-known in the Python world and especially in the wxPython user group. He often posts tutorials for beginners on his blog, and it's happened serveral times that I googled a technical question and found the answer in one of his tutorials.

Head over to Mike's blog to read the interview.

[ramrachum]

PythonTurtle makes it into Saudi Arabia's official state curriculum

PythonTurtle makes it into Saudi Arabia’s official state curriculum

I just heard some very exciting news.

Six years ago, when I was just starting out my development career, I made a little program called PythonTurtle. It’s a program that helps children learn how to program in the Python programming language, which is the programming language that I use it my day-to-day work as a web developer. I created PythonTurtle as a side project, because I saw there wasn’t a viable solution for children to learn how to program in Python. I figured there should be a solution, so I spent roughly two months of hard work building and releasing PythonTurtle.

Screenshot from the program:

What’s special about PythonTurtle is that it lets children learn programming in an exciting way that puts emphasis on fun and creativity rather than technical details.

When using the program, an illustrated turtle is displayed on the screen, and the children can program it to move around the screen and draw lines. The more programming concepts the children learn, the more impressive drawings they can create with the program. This gives them motivation to learn and improve their skills without feeling that it’s being forced on them by their schoolteacher.

PythonTurtle is based on an educational program called LOGO that was developed in the eighties; what I made is in fact a modern version, so instead of teaching programming in a didactic language, it taught programming in the Python programming language, which is a real language used in the industry today. The idea is to bring children closer to the techniques used in the real world, and possibly plant the seeds of a career in software development.

Because I was just starting out as software developer back then, I didn’t have the skills that I have today, and developing this software was hard for me. There were technical challenges (specifically modifying the wxPython shell to be able to command an auxiliary process.) These challenges were so hard, that it looked like I wasn’t going to solve them, and at a few points I considered giving up on the project entirely. I was asking myself, why am I even doing this? No one even knew I was working on this program, and no one seemed to care.

But I told myself that I’m creating something big here, and it’s important that I see this through to the end. So I did, and I overcame the technical problems.

I released the program as open-source under the MIT license, which means that every person on Earth could download it and use it free of charge. I decided to release it that way rather than as commercial software because I figured more children could use it if it was free, and that seemed more important to me than making a few bucks. I also liked the idea of contributing back to the open-source community, because so much of the software that I use every day is built on open-source software that was made by volunteers, so I was happy to contribute my share of open-source software.

I released the software for download and I submitted a link to the website to tech forums such as HN and Reddit, and over the next few days, the story blew up, and thousands of people visited the website. I was very happy and proud that people liked my project so much.

Over the six years since I’ve released the program, I’ve gotten many happy emails from teachers and parents who used the program to teach their children to program. It’s always heartwarming to get these emails. They come from all over the world: From the States, from the UK, from Africa, Australia, South America… I would occasionally also get emails from children themselves, and one time even from a 80-year-old man who said that he used my program to learn to program himself. I got more reports of adults enjoying using the program. Looking at the analytics for the website, I saw that PythonTurtle was downloaded almost 100,000 times, which made me very proud.

But last year, I’ve noticed something odd. I was checking how the site is doing on Google Analytics and saw that I’m getting a disproportionally large number of hits from Saudi Arabia.

Specifically, there was a big peak of Saudi visitors around January 2014, and than that peak appeared again in January 2015. I also got more feedback emails from people with Arab-sounding names. I investigated why, and found a Saudi forum where PythonTurtle was mentioned. The text was in Arabic, and I tried translating it to English using Google Translate, but the result was too hard to understand, so I let it go and didn’t investigate further.

Until a couple of days ago, I got an email from a teacher from Saudi Arabia about PythonTurtle. He told me that PythonTurtle is being used in all high-schools in Saudi Arabia! The ministry of education of Saudi Arabia has put PythonTurtle into the official state curriculum! This means that it’s being used by more than 4,000 schools which teach more than 700,000 students!!!

I’m very excited to have made a program that has helped so many students, and especially the students in Saudi Arabia. I’m an Israeli, and there are no diplomatic relations between Israel and Saudi Arabia. I’m an ignorant regarding the political affairs between the countries, but I’m happy to see that open-source software has no borders; if a developer in one country makes a program that can help people, it can be used everywhere and help people all around the world, regardless of the political situation.

[ramrachum]

The interactive homebrew encryption challenge

The interactive homebrew encryption challenge

(UPDATE: THE ENCRYPTION REMAINS UNBROKEN! I’m still waiting for the first hacker to figure out the algorithm and complete the challenge.)

There is a wide consensus among security experts that when choosing an encryption algorithm, it’s much better to choose a well-known public algorithm rather than a homebrew one. I completely agree with this approach.

However, I am curious about how easy it is to break a weak, homebrew encryption algorithm. When you’re faced with a seemingly-random piece of data, a bunch of ones and zeros that you have absolutely no context for, and you are only told that it is encrypted with no clue as to how the encryption algorithm works, how would you even know where to start at decrypting it?

To shed some light on this mystery, I’ve decided to do a little experiment here on my blog.

I’m challenging you to break a homebrew encryption algorithm that I’ve written.

Clarifications:

You will be given a ciphertext, and you’ll need to find the plaintext that was encrypted.

The algorithm will of course not be revealed to you. That’s the point of the contest. The plaintext itself is just a bunch of arbitrary text, serving as proof that you figured out the algorithm.

You are given an interface in which you can submit any plaintext  that you desire, and see its encrypted version.

I’ve written this algorithm for the purpose of this contest. I have no training in cryptography. I do not condone the usage of homebrew algorithms for encryption.

The winner is the first person to tell me what the plaintext of the ciphertext below is, either via email or in the comments.

A modest award of $50 will be given to whomever is the first to solve the challenge. Fine print: I maintain sole discretion to decide who is the legitimate winner, if any, and how to transfer the money. (Probably PayPal.)

The winner, if any, will be announced here on the blog. I’m hoping the winner will produce a write-up of how he cracked the encryption.

Here is the link to the encryption interface again.

Without further ado, the ciphertext:

7F1CA699D902CD5EB179E3674958DB0836A7EFC7031075F559 385F61C1C70EBF9F1E3A1EDD784541423D5C1F06B6A7CB4AD7 B6E52567F26095511EEEBE6AB241BE683B0F7A4D000EE7AF51 914B73028EBFA20CA64F539B399456FD71B71456F62A7D5D5C D4FB2D41F0C8B800B028330CDA79EB3A1136D267C97EA55278 63AA057E72D10F5BD374B1D280C741A1E5A0C58847FAE73F11 E64A0530322B095803ABEAA9395E7334B78F828A5804A48B50 73F726C038B338CAAE09FF3843A65110204F9F2848E1860567 2CB001A07A8DEC74FF8C4F12C5675FC5F6A9A6A253E996DF40 DF23E7936B0508DB9570F45FD9ECB4F7E288BFBCCFD56453E0 8B3F78C190CF9DCC91431031B21CA6F04B3E594244D8B375AF F4C6E6A98689CE42EC3CAA92092A8509653E842B2BA68C4775 9FD01B85004568F41E1D3B68DBBD94DF0EE2CFF0DD35D527FB E6BE2C14064CAD72EEADF41769E84A64C39A915E4156EA9F20 B213D6C85BD713F48614000E484FB3410BBC8BECD509F0F51D E5E644D4B1A61C8AA851228046EDF0ABD9A68B593E5555FF2D 348C5B54977B8293F2C9EA40D351D69F862496A664B589E6EC 995733248C4EEDFE94B34521F5D20D5C8A083209128B3CD015 82EDAC1FC7CF3FC5A5BA5FF03EBD94A754AD71255F06E84302 06E10BB32EF7DF17DCDA1AA58D315797AC25F6B1E666D6198F 15A257A3094565860448D2E2657F360F3AECBA2AC93EE74762 6A8391FBA928C23261B83AB46043551F4C45CE4B334D79E7F7 AC8CC6C71A0496D425282BDBDC4BE188A89ADC72B610553741 2E458894C661D5BE88749A56E2796B585B8B1C50249A1862FA 4F823B56597BDB1B7325A6B3FD753AB5075656F1E6F34C51BF 16AEF334BFF1C92707114111D72273CE745337D64C5A9D37BD 48E3A4F1360CD023E0665B5B727F42EED30929F51DA5E56883 EDCB6B9F753D7694A0DC232AFF917A3F558E4A06A4303A5A78 599AA6F797800E1BD6C83A0C46F611B21F3DA97914F07A0013 993225AE350B61EF0321EBD141DC25D56C46F7628D92411A89 33AA4E650FBA3011FACA44C20CF9B8D2C3C4D2308B97B928C8 53E9A5398663D8DB12980BC15D536F00BFE8EA53920F8467AE 8962D1D52271F300D3C4E87EFFECBFAAE0CAC64D1B7FEDCF80 9DFF89E85FB0E326442B3BB2332C8A91C300E07A188A83DE1A 1B4B077B6787FBED987B8D0C12A1A2F868A8C84D6570CAE519 F60429E2AD6BEC6ADF39B8C390804B7D5B4CACE48851E694C3 F798FE34286A9E7DD5BF65EBCE3CC28287770A96FA356C37F7 02FCEFE617E7AB088B7E9A27445EAAC77C32D1E8C684DACCDA 386AD6C3CFFE148C98320910433C98BAFD3218A0F74CF0537B AE6D9A36E434116B8B087930F2B42A11575C587163D2944D1C BE4C61DD3691417D8A07C3CB51EC2126AABB1B97F78AFCA96B 3C76

Let the hacking begin!

[ramrachum]

Can someone break my weak home-brew encryption?

Can someone break my weak home-brew encryption?

A newer, interactive version of this challenge is here.

There is a wide consensus among security experts that when choosing an encryption algorithm, it’s much better to choose a well-known public algorithm rather than a home-brew one. I completely agree with this approach.

However, I am curious about how easy it is to break a weak, home-brew encryption. When you’re faced with seemingly-random piece of data, and only told that it is encrypted with no clue as to how the encryption algorithm works, how would you even know where to start at decrypting it?

To shed some light on this mystery, I’ve decided to do a little experiment here on my blog.

I’ve encrypted a piece of information using a home-brew algorithm that I came up with and implemented in a few minutes. I won’t say much about the algorithm (since the point of this exercise is breaking the algorithm without knowing what it does) but I will say that it’s the result of several steps, each of which is easily breakable.

The data is posted below in hex. I’m very curious to see whether anyone would be successful at decrypting it, and even more curious as to how that was done.

Edit: Just to make it 100% clear: I will never use something like this for any real encryption need. This is just a puzzle. Never use anything like this to encrypt something even remotely valuable. I do not condone using home-brew encryption algorithms.

Without further ado, the encrypted data:

5A9E938B9A9BA0A0D5B9DB42EDD9A0F4 AFAF2780C9D941ED0BE6C8E6DD0DD082 05845029B203AE4A2C4169BFAEB6FDD9 AD461C685A700427C0E440EDE287FD31 7710ACA70B5C059D3B3EA6F58880EB70 4132D80D9F596BB1DC5922D3331BEFED 7D4C9146091F8093109EF21DE22B387C 10ADA38DF91FFBE4305B68B87A563447 AA97364600CC682F2328E8733BBA252E 3C7F93A1805B6593834F647B5BB34356 061B30DBE73734A62426816D49BB185D C8C6D5D5FF87D5C6FFDD7DE4651FAB43 1F61AD0D4E5E5AD211FEB1D255E6D2BF 906D963C04DCC7BF2914263EF4866BFB 2A99CB704A568C0E23D66215C922F581 8CC31947C57B9F519161C3DBBA19C4AE F18EE25281999E4C867B69838870811F 307BB3368D884E959EABC97092DD6975 232B8642424B66B67A326F6277231D98 18CBB796693D28BD92BE26956241589B BC28C98C903D9D6AB976FB55D57B0711 681232107038820F0AA96BD7B11ED190 1897FA07905304EA128355A3BE409379 C2484D40DB45F3A5C535AF736F492B92 63E46B078D3B74092656E3D1BEC2FCC4 46895AAE93FE7CA6D281C84E7AEAEDE4 98F707F9829324AD9C3E7C9A4017B710 CC6DDBB048699E1346AADD538853F037 7B1E139212A1055F331CAF1027ED4955 3F0C2DA2CF2C1D72173B77287893146D 93BEA605DD46B6A3BD77A95F1FF66A56 2B8A2D11804907DAFA6F64DAFB1F81A1 D70187301CE1DA816C11D5E87FBED853 9788333E1426FCF6926236B42C72A6E0 39FBB728C37633A2AD4088D42E13E005 F544D0ACE4ED0AF8EDC82D0F0D5CBE07 93096E3AB1484A2E1622077E458CE9B0 D3BB4E128DB3112706E1E55EC85D41DC 164C8177D296BCCB87FF18295E54F0A7 3A09ED54D9DC0406D97B4E2529BFB3A4 22484AE361AC1970818219409B464A9B 2BD4269F4B5901A389DF30CF7A1D6CF5 8BB39C6BA76B0B0EAEF64F516C15E166 D9BE6A256985601FBD1B2FC7FA0DDFF6 7D4989B264DE32A13B6A9385E0581D90 DB10B0C71FE99C9121B0AEC58303C61B B4F74F0E35FB4D1AE38836630C368A92 C1C4E5ACBB4B1B390246503A174ACFEA 08674C50BA7347558BC4F967BF154EDE 67ECEABEF97CE930F2539906F8A97B8C 3DEBE7C0AC8EFEDC9AC25AE1FDA9C941 3B16DECC5EDB562CE7AC386AB25D9662 275FE07DC8F59468C3C918FEC8BC6830 A3DE122885D834104F71707458D91917 6105614B1156884C5CD9D1BBCC1F25D2 6EE91F048FB9F9D3B9504B85D25CB317 36A5030C8EFD2EDA90C06FFA0AA44966 1B6259678FADFE8C27820398E867A8BA 45438A4802828ACFA8C0B6A89C32ED60 8286FD1125E99DD4B40AFFC3306E9BB9 242D7F4E026FA448A00D8D3F2E948D09

[ramrachum]

Concise explanation of why programming is fun

Concise explanation of why programming is fun

I’ve been asked many times by friends, acquaintances, family members and potential romantic interests the question: Why do you like programming so much? What’s so fun about programming?

Naturally, the people who usually ask this are those who do not have a technical background, and that makes it a bit difficult to answer without giving them a crash course in programming.

Here is my attempt at a concise answer that expresses the joy of computer programming without assuming any kind of technical background.

Imagine you’re building a house with brick and mortar. You lay down one brick. You lay another next to it. You lay yet another next to the second one. You keep laying more and more bricks, until you fill up the entire length of the wall. Now you start on the second line. You lay the first brick. You lay the second one. And on and on until you complete the second line. Then the third line and the fourth and the fifth, all the way up to the top. You finished building a wall! Now to build the next wall in your house. You lay down a brick. You lay another next to it…

Now, imagine the same exercise, except this time we’ll do it similar to how a software developer writes a program. You lay down one brick. You lay another one. You keep laying bricks until you finish the line. Achievement unlocked: You now have unlimited brick lines. You lay those brick lines on top of each other until the wall is complete. Achievement unlocked: You now have unlimited walls. You lay down 4 walls next to each other. Achievement unlocked: You now have unlimited houses. You build 100 houses in the same area. Achievement unlocked… Ad infinitum.

Of course, the metaphor is not perfect, and not all is happy-fun-time in programming-land. But I think this hits pretty close to what gives me the thrill about programming.

[ramrachum]

The `coverage` module celebrates 10 years today!

The `coverage` module celebrates 10 years today!

Today, December 4th 2011, marks the tenth birthday of the excellent coverage module! Here is the changelog entry showing it was created exactly ten years ago, on December 4th, 2001 by Gareth Rees.

The coverage module is one of the most sturdy and reliable pillars of my development toolset. It’s a hallmark of a great tool that you never have to spend too much with it– When I want to check the statement-coverage of my tests, I just use coverage (invoked via its Nose plugin), it just works, and then I can go back to writing my tests and ensuring they cover a big part of my code. You don’t have to spend too much time with it for the simple reason that it very rarely breaks– Which is not a trivial accomplishment for a tool that acts on every, single, line of code in your project.

Kudos to Ned Batchelder for his excellent maintenance of the coverage module! I’m a relatively new user of coverage, having used it only for about 2 years, if I remember correctly. So long-time users of coverage may have a bigger perspective on this module than I do. But in the relatively short time that I’ve been using it, Ned has been an exemplary maintainer, and the few tickets that I created on the bug tracker received his attention. Ned also made sure that coverage supported Python 3.x before it was cool, and on every release he takes the time to create Windows binaries of coverage to help Windows users avoid having to compile it themselves.

If every maintainer was as dedicated as Ned, the Python community would be in a better place today, and we would have probably already seen Python 3.x going into mainstream.

Thank you, Gareth and Ned, for this wonderful module!