14domino

[14domino]

Our site is looking prettier now. My friend Conrad is doing the design for the site; the above is a very early version of what he's already designed, and we have a team working to bring it to life. I'm mostly in charge of the backend, but doing a little front end work on the API. We finally got a full game timer working; I based the code mostly on lichess's. Why not use a fully functional thing for inspiration?

Josh C and I battle tested it for a bit with 1-minute games. (The card at the bottom right saying 15 0 is wrong, but we'll fix that later obvs#). Also, there are some silly phonies on the board :P

No real issues, everything worked smoothly enough. There is a ton of work left to make this the holy grail of Crossword Game, but I feel great about the fact that a full Crossword Game can be played.

We are splitting up the work into manageable chunks to try to release this later this year, at least as a beta. Account creation logic, friends, chat, beautiful scorecard, a functional lobby, a good rating system, etc are on the docket for the next few months, and then analysis features, AI bots, built-in tournaments, etc will hopefully come in the near future.

Exciting!

[14domino]

liwords

A small team of us have been working very hard on a project that I internally call "liwords". The inspiration is obviously lichess.org. Why not do for crossword board games what lichess did for chess?

This has been a dream of mine for a few years since I first saw that wonderful site. Although we are very far from being done, I hit a minor milestone recently on Twitch (I sometimes stream coding on Twitch).

https://www.twitch.tv/videos/636128298

There you can see several functions working (don't mind the UI, which is all over the place, some parts pretty, some parts broken -- we are working on fixing this).

The functions that work:

Seek a game shows the seek on both player screens

Clicking on a game being sought redirects both players to the game

Tiles are dealt out and the game begins.

You can type your tiles onto the board

The game logic is based on my Macondo project so it basically should work fine, but hooking it up to the frontend takes a looooong time. We are still quite far from being done, but making a little progress every day after work and on weekends.

[14domino]

Macondo Dev Blog - simming

I'm going to log more of my progress on the apps that I wrote about in an earlier post, in an attempt to:

1) make myself more likely to work on these apps 2) write a log for me and others and drum up some excitement! ;)

Monte Carlo simulation is basically working on Macondo. I expect that since I just got it working, that I'll discover some bugs and special cases, and there's so much more I want to do with it, but for now I'm excited that I got it working. As a nod to Scrabble expert Ron Tiekert, who might have been the first person to invent Scrabble simulations back in the 1980s, I plugged in the rack that he used to explain this concept, AAADERW.

In the 1980s many top Scrabblers were proponents of turnover theory; that is, they believed that the more tiles you turn over, the better. The theory certainly makes sense to an extent, if your rack doesn't have a bingo, why not race to the blanks, Ss, and other good tiles? So with the rack above, many experts might have played AWARD or WARED, keeping a rack of AE or AA (and nowadays, ADWARE is another good option). But Ron explained to Maven author Brian Sheppard that he preferred opening with AWA, and demonstrated to him how picking 50 random racks after AWA results in a higher score next turn than picking 50 random racks after the other longer plays. Brian explains that this was like "manna from heaven" and proceeded to co-invent the gold standard in Scrabble AIs so far - Monte Carlo simulations as applied to the game of Scrabble. This also disproved turnover theory, again, to an extent. The counter-argument goes that blowing up a potentially nice rack makes it just as likely that you would draw bad tiles as good tiles. The counter-argument to that is that it's situation dependent; turnover certainly has a place when the tile bag looks great, especially towards the end of the game if the blanks are still unseen, and you suspect your opponent doesn't have one, and so forth.

Here are some results after plugging in AAADERW as an opening rack, and simulating the top 10 moves for about 3183 2-ply iterations (100 seconds overall; I should be able to speed this up significantly with more cores and optimizations):

**Ply 1 (Opponent)** Play Mean Stdev Bingo % -------------------------------------------- 8C ADWARE 38.379 21.916 21.081 8H AWARD 36.233 22.763 19.384 8G ADWARE 39.977 23.149 21.426 8H ADWARE 38.391 23.625 21.426 8H AWA 30.548 19.686 14.923 8F AWA 30.111 19.716 14.860 8D ADWARE 39.559 22.138 21.112 8D AWARD 33.883 22.050 18.002 8E ADWARE 35.946 22.839 20.390 8D WARED 37.001 23.323 20.672 **Ply 2 (You)** Play Mean Stdev Bingo % -------------------------------------------- 8C ADWARE 39.689 21.931 19.981 8H AWARD 38.933 22.196 20.515 8G ADWARE 40.635 22.082 19.541 8H ADWARE 39.750 21.416 19.133 8H AWA 45.639 25.015 37.041 8F AWA 45.594 24.901 36.789 8D ADWARE 40.212 21.328 18.410 8D AWARD 37.740 22.550 20.138 8E ADWARE 38.484 21.037 18.724 8D WARED 31.642 17.380 9.237

I still need to calculate the overall "equity" for the play; basically the spread difference, but the results seem close to Quackle's; for example with roughly the same number of iterations, Quackle has:

Ply1 (opp): 8C ADWARE 39.3219 23.9456 25.2424 8H AWA 31.5039 21.7387 17.6103 Ply2 (us): 8C ADWARE 40.5299 24.0505 22.8965 8H AWA 46.0031 26.7589 40.4442

A few things I notice here is that I might still have some bugs; my bingo % for opp after ADWARE seems a bit low. I know much of the time it's close to 25% when you are going 2nd, and there are some nice letters for them to bingo through. The mean scores are close though, so I'll investigate more.

In any case, after this my plan is to get multi-core simming working and some sort of graphical logging (for heat maps), and then work on a pre-endgame engine. My endgame engine is finally bug-free, I believe, but quite slow since it's exhaustive. It still should work great for the majority of endgames, but at some point I'll have to rewrite some pieces of it to make it faster.

[14domino]

I have a lot of ideas for improving the state of tournament and recreational Scrabble, mostly revolving around the technological aspect of things. I am aware that there are many other ways in which it is lacking, but my expertise is in technology, and that's probably the best way I can make an impact.

Let's start with some projects I have already built:

Aerolith

https://www.aerolith.org is my word study website. It's been around in some form or another since 2007; I believe I launched it as a downloadable desktop app around the time of the Players' Championship. I rewrote the whole thing in 2011 as a web app, a considerable undertaking as I knew nothing about web programming, but the experience acquired here helped me obtain a job as an early employee-turned-cofounder at a YC company (Leftronic), which was then acquired a few years later, giving me some minor version of success.

I've used it as a bit of a technological testbed; I added and removed sockets at some point, I've rewritten it with the latest in JS tech (React.js at the time, although it looks like that's here to stay for a bit longer), and played with backend deploy technologies like Kubernetes (mostly a mistake, honestly, especially for such a small site). I get a few hundred users a month, around 450 monthly actives and 150 daily actives last I checked. It's a nice site that I'm pretty proud of, and is mostly self-sustaining in that I get just enough in donations to cover running it for free.

As far as the usefulness of it, it's a bit debatable. I certainly like using it and have learned a lot of words from it, and many people enjoy it and tell me about it, but it has at least a few glaring omissions. The one that comes to mind is some programmed way to study; i.e., something like spaced repetition / cardboxing. I've collected a ton of data on word difficulty that I could use, but I haven't had much time to add more features to my app recently. The deployment of it is also suffering, and I need to greatly simplify it.

For the future, the main things I want to add to it are spaced repetition / some sort of intelligent way to study words, which would keep per-user and global statistics on words that are easy or hard and know how to ask these of the user. I think this would make the tool much more useful. There are also a ton of small improvements, like showing previously guessed anagrams per question, allowing user to easily continue when their time runs out, untimed quizzes, etc.

Macondo

Macondo is not released yet; it is intended to be a machine-learning version of Quackle. I am still quite early in the development of it, although I have made a bunch of progress. Mostly, it can play games against itself with some preliminary "superleave" values for every rack. We need to determine these superleave values in a better way and have them converge to, or at least be close to Quackle values (or explain why they're not).

Machine learning hasn't been done with Scrabble yet, but I think this field is ripe for innovation. I think a proper machine-learning enabled AI can beat Quackle's Championship Player at least 60-70% of the time. The ideal place to showcase the finished AI would be in a best-of-X match against a top player (ideally Nigel Richards!) with some sponsored prize money on the line. I think this would provide great publicity for the game of Scrabble. Additionally, the tool would make huge strides in making players even better.

Some improvements I would add to it as well:

Automatic rangefinding

Heat maps for simulations, other statistics

A great pre-endgame solver, with a UI clearly showing winning and losing variations, etc

A great endgame solver (this is a prerequisite for the pre-endgame solver). The Quackle one is good but has issues with stuck tile endgames. I have spent a ton of time building a minimax-based solver with alpha-beta pruning; it's quite slow and buggy. I think I can significantly improve its speed.

Lots of graphs (showing values of tiles as the game progresses, winning % as game progresses - for easy analysis of famous games, etc)

It's still quite early and it's hard to find time to devote to both projects and my full-time job. Progress can be found here: https://github.com/domino14/macondo

Crossword Online

Or whatever we want to call it. The state of the art for playing online Scrabble sucks to put it mildly. ISC has an ancient interface, may actually have legitimate randomizer issues (I don't fully buy it but I've heard enough anecdotes and some data), and its phone app is not very functional. EA Scrabble is awful and is going to be discontinued. Its replacement is not getting rave reviews. We need something akin to lichess.org for Scrabble - this beautiful online Chess site is mostly a one-man job it seems. It is my dream to build something like this for Scrabble, but a giant undertaking in terms of time. I actually started on a game viewer and made a lot of progress on the front end for this app, but have almost no server code, and abandoned it once I saw ISC was releasing a phone version. However, after seeing the phone/browser version of ISC it's clear we still need a good solution.

From another perspective, I think an app that allows us to have proper tournaments would be great for the environment. I don't think it's sustainable in the long term for people to be flying everywhere constantly. I'm not saying the world will turn into Ready Player One anytime soon, but with an awesome app and proper anti-cheating technology we could start having some more high-profile tournaments without players having to travel to each other. I do, however, still prefer the analog feel of Scrabble, the actual tiles and board.

Once I decide to tackle this again, I should be able to salvage a lot of my front-end viewer code, and combine it with Macondo to provide a back-end engine. I'd just need to write multiplayer code. The app should support both live games and correspondence games.

Phone apps for everything

Conrad has designed some nice mocks for Aerolith Mobile, and the Crossword app should also have a mobile app. I can probably use React Mobile to make it easy to cross-develop for Android / iPhone. I've never built a phone app before, but can always learn!

Aeroblogs, articles, etc

People write ephemeral tournament reports / blogs / strategy articles nowadays as Facebook posts. That's not ideal. They're hard to search, and if you're not on Facebook when they're written you might never see them. LiveJournal was cool while it lasted but people moved on. I think we need a proper blogging platform with actual Scrabble integration (imagine pasting a board diagram in ASCII and it generates a viewable board, with comments, etc, or links to tournaments on cross-tables, etc etc).

I also want to have strategy articles and video content easily available for search on this site. I'm thinking it could be something like channelfireball.com (for Magic The Gathering). We'd need to get content makers, but I think we have a few people who are willing to do that.

What do I need?

Time and funding. Even though I have some time on weekends, I need to be better at managing it. I am considering setting up a Patreon to see if people are interested. None of this makes much sense if people are not interested - after all, people vote with their wallets.

[14domino]

Building an emulator

I have been coding in one form or another since I was around 8 years old, on my brother's calculator that had around 450 bytes of program memory, and I've been videogaming since I was 6 (my dad bringing an NES from the US back to my native Caracas was one of my happiest memories). But why not combine both those loves?

I've been wanting to make an emulator for ages but for some reason or another I've kept putting it off. I'm still always busy but I figure it's time to start a cool new project. My target will be emulating the SNES, probably my favorite console of all time, to the extent that it can play FFVI, Yoshi's Island, and Mario RPG as well as I can make them - the latter of those two use different coprocessor chips so those would be fun to emulate as well. As an electrical engineer by trade I do understand roughly what it would take to write a console emulator.

However, I hear the SNES is notoriously difficult to emulate. The bsnes creator has made it his life's magnum opus to write a perfect SNES emulator and it seems extremely involved. I don't necessarily need to emulate processor bugs in Speedy Gonzales, but something that can play games acceptably is probably fine for now; I just need not to let my perfectionism take over.

To start with, before anything, I plan to emulate the Game Boy, which should get me the practice needed to write something more involved. Apparently, it's rather easy to write a Game Boy emulator, and this was another console I played the hell out of (Link's Awakening FTW). I plan to use the Go language, which is one of my favorites, and whatever display libraries are used nowadays (OpenGL? even though apparently the Mac won't support it soon?). I'll post with more updates when I actually start.

[14domino]

Aerolith.org is temporarily down for maintenance

Hi all, something seems to be really wrong with the server and I've taken it down for maintenance. When it comes back, unfortunately multiplayer won't be a feature (for the time being). Something with the packages that I am using makes it destabilize the whole server, and very few people were using it, so I trust this won't be an issue.

Aerolith should be back within 24 hours of this post. Sorry again!

[14domino]

docker run -it -p 8181:8000 -v /home/ubuntu/word_db:/db:ro --env-file webolith/config/config.env domino14/webolith:latest

(and 8180 instead of 8181)

on home machine

ab -kc 100 -n 1000

For haproxy stuff

apt-get install ruby gem install haproxyctl

Need a haproxy.cfg file that exposes the socket. Need latest HAProxy:

https://haproxy.debian.net/#?distribution=Ubuntu&release=trusty&version=1.6

Take down a server cleanly

sudo haproxyctl set server servers/server2 state drain

Need to set up health checks, and figure out when a server is done draining

[14domino]

A cool post on doing proper database migrations.

[14domino]

LOL

This is hilarious:

http://www.openbsd.org/cgi-bin/cvsweb/src/lib/libssl/ssl/Makefile?rev=1.29;content-type=text%2Fx-cvsweb-markup

The CFLAGS in the commit contain: -DOPENSSL_NO_HEARTBEAT

The actual setting to disable the SSL heartbeat is -DOPENSSL_NO_HEARTBEATS (note the S at the end).

This commit was apparently OKed by two other people.

This right here tells you why code is hard. Hubris anyone? Is there a coding equivalent to Muphry's law?

[14domino]

Facebook insights have different return formats.

https://graph.facebook.com/me/insights/page_impressions_unique/day?access_token={{TOKEN}}&since=1388649600&until=1391241600

1388649600 = 1/2/2014 12:00:00 AM GMT-8

1391241600 = 2/1/2014 12:00:00 AM GMT-8

returns something that looks like this:

{ /* previous, next page, etc keys */ "data": [ { "values": [ { "end_time": "2014-01-03T08:00:00+0000", "value": 4 }, { "end_time": "2014-01-04T08:00:00+0000", "value": 1 }, { "end_time": "2014-01-05T08:00:00+0000", "value": 0 } .... /* More 0 values here until Feb 2 */ ], "period": "day", "description": "Daily: The number of people who have seen any content associated with your Page. (Unique Users)", "title": "Daily Total Reach" } ] }

Whereas

https://graph.facebook.com/me/insights/page_views/day?access_token={{TOKEN}}&since=1388649600&until=1391241600

returns:

{ /* previous, next, etc keys*/ data: [] }

It seems at some point that Facebook has fixed the problem where some insights are always delayed by two days, because I obtained up-to-date data for "page_impressions_unique" (as a matter of fact, I have data for January 4 at midnight even though right now it's January 3rd.. hm.. and the timezone is supposed to be PST for Insights data). But for page_views there is no data available yet, so I get a blank array. So why don't I get an array full of 0s, Facebook?

Of course, I probably should not be requesting data in the future. That bug is my fault, and it was uncaught because Facebook always returned no data if it wasn't available until sometime recently apparently.

In any case, this bug sucks. The fix is probably to request data only until the current day's midnight in PST, but Facebook should be more consistent with its responses and not change them randomly. And yes, I understand that nowhere in the specification does it say what should happen if you request data for dates in the future.

APIs are broken.

[14domino]

stupid Python logging woes

Recently moved a project to Django 1.3 and I kept seeing duplicate messages for all of my logger.x messages, even though my LOGGING dictionary looked like: LOGGING = { 'version': 1, 'disable_existing_loggers': True, 'loggers': { 'base': { 'level': 'DEBUG', }, } } I simplified it to this to see if there were any issues, and although I define no handlers for the 'base' logger, it still was printing out logging messages. I did a grep on the entire virtualenv libraries for basicConfig or BASIC_FORMAT to see if any of my external libraries were misbehaving and didn't find anything. I went through my codebase again and got rid of some extraneous logging.debug in some files and that fixed it. So always use import logging logger = logging.getLogger(__name__) logger.debug(....) instead of logging.debug directly!

[14domino]

Let's try my first post

 YOM is better than the bingo CO(E)MpLOY