nimmerwoner

[nimmerwoner]

Compiling GDAL with OpenCL on Ubuntu

I needed to apply GDAL operations to very large rasters, so I needed GPU acceleration for these. Our cloud infrastructure has GPUs available, but unfortunately Ubuntu's prebuilt GDAL binaries come without.

I found instructions on the internet how to include it, but they were outdated. The below instructions worked for me on Ubuntu 16.04 compiling GDAL 2.4.2. I strongly suspect the approach works the same on later Ubuntu versions, and only different in flags for later GDAL versions.

I based my method on this and this source.

# Add the nVidia cuda repo curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-ubuntu1604.pin sudo mv cuda-ubuntu1604.pin /etc/apt/preferences.d/cuda-repository-pin-600 sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/ /" # Install the necessary Ubuntu packages; I needed cuda-9-0 # Note: opencl is in a separate package, make sure it's the same version as your nvidia driver is (modprobe nvidia; modinfo nvidia) apt update && apt install cuda-9-0 nvidia-libopencl1-455 # Download and unpack GDAL wget http://download.osgeo.org/gdal/2.2.4/gdal-2.4.2.tar.gz tar -xvzf gdal-2.4.2.tar.gz cd gdal-2.4.2 # Set ENV vars to point GDAL to Cuda/OpenCL export LD_LIBRARY_PATH=/usr/local/cuda/lib64 export PATH=/usr/local/cuda/bin:$PATH # Compile with Cuda and OpenCL support # Note that --with-opencl-lib=cuda is necessary ./configure --prefix=/opt/gdal \ --with-threads \ --with-opencl \ --with-opencl-include=/usr/local/cuda/include/ \ --with-opencl-lib="-L/usr/local/cuda/lib64 -lOpenCL" make sudo make install

[nimmerwoner]

Getting haskell-gi started: writing hello world

Curious how to write a proper gtk application in haskell? Unfortunately with gtk2hs now deprecated most available documentation is too. So what follows is a haskell translation of Vala's documentation on how to start a gtk3 program.

It's reasonably self-explanatory. The hardest work is finding where these functions are exported. Luckily it's all in the import list now. Make sure you have gi-gtk and gi-gio in there.

{-# LANGUAGE OverloadedStrings #-} module Main where import Data.Text import GI.Gtk import GI.Gio (onApplicationActivate, applicationRun, ApplicationFlags(..)) main :: IO () main = do app <- applicationNew (Just "com.example.application") [ApplicationFlagsFlagsNone] onApplicationActivate app (activate app) applicationRun app Nothing return () activate toepassing = do w <- applicationWindowNew applcation windowSetDefaultSize w (fromIntegral 400) (fromIntegral 400) windowSetTitle w ("Example application") l <- labelNew (Just "Hello, GTK") containerAdd w l widgetShowAll w return ()

[nimmerwoner]

Working around Failed to load interface for ‘Data.GI.CodeGen.CabalHooks’

While trying to to get the latest version of haskell-gi to work, I stumbled on this error in the stack solver lts-8.16: Failed to load interface for ‘Data.GI.CodeGen.CabalHooks’.

What solved it for me was moving a few version back, namely to 8.12. No modifications to my stack.yaml or package versions were necessary.

[nimmerwoner]

Speeding up haskell-gi build times

If you use haskell-gi (automatically generated gtk bindings for haskell), you might have noticed that build times can get very very long, especially if you develop on a laptop. Luckily you can disable stuff to make compilation and development much faster!

So the cause of the abnormally slow build speed is that haskell-gi makes use of OverLoadedLabels. This allows library users to write #add container widget instead of containerAdd container widget. Very useful, but unfortunately too heavy for some computers (like my laptop).

This can be disabled for each haskell-gi package individually, by setting the build flags overloaded-methods, overloaded-signals, and overloaded-properties to false. An example by Leksah can be found here.

The trick is copy the extra-deps and flags sections to your stack.yaml. Remove packages you don't use from both sections. In my case gi-soup, gi-webkit2, and gi-gtksource, for example.

Now recompile! It should be much faster.

[nimmerwoner]

Gtk2Hs: Creating modal dialogs

So dialog windows are pretty useful for presenting information and asking for user input. That’s why Gtk3 (and earlier) support such windows, but unfortunately the documentation for Gtk2Hs, the Haskell bindings to Gtk, are partially out of date and don’t apply always anymore.

I make thankful use of this library however, that’s why below I’ll quickly discuss how to actually create a modal dialog with Gtk2Hs.

The idea

I’ll start with immediately giving you the code, because that’s what you want anyway:

import Graphics.UI.Gtk3 openDialog :: WindowClass w => w -> IO () openDialog window = do     dialog <- dialogNew     set dialog [windowModal := True, windowTransientFor := window]     dialogRun dialog

The current documentation recommends you use dialogNewWithButtons and then apply the function setModalDialog. These functions don’t exist anymore, though.

What you actually need are the function set, to set attributes, and the attribute functions windowModal and windowTransientFor. You also need the parent window of the dialog if you want it to be blocked.

Put the pieces together in the above way and you get a dialog.

Note

The above remark about the out of date documentation applies for Gtk2Hs 0.14, which is also the version for which the above code work.

[nimmerwoner]

Giving a face to neglected books: Retrieving book artwork from the net with Haskell

Books nowadays get beautiful covers that are perfect for recognizing them in the blink of an eye. Just look at the below example, which books can you find more easily?

The blue icons could be any book. The books with a thumbnail, however, are easily recognizable. In this article, therefore, we will see how to programmatically search Google Scholar for artwork. Downloading the image and inserting it in the ebook would be a next step.

The plan

So how would we achieve this? First we find identifying information about the book in its metadata, as explained previously in another blog post. Then we question Google Books what they know about this book. We analyze their response to search for the id they use. We can later use this ID to download the cover. Schematically, this becomes:

Search book for identifying info (explained here)

Google the information

Search their response for the book’s id

Download picture (not explained in this blog)

Imagine that we are trying to search the cover of De schaapherder, the above mentioned book. If we follow the above plan, it goes like this:

We take “de schaapherder” and “oltmans” (the second book from the right in the picture);

We use google’s api to google it. We can do this by appending the above words to the url https://www.googleapis.com/books/v1/volumes?q=

From the json response we take the volume ID;

We use that ID to download the related picture by appending that ID to this url: https://books.google.com/books/content?printsec=frontcover&img=1&zoom=2&source=gbs_api&id=

Supporting code

Now we’re almost getting to the real meat of the job. First we need to install some libraries to ease our work, namely wreq and aeson. The first will allows us to do http requests with a simple get command (that is, read internet pages). Aeson is a json parser needed to understand Google. Install them like you normally do, through cabal or stack.

We also need some standard library stuff like Data.Text and .Vector.

Add the following lines to the top of your file:

import Network.Wreq import Data.Aeson.Lens import Data.Text import qualified Data.Vector as V

The code

We’re finally here, the actual code! Converting the above steps 2 and 3 into Haskell code, we get:

-- Google’s url googleBooksUrl = "https://www.googleapis.com/books/v1/volumes?q=" -- Search Google for a book googleVolumeId :: Text -> IO (Maybe Text) googleVolumeId keywords = do   responseJson <- googleABook keywords   let items = responseJson ^. responseBody ^. key "items" . _Array   let maybeId = (items V.! 0) ^? key "id" . _String   return maybeId -- The actual HTTP request; it also replaces spaces with + to make a correct url googleABook keywords = do   let url = (++) googleBooksUrl $ unpack $ replace " " "+" keywords   get url

The function googleBooksUrl is only there as a constant for the url we need to consult. The last function, googleABook, is the actual HTTP request.

The function googleVolumeId weaves it all together. It commands the search, parses the json in the response, and returns an ID if it finds one.

Now for the technical explanation! Specifically, the strange operator ^. tell the next function to look into the result of previous. So responseJson is Google’s response to our query. ResponeBody takes the body, then key “items” looks in there for a json key-value pair named “items”. The key’s corresponding value is converted to an array by _Array and assigned to items.

Next it’s necessary to only access the first item of that array with index number 0 and search in there again for a key-value pair, but named “id” this time. That value is converted to a string (a text value) by _String.

But because we’re not sure whether this step will succeed, we use the ^? operator. This makes sure that if the array is empty, trying to access it will not crash the program. It’s also what causes us to return a Maybe value instead of a normal value.

Warnings

Be warned that for brevity the above code doesn’t anticipate errors. Using it as-is, unmodified might crash your program. Secondly, while Google doesn’t seem to mind consulting their api for low traffic purposes, for high traffic you might need to register and get a special key with them.

Continuing

So now you have the Google Books ID, what next? You can append it to the earlier mentioned url in step 4 to actually download the image. To give you a hint, you can again use Wreq to perform the download. Then you only to write it result out to a file. Good luck!

[nimmerwoner]

Haskell's epub cover extraction

Problems

Unfortunately, this is not as trivial as it might seem. This is because, firstly, there are many ebooks in circulation in the old version 2 epub format which didn't specify how a cover should be included. As a consequence there is no single standardized guaranteed-to-work method to find a cover.

Secondly, even though version 3 of epub does define a standardized way to define the book cover, it is not yet supported by the Haskell library I use: epub-metadata.

So is there no way to achieve this after all? Actually, there is! But it's a work-around and not 100% reliable. It does work for both v2 and v3 epubs, however.

The algorithm

Instead of relying on the epub's self description, we'll search for files that are likely candidates to be a cover. These are the steps:

Unzip the epub and make a list of all files in there

Filter everything out that is not a gif, jpg, or png image

Filter everything out that does not have the word cover (or any thinkable translation) in the name

If no file is left, no cover is included; if more than 1 is left, take one randomly; if exactly 1 is left, you have found your cover

In my experience this is pretty reliable, because most books that include a front image call it something with the word cover in it, even many non-English do.

If a book is written in a different language, it might also be worth it to look for that language's word for cover. I also look for the Dutch words kaft, omslag, wikkel and voor.

The code

The implementation of the above algorithm in Haskell then becomes the following. It uses the packages of lib-zip, bytestrings, and list to provide several convenience functions.

import Codec.Archive.LibZip import qualified Data.ByteString.Lazy as BL import Data.List (isInfixOf, isSuffixOf) guessExtractCover :: FilePath -> IO (Maybe String) guessExtractCover archivePath = do   bytes <- BL.readFile archivePath   let covers = filter isCover $ toArchive bytes   case covers of     []  -> return Nothing -- No cover     c:_ -> return $ Just c -- Return the first of all found covers isCover :: String -> Bool isCover s = hasCorrectExtension && containsKeyword   where exts = [".jpg", ".jpeg", ".gif", ".png"]         keys = ["cover", "front", "omslag", "wikkel", "voor", "kaft"]         hasCorrectExtension = seriallyCompare isSuffixOf exts         containsKeyword = seriallyCompare isInfixOf keys         seriallyCompare function needles = foldr ((||) . flip function (map toLower s)) False needles

I hope this helps any one! It's not guaranteed to find an included cover (if it doesn't have the right file name for instance), and it might even return the wrong image (if the filename in some way includes the word cover as a part of a larger word or sentence).

It is the most easily implemented and reasonably accurate method, however. Hopefully, it helps any one.

[nimmerwoner]

Digital DIY: your own ebook #2

Last time I wrote a partial blog post on rolling your own ebook. This post is the second half finishing it.

Last time we started with an ebook template provided by me and a text and cover file provided by you. We saw how to put those together to create an actual ebook. We are not done yet, however.

In this article I will explain how to create the table of contents (necessary for ereaders to navigate the ebook). Furthermore I'll show how to add descriptive information so ereaders (and other software) know what book by whom this actually is.

Finally we'll perform the last step of creating a special zip archive that gets recognised as an ebook. Let's get started!

3. Writing the table of contents

So in the epub standard, the table of contents is actually defined twice. Once in the <spine> element in Content/Content.opf and once in a specialized table of contents file, defined in the <manifest> element.

The important difference is that the <spine> only references separate files and not necessarily in sequential order. The specialized toc-file on the other hand, can have as many headers and subheaders as you want. It also is ordered, which means that it describes the right order to read it in.

Now, to get to work! First we'll do the <spine>.

First open up the opf file (Content/Content.opf). Scroll to the <manifest> element.

Get the identifier of the file with your writing. You can find this in the id attribute of that file's <item> element. Last time I chose the id content.

Now scroll all the way down where you'll find the <spine> element. You'll also see an empty <itemref> element there, which is the one you need to fill in.

In the empty <itemref>, write the earlier found identifier between the double apostrophes of idref="". This element is called itemref because it refers to the item you define in manifest, see?

Done! If you added more files that you want included in the book flow, repeat these steps for each file.

You'll see that there's already another <itemref>. This element references a page that includes Content/images/cover.jpg so that ereaders first show the book's cover when opened. If you don't want this, remove both the element (in <spine> and <manifest>) and the actual file Content/book-cover.xhtml.

Now to continue with the table of content that the ereader will actually show. This is the one that fits more with the traditional idea of a table of contents.

Open Content/nav.xhtml.

For each top level header, write that header's title between a <li><a href=""></a></li> which you insert after <ol>. Chapter one would like this: <li><a href="">Chapter 1</a></li>. This describes the headers and their order, but not yet which part they link to.

When your reader clicks on a title in the table of contents the ereader is supposed to directly open up that page. To support that, you need to fill in the href="" in <a>, linking it to the an <a> in a file. This works like it normally does in html. So, linking to the header Inleiding in the file Content/book-content.xhtml becomes: <a href="book-content.xhtml#inleiding">. This only works because there's an element <a id="inleiding"> around the Inleiding header in Content/book-content.xhtml.

If you want subheaders, insert a new ordered list element <ol> between the list item <li> that is these subheaders' parent header. Refer to the example book for an illustration.

4. Descriptive information, or metadata

So you're actually already almost about done. The only problem is that it's not clear yet what this book's title is, or who wrote it, nor what language it's in. Devices like ereaders and ebook collection managers won't know what to do with your book.

Properly describing the book allows them to beautifully presented, however. That's what we'll do now. This kind of descriptive information is also called metadata, or data about the data itself.

The minimum required information the book's title, language, a unique identifier and the last time of modification. Other useful information is the author's name and the rights the book falls under.

Let's go!

Open Content/Content.opf. You see have the <metadata> element.

Fill in the <dc:language> element your language's code. For Dutch as spoken in the Netherlands for example, that's nl-NL, while Belgian Dutch would be nl-BE.

Fill in the book's title in <dc:title>. This can be whatever you want

The <meta> element with property="dcterms:modified" expects a date in the YYYY-MM-DDT00:00:00Z format. The T and Z are literal, the rest needs to be replaced with the year, month, day and time of last modification of the book. Now, for example, would be 2016-11-27TT-17:21:00.

The unique identifier is a bit more tricky. This needs to be a code that's completely unique, around the whole world, for sure. There are several ways to do this. One is using your domain name. But here we'll use a very long, universally unique identifier randomly generated here. Copy the long string of characters from that site and put it in <dc:identifier>.

Those are the required elements and you'd have a working epub after zipping it in the next step. We'll add some more information however.

Write the name you want to publish your work under in <dc:creator>. This element has an xml:lang attribute, because the same author's name can be written in multiple languages. This allows a Japanese author's name to be written in both Japanese kanji and the Latin alphabet so that ereaders can decide which to show their user. This language is often the same as the book's language for simple publications.

Look up the <dc:date> tag to write in which year the book was published, for example 2016.

Find the <dc:rights> tag and write down the name of the license you want to publish it under. This could be Public Domain, any of the Creative Commons licenses or anything else.

There you go! That's all the most relevant info. Now to finish up.

5. Zipping it all up into your ebook

This part is easy and doesn't need much thinking if you have access to Linux command line zip utility. You merely need to go to the same directory as Content, mimetype and META-INF and execute the below command:

zip -rX ebook.epub mimetype Content/ META-INF/

The trick, and very important detail, here is the X instruction and that mimetype is the first file in the list. Mimetype is what tells other software that this is not just a normal zip archive, but actually an ebook.

This is also why the X instruction is important, because it prevents the zip program from modifying the mimetype file, which is essential.

You should now have a new file called ebook.epub. This is your ebook! Go forth and publicize!

Going from here

So what now? Well your ebook might not be that beautiful yet. You can improve styles, layout and looks with CSS.

You might also be interested in this Epub validator tool that verifies whether a given epub is faultless.

Lastly, even though these articles outlined how to roll an epub by hand, there are actually more automated ways to go about it. The ebook manager tool Calibre has a converter that can convert from and edit many formats. Another project with similar features is Pandoc.

Good luck!

[nimmerwoner]

Digital DIY: your own ebook #1

So you’ve finished a beautiful piece of writing – a Nanowrimo piece, a thesis, or maybe even just a bundle of your family’s Sinterklaas poems – and now you want to publish it online, right?

Here I’m going to discuss how to manually create your own ebook in the epub format. It’s easier than you think!

In this tutorial I work on Linux. For Windows and Mac the tools might be different, but the procedure is the same. I assume the reader is familiar with Linux, but not with epub or xml.

This will be a multipart blog post. Expect the next steps next week!

0. Necessities and ingrediënts

The tools you need are:

a text editor like gedit; and

a zip compression utility; I’ll use the command line zip program available on Linux, because it’s easiest to explain. Other operating systems have these alternatives.

Furthermore, you’ll want the following files:

Your writing converted to an xhtml document. Many text editors, like LibreOffice and Microsoft Office, can export to xhtml;

An image (jpeg or png) that will serve as your cover;

This archive with the framework of an epub. Download it and extract it somewhere into a temporary folder.

1. Preparatory work

We’re now going to place all the files in the necessary places.

In the extracted archive’s folder you’ll find two subfolders: An empty book is what you’ll work with; An example book is the result of this tutorial's and included for you to have an example.

Put the image you want to use as a cover page into the images folder in the Content folder (so that's Cover/images) and give it a simple name, like cover.jpg.

Copy the xhtml document into the Content folder as well and give it a good name. I have used doc-content.xhtml so it gets sorted next to book-cover.xhtml. This helps keep an overview of related files.

2. Connecting files

Now it's time to inform the ebook about these files and their function.

First we'll register the cover.

Open Content/Content.opf with your text editor. This files is the central registry of every file in the epub. Each file you want included should be listed in here in the <manifest> tag. You'll find that section in the middle with several <item> tags.

We're going to register the cover image. The <item id="cover"> tag does that, but it's incomplete. The file path is missing. We're going to add it by filling href="" with images/cover.jpg (insert it between the two "). You don't need to prepend the path with the parent folder Content because epub assumes all files are in there.

We're not done registering the image! Epub needs to know whether your image is a png or jpg in the media-type="" attribute. if the first, insert image/png between the double apostrophes, if it's the latter insert image/jpeg there.

Phew. Was that hard? Refer to the screenshot above if you're confused. Now to do the same for the book content.

If you closed (and hopefully saved!) Content.opf, reopen it. Use the empty <item> tag to register the book's xhtml file.

Put the file path in href: book-content.xhtml

Add the correct file type into the media-type attribute: application/xhtml+xml

Insert a unique identifier in the id attribute, for example "content".

Save!

To be continued

That's it for this week! You've colleceted the necessary files, placed them correctly and connected them to the ebook register.

Next post we'll look at writing the table of contents, adding identifying information (title, author, ...), and zipping it it all up to create the actual ebook.

[nimmerwoner]

Reading epub metadata in Haskell

For the ebook manager (provisionally called Boekerij) I’m writing, I want to be able to properly categorize and display the books in the collection. To visualize my intention, in the short term I want to do this:

This can be done using the metadata that accompanies an ebook. Metadata is not the book itself, but information about the book, like language, title, author(s), and publishing date.

In this article I will detail what I found out about how metadata is included in epubs and how I attempted to access that in Haskell, the programming language I use.

There are multiple versions of the Epub standard, most importantly version 2 and version 3. In both versions, the metadata is documented in a xml file embedded within the ebook.

For Haskell the best library seems to be epub-metadata. It works with both versions and is quite complete. With it, extracting metadata is a breeze:

readEpubMetadata :: FilePath -> IO Metadata readEpubMetadata epub = do   result <- runErrorT $ do     xml <- getPkgXmlFromZip epub     getMetadata xml   either (\_ -> return emptyMetadata) return result

In the resulting Metadata data structure, the relevant information can be easily accessed by name:

let authors = metaCreators (readEpubMetadata “/path/to/book.epub”) let titles = metaTitles (readEpubMetadata “/path/to/book.epub”)

More functions for accessing specific information as defined by the standards mentioned up above can be found on the linked documentation page. What's important, however, is that I now know how to get this data from the epub programmatically.

The next, more difficult, step will be to extract the cover image. Epub2 didn't standardize this well, so it's hard to implement. Epub3 has a beautiful and simple declaration of the cover image, but... this isn't supported by the library epub-metadata. But more on this next time!

[nimmerwoner]

The budding of a better e-book manager for Linux

I’m a happy owner of an e-reader. Because it has a screen using e-ink it's very comfortable to look at and it doesn’t disrupt my sleep if I read before bed. The letters are beautiful as well and being able to carry gigabytes of books in my hand is an immense luxury. There is however, one downside.

                                         Calibre – it works, but...

I find Calibre, the best available e-book manager to me, unattractive and frustrating to use. It presents your book collection as a mere list, which is ugly and doesn’t help recognizing books. Furthermore it shows so – many – buttons or other control elements. And each control requires extra interaction to make it do what it should do in one click!

So I’ve decided to write my own. A simple one, maybe, but one beautiful, clear and highly automated. On this blog I will describe design and tech choices I made, or the research I did beforehand.

[nimmerwoner]

3, 2, 1 ... deze tumblr is nu gelanceerd!

Leuven, we hebben een probleem: we hebben noch een thema noch een onderwerp...