#code snippets are in purple to differentiate them from regular text

[kitkatt0430]

Customizing FanFicFare on Calibre

I have another adventure in #ficArchiving for those interested.

So you’ve set up Calibre with the FanFicFare plugin and have been downloading fanfic. But you may have hit some snags. Fics with higher ratings come with age check/adult only warnings. Some fanfics are locked to the archive so that logged in users are the only ones who can access it. Fanfiction.net is hit or miss thanks to its use of Cloud Flare - and it’s mostly a miss. And probably some other issues too.

It’s time to dig back into FanFicFare’s configuration settings and start making some changes to the personal.ini file used to override the defaults.

First, you’re probably wondering what the personal.ini file even is. So open up Calibre, head over to the FanFicFare menu, and select the Configure FanFicFare option to reach the configuration modal. you’ll want the second tab on the left, labeled personal.ini. You’re probably not feeling all that enlightened by the screen that greets you, but at least now you’re in the right place to see what the file I’m talking about looks like.

Now the FanFicFare FAQs page doesn’t outright define what the personal.ini file is, but it does a pretty good job of eventually putting it into context the more you read about how FanFicFare works. But, essentially, the personal.ini is a file that will allow you to personalize the way FanFicFare accesses fanfiction and downloads it by redefining specific variables or adjusting code snippets. These personalized settings can be applied to all the fanfiction archives FanFicFare can access - which is a lot - or set up changes that affect only a single archive.

If you want to know what the default settings are, you can click the button to open the default.ini file. This file gives a detailed rundown of all available options and what the current settings are both on the universal (all archives) level and on the single archive level, since every site will work a little different with how the data is scraped and how FanFicFare determines which data is relevant and which isn’t. There’s a lot of information in the default.ini file and while it can be interesting to see what all those options are, don’t feel like you need to read through the whole thing right now.

Let’s switch over to the personal.ini file now. It should look something like the example.ini provided on the FanFicFare GitHub repository.

Much shorter and more readable than the default.ini file and all the options are currently commented out since you haven’t actually set any personal settings yet. You can probably already see the answer to fixing the first scenario I noted at the start of this tutorial, explained in the comments near the top of the example.ini file.

(Note: I'll be coloring code snippets in this color here to differentiate them from the rest of the post since tumblr doesn't do code blocks.)

If you’re having trouble accessing age restricted content, then you can uncomment #is_adult:true in the [defaults] section by removing the # at the start of the line.

If you removed all the commented code in the defaults section and leave only your personalized changes, it would look something like this:

[defaults] is_adult:true

Click the OK button on the personal.ini editor to provisionally save the changes and the OK button on the configuration modal to apply them. If you don't hit the OK button in both places, then your changes will not actually be saved and you'll have to go back and make them again.

Now if you test downloading a fanfiction from Ao3 that’s rated M or E, you should see it download correctly this time. Because this change was made in [defaults], it applies to all fanfiction archives; if you prefer you could set it on a per archive basis like so:

[www.tthfanfic.org] is_adult:true

Of you could have it set at the [defaults] level as true and override it as false on a per archive basis.

[defaults] is_adult:true

[www.tthfanfic.org] is_adult:false

Alternatively, if you want to make sure that the setting cannot be overriden, you could add it to the final override level like so:

[overrides] is_adult:true

Okay, so there’s one setting down, more to go. Now let’s talk about archive locked fanfictions.

Head back into the FanFicFare configuration modal and pull up the personal.ini editor again. Let’s say there’s a fanfic you want to read offline, but it’s locked to the archive. You can only read it if you’re logged in and FanFicFare can’t exactly log into your account. Or can it?

Actually, yes, FanFicFare can log into your account to download archive locked fanfiction. Now, you might be feeling a little - understandably - wary about handing your login credentials to FanFicFare’s personal.ini file. What happens to them once they’re in there? Are your accounts still going to be safe?

The answer is, yes, your accounts will be safe. The login information is only used when logging directly into the site using a headless browser on your computer - a headless browser meaning that it’s never visible on screen. Because of this, the actual login data and any associated login cookies created stay on your computer. Beyond the actual login process, the username and password information never go beyond your computer and remain entirely secure. To revoke FanFicFare’s access to your account, just remove the username and password from the file.

The plugin itself does not back up your personal.ini file outside of your computer nor does the plugin log any information from that file externally. So adding your log in credentials to the personal.ini file is currently quite safe.

Since a username and password are likely going to be unique for every site you have an account for, this is a setting you’ll want to add on a per archive basis only. When you add that data, it should look something like this:

[archiveofourown.org] username:<putYourAo3UsernameHere> password:<putYourAo3PasswordHere>

[www.tthfanfic.org] username:<tthFanficUsernameHere> password:<tthFanficPasswordHere>

Now click OK on the personal.ini editor to save the changes and OK on the Configuration modal to apply them. If you test downloading an archive locked fanfiction from Ao3 now, FanFicFare should be able to do so using your username and password to log in first. If it goes wrong, reopen the personal.ini file and verify the username and password are correct. Then double check the archive url, making sure to compare to what’s used in the default.ini file to ensure that FanFicFare is associating the username and password with the archive correctly. Also make certain there is no extraneous space between username: and your actual username or password: and your actual password.

Huzzah! At this point most fanfics requiring age checks or archive access are now available for you to download and manage using Calibre and FanFicFare. There will be exceptions, however. And the biggest one? FFnet.

Since moving to Cloud Flare, FFnet has kept it’s protection settings pretty high. When reading on the site directly, no doubt you’ve noticed there’s often a Cloud Flare redirect going on and sometimes even an ‘are you a robot’ check. Even the Internet Archive has trouble managing to back up anything from FFnet - most of the time it just… can’t. Similarly, FanFicFare struggles to download off of FFnet with it’s default settings. More often than not, you’ll see it fail with a 403 response in the logs when a download job completes.

But there is a way around this using FanFicFare. It’s not the prettiest solution, but it does get the job done.

The first thing you want to do is identify what your default browser is. If your default browser isn't Firefox, I recommend making it so specifically because Firefox is a good browser that is highly customizable and respects your privacy.

The reason why you want to identify your default browser is because we're going to be setting up some options for pulling fic data from the browser cache. There's a pretty good tutorial for this on the FanFicFare GitHub, which I'll be referencing here.

Head back into your personal.ini file and we'll do some more doctoring. First lets get that browser cache setting assigned.

[defaults] is_adult:true browser_cache_path:C:\Users\YourUser\AppData\Local\Mozilla\Firefox\Profiles\ZjwI7Fo4.default\cache2

Under [defaults] you'll want to add the browser_cache_path as shown above. I'm using the windows example for the Firefox cache from the FanFicFare tutorial here, but you'll want to fill in the actual path from your own computer. Make sure that information gets saved to your personal.ini and then we'll be making the next set of changes.

Now that FanFicFare knows where to find your browser cache, you want to have it start pulling fanfic data from FFnet from there. But you don't necessarily want it doing that for every archive, right? So we'll be making this changes for just FFnet.

[www.fanfiction.net] use_browser_cache:true

With this setting saved, FanFicFare will pull any cached data for the fanfic you're wanting to download from FFnet. Which is great - but only if you've already got every chapter for that fic in your cache right now. Otherwise it'll still try to go grab that missing information itself.

So the next settings you want to add for the FFnet archive is use_browser_cache_only:true and open_pages_in_browser:true. It should look like this in your personal.ini:

[www.fanfiction.net] use_browser_cache:true use_browser_cache_only:true open_pages_in_browser:true

Now FanFicFare will be grabbing data from the browser's cache and only the browser's cache and, so that you're not having to go open every single page before downloading, if there is missing data in the cache the FanFicFare will open the page in the browser which will allow it to pull that fresh data from your cache.

This does, however, mean that FanFicFare will no longer be running headless for this specific archive. It will pop open the new pages on your default browser on your screen and effectively take over your browser for the short (or possibly long for fics with lots of chapters) amount of time it takes to download to cache every piece of data FanFicFare uses to build the ebook. You can still do whatever you were doing - as soon as the page FanFicFare opens finishes loading you can close it as that data will now be in the cache for the plugin to use - but it is going to be a pain. Best to not expect to use your computer for anything else when downloading off FFnet at this point.

However, we're not quite done yet. At it's usual speed, FanFicFare is a bit fast for CloudFlare and you're going to want to keep an eye on it to make sure if a robot check happens that you have enough time to click the little checkbox before FanFicFare declares this venture (downloading a fanfiction) a failure.

slow_down_sleep_time is a variable that takes integers (numerical values) and essentially puts a delay between opening the new page, accessing the cache data, and then moving to the next page.

[www.fanfiction.net] use_browser_cache:true use_browser_cache_only:true open_pages_in_browser:true slow_down_sleep_time:10

With the slow_down_sleep_time here set to 10 seconds, it's a short delay between the plugin telling the browser to open the page and declaring that it cannot access the necessary cached data. It should be enough time to allow you to manually bypass the robot checks if necessary while not turning the download process into a ridiculously lengthy endeavor. At this rate a six chapter fanfic will take about a minute to access every chapter before it moves on to the process of wrapping all that data into an ebook, which isn't really that bad. The longer the gap between the plugin opening new pages in the browser, the less likely FFnet is to assume you're a robot - so you can go for a shorter gap of you want but know you're more likely to get periodic checks that way and if you're not monitoring the process then it's also more likely to fail (and you'll have less time to click that little checkbox).

Alright, save your settings and give downloading off FFnet another try. If you're downloading a fic you just finished reading, you might see nothing happen at all while the job runs in the background. Or you might see it access all the chapters for the fic in your open browser.

As mentioned, not the most elegant solution out there, but it does get the job done. If you're wanting the metadata to auto-fill for you or specifically to track unfinished works as they update, then this is still going to be the best solution for downloading your fanfics. If you're okay with filling in the metadata yourself and aren't looking to track the fic for new updates, then there are other options that will be discussed in future blog posts.

Those three are the issues you're most likely to run into while downloading fanfic, but not the only ones. The FanFicFare FAQs are pretty useful for figuring out what may be going wrong with any issues you may run into and can direct you on how to request new features or report any bugs you might encounter.