#i had to manually download a file and convert it to a png myself

[sleepy-seal]

i could honestly tear into one of these rn

[hydrus]

Version 282

windows

zip

exe

os x

app

tar.gz

linux

tar.gz

source

tar.gz

I had a great week. I fixed some things and added regex tag parsing to import folders.

flash fixed and debug exes

I accidentally broke flash embed windows for Windows users last week. It seems the new version of the program I use to make the Windows build was not linking something correctly, so many users were unable to play flash without a stream of errorspam, and some users could not even boot.

I have rolled back my PyInstaller to the old version and everything flash seems to be working again. I will be testing flash embeds every week from now on to make sure this doesn't slip through again.

Also: I think I have figured a way to include the debug executables in the regular builds. They are just called client_debug and server_debug. If you cannot boot the client in future, please try running the debug exe and screenshotting the additional error information it provides. (I'll write a .txt file in the base directory next week or so to explain this further)

faster thumbnails

This is just a little thing really, but I 'smoothed' out the thumbnail rendering pipeline a bit. The client should generate some thumbnails much faster, particularly when they are first viewed.

pixiv broken

It seems that Pixiv recentishly changed how their gallery pages work (they now load in a stream as you scroll down, via some javascript or json thing). Unfortunately, the existing hydrus pixiv parser cannot deal with this and was as a result not parsing these results (I think it was just getting a handful in the 'popular' row and then quitting).

I have hidden the pixiv downloader links in the client and paused any existing pixiv subscriptions. You'll also get a little popup about it on update.

I would like to say that the new hydrus downloading engine will be able to handle the new system, but I have not looked into it deep enough to be certain. I hope it can, and will look into it more when it finally comes time for us to rebuild the existing parsers in the new engine.

Pixiv do have an official json API, but it is an OAuth thing, which is an entirely different kettle of fish for our purposes.

EDIT: Someone just told me that artist pages are still using the old system, so I will revisit this next week to make sure and reactivate them if they are all ok.

filename tagging options

As has been long-planned, I have extracted the regex and other 'filename' tagging options you can set after the 'import files' dialog into their own object and applied them to import folders!

If you are into import folders or the regex parsing, please check it out under the manage import folders dialog--a new simple listctrl replaces the old '.txt parsing' button. You can't set 'tags for some files' or the '#'-based tags (because those require manual input), but you should be able to set anything else on a per-service basis.

I have tested this and it all seems to work correctly, but it was a complicated and semi-big rewrite, so please let me know if you run into trouble. Maybe try it out on a small example before you go full on!

string converter

This is an advanced thing for people interested in making scripts and downloaders with the parsing engine.

I realised this week that the login manager and downloader overhaul actually have a lot of overlap in the work that needs to be done, particluarly in the parsing engine. Rather than finish the login and domain managers first and then move on to the downloader overhaul, I have decided to work on all three at the same time, making small improvements as the new parsing engine can do more things.

So, this week, I finished a new 'StringConverter' class, that does simple string transformations--adding or removing text, encoding to hex, reversing, that sort of thing. I added it to the html formulas in the existing scripting system and expect it to put it in a couple of other places for when you are in the situation of, 'I can parse this data, but I don't need the first six characters.' I don't expect the existing file lookup scripts need this too much, but feel free to play around with it. I think I'll add a 'regex' transformation type in future.

I also added it to the 'file identifier' section of the initial file lookup request. Now, rather than the file's hash bytes being encoded to hex or base64 from the dropdown, it occurs through a string converter. There has been a persistent issue of wanting to generate an 'md5:[md5_hash]' argument, and I believe this will now fix it. I do not know the exact specifics here so I won't write the script myself, but I'm happy to work with people on the discord or email or whatever to figure out a solution here.

full list

rolled back to an older version of pyinstaller that does not seem to have the embedded flash window problems windows users experienced

added an error handler to wx.lib.flashwin import--if it fails to import, the client will print the error to the log and thereafter show 'open externally' buttons instead of the embedded flash window

created a 'filename tagging options' object to handle the instance-non-specific auto-tagging parts of the filename tagging dialog

moved all the appropriate tag generation code to this new object

extracted the simple and advanced panels in the filename tagging dialog to a separate panel

wrote a new wrapper panel to edit filename tagging options objects directly

cleaned up some tag generation code--it'll now all be siblinged/parented/censored the same way

import folders now use the filename tagging options object, harmonising this tag parsing code

edit import folder gui now has an add/edit/delete listctrl to manage the new tag_service->filename_tagging_object relationship (this replaces the old .txt management button and summary text)

finished the StringConverter class in the new parsing engine. it applies a series of transformations to text

wrote a panel to edit string converters

wrote a panel to edit string converters' individual transformations

updated html formulas to use string converters instead of the old cull_and_add system

html formula edit panels can now edit their new string converters

file lookup scripts now 'stringconvert' their file identifier strings--this should allow the 'md5:md5_hash' fix we've been talking about (i.e. by first encoding to hex and then prepending 'md5:')

the help->about window now shows the client's default temp directory

thumbnail regeneration--particularly full-size regen and resized png gen--should be much faster in the client

the debug exes are now included in the windows build

the debug exes are now included in the non-windows builds

the test exe is no longer included in the windows install (can't remember it ever being useful, and it is like 10MB now)

unfortunately, a recentish change to how pixiv serves gallery page results has broken the hydrus pixiv downloader. this will have to wait for the downloader overhaul, and even then, it might be _slightly_ tricky. pixiv downloader entries are now hidden, and existing pixiv subscriptions will be paused on update

the thumbnail select submenu now clarifies 'all (all in x)' if all are in the inbox or archive

you can now archive/delete filter from the thumbnail menu on a single file

lowered the http session inactivity timeout even more, to 45 minutes

fixed a couple of instances of the new subscription dialogs being unable to mass-retry failures

ffmpeg parsing errors now print more info about broken file path

some daemons will be snappier about shutting down on application shutdown

took out the sometimes invalid 'booting' phrase in the during disk cache population status report

the client will now warn you about page instability at 165 pages after one user reported instability at 175. 200 is still the strict limit.

downloader pages and subscriptions will fail more gracefully if their downloaders cannot be loaded (e.g. they used a since-deleted booru)

fixed listctrl panel button enabled/disabled status when the child listctrl starts empty

the new listctrl can now select data in a cleverer way

misc fixes

misc refactoring

next week

I want to get the 'StringMatch' object finished, and maybe the accompanying 'URLMatch' done as well. These will verify that strings and urls match certain rules, and will help with login and domain and downloader verification and veto states.