#i switched to linux from windows and had some trouble with the button mapping and pen pressure

[01tacocat]

[ID: A drawing of a teal and light green striped undertide dragon. He has green swirl patterns on his wings and frills and a light green underbelly. End ID]

[hydrus]

Version 421

windows

zip

exe

macOS

app

linux

tar.gz

I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works better™.

highlights

I fixed a problem importing files with 'only add tags that already exist' checked in tag import options. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.

When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.

Thanks to a user's contribution and help, the 'discord drag and drop' BUGFIX mode, which is really a 'export files to a temp location before starting the drag and drop' mode, now supports the same 'filename pattern' you see in the export files dialog. Check it under options->gui. This is experimental, but the first step to automatic export with nice filenames. If you care about export filenames, let me know how it works for you, and what sorts of filenames you want to make in future. I really want to update the old 'filename pattern' system to something with conditional 'if' statements.

If you are a new user and had trouble syncing to the PTR in the past week with an error about an invalid tag, you got caught by my new 'invalid tag catching' code being overzealous. This is now fixed, so if your PTR automatically paused due to the error, please hit services->review services->PTR and then unpause it. When a repository gives an invalid tag (we think there are about 17 really broken tags in the PTR atm, from older days), the client now recovers.

I played around with core database modes this week. Default values are now tuned a little better for modern use. Unfortunately, the planned new mode for HDD users did not pan out due to architectural problems, but I think that HDD users will experience better write performance nonetheless. For those who want to experiment more, I have finally properly documented the different launch switches for hydrus here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

full list

misc:

thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation

fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again

fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices

prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start

when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)

the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later

the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on

fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist

I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search

when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast

fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later

reduced overhead of tag filtering, which should improve display speed of taglist for very large pages

parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster

.

duplicates search improvements:

potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going

the search distance in duplicates pages is now synchronised across all pages--when one updates, they all do

all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often

misc cleanup for duplicates page

.

database modes:

a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

added the '--db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY

ensured --db_synchronous_override was hooked up correctly

the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code

fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true

db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_

.

technical database nonsense:

PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn

hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done

hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see

in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations

when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour

after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%

the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated

improved launch switch code generally

boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values

the client and server no longer disconnect from the db to check whether it is possible to vacuum databases

next week

There are two more work weeks in the year. I will continue working on small jobs and cleanup, but the main focus is now to update ancient network and service code to improve client-PTR sync and communications. I'll fix up buggy janitor tools and hopefully add some nice filters so clients and servers can better manage what they sync.

[hydrus]

Version 255

windows

zip

exe

os x

app

tar.gz

linux

tar.gz

source

tar.gz

I had a great week, getting a lot done. There are many improvements to the duplicate system and some important notes for anyone who downloads from gelbooru.

you can now move tabs!

Right-click on a page tab on the main gui and hit 'move left/right' to move it left and right!

I can't believe I didn't think of this before, but it was a hell of a lot easier than writing custom drag and drop code!

the gelbooru downloader is not healthy

Like several other downloaders in the program, now the gelbooru parser is now also having trouble. I fixed some gelbooru ssl and redirect problems this week, but the gelbooru guys themselves have also changed how their gallery thumbnail URLs work in a way that is not helpful to hydrus, particularly for subscriptions.

The gelbooru downloader page seems to fetch files and tags fine, although the urls associated with the files are not great.

For subscriptions, I have attempted to mitigate the url problem in the update code--you will receive a popup on update if you are affected--but if you had a gelbooru subscription fire in the past week (it then likely had an error due to the above redirect problem), you should check it in the manage subscriptions dialog. If the sub has twice as many urls as it should, you should go into the subscription, click the icon button that launches the list of urls and then either 'skip' or just delete the new ones.

If you have too many subs to fix or are just not confident playing around with this stuff, just pause all your gelbooru subscriptions. I will put more work into this next week, which I suspect will include a retroactive purge of these new ugly redirect urls.

This is another reason to overhaul the downloader engine, which will then be more flexible and easier to fix when these things happen. I will start it in a few weeks, as soon as I am done with the current duplicate stuff.

duplicate stuff

I've polished the duplicate filter substantially. Several miscellaneous buttons and unusual workflow cases now work properly, and shift-click should drag media around. Also, the background will brighten and darken as you flick between the pair to give better feedback, and the pairs it presents you will be selected faster and more intelligently.

And as an experiment, I've made it so pairs that have the same resolution will stay at the same zoom and pan when you flick between them. I'm really pleased with how this works, so I may extend it to more classes of pair, like those with the same aspect ratio.

full list

the duplicate filter now supports shift+left-click to drag, like the archive/delete filter (this remains hardcoded for now)

if a pair in the dupe filter has the same resolution, they will now maintain zoom and pan when switching back and forth (I might increase this to work for same ratio as well, let me know how it works in the real world)

the duplicate filter will show a lighter/darker background colour as you scroll the current pair

the way lighter/darker and alternate lighter/darker colours are calculated is now centralised and should be more reliable in edge cases

improved the dupe filter pair selection algorithm--it now chooses pairs more reliably under edge case conditions and prioritises decision-groups that have high potential decision value. it should also run a little faster

increased the dupe filter batch size to 250, let's see if it causes any problems

the close button on the dupe filter's top hover window now works

fixed the duplicate filter disappearing pairs that were skipped (meaning you could not go back to revisit them)

fixed a frequent deadobject error when the dupe filter closes

the shutdown 'maintenance due' test is less sensitive to dupe search tree rebalancing, which typically only takes half a second

the archive/delete filter now uses the new shortcuts system for both keyboard and mouse input

the archive/delete filter now intercepts archive or delete commands from different sources ('media' shortcuts, top hover frame button presses) more reliably and converts them into filter actions (hence moving on to the next file)

you can now move pages one to the left or right from their menu!

on the top-right hover window (and the background underneath), ratings are now on the top, and hence will always be in the same location as you scroll through your media regardless of known urls, remote location, or inbox status

autocomplete searches for tags with apostrophes, quote marks, braces, brackets and paretheses should be more reliable

urls are now associated with files through the same 'content' pipeline as tags and ratings and so on

gui-level media is now aware of the 'new url content update' event and will update and redraw itself appropriately

fixed deviant art nsfw parsing, but it might not hold for long. proper fix here is to wait for the downloader overhaul

networking engine now uses the 'requests' module's CA .pem (which the overhaul will be moving to anyway), which should reduce the frequency of ssl verify failures (gelbooru on the relatively new 'Let's Encrypt' CA had this problem for many users)

fixed the networking engine redirect parsing for gelbooru's unusual location header. unfortunately, gelbooru is still giving mickey-mouse garbage redirect urls from its main thumbnail pages, for which in this engine there is no immediate fix

neighbouring .txt tags will now be properly cleaned and sibling-collapsed in the path tagging dialog

neighbouring .txt tags will now be properly sibling-collapsed in the import folder workflow

making a media viewer borderless fullscreen and back will now recenter the media (previously, the current drag delta was not reset, so this frequently put media off-screen)

fixed a serious issue where the media viewer could lock the client up on opening with a video if its gui options set 'remember size' to false

tag import options objects will cleanse themselves of missing services on options save/client shutdown

manage tags now defaults to cross-referencing 'my files' on the 'local tags' domain, where 'remote' (i.e. deleted) files' tags are not useful

reduced memory use when importing large pngs with transparency

improved adminside petition processing gui reporting

servers will now cap the size of their mapping petitions so as not to ovewhelm the admin processing them (they now won't be both >20 tags and >1000 total row weight)

the media viewer's manage tags frame now listens for content updates from outside, so if you alter an in-view file's tags (such as with a shortcut key), the manage tags dialog will update as it happens

moved a number of the buttons on the top hover windows to the new unified internal command engine (which the new shortcut system also uses)

disk cache maintenance uses fewer resources but now occurs in the foreground (and should hence more reliably maintain the cache)

critical repository service id lookup errors will now automatically reset the repository's processing cache and better inform the user of what has happened. if you see this, please let me know the details and how this error fired in the real world

updated and reinstated the ipfs service panel in review services

updated some ipfs service code for the new service system

misc refactoring and cleanup

more cleanup and deletion of redundant old pubsub command code

updated ubuntu build machine to 17.04 and opencv 3.2

next week

I have 41 things left to do for the duplicate work, and I want to get that to 0 as soon as possible. Beyond some more gelbooru triage, I want to focus on duplicate stuff and a few bugs that I didn't have time to get to this week.

I'm behind on messages, so I'll make some time to catch up as well.

[hydrus]

Version 259

windows

zip

exe

os x

app

tar.gz

linux

tar.gz

source

tar.gz

I had a great week. Some things work better, some things work faster!

advanced mode

After several different discussions with users, I am prototyping an 'advanced mode' this week. The idea is that when it is off, the more complicated and dangerously powerful buttons and menu entries will be hidden from view so a fresh client will have nothing that isn't obvious. I have hidden a few things already--like the new duplicate thumbnail menu and the service-wide update buttons--and I intend to add more.

It defaults to off, so experienced users will want to switch it on as soon as they update under help->advanced mode. I will be doing more with this in future, so please let me know if it gives you trouble or there is anything you would wish were hidden when you first started.

some duplicate stuff

Thanks to some feedback from users, I polished a couple more things with the duplicate system. Setting larger numbers of duplicates will now happen a lot faster, and the new thumbnail menu lets you edit the default merge options. Also, when manually deleting a file in the duplicate filter, it will ask you if you actually want to delete both of the files in the pair.

I have also renamed 'exact match' to 'same quality' across the program--and updated some help and tooltip text to better explain it. I hope this will reduce some confusion and hesitance about setting this status, which wasn't initially meant to be so precise. Please set this status when the two files are duplicates and neither is obviously better than the other.

media results should load faster

I've written a semi-clever optimisation to speed up the loading of thumbnails in the program (where it says 'Loading 256/512...'). File tags are now usually loaded from a faster cache than the main mappings store. My rough testing says files load about twice as fast as before--both when the database is in and out of memory--but I would be interested in hearing your experiences.

If this works out well, we can see about reducing the size of the disk cache in future, as the full mappings cache is far less frequently consulted. I might even be able to compact the main mappings database even further.

downloader engine overhaul started

I started the downloader engine overhaul this week. I have a decent plan of the first phase, which will mostly concern low-level stuff like unifying network access across the program and adding session/cookie support and per-domain bandwidth awareness and tracking for every request. I also sketched out the basic skeleton in code, and I feel good about it.

I expect to work on it another couple of weeks and then start converting existing network code over to the new engine.

full list

planned out new networking engine and started the principal objects

renamed the 'exact match' duplicate status to 'same quality', to reduce confusion on what is appropriate for this status

the duplicate filter, on hitting the delete key, now offers the option of deleting both files

the duplicate system now combines duplicate status setting and the consequent batch of content updates into the same database transaction, speeding things up

the duplicate system now batches multiple duplicate status setting into a single transaction, massively speeding up large filters or thumbnail status set actions

misc duplicate help tweaks

you can now edit the default duplicate merge options from the new thumbnail duplicate menu

the duplicates page's jobs are less demanding on gui time and take better breaks if something else happens

renamed the new dupe system predicate to 'system:num duplicate relationships' to clarify what it searches

for normal queries, current and pending mappings are now fetched from a faster mappings cache. you should see faster result building across the board, particularly on fresh boots or otherwise slow-disk-access systems

added a prototype 'advanced mode' (defaulting to off, so experienced users will want to turn it on) under the help menu that will enable menu items that are often not helpful to new users. I will add more things to this in future, suggestions welcome

the new thumbnail menu dupe relationship set stuff is now considered advanced

thumbnail menu find similar files is now considered advanced

the thumbnail menu copy hash entries are now considered advanced

advanced content update buttons (on manage tags and review services) are now considered advanced

added an advanced mode 'open file location' entry to the thumbnail share menu, which will open the file in your OS's file explorer (not available on Linux)

added an advanced mode 'correct video frame count' thumbnail menu entry that will force-apply last week's more accurate video frame counter to correct and videos that render too fast and then cut off

fixed many entries on the media viewer menus, which were being blocked by an over-eager 'can continue' test and hence silently failing

fixed the youtube downloader on Linux and OS X--both now use youtube-dl

fixed an issue where the GetLighterDarkerColour function was producing very bright alternates to very dark colours (meaning dark dupe filters were having their bright background text rendered unreadable)

improved video frame number parsing accuracy

improved the accurate version of video frame number parsing accuracy, particularly for longer videos

the network engine now reports 5xx http status codes as ServerException to better contextualise to the user what went wrong

adminside mapping petitions are now sub-ordered by tag

adminside sibling/parent petitions are now ordered by the 'older' tag and sub-ordered by the 'newer' tag

censorship taglists are now roughly sorted

reduced default shutdown work max time to five minutes

improved how subprocesses are started

misc cleanup

misc refactoring

next week

I am going to take it easy next week. I need a little time off after some stressful months, and it just happens to be E3 as well, so I'd like to spend a little time shitposting over some streams. I still expect to put out v260 next Wednesday, but it will probably be mostly bug fixes!