hydrus

[hydrus]

Version 630

windows

zip

exe

macOS

app

linux

tar.zst

I had a great week. Some bugs are fixed, and the duplicate filter loads faster.

full changelog

highlights

The duplicates filter now fetches pairs using the new fast auto-resolution fetch tech. It should be able to produce a batch of work quickly even when you have a slow search that has something like 'system:has url class'. I also improved its logic so it presents you with more related pairs in a batch and auto-skips more efficiently. Please let me know what works well and badly here.

I also figured out the start of a proper pair sorting regime. It'll give you 'big files first' pretty much, this week, but I'll figure out a widget to change it to 'most/least similar first', 'biggest filesize difference first', 'completely random', whatever we can think of, in the near future. I'm also confident I can add a checkbox for a 'process whole groups at a time' mode.

If you have laggy mpv on Linux, particularly when transitioning to a new media when near the end of the current media, check the changelog! There's a line you can edit out of the default mpv.conf (in your db dir) that may fix this aggravation.

If you have wanted but been unable to map 'shift+tab' to a shortcut, try again this week. It may work now, calling the key 'backtab'.

Thanks to a user, the pixiv downloader now fetches 'series' and 'chapter' tags for manga works that have them. I'm not sure if the 'series' here is more often what we might call 'title', so if you use pixiv a lot, let me know what you think.

next week

More like this, but I also want to do something fun, if I can fit it in.

[hydrus]

Release Tomorrow!

I had a great week. I fixed some bugs and got the manual duplicate filter working on the new fast-fetch routine. It loads up pairs much faster for difficult searches and presents its pairs in a much more sensible order.

The release should be as normal tomorrow.

[hydrus]

Version 629

windows

zip

exe

macOS

app

linux

tar.zst

I had a great week mostly fixing bugs.

full changelog

highlights

The top-right hover window is fixed in the duplicate and archive/delete filters!

The duplicates auto-resolution system has had several bad or laggy timer calculations fixed. It will now take better breaks between work and will not hit you so much if it needs to do CPU work. Also, the 'A and B are visual duplicates' test gets another upgrade--it should be able to detect larger but paler watermarks as definitive alternates.

We get better AVIF support this week. All your AVIFs are going to regen their thumbs and check for exif and so on again. Users who run from source may like to rebuild their venvs this week to get this.

Also, the builds last week were having trouble parsing dates. I think I've fixed it!

next week

I didn't have time to make duplicate filters load fast this week, but that's still something I want to get done.

[hydrus]

Release Tomorrow!

I had a great week mostly fixing bugs, particularly issues with the top-right ratings hover, export filenames, date parsing in the builds, and duplicates auto-resolution work timing. There's also improved AVIF support.

The release should be as normal tomorrow.

[hydrus]

Version 628

windows

zip

exe

macOS

app

linux

tar.zst

I had a great week cleaning old code and fixing/improving small things. There are also more neat ratings UI options.

This release folds in the library updates we tested last week. The test went well and there are no special install instructions, but let me know if you have any trouble.

full changelog

ratings

Thanks to a user, we have another round of ratings UI features. A new options->ratings page lets you edit the size of ratings in every location, including separate sizes for inc/dec ratings, and in the 'manage ratings' dialog, and you can now choose to collapse the thumbnail numerical ratings display to just '3/5 (one-star)'.

Under services->manage services, you can now add this '3/5' text to the left or right of any normal numerical ratings display. Also, thanks to another user, we have a bunch of new default ratings svgs. I think they look great.

Also, under options->media viewer, you can now choose to show the 'top-right' hover window in the preview viewer! So, you can now quick-set ratings with one click. Try this out--I like it so much that I think I'll soon make it default for new users.

duplicates quality of life

Duplicate pages now count up their searches' pairs with my new incremental duplicate fetch tech. Touching a duplicate page will no longer lock your database for a couple seconds, and there's pause and fast cancel-reset tech in there too. Should just work better all around, but let me know if it ever under-counts.

Semi-automatic duplicate rules similarly commit large approve/deny commands in pieces, rather than blocking the database for the whole job, and they update the UI with a 'approving: 16/55' label so you can see how they are doing. There's also a 'select all' button.

next week

I've got a lot of ideas, but I would like to get the incremental duplicate fetch tech working in the main duplicate filter fetch routine and the 'show some random' button too. I'd love to reduce the fetch time when I only need a handful of pairs, and if I can get this all working on the same unified and sane pipeline, I can finally think about changing the order pairs arrive.

[hydrus]

Release Tomorrow!

I had a very good week. I cleaned a ton of code, fixed bugs, and improved some quality of life. There are several more rating UI features, too.

The release should be as normal tomorrow.

[hydrus]

Version 627

windows

zip

exe

macOS

app

linux

tar.zst

I had a great week. The "A and B are visual duplicates" test is ready for real use.

full changelog

A and B are visual duplicates

I figured out a spatial edge-comparison component to the "A and B are visual duplicates" test this week. It worked out really well to detect small alterations that the pure colour comparison was struggling with, and I solved my remaining false positive examples reliably. This test is now ready for proper testing and is now an available comparator in duplicates auto-resolution.

I have added a new suggested rule in duplicates auto-resolution called 'visually similar pairs - eliminate smaller' that uses it. If you have been following along with this tech, please try it out and let me know how it goes. Leave it on semi-automatic, and be aware that this test is pretty CPU heavy--about one second per pair test. I'm interested in how the UI handles this sort of lag IRL, where the comparators goes wrong (e.g. do we want the filesize/resolution test to be x1.4?), and obviously if you encounter any more false positives, please send them to me. I've just played with it IRL and I think it might benefit from a 'A imported earlier than B' test too.

This was the last difficult part of duplicates auto-resolution. I still have lots I'd like to do, but it will all be smaller features and polish. I will revisit the help and do a grand launch for all users once we know this is all tuned well.

future build

Only for advanced users!

I am making another future build this week. This is a special build with new libraries that I would like advanced users to test out so I know they are safe to fold into the normal release. More info in the post here:

next week

I'm going to take an easy week and focus on simple code cleanup.

[hydrus]

Release Tomorrow!

I had a great week. The 'A and B are visual duplicates' system has a significant improvement and is enabled for testing in the duplicates auto-resolution system.

The release should be as normal tomorrow.

[hydrus]

Version 626

windows

zip

exe

macOS

app

linux

tar.zst

I had a good week. There's a bunch of fixes and some new stuff.

full changelog

highlights

The AUR (Arch Linux) release broke during my vacation. It was my mistake--I didn't catch something that was deprecated for way too long, and suddenly file imports wouldn't work--and I'm sorry! If you have a recent source installation or ran into the AUR issue particularly, please update to this v626 when you can and you should be good again.

I improved the accuracy of my 'A and B are visual duplicates' test again. Almost all my false positive and false negative examples are fixed. I still have a little more work to do, but I'm feeling pretty good about it overall. I am still interested in seeing any more bad pairs, either false positive or negative.

File export filenames are a bit smarter about maximum file and path length. If you do a lot of exporting with tags and your filenames regularly get clipped, it should work better now, particularly with unicode and patterns that produce subdirectories. There's a new setting under options->exporting, too, that lets you override how long your export filenames can be.

The default derpibooru downloader stopped getting tags. I think I've fixed it, but let me know if there are still any problems.

For a bit of fun, I added support for Paint.NET files (.pdn) this week. They should have resolution and thumbnails.

If you are an advanced downloader maker, please check the changelog about the 'url slash test'. I'm interested to know if an URL-handling change will break anything on your end.

next week

More duplicates auto-resolution work and polish.

[hydrus]

Release Tomorrow!

I had a good week. I fixed quite a few bugs and improved the accuracy of 'A and B are visual duplicates'.

The release should be as normal tomorrow.

[hydrus]

Version 625

windows

zip

exe

macOS

app

linux

tar.zst

I had a great week. I finished a bunch of small jobs.

full changelog

highlights

The subscriptions dialog is now much much faster when you do large edits like a 'separate' call or pasting many new queries into a sub. All lists that undergo mass edits are now faster, but this particularly applies to subs, where some jobs could add several minutes of lag--the same job now takes a couple of seconds.

The page tab menu has a new 'collapse pages' entry, which lets you collapse the pages 'to the right' into one new frozen search page (and closing the old pages). If you have ten importers in a row, this lets you collapse them all into one thing you can process with. Try it out!

file->shortcuts is gone! It is now under file->options->shortcuts. A bunch of the other options pages are cleaned and chopped up into easier-to-digest pieces.

The horrid manage->file relationships thumbnail menu has had a small usability rework, and there's a new 'delete all false positives within this file selection' command.

The 'A and B are visual duplicates' test is now careful to give a false result if either file has transparency--it isn't smart enough to deal with this yet.

The 'preview' and 'review' panels in duplicates auto-resolution do their behind-the-scenes jobs better and spend less time doing nothing or hanging the UI.

next week

I am now on vacation for a week. v626 will be on the 18th. Thanks everyone!

[hydrus]

Release Tomorrow!

I had a great week. As I hoped, I cleared out a bunch of bad code and finished small todo items. I fixed some laggy list updates, brushed up some options and menu layout, and improved the feel and speed of several duplicates auto-resolution panels.

The release should be as normal tomorrow.

[hydrus]

Version 624

windows

zip

exe

macOS

app

linux

tar.zst

I had a great week. Duplicates auto-resolution is easier to preview, and I have re-introduced a way to 'lock' a search page in place.

This release updates several important libraries in Windows and Linux.

full changelog

new build

The Windows/Linux 'future build' test last week went well, no reports of problems. It seems like Win 10 will still run the program, although I suspect a very old (i.e. un-updated) version may have trouble. If you cannot boot the build today, please consider running from source and choosing an older version of Qt in the interactive setup: https://hydrusnetwork.github.io/hydrus/running_from_source.html

If you use the zip or tar.zst, it does not seem like you have to do a 'clean install', but the build releases are changing a bunch of stuff so it is a good time to do one anyway: https://hydrusnetwork.github.io/hydrus/getting_started_installing.html#clean_installs

If you run from source, this is a good week to rebuild your venv. Users on python 3.13 no longer have to choose the (a)dvanced install.

locked pages

A long time ago, when you opened files 'in a new page' or from a subscription files popup, the page created would have no search controls. It would just be a static page that could hold files. It worked well as a 'scratchpad' to work on, but you could not easily search the files if you wanted to.

I replaced that mode with 'initialise the page with a system:hash predicate', which helped simplify things behind the scenes but makes it annoying to append new files or merge other pages into it, since the underlying system:hash stays stuck as what it was originally.

Today we fix this. All search pages now have a 'lock' icon button beside the tag autocomplete text input. Click this, and the current search collapses to a system:hash for the current files in view and the search interface is replaced with an unlock button. It will keep track of when you add or remove files, and if you unlock, the system:hash is of what is currently in view. Have a play with it, and you'll see how it works.

Whenever a new page is created with files--which usually means 'open in a new page' or a subscription file popup--it now starts in the locked state. The old 'no search enabled' behaviour is back, but if you want you can flip to a regular search with one click. Let me know how it goes!

auto-resolution preview

The duplicates auto-resolution 'preview' panel, when you were editing rules, was running way too slow. It could take thirty seconds to load up a count or a preview on a big client, every time you made a change. I've overhauled the whole thing to stream results in fast pieces, with pause buttons and faster cancel tech and better feedback. This system also now handles when the pair comparison takes a while to compute. It is pretty much all ready for 'A and B are visual duplicates'.

I fixed a couple more false positives in 'A or B are visual duplicates'. There's one edge-detection situation that I poured a ton of time into and still failed to catch, so I'm going to keep thinking about it. Please send in any more weird pairs you come across!

Auto-resolution rules in semi-automatic mode will now only queue up 512 items for 'ready to action'. This queue can take a while to build and resets any time you change the rules, so I'm limiting it to keep things snappy. You can change the limit or remove it entirely in the edit rule panel.

misc

Ratings look nicer and line up great again!

A bunch of number-tests across the program have new 'less than or equal to' and 'greater than or equal to' operators.

next week

I only have one week before my summer vacation week, so I'm just going to fix little stuff and clean some code.

[hydrus]

Release Tomorrow!

I had a great week. I fixed a major lag problem in duplicates auto-resolution, re-made a 'search page locked' mode that makes it easier to manage 'static' file pages, and we are folding in some important library version updates to the main build.

The release should be as normal tomorrow.

[hydrus]

Version 623

windows

zip

exe

macOS

app

linux

tar.zst

I had an ok week. I fixed some bugs and the 'A and B are visual duplicates' test is available for everyone in the duplicate filter.

full changelog

highlights

A user has submitted a significant new darkmode stylesheet, 'Paper_Dark', which you can find under options->styles. It has assets and will change the appearance of common widgets like checkboxes.

I recently improved how some PNGs render their colours. The process I wrote was eating a lot of memory on large files, which I fix today. It still runs a little slow, but I'm close to figuring out a nicer solution.

The 'A and B are visual duplicates' test last week went well. Thank you to those who found and submitted pairs it was failing at. I have improved the algorithm's precision and stated confidence, and it is now turned on for all users. In the duplicate filter, you will now see statements like 'definitely visual duplicates'. You can generally trust them, but if you find a pair that is completely different to what it predicts, particularly if it is highly confident they are visual duplicates when they are not, I'd love to see them!

future build

Only for advanced users!

I am making another future build this week. This is a special build with new libraries that I would like advanced users (and this week particularly Win 10 users) to test out so I know they are safe to fold into the normal release. More info in the post here:

next week

I would like to get 'A and B are visual duplicates' working in the duplicates auto-resolution system. This test needs extra CPU, so I'm going to have to adjust some preview UI to handle the per-pair delay better. I feel generally good about it, and I hope we'll be able to auto-resolve a lot of non-pixel-perfect dupes very soon.

Just as a general note, I'm taking my summer vacation week on June 4th, so two more releases before then.

[hydrus]

Release Tomorrow!

I had an ok week. I fixed and cleaned some things, improved some recent png memory spikes, and polished and added the 'A and B are visual duplicates' test to manual duplicate filtering for all users.

The release should be as normal tomorrow.

[hydrus]

Version 622

windows

zip

exe

macOS

app

linux

tar.zst

I had an ok week. There's some bug fixes and new duplicate tech for advanced users to test.

full changelog

highlights

The recent resizable ratings changes broke on a particular version of Qt (PyQt6). If you were caught by this, and perhaps even couldn't boot, sorry for the trouble! I do test PyQt6 every week, but this somehow slipped through the cracks.

Ctrl+Shift+O will now launch the options dialog. If your menubar gets messed up because of a setting, this is the new fallback.

You can now paste multiline content into the text input of a 'write/edit' tag autocomplete that has a paste button (e.g. in 'manage tags'), and it'll recognise that and ask if you want to essentially hit the paste button instead and enter those lines as separate tags. If you would do this a lot, a new checkbox in optons->tag editing let's you skip the confirmation.

I improved some PNG colour correction this week. I think it will make about one in twenty PNGs a small shade brighter or darker--not usually enough to notice unless you are in the duplicates system. If you notice any of your PNGs are suddenly crazy bright/dark/wrong, let me know!

A couple of new checkboxes in options->files and trash let you, if you have the archived file delete lock on, pre-re-inbox archived files that are due to be deleted in the archive/delete or duplicate filters. I don't really like writing exceptions for the delete lock, but let's try this method out.

duplicate test

If you are in advanced mode, the manual duplicate filter will have a new '(not) visual duplicates' comparison line in the right hover, along with some mathematical numbers. This is the 'A and B are visual duplicates' tech I have been working on that tries to differentiate resizes/re-encodes from files with significant differences.

I have tuned my algorithm using about a hundred real pairs, and I'd now like users to test it on more IRL examples. It sometimes gives a false negative (saying files are not visual duplicates, despite them being so), which I am ok with from time to time. What I am really concerned about is a false positive (it saying they are visual duplicates, despite there being a recolour or watermark or other significant change). So, if you do some duplicate filtering work this week, please keep an eye on this line. If it predicts something wrong, I would be interested in being sent that pair so I can test more on my end. Feel free to look at the numbers too, but they are mostly for debug.

Assuming this goes well, I will re-tune this detector, polish its presentation, and enable it for all users, and I will add it as a comparison tool in duplicates auto-resolution.

next week

I think I will keep chipping away at my duplicates auto-resolution todo.