#Deep Space Nine Upscale Project Update: ‘Sacrifice of Angels’

[componentplanet]

Deep Space Nine Upscale Project Update: ‘Sacrifice of Angels’

It’s been almost exactly a month since I published my last Deep Space Nine report, where I showed how different AI software could upscale the show to something approaching HD quality. Despite the ongoing pandemic, I’ve kept the Cascade Lake testbed and RTX 2080 crunching busily away, testing various permutations. Some folks have contacted me to express interest in working together, and I’ve learned some interesting things along the way.

I haven’t been able to find my DVDs, so I bought Season 6 brand-new and started working with that source. I chose Season 6 because it has some of the best space-combat scenes, including the largest battle ever staged in the Star Trek universe in the episode “Sacrifice of Angels.” SoA was the obvious episode to work with and the Defiant image above is from an upscaled encode. Here’s the full shot.

I had hoped that the DVD source would offer a better upscaling alternative than using already-encoded MKVs. I still believe it does, but guys, I have to tell you — the baseline DS9 source sucks.

I watched this show when it was broadcast on cable, on a new 24-inch TV my parents had just bought. It’s one of my all-time favorite shows, and watching it on DVD looks nothing like watching it on TV did 25 years ago. Obviously the base resolution is low, but that’s not the problem. The video is noisy, it’s much darker than I remember, there’s a clearly visible 3:2 pulldown/telecine effect, and distant vessels are often heavily aliased (meaning they crawl with jagged lines). The credits are particularly bad as far as image quality. If the rest of the show looked as bad as the credits, I’d never want to watch it. I may upload a few videos just to show how rough they are.

Click to enlarge

This is what 3:2 pulldown looks like. You’re literally seeing half a frame of information, which is why every other line is blank. There are a number of these moments in any given episode, and while they don’t prevent anyone from enjoying the show, they can be annoying.

Nana Visitor’s reaction to the DVD source quality… or a very lucky pause on my part. You be the judge.

What I’ve Been Working On (and Learned)

Here’s the honest truth: You can get a pretty good looking video if you just rip the DVD without deinterlacing or detelecine via Handbrake (use the H.264 Production Max preset) and then upscale it. While the half-frame transitions are noticeable and annoying, upscaling this way actually cleans up some areas that are otherwise quite jagged and “crawly” at specific points in the show. If you want a one-and-done solution and you aren’t bothered by the occasional half-frame, it’s a great option and I recommend it. The result is 70-80 percent of what I think is likely possible, best-case. If you rip the DVD using Handbrake’s detelecine option, it will solve the half-frame flicker, but at the cost of introducing additional aliasing that wasn’t present before. In my opinion, the telecined upscaled DVD looks better, on the whole, than the detelecined Handbrake output post-upscale.

My long-term goal with this project is to create a guide using as much free software as is possible (Topaz VEAI is obviously a paid purchase). I’m working with a reader who has done some incredible color balance changes, and I’m excited about what that might mean for the project.

What I’ve done for the past month? About 600GB of renders at 3-8GB each. I’ve been examining color grading with DaVinci Resolve, rescaling in that same application, various AviSynth filters for antialiasing, detelecine, and deinterlacing using algorithms like QTGMC. The truth is, I could accelerate the process if I focused on smaller clips, but I prefer to upscale the entire episode. That way, I can check any trouble spot or problem area in one area of an encode against all the previous settings I’ve tested, to see how that particular area was handled.

Another thing I’ve learned? The best version of Deep Space Nine would be constructed clip by clip, using optimized video processing settings for each. I have no intention of slicing and dicing episodes up by hand, but if there was an episode you truly loved, you could achieve some truly impressive results that way.

I want to show you a short clip from “Sacrifice of Angels.” First, the DVD source and second, the upscaled output with QTGMC applied. QTGMC is a deinterlacing filter, not a detelecine filter, and it works by creating additional frames. The final output does not have the hypersmooth look of interpolated sports video, but I’ve had trouble matching audio to the clip. There’s a lot of hands-on learning involved in this kind of project because ultimately, each video benefits from a different set of filters. For best quality, change both videos to top available playback source.

This is the original DVD source. Note how the nacelles on the Miranda-class starship (the two ships in the opening frames) shimmer. This is telecined output, which means they look much better in this video than they do if I detelecine the source using Handbrake. There’s a lot of noise in certain frames and some visible compression artifacts in others.

This is the upscaled video after I applied QTGMC deinterlacing to it via a buggy and difficult-to-parse application named StaxRip. It’s been incredibly useful to me in certain respects, but if I’m being honest, I’m trying to find an alternative because this app is rather ornery and difficult to use. It also only seems to output H.265. Figuring out how to use applications like AviSynth (current user level: Bad) is part of the experience. One of our readers, Shortstick, has contacted me to show off some of his own color grading work on DS9, with impressive results:

We are looking into how to combine efforts and further improve the show.

Why I’m Doing This

If you watched Deep Space Nine on TV growing up as I did, I have news for you: You never actually saw the work that VFX designers put into those battle scenes. Watching “Sacrifice of Angels” in upscale on a much larger display, I was struck by how incredible the shots were. The space battle in Insurrection may have had more expensive special effects and a few more years of CGI advances, but it didn’t involve half as many ships or as many complex maneuvers.

Until I started working on this episode, I never thought about how space combat evolved from Star Trek: The Next Generation to DS9. On TNG, battle is almost stately, with large ships firing at each other from static positions. The exception to this is stern chases, where the Enterprise is pursued by an opponent.

DS9 changed all that. The decision to introduce the Defiant and to make it a small ship completely changed the rhythm and flow of space combat. The Defiant wasn’t made for stately, sweeping broadsides — it’s an antimatter-powered flying gun that can absorb significant amounts of damage while it blows your ass into next week. Above all, the Defiant is fast, and Jadzia Dax is one hell of a pilot.

The entire battle “language” of Star Trek changed dramatically between TNG and DS9, largely on the backs of the VFX artists who were tasked with doing the work. DS9 didn’t just add more ships; it showed those ships doing more things, with background battles often as dramatic as the foreground shots. True, some people disliked the look and preferred the idea of a more spread-out fleet engagement, but I’m not one of them. I’m watching a show in which aliens with no concept of time live inside a stable wormhole. I don’t need the starships to stay far away from each other to enjoy the battle scenes.

Speaking of wormholes…

Watching this episode in standard DVD quality is like looking at a da Vinci painting with 500 years of grime on it. You can recognize the mastery of the work, but there’s a lot of schmutz between you and it. Thanks to advances in AI processing, we’re finally seeing consumer tools that can wipe the grime away — and not just for DS9, but for any number of additional shows. The artists that worked on these episodes deserve to have their work seen in something approaching the way it could have looked.

I’m never going to be able to make these old DVDs look as good as what Paramount could do. Heck, I’m never going to improve them as much as a professional video editor could do. But Paramount has no plans to upgrade DS9 itself, and that means the only way to restore the TV show to some semblance of how it could look is with a lot of elbow grease, filter testing, and one exhausted RTX 2080. I think the work deserves to be done, even if Paramount disagrees.

Interested in helping? Give me an email or sound off below. Got ideas or tips for using AviSynth? Get in touch.

Now Read:

Upscaling Star Trek: Deep Space Nine Using Topaz Video Enhance AI

Star Trek: Voyager Gets 4K Upscale Remaster via AI

Fan Works to Remaster Star Trek: Deep Space Nine in 1080p HD, Using AI

from ExtremeTechExtremeTech https://www.extremetech.com/extreme/308505-star-trek-deep-space-9-upscale-project-update from Blogger http://componentplanet.blogspot.com/2020/03/deep-space-nine-upscale-project-update.html

[componentplanet]

What No Man Has Seen Before: Remastering Deep Space Nine to Maximum Quality

I’m done.

Most stories don’t begin at the ending, but that’s the only place to start this one. I’ve been working to remaster Deep Space Nine for the past nine months, ever since AI-based video upscaling software began to hit the market. After I saw how much improvement could be wrung out of some old MKVs, I decided I’d start over, using the original, superior, Deep Space Nine DVD source. Nine months later, I’ve accomplished what I set out to do: Create a method of remastering and upscaling Deep Space Nine that didn’t rely on hand-combing episodes to fine-tune deinterlacing algorithms while compromising on image quality to the smallest extent reasonably possible. I’ll be demonstrating the results all throughout this article.

I have created a 23.976 progressive version of Deep Space Nine, codenamed Rio Grande. While I have not yet checked the method against the entire show, it worked well on test episodes from S2, S4, S5, and S6. Special thanks to Cyril Niderprim, who found the hilariously simple solution below, and a pox on the small mountain of more complicated scripts I’d written. While Rio Grande is capable of throwing errors — I have included an example of such — there’s an alternative if it tangles up an episode in a way you don’t like.

I have also achieved an identical-quality 59.94 fps conversion, codenamed Orinoco. Orinoco preserves motion correctly in all scenes and is included here as an insurance policy. While it matches Rio Grande on quality, it packs 2.5x the frames and takes 2.5x as long to process and upscale. Rio Grande should work — but if it doesn’t, Orinoco will. If either of these methods proves insufficient to deal with an episode, we’ll come up with a custom way of handling it and I’ll wind up writing something about it and/or updating this article to keep track of special cases.

I’ve done a couple renders of the credits to showcase two different AI upscaling models Topaz offers: Theia Detail (at default settings) and Gaia-CG. Gaia-CG is below:

This article is not a step-by-step tutorial on how to perform this process suitable for literally anyone to follow. That will be its own project. I will, however, provide enough information that anyone with a passing knowledge of AviSynth should be able to recreate both approaches.

Theia-Detail credits below:

There’s a missing bit of frame for several seconds when the Bajoran freighter is panning around the station. This is a unique error — I’ve never seen it anywhere else on the show — and it pops up a lot, nearly regardless of what filters you use to adjust an episode. It’s always unique to the credits; you don’t need to worry about random blocks of content missing from the sides of your video.

Maximum Warp

Rio Grande and Orinoco make as few compromises on image quality as is reasonably possible. Every aspect of this process is designed to preserve detail at the deliberate cost of hard drive space until the final encode, at which point you can be as lossy as you like.

All of the clips you’ll see below were encoded in 0.0 H.264 until losslessly upscaled frame-by-frame. I encoded these clips in H.265 at a CRF of 6, which turned out to be ludicrous overkill. Live and learn. There’s a freeze-frame, just-noticeable quality difference between CRF 14 and CRF 20, while 25 is pretty ugly.

Upscaler Models: Avoid Artemis, 200 Percent Upscales

I’ve encoded at least one example of every upscaler mode Topaz offers except for Artemis-LQ and Artemis-HQ. These algorithms are completely unsuited to Deep Space Nine, and while they no longer produce literal garbage output when run against the show, they do not react well to its content. Consider them fundamentally unsuited to the task. I don’t care for Gaia-HQ’s output against DS9 very much, though I included one example below. That leaves the two Theia models and Gaia-CG.

I’ve always preferred Gaia-CG, so that’s what the majority of the encodes are in, though I’ve also showcased the Theia algorithm family. The Theia models offer tunable parameters for noise, sharpness, and detail recovery, and these can be very useful if you’re attempting to denoise an ugly patch of wall or carpet.

The 200 percent upscale option should be avoided. It creates errors that were not originally present in the source in some cases and it offers meaningfully reduced image quality compared to the 400 percent version. Be advised that the repair process has a habit of creating a seam down the left or right side of the image frame — you can crop it out with no trouble, but it’s something to be considered.

Topaz offers the option to upscale losslessly by image or via lossy MP4. All of the upscale videos you’ll see here are lossless PNG conversions.

How Much Additional Detail Did I Recover?

The idea that sent me haring off on this mad quest was that I could extract additional visual data if I returned to the source DVD rather than relying on old backup files. Here’s an example of my best upscale in January:

That was then.

While some of these improvements come from low-level improvements to the AI upscalers themselves, the upscaler has to be able to lock on to enough information to upscale an image without making hash of it. Imagine a screenshot where everything was rendered at DVD resolution except for Odo’s hair, which was done in 4K. It would look odd, to say the least. The more detailer the upscaler can get its hands on, the better the chance it creates something gorgeous on the back end. Going back to the DVDs gave the upscaler more to work with in the final process.

This is now.

As a reminder of how far we’ve come: The following three screenshots are the Defiant as it appears on the native DVD, the best upscale result I could achieve in early February, and my current best result today:

The Defiant in all her DVD glory. At least it beats Netflix. Let’s see what we can do.

February, 2020: It’s a big improvement, but I believed better was possible.

That’s more like it. Image from Orinoco encode.

Here’s my favorite Defiant shot from the entire show. I was really proud of how this turned out in April. It’s still quite nice.

I still like the way this came out, but it’s not the best we can do.

This is one place where I’m obviously getting help from a tweaked AI upscaler model. Better processing might clean up the model a bit differently, but some of the features on the hull are slightly different shapes. This image benefits from a side-by-side comparison to the other at maximum resolution.

The gains are real. Every time I compare an upscale based on the old MKV files I’d created years ago against the benefit of going back to the DVD, the DVD wins. If you want the best version of Deep Space Nine, invest in the DVD set.

So How Good is It?

With careful processing and good upscaling, it’s possible to give Deep Space Nine a clarity that I think approaches that of what’s typically referred to as “HD” content, though it’s still limited to the NTSC color gamut as opposed to later standards like Rec. 709. At its worst — allowing for some deviations from perfection — it’ll still look like the best damn DVD you’ve ever seen. At its best — and I consider the shot of Sisko up there to be one of the best — I’d argue that he, at least, comes across in HD levels of detail.

The processing steps I’ve used repair some baked-in damage to the source, but not all of it. There’s a frame sequence in “The Way of the Warrior” that literally looks as though something was spilled on the film, and that’s not something I know how to fix. All of the encodes below are from Rio Grande.

Way of the Warrior: It’s No Illusion

“The Way of the Warrior” contains two of the best battle scenes of Season 4. In the first, Sisko takes the Defiant to rescue a Cardassian ship under heavy fire from the Klingons. You can see the VFX team experimenting with what amounts to an over-shoulder kill cam — not exactly the kind of thing Next Generation was known for. It’s obvious the VFX teams had a lot of fun with the Defiant.

The second major battle scene of the episode is below. There’s an excellent callback to an event from the show’s pilot, when Kira attempts to bluff a squadron of Cardassian warships into believing the station is far better armed than it actually is.

This time around, things go differently. Upscaled in Theia Fidelity first, at default settings, followed by Gaia-CG. Gaia-CG is what I’ve used for screenshots in the past:

I’ve included clips of normal human interaction in this article specifically to illustrate that these processing techniques don’t just work on starships and battle scenes. Deep Space Nine is about far more than just CGI combat — it just happens to have some really great examples of the latter, particularly by late-90s standards. Gaia-CG below:

Paradise Lost

The Defiant takes on the Lakota in this battle scene. Elements of Starfleet were in the process of slow-walking a coup attempt. This clip deliberately shows what happens when Rio Grande doesn’t get a scene entirely right. While this shot mostly works, it’s noticeably jerky at the end. I’ve confirmed that this only affects Rio Grande — Orinoco plays back perfectly smoothly in this situation. If the small amount of jerk doesn’t bother you, Rio Grande is your oyster. If it does, try Orinoco.

Far Beyond the Stars

One of the benefits of science fiction is that it allows creators to play fast and loose with the question of time travel. Here, Benjamin Sisko is flung back into the 1950s — at least, in his own mind — and finds himself face to face with the ugly racial politics of the era. Upscaled in Gaia-CG:

In the Pale Moonlight

The greatest episode of Deep Space Nine concludes with a chilling soliloquy. Upscaled in Gaia-CG:

The same scene, rendered in Theia Detail at 10 percent sharpening, 20 percent noise reduction, and 20 percent detail recovery.

Sacrifice of Angels

Y’all knew we’d be back to this episode, right? In First Fleet (below, rendered in Gaia-CG), the Federation closes to battle with the advance forces of the Cardassian / Dominion military. I quite like the Galaxy-class maneuvers that bookend the clip. The Galaxy-class often looked fat and ungainly from certain angles, while others showed it to great effect. It gets more of the latter and less of the former on DS9 than it did on TNG.

This initial engagement doesn’t provide the opening the Defiant needs in and of itself. Starfleet decides to try and force a breach through enemy lines, even though they’re outnumbered, 2:1. This is the scene I chose to render in a number of different ways. First, Gaia-CG:

Next, here’s Gaia-CG at CRF=20. The quality loss is modest here, but I don’t recommend going much higher. The problem with starting from DVD source is that there’s not a lot of detail to lose before quality starts suffering, and the interpolated/upscaled information is only just so good to start with. When you start pushing the quality downwards past a certain point, it drops like a rock. CRF=25 is way too high. I wouldn’t go lower than 20.

Gaia-HQ:

Theia Detail, default settings:

And finally, Theia Fidelity, at 20 percent sharpness, 30 percent detail recovery, and 20 percent noise reduction settings:

Although I’ve chosen to focus on the later seasons of the show, this encoding and upscaling method works on the early seasons, too. Just keep in mind that the actual source material for the early seasons isn’t as good as what we got in later years.

That concludes the demonstration portion of today’s program. Now, let’s talk about implementation.

How to Encode Rio Grande

To encode using Rio Grande, rip the DVD using DVD Decrypter and generate a D2V index file using DGIndex or an equivalent tool of your choice. Next, fire up whatever AviSynth front-end you use, if you use one at all. I use StaxRip, personally.

The AviSynth script is as follows:

TFM() TDecimate() QTGMC2 = QTGMC(Preset=”Very Slow”, SourceMatch=3, InputType=2, Lossless=2, MatchEnhance=0.75, Sharpness=0.5, MatchPreset=”Very Slow”, MatchPreset2=”Very Slow”) QTGMC3 = QTGMC(preset=”Very Slow”, inputType=3, prevGlobals=”Reuse”) Repair(QTGMC2, QTGMC3, 9)

That’s it. You also have the option to inject some noise back into the video in the process of running these scripts — season to taste, basically. Noise added to one of the QTGMC runs will still come through after repair. While Rio Grande isn’t 100 percent frame-perfect, it works perfectly on all but one of the 29.97 fps scenes I’ve tested against. If it proves to have more issues than anticipated, we’ll modify it — and that’s where Orinoco comes in. Technically, it’s possible to render this specific scene at 59.94 fps and then include it in the 23.976 fps version by assembling a VFR file when you compile your image sequence back into video. I might explore this in a future update.

The QTGMC runs above are required to achieve the improved visuals, and while I’m completely open to alternatives, nothing I’ve ever tried comes anywhere close to what this script can do. You can tune the above script for strength and processing time, but you’ll notice the quality hit from stepping down to “Slow” from “Very Slow” and “Medium” is unsuited to the task. If you like your content noisy and you don’t mind more combing artifacts (or have a different solution for dealing with them), “Slow” may be a better option.

The one downside to running QTGMC this way is that it can occasionally produce a small ripple in one part of a scene. I find it trivial to ignore and the only solutions I’ve found for it so far cause far more damage than the ripple itself. While I’d like to get rid of it, it’s currently a tolerated “feature.”

Defiant, rounding the station.

The “InputType=2” and “InputType=3” sequences are essential. Running a single QTGMC instance will not produce the same effect. Running two instances of “InputType=2” will not achieve the same effect. There are nine potential ways to combine input types in a repair run like this and I’ve tested every single one of them. The “2,3” combination is what works the best, from “Emissary” to “What You Leave Behind.”

Similarly, I’ve tested all 25 functional “Repair” modes AviSynth supports. You want ,9. If you don’t like that output for whatever reason, try ,1. Most of the others didn’t have much of an impact one way or the other, though a few of them broke content in interesting ways. I’ve revisited them more than once when testing new filters, just to see how final output varied by mode.

If you have suggestions on how to improve output quality further, by all means, drop me a line.

How to Encode Orinoco

Orinoco is not simple, but it works far better than I feel like I had any right to expect.

For starters, we’re creating two files, not just one. Encode your first version of your target episode using the following script:

TDeint(mode=1, type=2, tryweave=true, mtnmode=3, full=false, ap=10, aptype=2, slow=2)

This script orders TDeint to produce doubled frame output, to perform kernel interpolation, to attempt to repair a frame by weaving if the result is fewer combing artifacts than deinterlacing would create, and to only deinterlace interlaced frames. This preserves the progressive frames baked into the NTSC source. Type=5 was the only option that came close to Type=2 in overall image quality, and the bi-directional blending Type=5 introduces causes more frame blending issues at scene boundaries. Type 5 occasionally fixes a problem in Type 2, but it causes them far more often than it repairs them.

Now we set this clip aside and turn our attention to the other. Run the following script against your second clip:

QTGMC2 = QTGMC(Preset=”Very Slow”, SourceMatch=3, Lossless=2, MatchEnhance=0.75, Sharpness=0.5, MatchPreset=”Very Slow”, MatchPreset2=”Very Slow”) QTGMC3 = QTGMC(preset=”Very Slow”, prevGlobals=”Reuse”) Repair(QTGMC2, QTGMC3, 9)

While similar, this script is not identical to the script we used for Rio Grande. Rio Grande’s QTGMC implementation is run in progressive mode. For Orinoco, we want to double the frame rate, bringing our base 29.97fps up to 59.94fps. Now that QTGMC and TDeint are at the same frame rate, we run a repair against them, too. If you run two clips in “Repair” mode at two different frame rates, it’ll work… ish. You’ll know if you’ve done this if you’ve got a weird seam migrating across your video.

This is a bass-ackwards approach to video editing, at least as far as I’m aware, but there’s a method to my madness. The reason people don’t typically just inject a bunch of frames to bring VFR 23.976 / 29.97 content up to 59.94fps is that the process the interpolated and blended frames that boost the frame rate can also create visual errors and other problems. This creates a catch-22: Leave those frames out, and the single-rate output is choppy. Keep them in, and some of your scene boundary changes are going to look decidedly odd as a fireball mushrooms from someone’s head, or two completely different locations are blended together. It’s not a great situation.

But there’s a way to fix the problem.

clip1=FFVideoSource(“C:\DS9S6D2\Sacrifice-TDeint.mkv”) clip2=FFVideoSource(“C:\DS9S6D2\Sacrifice-QTGMC-ToPairWithTDeint.mkv”) Repair(clip1, clip2, 9)

The reason we create the interpolated QTGMC file is to serve as a sharp, clean source against the Sacrifice-TDeint file. The reason I used QTGMC for this is that — as I’ve said before — I’ve never found an equivalent method for creating clean footage. If you want less noise in your video, use “Very Slow” or “Slow” as an alternative. I do not recommend “Medium” for this purpose.

You can run QTGMC against this output again if you want, but the footage will start to look a bit overprocessed. The Theia models can compensate for this to some extent, thanks to a fairly effective (and tuneable) denoise algorithm. There are multiple examples of Theia models in the videos below.

Quality-wise, Orinoco and Rio Grande get you to the same place and I often cannot tell the difference between them when I compare them frame-by-frame (not counting interpolated frames, obviously). Rio Grande is to be preferred — it’s faster and simpler — but I cannot guarantee that it’ll work on all 176 episodes of the show. If you run into an episode with erratic or stuttering motion when its created via Rio Grande, try Orinoco as an alternative and let me know you’ve hit a snag.

Flaws and Shortcomings

Orinoco and Rio Grande are not perfect. I’m still hunting for a method of eliminating the ripple. Small as it is, it’s basically the one problem left (so far!) that I haven’t solved. The best method I’ve come up with so far involves using MCTemporalDenoise, and MCTD appears to be both entirely single-threaded and causes me other headaches in terms of source quality.

If you want to experiment with MCTemporalDenoise, I’d recommend using up to seven pixels but applying a relatively weak effect (strength 2-3). This will not completely eliminate it, but it helps in certain places. If you don’t like the impact this has on your visual quality, investigate the Theia upscaling models and try applying 10-20 percent noise reduction. While you don’t want to go nuts, you can lightly smooth away some defects. I don’t like using one filter just to squash problems another creates, and with any luck Orinoco won’t be needed in the first place.

While my goal has always been to create a one-size-fits-all template for this project, I’ll record any adjustments or changes that need to be made to Rio Grande or Orinoco to deal with problematic episodes, unless that episode is Move Along Home.

Next Time on… the Deep Space Nine Upscale Project

Now that I’ve finished this project, I get to actually start ripping episodes and encoding them. I’m going to write a set of tutorials that give an actual guide to how this works, build a benchmark for workstations, and solve any per-episode problems that need fixing. I’ve come up with a completely free method of upscaling that, while not as good as Topaz, presents a genuine improvement. I also want to look into PAL.

ViacomCBS doesn’t think Deep Space Nine is worth remastering. I say it’s wrong. There’s a limited amount of improvement we can squeeze from a DVD, which is why you don’t see me referring to the show as “4K” or “UHD”, even though it’s common to see those terms tossed around in projects like this. You can upscale Deep Space Nine to 4K resolution, but there’s no way (yet) to squeeze 4K quality out of a DVD source file. This is an imperfect solutions — but it’s light-years beyond any previous version of the show that’s ever existed. If it turns out that Rio Grande does throw more than very occasional 23.976 fps errors, we’ll find a solution for those episodes. There’s every indication that Nvidia’s upcoming GeForce RTX 3080 is going to make this kind of upscaling much faster, which will reduce the pain of a 59.94 conversion.

After today, no Star Trek fan has to wait for ViacomCBS to decide it makes financial sense to invest in its own back catalog. It’ll cost some money for the DVDs and the software, and a lot of compute time to get the work done (8-10 hours per episode on an RTX 2080 at 23.976, 2.5x longer at 59.94fps). Will most people do this? No. Most people will watch the show on Netflix, where the quality is much lower than what they’d get on DVD.

But as of today, nobody has to do that again. As of today, you can see the work of the VFX artists and actors who created in this show the way it was meant to be seen. Orinoco and Rio Grande aren’t perfect, but they illustrate just how much additional quality can be wrung from late-90s DVD source. GPUs are getting faster. AI upscalers are getting smarter. Even if the computer or GPU you own today isn’t capable of tackling a project like this, your next one very well might be.

May the Prophets guide you,

Hruska out.

I owe thank-yous to Stan Pennington, Gary Huff — man, Gary, I probably owe you a hearing aid — Cyril Niderprim, Robert Hallock, Andrew Swan, David Berry, Dan Greeney, Mark Renoden, the folks at Black Magic, my long-suffering fiancée who has listened to me talk about this topic for most of a year and contributed insights at numerous points, Ben Carter, Joe Robinson, multiple helpful individuals at Doom9, and a lot of other people. If I forgot names, I’m sorry. If you helped me, thank you. Most of all, thank you to the talented actors and artists who created Deep Space Nine in the first place. 

Now Read:

DS9 Upscale Project Update: What I’ve Been Working On

Deep Space Nine Upscale Project Season Finale: What We’ve Brought Ahead

Deep Space Nine Upscale Project (DS9UP): Technical Goals and FAQ

from ExtremeTechExtremeTech https://www.extremetech.com/extreme/314653-remastering-deep-space-nine from Blogger http://componentplanet.blogspot.com/2020/09/what-no-man-has-seen-before-remastering.html

[componentplanet]

DS9 Upscale Project Update: What I’ve Been Working On

It’s been several months since I wrote an update on my ongoing efforts to restore Deep Space Nine. I took a break from the project through much of June due to a move and an associated injury but jumped back into it in July and have been at work steadily since then. The majority have my time has been focused on understanding how shifting the episode into various alternative frame rates would impact motion smoothness and image quality.

In the past, I’ve written updates when I hit specific milestones I’d set for myself or discovered something I thought was interesting. This is more of a progress report. So, to begin: A bit of recap: I’m a lifelong Deep Space Nine fan who started this project in January and has pursued it since. I’ve been learning about video processing and encoding from scratch as I’ve worked, and according to everyone I’ve talked to, I didn’t pick a beginner-level project.

Deep Space Nine is a VFR (Variable Frame Rate) show, which means the DVD alternates between playing back at 23.976 fps and 29.97 fps at various points within the episode. This is a common format for late-1990s science fiction. Shows encoded in this fashion include Babylon 5, Stargate SG-1, Buffy the Vampire Slayer, Star Trek: The Next Generation (DVD-only), and Star Trek: Voyager.

The episode of Deep Space Nine I chose to treat as my test vehicle, “Sacrifice of Angels,” is about 14 percent 29.97fps footage and ~86 percent 23.976fps footage. The problem is, applications like AviSynth cannot edit VFR video and must convert it to CFR (Constant Frame Rate). Applications like DaVinci Studio Resolve can technically “handle” VFR files, in that it will ingest them properly, but the resulting output periodically pauses in a way I couldn’t find a clean solution for. For now, unless I figure that out, processing the show requires that it be converted to CFR as an initial step.

If you encode a VFR show at 23.976 CFR, the 29.97fps content will be cut to 23.976fps and the playback may not be perfectly smooth. In some cases, you won’t see any stutter because there’s not enough motion on screen for the frame decimation to be visible. There’s a several-minute block of 29.97fps content in “Sacrifice of Angels” when Dukat, Dumar, Weyoun, and the female Changeling are all talking at Ops. While there are a few telltale signs, you only really see it when Dukat walks around the table — and this is after both postprocessing and upscale.

The reason that Dukat’s hand and body are blurred as he moves is that, if you go frame-by-frame, what he’s doing looks like this:

I boosted the brightness a bit here, to make the shadow easier to see. Most of the frame looks normal, but you can see where Dukat’s hand is going to be in the next frame. The error is visible but small and confined to one part of the screen.

The fact that a lot of TNG-era Star Trek is conversation makes the frame rate shifts that much easier to deal with, but it’s still noticeable as heck when it happens. My goal has been to find an automated method of processing DS9 that would typically produce better motion during 29.97fps content sections. I spent the last few months playing with various methods of converting the show’s frame rate to see what the options would look like.

Source Sensitivity

The transmutative property of mathematics states that when you multiply two numbers together, it doesn’t matter what order you write the numbers in. 1 * 2 * 3 * 4 = 24. So does 4 * 3 * 2 * 1. Video processing is not transmutative. The order in which you apply filters changes what the final output will be. Video processing workflows need to be duplicated exactly in order to guarantee accurate results, up to and potentially including using the exact same application and filter versions.

There are a few reasons I’ve been exploring the outcomes for Handbrake and MakeMKV as opposed to using DVD Decrypter to create a VOB copy of the DVD data in 59.94 interlaced format.

First and foremost, I’ve yet to figure out how to get the video output quality to look anywhere near as good as what I’ve achieved with HB/MMKV without creating scripts for each episode. In point of fact, I haven’t completely figured out the episode scripting, either. This is what I get for taking my last programming course circa “I Want It That Way.”

While we’re being honest, I want it this way, but we don’t get everything we want, do we ViacomCBS? Image from the “What We Left Behind” DS9 documentary, showing what a remastered Defiant would look like.

My own best. Spoiler: The professional one is better.

Second, Handbrake offered some really simple options to batch up and test a huge range of file encode presets. In mid-July, I ripped “Sacrifice of Angels” more than 250 separate times in Handbrake in order to examine the impact of various quality control settings, H.264 flags, frame rates, and deinterlacing options. Third, I finally figured out how to hand StaxRip a set of flags that would synchronize the audio/video playback of a VFR MakeMKV file, and I wanted to experiment with it. Finally, part of learning something is figuring out what not to do. I make a lot of mistakes and I make some of them on purpose, just to see how various ideas change the final output.

I have spent a great deal of time during the last two months playing with various methods of changing frame rates. AviSynth has a number of filters for changing the frame rate and different source filters yield subtly different outputs. I’ve experimented with various methods of interpolating up to 119.88fps before trimming back down — either to a compromise frame rate like 59.97 or back to 23.976. I’ve done a lot of testing combining a pass through Davinci Studio Resolve through further processing with AviSynth, or before AviSynth, or after. I’ve experimented with various H.264 quality levels and specific presets to look at the impact these would have on the areas of troublesome motion in the show. To be honest, I worked out a strategy for what I wanted to encode and allowed the encoding to race ahead of my actual evaluations. I’m still evaluating what I’ve created. If any of these methods had yielded a single clear winner, I’d have said so, but I’ve certainly seen some intriguing differences among the data. I’ve even played with some of the AI-based methods of interpolation to see how they’d compare.

Separately from this, I’ve experimented with deinterlacing based on 59.97 VOB files. Even with script help from some of the community at Doom 9, I haven’t found a single, broad, fire-and-forget solution that gave me as clear an image quality as what I’ve gotten from MakeMKV and Handbrake. Part of the reason I chose to stick with these sources when evaluating motion is that I knew I’d already achieved something reasonably close to what I’d consider final quality. I wanted to hold that set of variables constant and experiment with the methods I’d already worked with, especially when I had trouble achieving the same image quality. Still hoping to find one, but that’s why I chose to focus my time where I did.

The Pros and Cons of 119.88fps

One way of solving the 23.976fps and 29.97fps playback problem is to shift content up to 119.88fps. The problem with 119.88 — well, one of them, because there’s not just a problem — is that you’ve definitionally quintupled your workload. If it takes 15 wall-clock hours of mixed CPU and GPU processing time to upscale an episode of 23.976 DS9, it’ll take ~75 hours for 119.88.

That’s not great. And to add insult to injury, you need a 120Hz display to watch the output without dropping half the frames.

I’m still messing around with 119.88, because so far I’ve gotten the best overall results in those troublesome patches at this frame rate, but it’s hard to imagine attempting to do the show this way. Ampere would have to be more than 2x faster than the GTX 1080 Ti to make the GPU processing times anything near reasonable.

Alternatively, one can attempt a frame rate between 23.976 and 119.88, and I’ve been doing some experimenting there as well. These frame rates all require either the film or video portion of the material to shift playback speed by a non-integer multiplier, which means there’s always some degree of detectable something. What varies is just what that something is, and how often it pops up. I’ve also tested the outcomes if you upscale the video first, then process it. The end results are pretty good, but the clock time penalty for processing 2560×1920 clips versus 720×480 clips is larger than the resolution increase alone would suggest.

Where This Is Going

My plan is to assemble a set of options that make some reasonable tradeoffs as far as motion smoothness versus processing time versus frame rate, with at least two and possibly three targets. I’ve also been experimenting with masking and antialiasing lately, including using a version of an episode with fewer aliasing problems as an external antialiasing guide for a version of the same episode optimized for smoother motion. And it works!

…ish.

One of the things I’ve learned is that when searching for a best-fit line that will safely adjust a television show, you may be very lucky to find a single method that works for an episode. Asking for a method that globally works well for 176 episodes is asking a lot.

Most of the time, what you get is…ish, and some things are a heck of a lot ‘ishier’ than others. The external clip concept is interesting, but after playing around with it for a little while I’ve got my doubts about whether it can work. There are scenes that it transforms as perfectly as I could ask for — and scenes that, uh, don’t.

This is what “Don’t” looks like.

I suppose a pertinent question to y’all would be: How much of what doesn’t work are you curious to see in the first place? I haven’t posted or talked much about failed experiments to date, and the reason this story doesn’t have more video is that I’m not sure what people would find interesting in the first place. It doesn’t seem all that interesting to just talk about what doesn’t work. If you find this sort of work-in-progress more interesting, or if you’d find it more interesting if I gave you more to look at, say so.

Now Read:

Deep Space Nine Upscale Project Season Finale: What We’ve Brought Ahead

Deep Space Nine Upscale Project (DS9UP): Technical Goals and FAQ

Upscaling Star Trek: Deep Space Nine Using Topaz Video Enhance AI

from ExtremeTechExtremeTech https://www.extremetech.com/extreme/313963-ds9-upscale-project-update-what-ive-been-working-on from Blogger http://componentplanet.blogspot.com/2020/08/ds9-upscale-project-update-what-ive.html

[componentplanet]

Deep Space Nine Upscale Project Update: Variable Frame Rate DVDs Can Burn in Hell

The following unhinged rant article is the sole opinion of its author. ExtremeTech does not discriminate against variable frame rate media.

In my previous two Star Trek: Deep Space Nine articles, I’ve discussed my efforts to upscale the content, provided some video and image samples, and discussed the process of slowly attempting to remaster the show using commercially available software. I have continued to make real progress on the show in terms of overall image quality, to the point that I’m turning my attention to fixing some other problems I haven’t been sure how to deal with.

Specifically: The frame rate.

In Which My Barely Video-Literate and Underslept Self Attempts to Explain the Mechanics of Hell

Star Trek: The Next Generation, Deep Space Nine, and Voyager weren’t recorded in anything as sensible as a steady 23.976fps. When these shows aired in the 1980s and 1990s, they were constructed from at least two different types of content: 23.976fps progressive content and 29.97fps interlaced content. Episodes of these shows do not run at one frame rate; they run at different frame rates depending on what’s happening on-screen. A show that runs at one frame rate, start to finish, uses Constant Frame Rate encoding (CFR). Deep Space Nine, TNG, and VOY all use VFR. Stargate SG-1 and Babylon 5 do the same thing. Most of the best nerd content of the late 1990s and early 2000s is locked behind media that go for this kind of manipulation.

One of the things I haven’t talked about, but have been privately confused by, is why so many of the filters I deployed in an attempt to fix DS9’s frame rate weren’t producing the results I wanted. When I reported in my last article that DS9 used 3:2 pulldown, in which three progressive frames of data are followed by two interlaced frames, I was right — and I was wrong.

The Deep Space Nine episode “Sacrifice of Angels” absolutely contains video footage that was mastered with 3:2 pulldown applied. It is not made of this footage. It contains interlaced frames that were there when the videos were transferred to DVD in the first place. It isn’t made of this footage, either. No, Deep Space Nine, or at least “Sacrifice of Angels,” is mostly 23.976fps (which looks fine), but with enough 29.97fps footage spliced into it to make it jarring.

Part of the problem is that interlacing and 3:2 pulldown are a lot more visible on modern LCDs than they were on older CRTs. Issues like this have cropped up in gaming as well. If you look back at reviews of these DVDs, they were lauded for their quality when the show came out in the early 2000s — and probably reviewed on CRT monitors. This, at least, is what a long-suffering video editor friend told me when I pestered him to answer my questions for three months straight a few hours.

Part of the problem is that TVEAI does a really good job enhancing half-field interlaced frames when they appear on-screen. They’re far more visible when upscaled than at native resolution.

This is what a telecined or interlaced frame looks like. You can clearly see that it’s just half a field of visual information, not the full frame.

Here’s another issue: A lot of video editing applications, including AviSynth and TVEAI itself, are none too fond of VFR content.

When I feed a MakeMKV-derived source file directly into Topaz Video Enhance AI, it detects more than 80,000 frames in an episode that ought to contain around 65,000 frames. TVEAI exposes no encoding options to the end-user, and the default output from content scaled in this mode runs at 29.97fps. If you were the kind of kid who enjoyed playing 33 1/2 RPM records at 45 RPM, you’ll be very excited by this result. If you weren’t, Captain Benjamin “Alvin” Sisko is not an improvement on the original version. (Assuming, at least, that I was nuts enough to pitch-shift the audio to match it).

If you don’t want to run content at 29.97fps, there’s another option — re-encode it at 23.976fps. Convert the interlaced frames, lower the frame rate, and you’ll end up with imperfect, highly noticeable jerks and jumps during fast-paced space combat scenes — which are exactly the ones we are trying to preserve. This is known as judder. Judder sucks.

Topaz Video Enhance AI is not the only application that gets confused by VFR content. I’ve been using StaxRip as a front-end for AviSynth+ and have been mystified as to why the program constantly detects a frame rate of 24.66fps when loaded with MakeMKV source. It turns out that 24.66fps is the average frame rate over the entire episode if you average together the relatively small number of 29.97fps scenes with the much larger number of 23.976fps scenes.

So. The goal is to duplicate the smoothness of the 29.97fps content in the stream without judder. I’ve actually made some real progress towards that goal — enough to say I feel like I’m nearing the end of the first stage of this process. (Color grading is going to be the second stage, oh joy of my heart.)

Where I’m running into trouble, however, is figuring out to remux the audio. I haven’t yet figured out how to synchronize audio to video playback at new frame rates, and the judder management isn’t perfect yet. This (near) final video is from several days ago. I’ve actually improved further on this since I ran the encode. And since if you’re reading this, you’re probably wanting to see the content, so I’ve got an interim version of the credits I’m happy to share. You’ll have to ignore the misaligned audio and you need to set YouTube to 4K for best image quality, even if you don’t have a 4K monitor.

Let’s compare this new video with the video I created back in February. There are a few things I want to specifically call out. Both videos were upscaled using Topaz Video Enhance AI, but Topaz has updated the application in the intervening period. This does not seem to have had a large impact on image quality, but may have tweaked it in some ways. The top video is April, the bottom video is from February. Run both of them in 4K mode or as close to it as you can get.

Speed: The old video runs at 49.32fps, while the new one is encoded at 24.66fps. I’m still grappling with StaxRip / AviSynth+ to get this problem out of the video, but 24.66fps is close enough to see what it’s going to look like, where 49.32fps looks artificially smooth to eyeballs used to slower video refresh.

Blur: The February video is over-sharp when it passes through the comet, while this April video is too blurry. I’ve already cleaned this up substantially since I uploaded the video a couple of days ago, so no worries on that score.

Aliasing: The February edition of DS9 is heavily aliased in certain places, like when the runabouts approach the logo in the first part of the credits. The new version shows far less aliasing. With that said, there are certain shots, like the Defiant’s final approach into the wormhole, where the ship is badly aliased no matter what I do. Haven’t figured a solution for that yet.

Shimmer: The February video makes it look as if DS9 (the station, not the show) is crawling with ants in certain places. The reason the April video looks a bit blurred is that my efforts to lock that problem down were a little too enthusiastically applied. Watch the station pans to see the difference here.

Judder: My new, enthusiastic, and least-wanted best friend has been substantially improved, but you can see some clear instances of judder in the April video, including when the runabouts pass in front of the camera and the Defiant passes the outer docking pylon. I’ve already got an encode that improves sharpness and the Defiant’s passage while yanking the frame rate down to 23.976fps rather than this dorky 24.66 faux-hybrid thing.

Overall, you can see the rate of progression from February to April in these two videos. February represents the beginning of my work, when I was just running an application through TVEAI to upscale. The April video represents the application of additional filters.

What Comes Next?

I figure it’s time I lay out a formal roadmap for where I’m going with this thing.

Video Quality: After three months of working on DS9 and literally hundreds of video encodes, I am hopeful that I’m nearing the end of the road, as far as video work. Judder is the last major problem to solve, to the extent that judder can be “solved” in the first place. I have attempted some rather … whacky experimentation to solve this problem. I launched 17 different encodes of the MakeMKV source file last night. The first 12 had serious names that reflected the settings I picked for them. #13 – #17 were named “F***It.mkv.” I’ll go downstairs later on and check the results before repairing to AviSynth documentation to attempt more testing.

I’d like to know more about antialiasing scripts than I do (most of them reference anime, not video content) and I need to run some tests to confirm that DAA is actually still getting me a visual improvement. I’d love to find a better AA solution to clean up a few niggling issues.

Color Reprocessing: I have decided to separate out any work I do on DS9’s color from the work I’m doing on the video. I haven’t dropped this aspect of the project, but I’m not picking it back up until I have the visual processing nailed down. Not everyone is going to want color tweaking in the first place, after all. Users should note that Topaz Video Enhance AI introduces some color shifting when it upscales. In the initial version of the application, this applied a subtle lightening effect in HQ-CG mode, as shown below:

It’s subtle, but you can see that the image on the right is slightly lighter than the one on the left.

Topaz recently updated the application to a new version that now appears to have a different impact on color, but I wanted to show the original. I’m keeping both the original and newer version of the application handy to cross-compare video output.

Audio: HELP WANTED: INQUIRE WITHIN.

More seriously, I expect to nail down the audio issues, too. I just can’t remember how I solved the problem last time.

Tools Used: Right now, MakeMKV, StaxRip 2.0.8 with AviSynth+, and Topaz Video Enhance AI itself. The front-end is a useful way to code AviSynth scripts. The goal is to create this project with as much free software as possible. Still evaluating a run through Handbrake, if only to try syncing the audio/video that way.

Once I’ve hammered the workflow into shape, I’m going to publish a step-by-step tutorial on how to do this on ExtremeTech. Honestly, it doesn’t take that much in terms of filters — well, not yet. I haven’t shared my specific settings yet because I’m still trying to figure out which variables and filters are the right ones to use. I don’t want to publish pieces of a guide that then become separated from the final product.

Finally: A great many people have reached out to me following the publication of my second article with encouragement, well wishes, and technical assistance. A longer list of thank-yous will accompany the final version of my article, once I’ve checked in with people and figured out how they want to be acknowledged. The help I have received from some of you has been vital to pulling off this project. Anyone with ideas on how to improve image quality or solve the judder problem is welcome to get in touch.

Once Deep Space Nine is complete, I’m moving on to Buffy the Vampire Slayer, another beloved show with an execrable 16:9 “conversion” that very few people seem to like. I’ll be using the 4:3 version of the show to see what kind of improvements I can bring to the Scooby Gang. We’ll also check in with the Blu-ray version of Firefly to see how Topaz Video Enhance AI handles upscaling already-HD content.

Now Read:

 Deep Space Nine Upscale Project Update: ‘Sacrifice of Angels’

Upscaling Star Trek: Deep Space Nine Using Topaz Video Enhance AI

Star Trek: Voyager Gets 4K Upscale Remaster via AI

from ExtremeTechExtremeTech https://www.extremetech.com/extreme/309653-deep-space-nine-upscale-project-update-variable-frame-rate-dvds-can-burn-in-hell from Blogger http://componentplanet.blogspot.com/2020/04/deep-space-nine-upscale-project-update.html

[componentplanet]

Star Trek: Voyager Gets 4K Upscale Remaster via AI

End users are waking up to the idea that AI-based products can demonstrate practical use cases for ordinary people. About a month ago, I covered my early work upscaling Deep Space 9 for the modern era — and I’m not the only one working on this sort of project. I was inspired by the work of CaptRobau back in 2019, and now a new individual has launched a project of his own, this time to remaster Star Trek: Voyager.

Reddit user theboomsterz remastered the TV show using Topaz Gigapixel AI initially before switching to Topaz Video Enhance AI. He’s published a couple of clips to show the net effect, and it’s interesting to see them now that I’ve been working on DS9 for the past few months. One bit that theboomsterz states in his Reddit thread is that he’s run some of these episodes through the application more than once. I admit, I haven’t tried that, but there is a theoretical method for doing so — Topaz VEAI has recently added the ability to run a video file through without upscaling it, in order to remove noise and other visual artifacts. I haven’t tested the impact of running it 3-4x in a row, though.

Here’s a second clip. This clip implies that Voyager suffers from the same problem Deep Space 9 does — namely, it’s quite dark. I’ve been collaborating with a few interested readers who wanted to contribute to the Deep Space 9 project by looking for solutions to this issue.

It’s great to see more fans taking up this kind of work. ExtremeTech, again, does not condone piracy or working from downloaded torrents — if you want to perform this kind of upscaling work, you should work from the original DVD source — if only because, believe me, you’re going to want every single bit of detail you can scrape out of these DVDs. It doesn’t amount to nearly as much as you might hope.

The big question, of course, is whether these various fan efforts have a chance in hell of persuading Paramount to actually do the job itself. The answer to that is probably no. I really hate to type that, because Paramount is capable of doing some absolutely stunning work. Here’s a clip from the DS9 episode Sacrifice of Angels that Paramount remastered for the What We Left Behind DS9 documentary.

Nothing anyone is going to be able to do with source footage is going to approach this kind of fidelity, and each series of ST brings it own challenges. Voyager was apparently filmed in variable frame rate, which theboomsterz discusses as a challenging issue to deal with. I’ve been grappling with some issues myself, but I’ll drop an update on my own progress into a separate story.

Feature image from the first upscaled video clip posted above. Video by theboomsterz.

Now Read:

Upscaling Star Trek: Deep Space Nine Using Topaz Video Enhance AI

Modders Are Using AI to Overhaul Old Games Textures, With Gorgeous Results

Fan Works to Remaster Star Trek: Deep Space 9 in 1080p HD, Using AI

from ExtremeTechExtremeTech https://www.extremetech.com/extreme/308434-star-trek-voyager-gets-4k-upscale-remaster-via-ai from Blogger http://componentplanet.blogspot.com/2020/03/star-trek-voyager-gets-4k-upscale.html