New Super Resolution AI – Enhance ~10x Faster!
❤️ Check out Weights & Biases and sign up for a free demo here:
📝 The paper "Deep Fourier-based Arbitrary-scale Super-resolution for Real-time Rendering" is available here:
📝 My paper on simulations that look almost like reality is available for free here:
Or this is the orig. Nature Physics link with clickable citations:
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Gaston Ingaramo, Gordon Child, John Le, Juan Benet, Kyle Davis, Loyal Alchemist, Lukas Biewald, Martin, Michael Albrecht, Michael Tedder, Owen Skarpness, Richard Sundvall, Taras Bobrovytsky,, Thomas Krcmar, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here:
My research:
X/Twitter:
Thumbnail design: Felícia Zsolnai-Fehér –
No more blurry videos of UFOs and other strange phenomenon, finally!
Nop! u didnt understand concept of this. Upscalers has bias like us.
@@emircanerkulthe joke flew above your head just like a ufo
this made my morning
Promise, next year the graphic design trend will be that 270p look and will be hard as hell to accomplish (well maybe not with ai but y’all get it) we scale up to scale back🔄
what a time to be alive!
We re just away about 2 more papers
This is all amazing, but honestly, the gaming industry would’ve been in much better shape if no AI upscalers were ever created. There are still optimised games, but nowadays instead of making the game run at 60+ fps by itself a lot of game devs just target at 60 fps with upscalers, or in other words you’d at best have 30 fps native. Really sad to see that as the norm now
You don’t have a clue how significant is AI upscaling.
That’s rose tinted glasses doing the talking. There have always been poorly optimized games. Consoles have been running below ‘native’ (or maybe call it target) resolution since long before DLSS and other AI upscalers became a thing.
For example, the PS3 was marketed as a 1080p console, but games mostly ran at 720p or even 576p to hit 30fps. Except it didn’t have a good upscaler, so you had to hope your TV was up to the task of getting a decent image. Same for the xbox 360. The next gen of consoles also often ran below 1080p, though usually 900p, 792p or 720p. I think that console generation is also when games started using dynamic resolution.
AI upscaling itself is not the culprit for poor optimization. AI upscsling is the scapegoat for budgets that can’t hire programmers, Epic’s lack of care for products other than Fortnite, and the structural dependency of developers on pre made assets (especially the UE marketplace). Also the RDNA2 GPUs on consoles are really weak compared to Nvidia GPUs, no amount of optimization will turn 10 teraflops into a PC level GPU. AMD really are falling behind Nvidia
True. I’ve been playing Marvel Rivals recently, the frame interpolation is a mess, granted my computer is not that good anymore, but everything looks fuzzy for me
@@Wobbothe3rd while it is obvious that pc gpus are much faster, 10 teraflops is plenty enough to make modern games run and look much better than pastgen games. And yes, AI upscalers are not the culprit, but they are the tool that allows gamedevs to disregard optimisations. Just look at GTA 5 – it came out on PS3 and XBOX 360 so it was limited to 512 mb shared ram/vram and 0.2 TFLOPS gpu, yet it featured massive open world with dynamic lighting etc. And nowadays there are games that require 10-20 times more processing power just to look the same or even worse.
Even with previous console upscalers developers had to implement them and still think about the limitations of the hardware, but now there’s this magical button with a label “make the game look nice” with lumen, raytracing, nanite etc behind it, and another button with a label “make it run smooth” with dlss, framegen etc behind it. AI upscalers are not the reason, they just allow the devs to take shortcuts for which the player pays (both literally and figuratively)
We’re still a long way off from replacing high-resolution textures with super resolution ML. The method shown here might be better than all before it, but the textures are the most obvious weakness. It looks a lot more ‘anime’ compared to the original.
Also this lacks motion vectors, it’s guessing the optical flow in motion. Obviously, that makes the model MORE impressive though!
There could be different upscalers per game. Specifically trained with only that game.
For example, GTA6 train their model with 8k and 720p versions and map those data together. In consumer GPUs they only need to run at 720p but upscaler upscales 8k with perfect clarity and correctness without loosing any details. This way GTA6 can achieve 8k resolution with just 1060TI gpu (which mostly used for physics calculations)
DLSS 1.0 was trained per game. It was a shitshow and nvidia changed tracks to a general upscaler model as soon as they could. I suppose a big studio with a inhouse game engine could do their own thing, but I think you’re underestimating how much effort that’d take a studio, especially since they’d have to start by hiring a dev team from scratch. Plus the expense of building your own datacenter or renting the GPU hours from Microsoft or Amazon.
Everything over 3 ms is way too long for a video game.
Wow, it even knows where to place warning signs on the wall 6:29 🙂
Yes it’s a bit weird. Look closely at 0:06, the signs are there but only appear when the camera gets closer to the wall, due to the low resolution.
I believe these models are trained specificaly on particular games, hence they know where what should be. Tbh it’s a good idea for handhelds for example. You could render 360P and neural upscale them to FHD/2k without strain.
I’m watching on 144p and I’m impressed
If I understood this correctly this is not going to work on 2D Images, but only for something that is rendered on a GPU because it needs the G-Buffer?
Why would computational resource be used to upscale when you can use the computational resource to just render at a higher quality. I never understood this frame generation stuff
When you render at a high (native) resolution, every single pixel’s lighting, geometry, and textures must be calculated independently—this adds up to a huge workload. Upscalers like NVIDIA DLSS avoid that by rendering at a lower resolution and then using a specialized neural network to fill in the missing detail. Because the network’s job is simply to predict extra pixels rather than compute them from scratch, it requires far fewer computations and thus uses much less GPU power—yet still delivers a final image that looks nearly identical (and in some cases even better) than native rendering.
Frame generation is not the same as upscaling, but is used along side it usually. Frame generation uses motion vectors and frame data to predict a totally new frame in between frame your GPU has already rendered. It is called frame interpolation and is pretty solid when implemented properly. You don’t have to use upscaling to use frame gen. You can render at native resolution and then use a very resource efficient neural network to predict new frames, so you get higher FPS without your GPU having to do much more work at all.
Upscaling also leads to more FPS, but only because you are rendering at lower resolution and then upscaling to use less GPU resources. Frame gen literally creates more FPS.
Because rendering at higher quality would take 10x the resources of upscale
Looks terrible in fast motion though??? Literally all videogames and most videos have a lot of motion
the thing is to get these insane upscales (like the sub-1 pixel grass) you have to train the NN directly on the scenes in question. Otherwise it would have no idea what those single pixels are supposed to be. This is the CSI “enhance” meme – it’s physically impossible without the NN knowing the ground truth for that specific case. You know what that means? Game devs would have to spend a bunch of $$$ and time training the upscaler on every little nook and cranny of their game, and every time they update the game they would have to re-train it!
Funnily enough, CSI came to my mind as well.
1:40 you are contradicting yourself… How can a cheapo GPU be able to process this on the fly…
Sounds like that’s just your presumptions.
This was a 7 minute bonus episode of two minute papers.
From the looks of the second scene, the technique seems to be creating extra detail based on external data. The warning sign on the second column started appearing (probably duet to LOD) in the LR example and the Ours example had it visible from the get go. The third column did not have the sign present in the LR at all, however Ours version has it. The sign is either repeated due to the sign on the first column, or it’s existence on the further columns is inferred from external data.
Don’t get me wrong, I believe that an upscaler that uses the uncompressed game assets to perform more accurate upscaling is a great idea, however it doesn’t seem to be a pure upscaler.
The biggest problem problem with those methods, are image stabilization between frames and movement. That is what DLSS and FSR still struggles today.
enhance resolution: CSI is no longer a joke
CSI was ahead of its time
This looks horrible sorry. But it’s impossible for AI enhancers to produce the real 100% upscaled look from 25%. The bushes were flickering and the text warps over time. Hard pass.
Wow! From 270p to stunning visuals? This super resolution tech could totally change gaming. How real-time ready is it?
I’ve been seeing some videos of Runway Gen 3 video-to-video turning video game gameplay and turning it into real life looking videos. Now those take about a minute to render but a video like this makes me feel like it might not be too far away from being real time