Frame interpolation is something that is found in the most advanced televisions of the moment, but the implementation of dedicated hardware for artificial intelligence together with the implementation of cloud computing is going to make this technique gain a crucial importance in systems of all. But how does it work and what does it consist of?
Frame interpolation is known as the ability of software and / or hardware to generate intermediate frames from the information of two or more previously rendered frames.
Depending on whether we are rendering an image in 2D or in 3D, the interpolation of frames is done in a very different way, the most common being the interpolation of images in 2D.
How Frame Interpolation Works
To perform the interpolation we need two final frames and therefore of which we have the final information.
In the case that we are talking about interpolating a video streaming we have two different methods:
- The interpolation is performed by advanced artificial intelligence systems that have been trained to detect the different objects in the same image and classify them. This is highly used in smart driving systems.
- Another method is to record 2D images using an infrared camera or one in time of flight, said camera is accompanied by a light emitter, the combination of both allows to have a depth map of each object, which helps to separate the different elements of the scene.
But in video games rendered with a real-time 3D engine (because a depth map is necessary to know which objects have preference when drawing) there is the possibility of creating a data map with the identification position of each object
The next step for the hardware is to create a velocity buffer. This is built taking into account the initial and final position of each previously identified object between the two frames, in this way the system can create an intermediate frame with all the elements that have been moved from one frame to another.
Applications and utility of interpolating frames
The clearest utility is in adding additional frames in videos with a low rate of frames per second, especially it is being used to restore old movies that were recorded at a very low speed or frames of them have been lost due to the deterioration of the passage of time .
But in terms of games, this one has a number of quite interesting applications. For example, when it comes to streaming games: whether we are streaming our games to others over the internet or playing on a game service, the ability to add intermediate frames helps to create a smooth feeling to the playable experience. and avoids the feeling that images are reaching the player too late.
The reason why this idea is going to end up being implemented in hardware is because the units in charge of encoding and decoding video are of a very similar nature to that used to accelerate artificial intelligence algorithms, which is why hardware designers it does not cost them an effort to implement it in their products.
It must be taken into account that the concept does not consist in that our games are rendered at twice the number of frames per second, but rather that their application is for internet streaming applications, and the power to transmit video content between devices in a wireless on a local network.