Biggest Bottlenecks in Video Data Processing

In video quality testing there are several different roles. A manual tester is responsible for video filming, a video data processor is in charge of video data processing, and a data validator takes care of data validation. Each one of these roles are really important, but in this blog post we will focus on video data processing. Specifically, the daily struggles experienced by data processing team members, how various mistakes/errors can be avoided, and how to make things easier for all who are involved in the project. Additionally, we will look at video data processing metrics and—most importantly—discuss the biggest bottlenecks in video data processing.

What is video data processing?

Video data processing refers to the process of converting data about video quality into meaningful information. It involves collecting, recording, organizing, storing, and adapting or altering data into useful information. But why is video quality—and as a result video data—important?

Nowadays, videos of all kinds are becoming more and more popular and video consumption is on the rise. According to data from Statista, over three billion internet users were watching streaming or downloaded video at least once a month in 2022 with the number predicted to rise in 2023. And chances are you come across some sort of video online—be it an ad, music video, tutorial, stream, product review, or any kind of video, really—at least once a day. 

Furthermore, more people are using video not just for entertainment, but for work purposes, too. In fact, 58% of companies use video conferencing in their day-to-day operations—and when important meetings and decisions are being made during these calls, video latency and bad video quality is the last thing people want to experience or worry about. This is where video data processing comes in. With video data processing, we are able to gather and deliver video quality data to stakeholders so that they can make improvements in their application and software. In turn, they can use this data to provide a more reliable product and user experience to their customers.

You might be interested in: Analyzing Video Latency Data (With Example)

What does video data processing look like?

We conduct video data processing in our dedicated audio and video quality testing laboratories. Basically, the process—from start to finish—looks like this:

Step 1: Download all necessary videos to a remote computer with a Linux operating system (because processing uses a lot of computer resources).

Step 2: Organize all videos in folders with specific names.

Step 3: Arrange/adjust processing scripts. 

Step 4: Begin processing all videos.

Step 5: Check results to find if all scripts read data correctly.

Step 6: Compare results with the video itself.

Step 7: If everything is good, download all the result files. If not, then recheck scripts where problems may have occurred and re-run processing.

Step 8: Merge results into files for faster copying.

Step 9: Copy data into sheets to get average values and graphs. 

Step 10: Submit results sheet for validation.

Step 11: If the data validation team finds some mismatches in data, check the scripts and repeat all the steps starting from step 3.

What metrics do we collect? 

When processing videos we collect different video quality metrics, such as FPS, BRISQUE, VMAF, SSIM, PSNR, VQTDL, video delay, and freezes. You can read about our full-reference quality metrics in this blog post. 

Biggest bottlenecks in video data processing 

While processing video data we may encounter many different situations where we need to make minor adjustments in the scripts or ask for new additional tests for retesting. However, in some cases it is not possible to check video data correctly, even when scripts are adjusted. Here are some of the bottlenecks we can encounter when processing video data:

The order in which the data is processed   

When processing tests, we need to remember the full-reference quality metrics (VMAF, SSIM, PSNR) as well as the VQTDL data we get from original videos, without cropping or changing the video in any other way. That’s why we usually process full-reference and VQTDL metrics first. In some cases, we can process FPS and freeze data without cropping the video, but this isn’t guaranteed and is more situational. After we process this data, if cropping is needed, we crop out necessary zones from the video by cutting out all of the web/app UI and proceed with gathering FPS and freeze data (if we couldn’t get this data without cropping). Then we process BRISQUE data and in the end BRISQUE/VQTDL filtering. If we didn’t collect full-reference and VQTDL data before cropping the video, that means we need to download the original video and start over again.  

Video cropping 

To crop videos, we usually use one of two methods. The first is with manual zone selection, while the second one automatically finds AruCo markers (see picture below) and crops out the video for further processing. You may be wondering why video cropping is considered a bottleneck, the answer is simple. We can recheck the order in which the data is being processed and if something from previous processes shows some errors, we need to get the original video and start over again because there are no possibilities to uncrop video. With AruCo cropper it’s simple, just run the process and it will automatically select the video zone. Keep in mind, however, that it’s mainly used for desktop (landscape) video tests. With a manual cropper, it’s not harder, it just takes more time. For example, imagine if you have 30 videos and for each video you need to manually select a zone so that there are AruCo markers, and if mistakes have been made, sometimes you need to download the original video and start over again. This process would take significantly longer than if you were to use the AruCo cropper from the start.

Brightness 

Due to factors like app behavior, some videos may appear too dark or too bright. While processing video data, it is important to be able to clearly see AruCo color markers. For example, if the blue color marker is too dark, the script won’t be able to read it correctly and may assume it is black or not read color at all, which will result in incorrect video delay data. This is why we have an option to change video brightness to brighter or darker, but it can take some time before we can get the correct brightness levels while manually changing settings. To save time and accelerate the process, the best scenario is for the video to be filmed with correct brightness levels so that no additional changes need to be applied. 

Let’s take a look at a video delay screenshot below. On the left, we can see the sender video and on the right, the receiver video. The brightness levels are good, so the color markers can be read correctly. 

BRISQUE scale parameters  

Another bottleneck that may affect the final results in video data processing are the BRISQUE scale parameters used. When running tests on mobile devices, portrait videos are usually used, while for tests on desktops, landscape videos are used. Additionally, video resolutions may vary, so the BRISQUE testing zone or ROI (region of interest) may change as a result. Therefore, we need to make sure that the ROI is correctly centered on the object in the video. After processing the video we can check if the selected ROI has been detected correctly for each test. If the test ROI is not centered on the object, we can adjust it manually. Sometimes, by default, we get unrealistic BRISQUE scores, so we have to adjust BRISQUE settings to get them more precise. 

Thankfully, TestDevLab has developed its own no-reference image quality metric—VQTDL—based on a neural network. It is specially made for video conferencing image quality evaluation and has been found to be much more stable and accurate than BRISQUE, saving time on processing and validation. For this reason, we are slowly introducing VQTDL in our projects.

Low video quality data is less accurate

This bottleneck is the hardest to handle, as lower quality video equals less accurate data and thus it takes more time to recheck all data, specifically to see if values haven’t disappeared and are objectively calculated. To save time in these instances, it’s better to copy original videos in another folder, in case you need to redo some tests again.

Video zoom in/ zoom out 

When filming test videos, there are times when AruCo color markers zoom in and out. Sometimes it’s hard to notice these zooms on the video itself, but in processed data there usually are some unnatural drops or spikes in graphs. Of course, we try to fix this issue depending on how much AruCo markers zoom in or out. For FPS it’s an easier fix but in video quality metrics it’s much harder to find a suitable solution. However, it all depends on the severity of zooming. In situations where the zooming is too intense, we need to ask for additional tests, so that we can save time and gather useful results. 

Disconnects 

In stream videos, there is a possibility that the receiver video disconnects from a stream by leaving a “black screen” where we don’t have any data about the receiver video and all necessary AruCo color markers. These kinds of disconnects need to be filtered out. However, it does not end with just using a filter—we need to recheck all videos manually to see if the data has been read/filtered correctly. Later, when gathering all test data, these disconnects are seen in final graphs.

Data copying 

After processing is done, we need to collect all the video data and copy it into spreadsheets for further analysis and data storing. For that, we have created some templates to save time. It’s also important to stay alert when copying data so that you don’t copy it into the wrong spreadsheet or test conditions, or copy only half of the data. It’s better to double check what kind of data you copy and where you copy it to, otherwise it can start a chain reaction of problems. The best solution for escaping human errors is to automate this process with scripts, so it automatically takes what’s needed and copies it to spreadsheets. Nevertheless, it all depends on the client’s test requirements, so we need to consider if it’s worth automating, as it may take too much time to automate such a process.

The bottom line

At the end of the day, everything comes down to time. Video data processing takes a lot of time, expecially if there are many video tests to process. From downloading videos to copying all gathered data into sheets so that we can send these data to validation, all these things require time. When performing video data processing, our goal is to get as many data metrics as possible (unless asked differently) and some metric values usually take more time to collect than others. So if you haven’t noticed any errors while processing videos, you will probably have to redo all the processing-related tasks over again. 

Here are the main things to take into consideration in order to streamline video data processing: 

• If you aren’t sure if there are mistakes/errors, ask your team lead or buddy for help/advice. 
• Don’t process all metrics unless specified.
• Always update your scripts when an update is available. 
• Stick to the same processing order to avoid mistakes. 
• Copy original videos into a new folder so that you don’t have to download them again. 
• Check all processed data before submitting for validation. 
• Before processing, always check video for unwanted popups, black screens, and other potential bottlenecks.

Do you have an application that depends on delivering high video quality? We can help you gather and process important video data. Get in touch with us to learn more about how we can help you.