Video Quality Testing by TestDevLab

Video quality testing

Whether it is VoIP, WebRTC video call, or a media broadcast service, you should know how the media quality of your solution changes under different network conditions and device configurations. Such videos usually are tested manually and checked by simply looking at them. Our solution for media testing automatically deconstructs video into frames and evaluates video quality based on multiple objective characteristics. You will get an insight of the product’s state as well as how it compares to competition.

GradientVideo quality testing
Video quality testing setup

Setup

We have created a dedicated video testing laboratory that includes:

  • Dedicated treated room with controlled environment for tests with constant brightness

  • Custom network routers that allow us to change various network configurations ( bandwidth limitation, packet loss, delay and all combinations of those parameters to simulate network across the Globe)

  • Tools to capture video either from desktop or mobile devices

  • Specific software and hardware setup for post-processing and analysing the results

Network conditioning

We can emulate various network conditions to create real-world network scenarios like low bandwidth, high latency, or jitter.

Jitter

Jitter is defined as a variation in the delay of received packets. Due to network congestion, improper queuing or configuration errors, a steady stream can become lumpy or the delay between each packet can vary instead of remaining constant. We can set custom Jitter configurations.

Packet loss

Packet loss occurs when one or more packets of data travelling across a computer network fail to reach their destination. Packet loss is either caused by errors in data transmission, typically across wireless networks, or network congestion. We can set any packet-loss scenarios.

Network conditions / speed

We can set any network bandwidth — the rate of data transfer, ranging from very low network bandwidth to almost unlimited network bandwidth.

Network switches/changes

Within our laboratory we can emulate network interruptions, changes between 3G, 4G, and Wi-Fi, re-connection scenarios, and other real-world scenarios, so application developers can better understand how their product behaves

Reports

Using various objective video quality metrics and representing them in understandable reports, we can help application developers understand the quality of their video product, how it compares to competitors, the changes in its quality under different network conditions and other use cases.

BRISQUE

Time (s)

Test 1

Test 2

Test 3

BRISQUE

BRISQUE is Blind/Referenceless Image Spatial Quality Evaluator, which helps us determine video quality score in range from 0-5

Milliseconds (ms)

Time (s)

Test 1

Test 2

Test 3

Audio & Video delay

Audio video delay shows real Audio / Video sync delay represented in seconds

Frames per second

Time (s)

Test 1

Test 2

Test 3

Frames per second

By embedding specific markers within each frame of the video, we can calculate real FPS metric

Resolution (p)

Time (s)

Test 1

Test 2

Test 3

Resolution

By embedding specific resolution markers within video, we can calculate actual resolution and how much compression and network quality affects actual video quality.

VMAF

Time (s)

VMAF 1

VMAF 2

VMAF 3

Full reference video quality assessment

VMAF – a recent addition to the perceptual video quality assessment algorithm family, developed by Netflix, quickly becoming an industry standard. It incorporates several elementary metrics with different weighting.

Freeze duration (s)

Freeze start (s)

Test 1

Test 2

Test 3

Freeze & Stall

QoE metric. Happens when video stops but audio still carries on. Usually seen for short time amount of time, but still visible for the user.

We provide also custom-made video quality assessment metrics

Sometimes our clients require a specific video quality testing solution such as continuous video quality monitoring that is implemented within Client's CI (Continuous Integrations systems) or adding some custom quality markers. We're open to discuss and implement also such custom-made video quality assessment solutions.

GradientCustom Metrics