Assessing the Video Quality of Tesla’s Camera System and Streaming Services
Tesla is a well-known company in the IT industry, thanks to its use of a screen to control most vehicle functionalities, giving you the sense of dealing with a giant tablet instead of the usual knobs and buttons found in other cars. This approach is commonly seen in most high-end tech, where functionality is added or removed via software updates instead of hardware changes. Following the tech sector trend to minimize hardware, Tesla is able to standardize, develop, improve and, of course, scale production of their product much more efficiently. At TestDevLab, we like to stay on top of all the latest happenings in the IT industry, and are dedicated to raising industry standards to the highest level of quality. With that in mind, Tesla is a great example of software development in a sector that has been slow to adopt features such as games, video calls, and streaming services, which have been available on other devices, like smartphones and tablets, for several years already. With Tesla planning to introduce video calls (via Zoom) in their Infotainment system, we want to assess the quality of this new feature in their base models. If you want to know more about Zoom quality for video calls, check out the Zoom Video SDK Performance Report, a comparative analysis conducted by TestDevLab. Tesla hasn’t yet released Zoom calls in their cars in Europe as we are writing this article, but we can still test the performance and assess the quality of their system by checking other features of the vehicle that relate directly to their camera system, hardware, and software. To do so, we performed testing on the 2023 Tesla Model Y Long Range, which is expected to be the highest-selling car this year.
Camera system quality
The first step in our assessment of the video quality in this particular Tesla model was to evaluate the camera system quality. To do this, we first measured the maximum frames per second that can be captured by the vehicle’s camera system. We used special media and software developed at TestDevLab within the Audio & Video department to do this. Our software doesn’t require rooting the device or modifying the software or hardware of the Tesla vehicle in any way to get real and accurate measurements that the user would see on the Tesla screen itself.
From the graph above we can see that the inside camera (the one that will be used for Zoom calls) has a maximum frame rate of around ~28 FPS, which means that the software/hardware is probably capped to capture a maximum of 30 FPS. Normal video calls using Zoom on computers deliver around 28-30 FPS on good network conditions, which means that Tesla vehicles should deliver a fairly good video call experience on the temporal quality side.But what is the maximum picture quality that it can deliver? We used VQTDL to measure the picture quality and we obtained a 4.1 MOS from a scale of 1-5, which is a very good video quality score, compared to regular webcams from laptops (normally video calls have around 4-4.5 MOS in unlimited network conditions).
The outside cameras also capture video at around 30 FPS, but it is quite remarkable that they can do this while rendering three video streams at the same time (side cameras + back camera). This means that they render more than 90 FPS while doing plenty of other things simultaneously in the background. Very good performance!
Also, it is fair to say that we lack information about the specifications of the display on this particular Model Y, which means that the output video might be limited by the screen hardware (and not the cameras), the vehicle processor, or the software handling those inputs/outputs.
Streaming services on screen
Our next objective was to determine the highest frame rate that can be displayed by the vehicle. To accomplish this, we used the built-in YouTube streaming service. We recorded the screen with an external camera with a high refresh rate, and analyzed the recordings. To establish a baseline, we did the same thing on a MacBook Pro. Here we got the following results.
From Graph 2, we can see that the MacBook maintains a consistent frame rate of almost 60 FPS, where the drops fall within the error bar of the measuring system. On the other hand, the Tesla has a stable average frame rate of 30 FPS. Again, we see that the maximum amount of frames is 30, which could mean that the screen itself is limited to this frame rate. This also means that playing videos with high activity, such as sports and action films, will feel a bit worse than it would on a different device, but they will still work fairly well.
When played in slow motion, we can see that the refresh rate of the Tesla screen is a bit uneven in the way they display the frames, being slow at some points and fast at others. This might be because of aliasing while trying to display a 60Hz video on a 30Hz output. Meanwhile, on the MacBook screen, the frames are displayed at a constant rate, providing a smoother and more natural user experience.
Sentry Mode performance on Tesla app
Sentry Mode is an interesting and unique feature that Tesla offers its customers and sets it apart from other car manufacturers. This feature not only allows you to record the surroundings and interior of your vehicle while parked, but it also provides you with a live camera feed streamed directly to the Tesla app on your phone. Think of it like your very own surveillance system. You can monitor your vehicle from anywhere. However, though this is an excellent feature, what is the quality of the media that is sent to your phone and, more importantly, how much data is sent and how does it behave under different network conditions? To answer all of these questions, we tested the performance of the Sentry Mode feature on the Tesla app.
First, we divided the video quality into two main categories—temporal and spatial. The former is determined by the frames per second and stalls, while the latter is determined by the picture quality and for this we used VQTDL again.We found that the maximum frames per second received is ~5, which is quite low—firmly under 15 FPS, the average frame count for general surveillance devices. Nonetheless, it is enough to see what is happening. But here we found another problem. Specifically, the picture quality was not up to standard, scoring below 3 MOS, while the camera has the capability to deliver a 4.1 MOS. In addition, the video quality can drop even lower if the video shows someone or something moving fast. This is most likely due to video compression being too high.
But let’s look at the network consumption before making any assumptions. We measured the network by capturing packets on the receiver's end (the mobile phone) and we filtered the packets that were sent to the app from the live feed just by checking the protocols and the IP from the sender’s side.We found that the bitrate is around 100KB/s which is quite low. We also noticed that both the app and the vehicle do not check if the network is good or bad before sending data. Therefore, they always send the same amount of information, which means that if the bandwidth goes down, below those 100KB/s, then it will start losing frames. But this is an unlikely situation, as the bandwidth needed is very low. The problem arises when the limitation is packet loss or jitter. In these situations, the live feed starts behaving poorly very soon. The live feed from the app doesn’t work in the same way as it does during a normal video call, where it duplicates the packets to solve this problem. Namely, whenever there is more than 5-10% of packet loss, the feed starts having problems.
The results are in: How do we rate the quality of Tesla’s camera system and streaming services?
So far the video quality of the system is good for the purpose that it is intended to serve. And it is quite impressive considering that there is almost nothing alike in the vehicle market, unless you install something extra to your own car. However, adding aftermarket parts usually leads to a worse user experience and eventually you end up not using it at all as it is often not integrated as good as an OEM part.There are some unknowns about the potential of this system for software updates that could improve its performance. However, it’s also possible that the hardware—the screen itself—may be a limitation. Namely, if the screen is not limited to 30Hz, the output from the streaming services could be upgraded to 60Hz via a software update. Looking at the processor, it seems that the current system with an AMD Ryzen and 8GB of RAM (for the Tesla Model Y/3) is good enough to give this improved performance with over-the-air (OTA) updates and optimizations. This means that the user experience might get better within the time of ownership.Nevertheless, we are still waiting for Zoom to be released by Tesla in Europe so that we can test the full capabilities of audio and video quality using different network conditions that one might find in crowded areas, like EV charging stations, for example. This way, we can have a full understanding of how reliable it is to have an important meeting in your Tesla vehicle. Stay tuned for further updates!
Want to test the audio and/or video quality of your software product? We can help. Contact us with your project details and we’ll get in touch.