Hi, I recently bought a DJI Osmo handle and a
Zenmuse X5R camera.
I have been facing two main issues with this setup. The first is that in the proxy files
audio and video are not synchronized. The second is a random gimbal drift.
I believe these problems are unacceptable for this price range.
The DJI Osmo handle is a control unit optimized for ground use that can mount a gimbal plus
camera system, which could also fit an Inspire drone.
The Zenmuse X5R is a camera that can record 4K raw videos. The camera has a micro four
thirds sensor with MFT mount, and is also mounted on a 3-axis gimbal.
I’m using the default DJI MFT 15mm F 1.7 lens.
The X5R camera records RAW videos in 4K DCI on a SSD hard drive. Videos are saved as a
sequence of Cinema DNG files.
Simultaneously to raw recording, proxy files are also saved into a micro SD card. Apparently,
audio can only be recorded in the proxy files.
Here is the audio input.
For recording the audio I am using the OSMO DJI FM-15 Flexi Microphone.
According to this website the fastest micro SD on the market at the moment (February 2018)
is the SanDisk Extreme PRO V30.
I got this SD card from Amazon for 43 euros. The card is indicated for 4K video recording.
It’s rated for 90 MB/s sequential write speed.
The Osmo system can be wireless controlled from an Android or iOS. device. It can also
be controlled with a cable for a faster video feed, although I noticed some delay.
This is the control app. The Osmo is running the most recent Firmware, version 1.4.0.70,
which was released on the 9th of May 2017.
We first calibrate the Osmo. Calibration was repeated every time a battery was replaced.
We format the SSD and the SD cards.
We set recording to 4K DCI and 23.976 fps.
As I said before, many Osmo users have reported issues with audio or video not being synchronized.
What I am trying to figure out with DJI is if my Osmo system is faulty or if this issue
affects any Osmos.
Let me briefly illustrate my setup.
On the right we have a Canon C200, where I record a reference video + audio with a VideoMicPro.
The framerate is set to 23.976.
Which is the same frame rate of the Osmo.
I also record audio separately with a Zoom H1.
The subject of the videos is a metronome app.
A metronome is a device that produces an audible beat at regular intervals. These devices include
synchronized visual motion.
Basically this app will blink and sound at the same time.
I also use a slate or clapperboard app. A slate is a device used to assist in synchronizing
video and audio in video production.
So, let’s start the tests. I start recording with the three devices, the C200, the zoom
H1, and the Osmo. Then I clap the slate to synchronize all videos.
The X5R raw files were exported with the DJI Camera Exporter running on a Windows 10 system.
This is the recorded data. We have a folder containing the raw X5R data from the SSD card,
a folder with the X5R proxies from the SD card, a folder with the C200 raw videos, a
folder with the C200 proxy videos, and a folder with the audio recorded with the Zoom H1.
We import the X5R proxy and our reference, the C200 proxy, in Premiere Pro. Both sequences
have the same frame rate.
We create a new sequence from the C200 proxy.
Then we import the X5R proxy into the sequence.
We try to synchronize the two videos with the automatic synchronization tool in Premiere
pro, but it fails.
This means that we need to synchronize manually using the most precise option, that is the
audio signal.
We use the audio waveforms and we synchronize with respect to the clap.
These are the aligned clips.
While the audio is well aligned, the video is not aligned. We can see this for example
when the slate closes at the beginning.
In the X5R proxy file the slate closes earlier than in the C200 file. This happens because
there is a delay between audio and video in the X5R proxy.
As the audio is later than the video, after we align the C200 and X5R footages, the video
of the X5R will be anticipated with respect to the video in the C200.
We can estimate the delay by looking at the clock in the slate.
The delay at the beginning is seems to be about 150ms.
We can quantify better if there are drifts in the audio/video in MATLAB.
As I mentioned before there is a drift in the gimbal position. Obviously, that drift
should not be there, otherwise what is the point of having a gimbal?
We need to fix this before doing quantitative analysis.
We open the sequence in After Effects and we apply Stabilize Motion, position and rotation.
For this we use two trackers.
As, you can see there is a huge drift.
I save the tracking coordinates for these two tracking points in an Excel sheet.
This is a plot of the first tracker trajectory. The camera was at about 70cm from the target.
The drift in pixels along the x direction is about 100 pixels, the drift along the y
direction is about 60 pixels.
This is a plot of the second tracked trajectory, which obviously looks like the first tracker.
We go back to premiere pro, where we have the stabilized sequence.
We create a new sequence from the X5R proxy and export it as a sequence of jpeg images.
The duration of the video from Premiere was 3 minutes, 38 seconds and 10 frames, this
means that I should expect 5242 frames or jpeg files.
So in the folder we have 5242 frames.
The original audio was recorded at 48KHz. We export the audio as a waveform file.
Let’s have a look at the audio first. We want to see if there is any irregularity in
the audio. We first check the audio recorded with the zoom H1 as an example.
We move to Matlab. Here we wrote a script for audio analysis.
We import the Zoom audio file. The sampling rate is 96KHz.
Audio was recorded as stereo. We extract one channel.
The time vector in seconds is obviously given by the sample number divided by the sampling
frequency.
We then consider a portion of this file and plot it. These peaks correspond to the beat
of the metronome. The two intermediate peaks correspond to the slate. In fact we clapped
twice.
We check the distance between two peaks to set the parameters for peak detection with
the ‘findpeaks’ function.
We extract the instants at which the beats occur.
This is the differential of the previous plot, which tells us that there is more or less
a constant gap between each beat. It also tells us that the gap does not really vary
over time, as shown by this tredline.
The bpm was 60, meaning that the beats are spaced 1 second.
The median is in fact 1 and the standard deviation is 70 micro seconds.
Next we check the audio we just extracted from the X5R proxy. Here the difference is
that the sampling rate is 48KHz.
We repeat what we did before.
We find a regular spacing between the beats. The average is 1 second, and the standard
deviation is higher than the one calculated before, 264 micro seconds.
Let’s have a look at the video sequence now.
First of all let’s make sure again we have the right number of frames. In the folder
we have 5242 frames. If we divide this by 24 frames per second we get 218.375 seconds.
From this we can work out the duration in minutes, seconds, and frame, which is correct.
This is the first frame.
We define 4 regions of interest (or ROIs) around the 4 circles that illuminate at each
beat. ROI1, 2, 3, 4.
We reduce each ROI from RGB to grayscale and plot these ROIs.
We define the time vector by dividing the sample number by the framerate.
We calculate the sum of the intensity within each ROI for all frames. The sum of the intensity
will spike at each beat.
We then plot the sums for each ROIs. Since each circle stays illuminated, we just need
to extract the moment in which the circle starts illuminating.
This corresponds to a positive peak in the differential of the sum signal, where we have
the maximum slope.
We plot the differential of the sums for each ROIs. Each peak is equally spaced of one second,
as expected for a metronome running at 60 bpm.
We locate the peaks and plot them in red.
The peak detection is almost perfect. We show this for the four ROIs.
We plot the detected peaks for the 4 ROIs. These correspond to the instants in which
the circles light up.
Then we merge these instants into a single vector and we sort its values.
This plot includes all beats for all circles. Each peak should be spaced 1 second.
From the differential of the previous graph we can see that there are a few errors in
the detection, which we can eliminate.
Finally we calculate the average of gap between these beats, which is 1, and the standard
error, which is almost 0.
We now put together the data we extracted for audio and video of the X5R proxy file.
These are the beat times for the audio.
These are the beat times for the video.
We restrict data to avoid the claps from the slate at the beginning and at the end.
We create a vector representing the difference between the beats detected in the audio and
the ones detected in the video.
This is a plot of this difference. It clearly shows that the delay between audio and video
cumulates over time. We start with a delay of less than 200ms and we end with a delay
of about 400ms after 200 seconds of recording.
We can fit a trendline to extract the delay rate in ms per seconds of recorded video.
The delay in seconds after 5minutes of video should be 0.46 seconds or equivalently 11
frames.
The delay in seconds after 10minutes of video should be 0.7 seconds or equivalently 17 frames.
As you can read from the Wikipedia page on audio-video sync for television applications,
the Advanced Television Systems Committee recommends that audio should lead video by
no more than 15 milliseconds and audio should lag video by no more than 45 milliseconds.
However, the ITU performed strictly controlled tests with expert viewers and found that the
threshold for detectability is -125ms to +45ms.
With the Osmo and the X5R we are way out of these limits.
In addition to these issues there is one more thing. The X5R proxy and the X5R raw videos
do not match in length.
Let me show you this in Resolve.
As we said the X5R raw videos were exported from the SSD card into the PC with the DJI
Camera Exporter.
The video we exported, according to this software lasts 218.886 seconds. However, the proxy
video lasts 218.375 seconds.
Note that both videos are interpreted as 23.976 frames per second.
So, in conclusion, I just demonstrated that my Osmo with the X5R camera has multiple
serious issues:
- Audio and video are not synchronized; - The delay between audio and video cumulates
over time; - The delay is 10 times out of the recommended
limits for audio-video synchronization; - The RAW videos recorded on the SSD card
and the proxy videos recorded on the SD card do not have the same length.
- Finally there is a gimbal drift
So, I have a question for DJI.
Is this an issue that affects all Osmos and all X5R cameras or is this an issue that affects
only my camera?
So is my camera faulty or this is a general problem for all cameras?
Please let me know and thanks for watching.