Multiple 1080p50 Channels Performance

#1
Hi folks,

Currently working on a project to use Caspar to do a video wall.
In a nutshell there's 6x 1080p50 outputs provided by 2x Decklink Duo 2's and 2 additional 1080p50 channels for video playback.
As the content spans multiple displays the file is played back on a non-output channel then routed to an output channel and scaled accordingly. Works surprisingly well.

However I seem to be getting awful performance in 2.0.7 with loads of dropped frames and stuttering in the video.
Performance is better in 2.2.0 Beta 1 but still seeing frame drops.
I've tried fiddling the pipeline-tokens setting but no joy.
The issue doesn't appear to be the Decklink consumer as such but more the Route Producer. Certainly that's where the diag window is showing the dropped frames.
Dropping to 1080p25 improves things but it's not really an ideal solution.
There are potentially up to 12 instances of the route producer running depending on the content. Interestingly it doesn't seem to get any worse/better if you have more/fewer.

Hardware is Intel Xeon W-2125 @ 4Ghz + 32GB RAM + Quadro P4000.
I've also tested on an Intel i7 6700K @ 4Ghz + 16GB RAM + Quadro P4000 and get worse results.
Both machines use SSDs in RAID 1 for OS/Caspar and RAID 0 for Media.
All files are ProRes HQ @ 1080p50.

Any thoughts or advice are very much appreciated!

Re: Multiple 1080p50 Channels Performance

#3
FredB wrote: What format is the source material? If it's compressed do you have 6 producers all trying to decompress?
The source material is ProRes HQ @ 1080p50. The ffmpeg producer is only used on the 2 playback channels.
Assuming Caspar is simply passing the raw frames via the Route producer then no further decompression should be happening I think?

So for example channels 1-6 have decklink consumers and use the route producer and channels 7-8 have no consumers but use the ffmpeg producer.

Code: Select all

PLAY 1-1 route://8
PLAY 2-1 route://8 (etc. Up to 6-1)
MIXER 1-1 FILL 0 0 2.9 0.5
MIXER 2-1 FILL -0.4 0 2.9 0.5 (etc. Up to 6-1)
PLAY 8 MY_FILE LOOP
So only channels 7 or 8 will be doing any real file playback.

Re: Multiple 1080p50 Channels Performance

#5
didikunz wrote:
25 Apr 2018, 09:29
AFAIK ProRes is quite performance hungry. have you tried with another codec? Does it also drop frames when you only show a still image?
Going to try that this morning and re-encode everything as H.264.
Massive improvement with the graph using a still image and the CPU begins to clock down. Will post back with further results!

Re: Multiple 1080p50 Channels Performance

#6
I did the similar project with 8 outputs instead of 6. Haven't used real-time scaling , instead pre-rendered each output file in 8 different HD 1080p files, and then played these files synchronously on 8 CasparCG Server instances. I've got perfect results, visually inseparable from one giant 8 x HD screen.

Re: Multiple 1080p50 Channels Performance

#7
Peterbbc wrote:
25 Apr 2018, 09:46
didikunz wrote:
25 Apr 2018, 09:29
AFAIK ProRes is quite performance hungry. have you tried with another codec? Does it also drop frames when you only show a still image?
Going to try that this morning and re-encode everything as H.264.
Massive improvement with the graph using a still image and the CPU begins to clock down. Will post back with further results!
So swapping to H.264 resulted in a massive performance gain! Still seeing some dropped frames on the i7 and the diag graphs get perilously close to the realtime mark...
itod wrote: I did the similar project with 8 outputs instead of 6. Haven't used real-time scaling , instead pre-rendered each output file in 8 different HD 1080p files, and then played these files synchronously on 8 CasparCG Server instances. I've got perfect results, visually inseparable from one giant 8 x HD screen.
Interesting. I did look at that as an option but was mildly concerned about sync dropping out.
We also have the slight issue where not all the content will always be going over all the displays all the time. It needs to be fairly dynamic which is a pain.
Would be interested to see some examples if you have any?

Re: Multiple 1080p50 Channels Performance

#8
Peterbbc wrote:
25 Apr 2018, 15:49
Peterbbc wrote:
25 Apr 2018, 09:46
didikunz wrote:
25 Apr 2018, 09:29
AFAIK ProRes is quite performance hungry. have you tried with another codec? Does it also drop frames when you only show a still image?
Going to try that this morning and re-encode everything as H.264.
Massive improvement with the graph using a still image and the CPU begins to clock down. Will post back with further results!
So swapping to H.264 resulted in a massive performance gain! Still seeing some dropped frames on the i7 and the diag graphs get perilously close to the realtime mark...
itod wrote: I did the similar project with 8 outputs instead of 6. Haven't used real-time scaling , instead pre-rendered each output file in 8 different HD 1080p files, and then played these files synchronously on 8 CasparCG Server instances. I've got perfect results, visually inseparable from one giant 8 x HD screen.
Interesting. I did look at that as an option but was mildly concerned about sync dropping out.
We also have the slight issue where not all the content will always be going over all the displays all the time. It needs to be fairly dynamic which is a pain.
Would be interested to see some examples if you have any?
I have only this low-quality-mobile-phone shots from years ago, don't know if you will be able to see it from Google sharing:
https://goo.gl/DfYQH7

You can see 5 HD screens in one part o the studio + 3 HD screens in other part, all playing giant 15360px X 1080px video, pre-processed to eight regular HD videos in After Effects. TV station made this different backgrounds for every show they are doing in that studio, and operator can control each screen individually to display photo or video illustrating what it has been talked about in the show. This system is many years in production and haven't dropped single frame ever, all screens are frame-synced all the time regardless of the fact eight different CasparCG servers are playing them.

Re: Multiple 1080p50 Channels Performance

#9
itod wrote:
26 Apr 2018, 13:54
Peterbbc wrote:
25 Apr 2018, 15:49
Peterbbc wrote:
25 Apr 2018, 09:46


Going to try that this morning and re-encode everything as H.264.
Massive improvement with the graph using a still image and the CPU begins to clock down. Will post back with further results!
So swapping to H.264 resulted in a massive performance gain! Still seeing some dropped frames on the i7 and the diag graphs get perilously close to the realtime mark...
itod wrote: I did the similar project with 8 outputs instead of 6. Haven't used real-time scaling , instead pre-rendered each output file in 8 different HD 1080p files, and then played these files synchronously on 8 CasparCG Server instances. I've got perfect results, visually inseparable from one giant 8 x HD screen.
Interesting. I did look at that as an option but was mildly concerned about sync dropping out.
We also have the slight issue where not all the content will always be going over all the displays all the time. It needs to be fairly dynamic which is a pain.
Would be interested to see some examples if you have any?
I have only this low-quality-mobile-phone shots from years ago, don't know if you will be able to see it from Google sharing:
https://goo.gl/DfYQH7

You can see 5 HD screens in one part o the studio + 3 HD screens in other part, all playing giant 15360px X 1080px video, pre-processed to eight regular HD videos in After Effects. TV station made this different backgrounds for every show they are doing in that studio, and operator can control each screen individually to display photo or video illustrating what it has been talked about in the show. This system is many years in production and haven't dropped single frame ever, all screens are frame-synced all the time regardless of the fact eight different CasparCG servers are playing them.
Wow, that's very impressive!
I'll have to look at doing it that way as a future option, currently a bit close to launch to rework how the graphics are delivered (I.E. as a single file) and then handled in Caspar.
But lots of time after launch to tweak that sort of thing as I've had a few other ideas along the way.

As an aside performance has improved since tweaking Windows to disable anything unnecessary and moving over to H.264 (thanks didi) so success there I think!
cron