Bug#730810: psychopy: Movie support suboptimal

[Mario Kleiner]

On 30/11/13 02:38, Yaroslav Halchenko wrote:
I guess this is hardware independent? (i.e. you have tried on few
boxes/laptops with the same success)
have you asked PsychoPy guys themselves?

O mighty Mario,  would you happen to have ideas where to look at?


This is pure guesswork, but as you asked ... ;-)

I don't know how PsychoPy does this or how avbin works, but my best 
guess from the info below would be that PsychoPy could be inefficient in 
converting video frames to OpenGL textures for rendering, so it can't 
keep up with avbin? Assuming of course that PsychoPy or some backend 
library converts video frames to OpenGL textures for presentation?

In any modern movie player i know of, if there are any audio tracks 
played back, the principle is that audio timing drives video timing and 
the player drops or holds back video frames to keep audio and video from 
drifting apart. If avbin does the same, then it would drop video frames 
if the client can't keep up with fetching/processing them, to keep 
audio-video sync. If there wouldn't be any audio it might just wait for 
the client to catch up - video would not drop frames but play too slowly 
or jittery.

You could check if the problem goes away/is reduced if you use movies 
with very small frame sizes, so inefficient conversion into OpenGL 
textures doesn't matter. And/Or try on graphics cards from different 
vendors/different drivers etc.

But this is all just a guess based on how Psychtoolbox does it. One can 
easily have over an order of magnitude of performance difference in the 
video->OpenGL conversion stage, depending on implementation and/or 
graphics driver. I spent multiple months optimizing that part of ptb, so 
this can be non-trivial.

best,
-mario

On Fri, 29 Nov 2013, Michael Hanke wrote:

Package: psychopy
Version: 1.77.02.dfsg-1
Severity: normal

Hi,

using movie stimuli in Psychopy doesn't really work smoothly. Whenever a
movie with sound is played I experience dropped frames. I am attaching a
little benchmark that can be used to show this (done in the builder,
just select a movie file and run it). This issue is only present when
sound is actually played. When it is not played, even when a sound
stream is present in the file and avbin just 'ignores' it, the stimulus
is smooth.

All files and codec combinations I tested play smooth in avplay, vlc,
mplayer and others. I see this issue on amd64 and i386 -- independent of
whether Psychopys own system check reports 'dropped frames' on a
particular system or not. I tested Debian Jessie and Ubuntu 13.04, both
show the problem.

Psychopy upstream recommends avbin 5 for movie stimuli, but Debian never
had it (according to snapshots.d.o). Psychopy upstream also recommend
uncompressed PCM audio -- that makes no difference in comparison to,
e.g. FLAC.

Interestingly, I see a similar jerky playback in openshot. However, when
using the MELT player (from the framework that openshot is based on)
everything is smooth.

This is also not an obvious IO issue. Even if I cache the entire movie
file in memory, the frame drops are still present. System load at
playback is negligible.

Any advise would be very much appreciated.

Thanks,

Michael


-- System Information:
Debian Release: jessie/sid
   APT prefers testing
   APT policy: (650, 'testing'), (600, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.10-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages psychopy depends on:
ii  python             2.7.5-5
ii  python-configobj   4.7.2+ds-5
ii  python-lxml        3.2.0-1+b1
ii  python-matplotlib  1.1.1~rc2-1
ii  python-numpy       1:1.7.1-3
ii  python-opengl      3.0.1-1
ii  python-pygame      1.9.1release+dfsg-8
ii  python-pyglet      1.1.4.dfsg-2
ii  python-scipy       0.12.0-2+b1
ii  python-support     1.0.15

Versions of packages psychopy recommends:
ii  ipython          0.13.2-2
ii  libavbin0        7-1.4
ii  libxxf86vm1      1:1.1.3-1
ii  python-imaging   1.1.7-4
ii  python-openpyxl  1.5.8-1
ii  python-pygame    1.9.1release+dfsg-8
ii  python-pyglet    1.1.4.dfsg-2
ii  python-pyo       0.6.8-1
ii  python-serial    2.6-1
ii  python-wxgtk2.8  2.8.12.1+dfsg-1

Versions of packages psychopy suggests:
pn  python-iolabs  <none>
pn  python-pyxid   <none>

-- debconf-show failed

<PsychoPy2experiment version="1.77.01" encoding="utf-8">
   <Settings>
     <Param name="Show mouse" val="False" valType="bool" updates="None"/>
     <Param name="Save csv file" val="False" valType="bool" updates="None"/>
     <Param name="Monitor" val="testMonitor" valType="str" updates="None"/>
     <Param name="Enable Escape" val="True" valType="bool" updates="None"/>
     <Param name="color" val="$[0,0,0]" valType="str" updates="None"/>
     <Param name="Window size (pixels)" val="[1024, 768]" valType="code" 
updates="None"/>
     <Param name="Full-screen window" val="False" valType="bool" 
updates="None"/>
     <Param name="colorSpace" val="rgb" valType="str" updates="None"/>
     <Param name="Save log file" val="True" valType="bool" updates="None"/>
     <Param name="Experiment info" val="{u'session': u'001', u'participant': u''}" 
valType="code" updates="None"/>
     <Param name="Save excel file" val="False" valType="bool" updates="None"/>
     <Param name="Save wide csv file" val="True" valType="bool" updates="None"/>
     <Param name="Save psydat file" val="True" valType="bool" updates="None"/>
     <Param name="expName" val="movie_benchmark" valType="str" updates="None"/>
     <Param name="logging level" val="exp" valType="code" updates="None"/>
     <Param name="Units" val="use prefs" valType="str" updates="None"/>
     <Param name="Show info dlg" val="True" valType="bool" updates="None"/>
     <Param name="Saved data folder" val="" valType="code" updates="None"/>
     <Param name="Screen" val="1" valType="num" updates="None"/>
   </Settings>
   <Routines>
     <Routine name="movie_routine">
       <MovieComponent name="movie_clip">
         <Param name="opacity" val="1" valType="code" updates="constant"/>
         <Param name="name" val="movie_clip" valType="code" updates="constant"/>
         <Param name="movie" val="fg_av_test.mov" valType="str" 
updates="constant"/>
         <Param name="stopVal" val="10" valType="code" updates="constant"/>
         <Param name="durationEstim" val="" valType="code" updates="None"/>
         <Param name="pos" val="[0, 0]" valType="code" updates="constant"/>
         <Param name="forceEndRoutine" val="False" valType="bool" 
updates="constant"/>
         <Param name="startEstim" val="" valType="code" updates="None"/>
         <Param name="units" val="from exp settings" valType="str" 
updates="None"/>
         <Param name="startType" val="time (s)" valType="str" updates="None"/>
         <Param name="ori" val="0" valType="code" updates="constant"/>
         <Param name="stopType" val="duration (s)" valType="str" 
updates="None"/>
         <Param name="startVal" val="0.0" valType="code" updates="None"/>
         <Param name="size" val="" valType="code" updates="constant"/>
       </MovieComponent>
       <CodeComponent name="code">
         <Param name="Begin Experiment" val="from datetime import datetime&#10;import numpy as np" 
valType="code" updates="constant"/>
         <Param name="name" val="code" valType="code" updates="None"/>
         <Param name="Each Frame" val="cur_movietime = movie_clip._movie._last_video_timestamp&#10;if last_movietime and cur_movietime &gt; 
last_movietime:&#10;    n_movieframes += 1&#10;    incr = cur_movietime - last_movietime&#10;    movie_frame_increments.append(incr)&#10;    
movie_lags.append(cur_movietime - t)&#10;    if lastframe_time:&#10;        realtime_frame_increments.append(t - lastframe_time)&#10;    lastframe_time = 
t&#10;last_movietime = cur_movietime" valType="code" updates="constant"/>
         <Param name="Begin Routine" val="n_movieframes = 0&#10;last_movietime = None&#10;lastframe_time = 
None&#10;movie_frame_increments = []&#10;realtime_frame_increments = []&#10;movie_lags = []" valType="code" 
updates="constant"/>
         <Param name="End Routine" val="" valType="code" updates="constant"/>
         <Param name="End Experiment" val="import pylab as pl&#10;pl.subplot(1,2,1)&#10;pl.hist(realtime_frame_increments, 
bins=100)&#10;pl.title('Distance of two movie frames in real time')&#10;pl.xlabel('Time in secs')&#10;pl.subplot(1,2,2)&#10;pl.hist(movie_lags, 
bins=100)&#10;pl.title('Lag between movie time and real time for all frames')&#10;pl.xlabel('Time in seconds')&#10;print &quot;Movie frames played:&quot;, 
n_movieframes&#10;print 'Mean movie frame increment:', np.mean(movie_frame_increments), 's (', 1./np.mean(movie_frame_increments), 'fps )'&#10;print 'Unique movie frame 
increments', np.unique(movie_frame_increments)&#10;pl.show()" valType="code" updates="constant"/>
       </CodeComponent>
     </Routine>
   </Routines>
   <Flow>
     <Routine name="movie_routine"/>
   </Flow>
</PsychoPy2experiment>




--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org