PR #23736 opened by philipl URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23736 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23736.patch
vf_libplacebo is unique in that it can function as both a Vulkan filter, and a software filter, depending on how it the filter graph is configured. While this is very flexible, it create a problem in situations where the filter does not receive a Vulkan hw frames context up front. When that doesn't happen, it will initialise its own standalone pl context, and then fail to interoperate with a context provided via the input link. This then leads to graph failures. There are two primary scenarios where this existing logic breaks what should be valid configurations: * When the global filter hw device is a cuda device and we use something like `hwupload=derive_device=vulkan` to pass frames from cuda to vulkan * In mpv (and probably other media players) which don't set the global filter hw device at all. In this case, it's impossible to configure vf_libplacebo to use a hw frames ctx, even if it's using Vulkan for everythng else. This prevents the use of vf_libplacebo in any fully hardware accelerated pipeline in mpv There are various ways we could imagine addressing it - such as allowing the filter to discard the initial pl context and recreating it based on passed in device, but it's easier to reason about if we add a flag that explicitly tells the filter that it should inherit the device context from the input link. This puts the filter into a mode that works like all the other Vulkan filters. This requires explicit configuration from the user, but the intent is clearer, and the user can always know when it's necessary as they define the filter graph. # Summary of changes Briefly describe what this PR does and why. <!-- If this PR requires new FATE test samples, attach them to the PR and list their target paths below (relative to the fate-suite root). Attached filenames must match the sample's filename: ```fate-samples # e.g. vorbis/new-sample.ogg ``` --> >From fb4026a410cbe35eda5a882f2a35cb7b85e7fbe4 Mon Sep 17 00:00:00 2001 From: Philip Langdale <[email protected]> Date: Mon, 6 Jul 2026 13:07:46 -0700 Subject: [PATCH] avfilter/vf_libplacebo: add flag to inherit the input's Vulkan device vf_libplacebo is unique in that it can function as both a Vulkan filter, and a software filter, depending on how it the filter graph is configured. While this is very flexible, it create a problem in situations where the filter does not receive a Vulkan hw frames context up front. When that doesn't happen, it will initialise its own standalone pl context, and then fail to interoperate with a context provided via the input link. This then leads to graph failures. There are two primary scenarios where this existing logic breaks what should be valid configurations: * When the global filter hw device is a cuda device and we use something like `hwupload=derive_device=vulkan` to pass frames from cuda to vulkan * In mpv (and probably other media players) which don't set the global filter hw device at all. In this case, it's impossible to configure vf_libplacebo to use a hw frames ctx, even if it's using Vulkan for everythng else. This prevents the use of vf_libplacebo in any fully hardware accelerated pipeline in mpv There are various ways we could imagine addressing it - such as allowing the filter to discard the initial pl context and recreating it based on passed in device, but it's easier to reason about if we add a flag that explicitly tells the filter that it should inherit the device context from the input link. This puts the filter into a mode that works like all the other Vulkan filters. This requires explicit configuration from the user, but the intent is clearer, and the user can always know when it's necessary as they define the filter graph. --- doc/filters.texi | 15 ++++++++++ libavfilter/vf_libplacebo.c | 60 +++++++++++++++++++++++++++++-------- 2 files changed, 63 insertions(+), 12 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 5497421f34..34219890d4 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -16908,6 +16908,21 @@ exist. -vf "libplacebo=shader_cache=/tmp/pl-shader-" @end example +@item inherit_device +Take the Vulkan device from the hardware frames arriving on the input, instead +of creating a private one from the filter's own hardware device. This is +useful when the input frames originate on another API (for example CUDA frames +uploaded to Vulkan by @code{hwupload}) and the filter's device is not itself a +Vulkan device: libplacebo would otherwise spin up a separate Vulkan device that +the input frames do not belong to, and the frames could not be mapped. + +When enabled, the filter requires Vulkan hardware frames on the input and will +fail negotiation otherwise. Disabled by default. + +@example +-init_hw_device vulkan=vk:0 -init_hw_device cuda@@vk -filter_hw_device cuda0 @dots{} -vf "hwupload=derive_device=vulkan,libplacebo=inherit_device=1" +@end example + @item colorspace @item color_primaries @item color_trc diff --git a/libavfilter/vf_libplacebo.c b/libavfilter/vf_libplacebo.c index a7b8a38f7b..d54bc05901 100644 --- a/libavfilter/vf_libplacebo.c +++ b/libavfilter/vf_libplacebo.c @@ -200,6 +200,7 @@ typedef struct LibplaceboContext { pl_cache cache; char *shader_cache; + int inherit_device; int have_hwdevice; @@ -506,13 +507,13 @@ static int parse_shader(AVFilterContext *avctx, const void *shader, size_t len) static void libplacebo_uninit(AVFilterContext *avctx); static int libplacebo_config_input(AVFilterLink *inlink); -static int init_vulkan(AVFilterContext *avctx, const AVVulkanDeviceContext *hwctx); +static int init_vulkan(AVFilterContext *avctx, AVHWDeviceContext *vkdev); static int libplacebo_init(AVFilterContext *avctx) { int err = 0; LibplaceboContext *s = avctx->priv; - const AVVulkanDeviceContext *vkhwctx = NULL; + AVHWDeviceContext *vkdev = NULL; if (s->normalize_sar && s->fit_mode != FIT_FILL) { av_log(avctx, AV_LOG_WARNING, "normalize_sar has no effect when using " @@ -593,12 +594,17 @@ static int libplacebo_init(AVFilterContext *avctx) RET(av_parse_video_rate(&s->fps, s->fps_string)); if (avctx->hw_device_ctx) { - const AVHWDeviceContext *avhwctx = (void *) avctx->hw_device_ctx->data; + AVHWDeviceContext *avhwctx = (void *) avctx->hw_device_ctx->data; if (avhwctx->type == AV_HWDEVICE_TYPE_VULKAN) - vkhwctx = avhwctx->hwctx; + vkdev = avhwctx; } - RET(init_vulkan(avctx, vkhwctx)); + /* With inherit_device and no Vulkan device of our own, defer all Vulkan + * setup until config_input(), where we adopt the Vulkan device the input + * frames live on. Creating a private device here would put us on a + * different device than the input and make the frames unusable. */ + if (!(s->inherit_device && !vkdev)) + RET(init_vulkan(avctx, vkdev)); return 0; @@ -664,10 +670,11 @@ static int copy_pl_queue(const AVVulkanDeviceContext *hwctx, return 0; } -static int init_vulkan(AVFilterContext *avctx, const AVVulkanDeviceContext *hwctx) +static int init_vulkan(AVFilterContext *avctx, AVHWDeviceContext *vkdev) { int err = 0; LibplaceboContext *s = avctx->priv; + const AVVulkanDeviceContext *hwctx = vkdev ? vkdev->hwctx : NULL; uint8_t *buf = NULL; size_t buf_len; @@ -682,7 +689,7 @@ static int init_vulkan(AVFilterContext *avctx, const AVVulkanDeviceContext *hwct .features = &hwctx->device_features, .lock_queue = lock_queue, .unlock_queue = unlock_queue, - .queue_ctx = avctx->hw_device_ctx->data, + .queue_ctx = vkdev, .queue_graphics = { VK_QUEUE_FAMILY_IGNORED }, .queue_compute = { VK_QUEUE_FAMILY_IGNORED }, .queue_transfer = { VK_QUEUE_FAMILY_IGNORED }, @@ -1286,6 +1293,16 @@ static int libplacebo_query_format(const AVFilterContext *ctx, const AVPixFmtDescriptor *desc = NULL; AVFilterFormats *infmts = NULL, *outfmts = NULL; + if (!s->gpu) { + /* Device deferred to config_input (inherit_device): we have no GPU yet + * to enumerate software formats against, and this mode requires Vulkan + * input, so advertise Vulkan only. */ + RET(ff_add_format(&infmts, AV_PIX_FMT_VULKAN)); + if (s->out_format == AV_PIX_FMT_NONE || av_vkfmt_from_pixfmt(s->out_format)) + RET(ff_add_format(&outfmts, AV_PIX_FMT_VULKAN)); + goto done; + } + /* List AV_PIX_FMT_VULKAN first to prefer it when possible */ if (s->have_hwdevice) { RET(ff_add_format(&infmts, AV_PIX_FMT_VULKAN)); @@ -1317,6 +1334,8 @@ static int libplacebo_query_format(const AVFilterContext *ctx, RET(ff_add_format(&outfmts, pixfmt)); } +done: + if (!infmts || !outfmts) { err = AVERROR(EINVAL); goto fail; @@ -1360,8 +1379,10 @@ fail: static int libplacebo_config_input(AVFilterLink *inlink) { + int err = 0; AVFilterContext *avctx = inlink->dst; LibplaceboContext *s = avctx->priv; + FilterLink *l = ff_filter_link(inlink); if (s->rotation % PL_ROTATION_180 == PL_ROTATION_90) { /* Swap width and height for 90 degree rotations to make the size and @@ -1371,13 +1392,26 @@ static int libplacebo_config_input(AVFilterLink *inlink) inlink->sample_aspect_ratio = av_inv_q(inlink->sample_aspect_ratio); } + /* Deferred Vulkan setup (inherit_device): the device was not created at + * init; adopt the one the input frames live on. query_formats advertised + * Vulkan only, so a non-Vulkan input here is a configuration error. */ + if (!s->gpu) { + if (inlink->format != AV_PIX_FMT_VULKAN || !l->hw_frames_ctx) { + av_log(avctx, AV_LOG_ERROR, "inherit_device requires a Vulkan " + "hardware frames context on the input.\n"); + return AVERROR(EINVAL); + } + RET(init_vulkan(avctx, ((AVHWFramesContext *)l->hw_frames_ctx->data)->device_ctx)); + } + if (inlink->format == AV_PIX_FMT_VULKAN) - return ff_vk_filter_config_input(inlink); + RET(ff_vk_filter_config_input(inlink)); + else + /* Forward this to the vkctx for format selection */ + s->vkctx.input_format = inlink->format; - /* Forward this to the vkctx for format selection */ - s->vkctx.input_format = inlink->format; - - return 0; +fail: + return err; } static inline AVRational max_q(AVRational a, AVRational b) @@ -1524,6 +1558,8 @@ fail: static const AVOption libplacebo_options[] = { { "inputs", "Number of inputs", OFFSET(nb_inputs), AV_OPT_TYPE_INT, {.i64 = 1}, 1, INT_MAX, .flags = STATIC }, + { "inherit_device", "Inherit the Vulkan device from the input's hardware frames context", + OFFSET(inherit_device), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, .flags = STATIC }, { "w", "Output video frame width", OFFSET(w_expr), AV_OPT_TYPE_STRING, {.str = "iw"}, .flags = STATIC }, { "h", "Output video frame height", OFFSET(h_expr), AV_OPT_TYPE_STRING, {.str = "ih"}, .flags = STATIC }, { "fps", "Output video frame rate", OFFSET(fps_string), AV_OPT_TYPE_STRING, {.str = "none"}, .flags = STATIC }, -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
