I've noticed that there is some kind of discontentment with current state of
the libavfilter directory and a wish to reduce the number of mainstreamed
filters there, relying on some kind of interface for keeping non essential
plugins out of tree.
Al noticed was the desire to make it easier for 3rd parties to integrate to
ffmpeg without requiring mainstream changes.
I've been working on a Rust [1] based filter which I do not expect to be
mainlined in ffmpeg any time soon, so having such ouf of tree support would
benefit me a lot as well.
The approaches I've seen so far are (there are probably other approaches,
please suggest them here):
- runtime loadable filters: this would require the current private filter API
to be refactored as a public API, and would need to guarantee some ABI
stability to make it usable in real life by real filters. This seems to be a
lot of work, with questionable benefits and a huge burden to keep API and ABI
stability.
- filters implemented in a DSL, at runtime: think on offering for instance a
Lua API for the filters. This would allow better control of the API, but
require writing, possibly manually, such bindings from C to Lua (or whichever
chosen language). Due to the nature of the filters, this approach would likely
have big performance overhead on the filters implementations, on the C<->Lua
translation layer. There is extra burden on keeping the bindings too.
- sandboxed filters, using webassembly. This shares similar issues with the DSL
approach above, but at least would allow filters to be written in any language
that compiles to WASM. It would also share the disadvantages of the runtime
loadable filters, as it'd require some stable ABI to be useful. It seems (I am
probably wrong; have no experience using WASM) that in WASM sharing memory
between host and virtual machine is not an easy task yet, requiring either
frequent memory copying between host and guest, which would kill performance.
WASM specs seem to be changing to address similar issues (the GC proposal), but
there seems not to be anything ready, stable and that directly addresses the
aforementioned issues.
- out of tree filters, integrated at build time: some projects such as LLVM use
it. This approach does not require offering any API or ABI stability, being up
to the 3rd party developer to guarantee it compiles with a certain version of
FFmpeg. It seems that this approach would also provide the shortest path for
extracting currently filters that are not are used by most users or require C++
(dnn, decklnk) to external repos and developed independently from the main
FFmpeg repo.
This is not an RFC, just some brainstorming. I believe that the out of tree
approach is the best fit for the current codebase, but it comes with
challenges/issues:
- build system integration
From someone compiling FFmpeg, I'd like something as simple as `./configure
--extra-filter=/path/to/foo --extra-filter=/path/to/bar`. This would then look
at such directories and report an error in case dependencies are not met, etc.
The extra filters would then be built directly into
libavfilter.so/libavfilter.a.
On the filter perspective, we'd need support for describing the filter
properties, dependencies, etc. I am thinking right now on something we can
easily integrate into the current build system, like a `filter-manifest.json`
file, in this example declaring two filters:
```
[
{
"name": "example",
"description": "An example filter",
"license": "LGPL3",
"symbol": "ff_vf_example",
"ldflags": "-lsome_external_lib -lblah",
"cflags": "-DFOO=123 -I/usr/local/include/blah",
"check_command": "is_foo_installed.sh"
"pre-make": "command-to-run-before-make"
},
{
"name": "blah",
"description": "baz",
"license": "LGPL3",
"symbol": "ff_af_blah",
"pkg-config": [
"foo",
"bar"
]
}
]
```
It would require `jq` to be installed, for querying JSON directly from `sh`.
Additionally, there there should be a Makefile in the directory, for the build
process itself, but I am not well versed on make and the FFmpeg build system to
suggest how it'd look like right now.
I know it sucks to create a new format, but it seems for me a good balance
between a bunch of files spread in a directory + awk/sed/make vs requiring
FFmpeg to use a different build system such as meson on cmake.
I haven't put much thought into the filter description though, so I might be
very mistaken and there are easier ways within the current FFmpeg codebase.
- Licensing
Some people will be interested in building proprietary filters, so I don't know
how much building them directly into libavfilter library would be compatible
with FFmpeg licenses. Many companies already link FFmpeg into proprietary
products, by building only LGPL code and linking to FFMpeg dynamically, so
maybe not much would change by putting their code directly into libavfilter.so
- Stability and support
Building 3rd party code directly into libavfilter.(so|a) could cause users of
such FFmpeg builds to report issues caused by the 3rd party code to FFmpeg
developers, increasing the burden on them. I think this is mostly a social
issue and don't have a possible solution for it. But I wonder how much
different this is from the current status of Linux distributions building,
shipping and supporting FFmpeg.
- Documentation
It would be great to be able to also generate documentation from such extra
filters in the same fashion as the mainstream filters.
I would like to know if any of this sounds interesting to you folks and whether
anyone is already working in solving the original problem. Advice from more
experienced devs is also appreciated.
In case of no objections, I'd like to start experimenting with it, working on a
prototype/POC.
I am also interested in knowing if other anyone else interested in working on
it too.
Regards,
Leandro
[1] https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/11
_______________________________________________
ffmpeg-devel mailing list
[email protected]
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
[email protected] with subject "unsubscribe".
