1# Video (experimental) 2 3The virtio video decoder and encoder devices allow a guest to leverage the host's 4hardware-accelerated video decoding and encoding capabilities. The specification ([v3], [v5]) for 5these devices is still a work-in-progress, so testing them requires an out-of-tree kernel driver on 6the guest. 7 8The virtio-video host device uses backends to perform the actual decoding. The currently supported 9backends are: 10 11- `libvda`, a hardware-accelerated backend that supports both decoding and encoding by delegating 12 the work to a running instance of Chrome. It can only be built and used in a ChromeOS environment. 13- `ffmpeg`, a software-based backend that supports encoding and decoding. It exists to make testing 14 and development of virtio-video easier, as it does not require any particular hardware and is 15 based on a reliable codec library. 16 17The rest of this document will solely focus on the `ffmpeg` backend. More accelerated backends will 18be added in the future. 19 20## Guest kernel requirements 21 22The `virtio_video` branch of this [kernel git repository](https://github.com/Gnurou/linux) contains 23a work-in-progress version of the `virtio-video` guest kernel driver, based on a (hopefully) recent 24version of mainline Linux. If you use this as your guest kernel, the `virtio_video_defconfig` 25configuration should allow you to easily boot from crosvm, with the video (and a few other) virtio 26devices support built-in. 27 28Quick building guide after checking out this branch: 29 30```sh 31mkdir build_crosvm_x86 32make O=build_crosvm_x86 virtio_video_defconfig 33make O=build_crosvm_x86 -j16 34``` 35 36The resulting kernel image that can be passed to `crosvm` will be in 37`build_crosvm_x86/arch/x86/boot/bzImage`. 38 39## Crosvm requirements 40 41The virtio-video support is experimental and needs to be opted-in through the `"video-decoder"` or 42`"video-encoder"` Cargo feature. In the instruction below we'll be using the FFmpeg backend which 43requires the `"ffmpeg"` feature to be enabled as well. 44 45The following example builds crosvm with FFmpeg encoder and decoder backend support: 46 47```sh 48cargo build --features "video-encoder,video-decoder,ffmpeg" 49``` 50 51To enable the **decoder** device, start crosvm with the `--video-decoder=ffmpeg` command-line 52argument: 53 54```sh 55crosvm run --disable-sandbox --video-decoder=ffmpeg -c 4 -m 2048 --block /path/to/disk.img,root --serial type=stdout,hardware=virtio-console,console=true,stdin=true /path/to/bzImage 56``` 57 58Alternatively, to enable the **encoder** device, start crosvm with the `--video-encoder=ffmpeg` 59command-line argument: 60 61```sh 62crosvm run --disable-sandbox --video-encoder=ffmpeg -c 4 -m 2048 --block /path/to/disk.img,root --serial type=stdout,hardware=virtio-console,console=true,stdin=true /path/to/bzImage 63``` 64 65If the guest kernel includes the virtio-video driver, then the device should be probed and show up. 66 67## Testing the device from the guest 68 69Video capabilities are exposed to the guest using V4L2. The encoder or decoder device should appear 70as `/dev/videoX`, probably `/dev/video0` if there are no additional V4L2 devices. 71 72### Checking capabilities and formats 73 74`v4l2-ctl`, part of the `v4l-utils` package, can be used to test the device's existence. 75 76Example output for the decoder is shown below. 77 78```sh 79v4l2-ctl -d/dev/video0 --info 80Driver Info: 81 Driver name : virtio-video 82 Card type : ffmpeg 83 Bus info : virtio:stateful-decoder 84 Driver version : 5.17.0 85 Capabilities : 0x84204000 86 Video Memory-to-Memory Multiplanar 87 Streaming 88 Extended Pix Format 89 Device Capabilities 90 Device Caps : 0x04204000 91 Video Memory-to-Memory Multiplanar 92 Streaming 93 Extended Pix Format 94``` 95 96Note that the `Card type` is `ffmpeg`, indicating that decoding will be performed in software on the 97host. We can then query the support input (`OUTPUT` in V4L2-speak) formats, i.e. the encoded formats 98we can send to the decoder: 99 100```sh 101v4l2-ctl -d/dev/video0 --list-formats-out 102ioctl: VIDIOC_ENUM_FMT 103 Type: Video Output Multiplanar 104 105 [0]: 'VP90' (VP9, compressed) 106 [1]: 'VP80' (VP8, compressed) 107 [2]: 'HEVC' (HEVC, compressed) 108 [3]: 'H264' (H.264, compressed) 109``` 110 111Similarly, you can check the supported output (or CAPTURE) pixel formats for decoded frames: 112 113```sh 114v4l2-ctl -d/dev/video0 --list-formats 115ioctl: VIDIOC_ENUM_FMT 116 Type: Video Capture Multiplanar 117 118 [0]: 'NV12' (Y/CbCr 4:2:0) 119``` 120 121### Test decoding with ffmpeg 122 123[FFmpeg](https://ffmpeg.org/) can be used to decode video streams with the virtio-video device. 124 125Simple VP8 stream: 126 127```sh 128wget https://github.com/chromium/chromium/raw/main/media/test/data/test-25fps.vp8 129ffmpeg -codec:v vp8_v4l2m2m -i test-25fps.vp8 test-25fps-%d.png 130``` 131 132This should create 250 PNG files each containing a decoded frame from the stream. 133 134WEBM VP9 stream: 135 136```sh 137wget https://test-videos.co.uk/vids/bigbuckbunny/webm/vp9/720/Big_Buck_Bunny_720_10s_1MB.webm 138ffmpeg -codec:v vp9_v4l2m2m -i Big_Buck_Bunny_720_10s_1MB.webm Big_Buck_Bunny-%d.png 139``` 140 141Should create 300 PNG files at 720p resolution. 142 143### Test decoding with v4l2r 144 145The [v4l2r](https://github.com/Gnurou/v4l2r) Rust crate also features an example program that can 146use this driver to decode simple H.264 streams: 147 148```sh 149git clone https://github.com/Gnurou/v4l2r 150cd v4l2r 151wget https://github.com/chromium/chromium/raw/main/media/test/data/test-25fps.h264 152cargo run --example simple_decoder test-25fps.h264 /dev/video0 --input_format h264 --save test-25fps.nv12 153``` 154 155This will decode `test-25fps.h264` and write the raw decoded frames in `NV12` format into 156`test-25fps.nv12`. You can check the result with e.g. [YUView](https://github.com/IENT/YUView). 157 158### Test encoding with ffmpeg 159 160[FFmpeg](https://ffmpeg.org/) can be used to encode video streams with the virtio-video device. 161 162The following examples generates a test clip through libavfilter and encode it using the virtual 163H.264, H.265 and VP8 encoder, respectively. (VP9 v4l2m2m support is missing in FFmpeg for some 164reason.) 165 166```sh 167# H264 168ffmpeg -f lavfi -i smptebars=duration=10:size=640x480:rate=30 \ 169 -pix_fmt nv12 -c:v h264_v4l2m2m smptebars.h264.mp4 170# H265 171ffmpeg -f lavfi -i smptebars=duration=10:size=640x480:rate=30 \ 172 -pix_fmt yuv420p -c:v hevc_v4l2m2m smptebars.h265.mp4 173# VP8 174ffmpeg -f lavfi -i smptebars=duration=10:size=640x480:rate=30 \ 175 -pix_fmt yuv420p -c:v vp8_v4l2m2m smptebars.vp8.webm 176``` 177 178[v3]: https://markmail.org/message/dmw3pr4fuajvarth 179[v5]: https://markmail.org/message/zqxmuf5x7aosbmmm 180
