I wanted to use my RPI4 for similar purposes but noticed a few limitations which I documented in my blog in detail: michaelspinboard.blogspot.com .
Summoning up my experience: it works and performs well if you don't care about crypto since the soc lacks AES extensions. Also using a SATA SSD as data storage drive didn't work out as many of the 2.5" drive cases still use outdated controllers (I tried three of them...).
I didn't cover hardware video encoders and decoders in my blog post but I successfully managed to get h264 encode/decode running after recompiling the kernel on Ubuntu and enabling an additional kernel module. But it looks like they are switching over to v4l2_m2m in kernel 5.4 which still takes time to majure . (but it is technically a good decision to use an open, modern api).
I still like the RPI4, it's an amazing single board computer. Especially considering the price. But for my own setup I decided to use an AsRock DeskMini A300 instead.