Kernel's aren't important. Specifically, they are not business-differentiating. That means they don't make your product look better than your competitor's product. Users never see them, never interact with them. Kernel performance isn't important. It is a tiny percentage of the time used by the application. IPC cost isn't important. It was when processors ran at 60 MHz, but we are long past that and IPC performance is a tiny fraction of network performance, which nearly every application _is_ dependent upon. Filesystem performance isn't important. It was in the day of rotating disks, which had seek delays and rotational delays that solid-state storage doesn't have.
But you still need a kernel, and it has to be reliable, and security is one of its features but a secure kernel won't help you with a less-secure application.
So, what do you do? Use one of the Open Source kernels. Besides Linux there is BSD, and there are several others. Spend as little of your time and money on these as you can, then spend the rest of your time and money developing things that will differentiate your product from your competitors, because that's the only reason anyone will buy them. Not a kernel.
I've seen this same mistake many times now. Nokia did it with Symbian (remember them?) HP spent a Billion putting IPV6 and other modern features in HP/UX while IBM was going to Linux. Who do you think won?