I have a couple of methods that I use when a moulding has an odd shaped outside or an odd shaped inside.
I still join about 20% of the frames that we make in vises before vnailing.
We have about 10 miter vises and 2 vnailers.
If I was at work I would post pictures of how I join odd shaped frames.
One method that I use is to attach square pieces of wood with small drywall screws to the back of the legs that I'm joining near the mitered corners.
The blocks have to be placed accurately or you won't get a proper 45° at the corner.
Then I glue the corners and clamp on the square wood pieces that I've attached to the back of the moulding, rather than trying to clamp on the odd shaped moulding.
Then when the frame is glued up, I remove the blocks from the back and vnail.
When I vnail these odd shaped mouldings, I back out the front clamp on the vnailer so it doesn't push the moulding up or dent the outside of the frame.
I just hold the already glued and joined frame and vnail.
Roma used to have a large profile that had a round outside edge....
This method also worked on those.
It will also work on mouldings that are too wide for the vise.
One other method that I use, and it's also with a miter vise, is to use small pieces of the Arlo 1/8" solid plastic frame space between the outer vise jaw and the moulding.
That way it is only clamping on the very bottom edge of the outside of the moulding and not pushing so much on the slanted or odd shaped area.
This one is a little hard to describe without pictures...
You can also try to just join in the vnailer while eliminating the front clamp so it's not pushing and just use your hands as clamps to hold while vnailing but I prefer to join in a vise first.