I would like to use the question and answer layout as well as the extended layout on the same installation
Yes it possible as you can see here on our support forum. You should have two top Categories (top parent forums). Edit those and set first Extended Forum Layout, Second Q&A Forum layout. Doc: https://wpforo.com/docs/root/categories-and-forums/forum-manager/add-new-forum/#forum-options
You just need to know that the forum layout is set for the topic category and all forums of this category will extend the layout. So if you should have two top categories with different layouts.