The amount of memory allocated for generating PDF output with the built-in Apache FOP processor is controlled by a different setting available in <oXygen/> XML Author Preferences: Memory available to the built-in FOP. In case of Out Of memory errors this is the setting that must be modified for allowing more memory for the built-in FOP.
For external XSL-FO processors configured in Preferences -> XML -> XSLT/FO/XQuery -> FO Processors and for external XSLT processors configured in Preferences -> XML -> XSLT/FO/XQuery -> Custom Engines and for external tools configured in Preferences -> External Tools the maximum memory must be set in the command line of the tool with a parameter -Xmx set to the Java virtual machine.