The amount of memory allocated to generate PDF output with the built-in Apache FOP processor is controlled by the FOP memory setting available in Oxygen XML Author Preferences: Memory available to the built-in FOP. In the application throws an Out Of Memory error (OutOfMemoryError), this is the setting that must be modified to allow more memory for the built-in FOP.
For external XSL-FO processors configured in and for external XSLT processors configured in and for external tools configured in the maximum memory must be set in the command line of the tool with a parameter -Xmx set to the Java virtual machine.