Quantcast

Out of Memory error

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Out of Memory error

vsyamala
Hi,

I am trying to generate PDF from 45MB .fo file, fop(0.95) produces a pdf of about 12MB and it fails with "out of memory" errors. XMx and XMs options are set for 2048MB(that's max I could go on the machine). And this .fo file is referring to about 300 images of appx 48KB each. FOP should be able to generate PDF. I am not sure if images are causing this issue. Does anyone know if images are the issue? Here is the error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.io.ByteArrayOutputStream.write(Unknown Source)
        at java.io.OutputStream.write(Unknown Source)
        at org.apache.fop.pdf.PDFStream.add(PDFStream.java:60)
        at org.apache.fop.render.pdf.PDFRenderer.concatenateTransformationMatrix
(PDFRenderer.java:839)
        at org.apache.fop.render.AbstractPathOrientedRenderer.renderReferenceAre
a(AbstractPathOrientedRenderer.java:539)
        at org.apache.fop.render.AbstractRenderer.renderBlock(AbstractRenderer.j
ava:560)
        at org.apache.fop.render.pdf.PDFRenderer.renderBlock(PDFRenderer.java:13
29)
        at org.apache.fop.render.AbstractRenderer.renderBlocks(AbstractRenderer.
java:526)
        at org.apache.fop.render.AbstractRenderer.renderBlock(AbstractRenderer.j
ava:573)
        at org.apache.fop.render.pdf.PDFRenderer.renderBlock(PDFRenderer.java:13
29)
        at org.apache.fop.render.AbstractRenderer.renderBlocks(AbstractRenderer.
java:526)

Thanks,
Vsyamala

 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Out of Memory error

Griffin,Sean
Vsyamala,
There are a variety of factors that could contribute to your OOM error, including, most significantly, the size of your page sequences within you FO, the types of images that you're embedding, the version of FOP that you're using, and whether you're writing to the PDF in memory or into a file.  300 images @ 48KB each doesn't sound extreme, but a 45MB FO file is pretty big.  It's too hard to say whether a 45MB FO file with 300 images at 48KB each should be able to run in 2GB memory...there's just too many variables that aren't mentioned.  So I would recommend a divide-and-conquer approach.  Remove the image references from the FO and try again.  Still run out of memory or come close to it?  If so, it must not be the images.  Different versions of FOP handle images in different ways.  For example, in v0.20.5 (maybe 0.95 as well), JPG images were directly embedded into the PDF whereas PNG images were transcoded into bitmaps and compressed into the PDF.  As a consequence, JPG images used a lot less processor power but created, generally, larger PDFs.

I just noticed from your stack trace that it appears you're writing to a ByteArrayOutputStream.  That could be a big problem right there.  You're wanting to store the entire 12+MB PDF in memory in addition to the memory needed for FOP to function?  A much better option would be to write that file to disk.  Likewise, are you feeding the FO into FOP from a memory stream, a file stream, or as the result of an XML/XSLT transformation?  Of all options, if you're rendering files this big, you should use the FileInputStream method.

Sean

-----Original Message-----
From: vsyamala [mailto:[hidden email]]
Sent: Thursday, December 11, 2008 1:10 PM
To: [hidden email]
Subject: Out of Memory error


Hi,

I am trying to generate PDF from 45MB .fo file, fop produces a pdf of about
12MB and it fails with "out of memory" errors. XMx and XMs options are set
for 2048MB(that's max I could go on the machine). And this .fo file is
referring to about 300 images of appx 48KB each. FOP should be able to
generate PDF. I am not sure if images are causing this issue. Does anyone
know if images are the issue? Here is the error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.io.ByteArrayOutputStream.write(Unknown Source)
        at java.io.OutputStream.write(Unknown Source)
        at org.apache.fop.pdf.PDFStream.add(PDFStream.java:60)
        at
org.apache.fop.render.pdf.PDFRenderer.concatenateTransformationMatrix
(PDFRenderer.java:839)
        at
org.apache.fop.render.AbstractPathOrientedRenderer.renderReferenceAre
a(AbstractPathOrientedRenderer.java:539)
        at
org.apache.fop.render.AbstractRenderer.renderBlock(AbstractRenderer.j
ava:560)
        at
org.apache.fop.render.pdf.PDFRenderer.renderBlock(PDFRenderer.java:13
29)
        at
org.apache.fop.render.AbstractRenderer.renderBlocks(AbstractRenderer.
java:526)
        at
org.apache.fop.render.AbstractRenderer.renderBlock(AbstractRenderer.j
ava:573)
        at
org.apache.fop.render.pdf.PDFRenderer.renderBlock(PDFRenderer.java:13
29)
        at
org.apache.fop.render.AbstractRenderer.renderBlocks(AbstractRenderer.
java:526)

Thanks,
Vsyamala

 
--
View this message in context: http://www.nabble.com/Out-of-Memory-error-tp20962360p20962360.html
Sent from the FOP - Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

----------------------------------------------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Out of Memory error

vsyamala
Hi Sean,

Thanks for responding. I am using FOP 0.95 version and generating .fo file from XSL transformation using xsltproc processor:

xsltproc --nonet --novalid --output Part5.fo PDFPublish_External.xsl PDF_Book_Part5.xml

And then using Fop 0.95 version to convert .fo file to .pdf

call Fop -q -r -c fop.xconf Part5.fo PDF_Part5.pdf

I am not sure whether I am using memory stream or file stream. How do I check that? How should I use the FileInputStream method or write to a file instead of disk?

Thanks,
Vsyamala


Griffin,Sean wrote
Vsyamala,
There are a variety of factors that could contribute to your OOM error, including, most significantly, the size of your page sequences within you FO, the types of images that you're embedding, the version of FOP that you're using, and whether you're writing to the PDF in memory or into a file.  300 images @ 48KB each doesn't sound extreme, but a 45MB FO file is pretty big.  It's too hard to say whether a 45MB FO file with 300 images at 48KB each should be able to run in 2GB memory...there's just too many variables that aren't mentioned.  So I would recommend a divide-and-conquer approach.  Remove the image references from the FO and try again.  Still run out of memory or come close to it?  If so, it must not be the images.  Different versions of FOP handle images in different ways.  For example, in v0.20.5 (maybe 0.95 as well), JPG images were directly embedded into the PDF whereas PNG images were transcoded into bitmaps and compressed into the PDF.  As a consequence, JPG images used a lot less processor power but created, generally, larger PDFs.

I just noticed from your stack trace that it appears you're writing to a ByteArrayOutputStream.  That could be a big problem right there.  You're wanting to store the entire 12+MB PDF in memory in addition to the memory needed for FOP to function?  A much better option would be to write that file to disk.  Likewise, are you feeding the FO into FOP from a memory stream, a file stream, or as the result of an XML/XSLT transformation?  Of all options, if you're rendering files this big, you should use the FileInputStream method.

Sean

-----Original Message-----
From: vsyamala [mailto:vsyamala@gmail.com]
Sent: Thursday, December 11, 2008 1:10 PM
To: fop-users@xmlgraphics.apache.org
Subject: Out of Memory error


Hi,

I am trying to generate PDF from 45MB .fo file, fop produces a pdf of about
12MB and it fails with "out of memory" errors. XMx and XMs options are set
for 2048MB(that's max I could go on the machine). And this .fo file is
referring to about 300 images of appx 48KB each. FOP should be able to
generate PDF. I am not sure if images are causing this issue. Does anyone
know if images are the issue? Here is the error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.io.ByteArrayOutputStream.write(Unknown Source)
        at java.io.OutputStream.write(Unknown Source)
        at org.apache.fop.pdf.PDFStream.add(PDFStream.java:60)
        at
org.apache.fop.render.pdf.PDFRenderer.concatenateTransformationMatrix
(PDFRenderer.java:839)
        at
org.apache.fop.render.AbstractPathOrientedRenderer.renderReferenceAre
a(AbstractPathOrientedRenderer.java:539)
        at
org.apache.fop.render.AbstractRenderer.renderBlock(AbstractRenderer.j
ava:560)
        at
org.apache.fop.render.pdf.PDFRenderer.renderBlock(PDFRenderer.java:13
29)
        at
org.apache.fop.render.AbstractRenderer.renderBlocks(AbstractRenderer.
java:526)
        at
org.apache.fop.render.AbstractRenderer.renderBlock(AbstractRenderer.j
ava:573)
        at
org.apache.fop.render.pdf.PDFRenderer.renderBlock(PDFRenderer.java:13
29)
        at
org.apache.fop.render.AbstractRenderer.renderBlocks(AbstractRenderer.
java:526)

Thanks,
Vsyamala

 
--
View this message in context: http://www.nabble.com/Out-of-Memory-error-tp20962360p20962360.html
Sent from the FOP - Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org

----------------------------------------------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Out of Memory error

Griffin,Sean
Vsyamala,
I was assuming you were embedding FOP into your application and calling directly through Java.  Since it appears you're launching from the command-line, it's up to the FOP command-line program on how it streams the output.  Since you're specifying file names as your I/O, it stands to reason that you'd be reading from/writing to File I/O streams, so the use of a ByteArrayOutputStream in the PDFStream.add() method might be a red herring and perfectly normal.

If you haven't already, you'll want to read this, which may point you in a direction to finding the source of your problem: http://xmlgraphics.apache.org/fop/0.94/running.html#memory.

Sean


-----Original Message-----
From: vsyamala [mailto:[hidden email]]
Sent: Friday, December 12, 2008 8:12 AM
To: [hidden email]
Subject: RE: Out of Memory error


Hi Sean,

Thanks for responding. I am using FOP 0.95 version and generating .fo file
from XSL transformation using xsltproc processor:

xsltproc --nonet --novalid --output Part5.fo PDFPublish_External.xsl
PDF_Book_Part5.xml

And then using Fop 0.95 version to convert .fo file to .pdf

call Fop -q -r -c fop.xconf Part5.fo PDF_Part5.pdf

I am not sure whether I am using memory stream or file stream. How do I
check that? How should I use the FileInputStream method or write to a file
instead of disk?

Thanks,
Vsyamala

----------------------------------------------------------------------
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Out of Memory error

Andreas L. Delmelle
On 12 Dec 2008, at 15:28, Griffin,Sean wrote:

Hi Sean, Vsyamala,

> I was assuming you were embedding FOP into your application and  
> calling directly through Java.  Since it appears you're launching  
> from the command-line, it's up to the FOP command-line program on  
> how it streams the output.  Since you're specifying file names as  
> your I/O, it stands to reason that you'd be reading from/writing to  
> File I/O streams, so the use of a ByteArrayOutputStream in the  
> PDFStream.add() method might be a red herring and perfectly normal.

Indeed. In general, a stack trace is next to useless for an OOMError.  
It will only tell you at what point in the process the error occurred,  
but it doesn't really say anything about the actual cause.
If the error were to occur during a StringBuffer.append(), chances  
would be very slim that it's actually the StringBuffer that is  
responsible. It just means that append() needed to allocate a few  
extra bytes of memory, but other objects/code already used up the  
available heap space.

In this case, the PDFStream.add() method is FOP-internal, and IIC, no  
cause for concern. With a FO file of 45 MB, I'm suspecting the issue  
is caused by the fact that the entire document consists of a single  
fo:page-sequence, which is a very well-known limitation at the moment.  
If that is the case, the only workaround would be to introduce more  
structure in the document, so that you get sequences of 10-15 pages  
max. Well, even 40-50 pages should work nicely, and with 2GB of heap,  
I guess this could even be a lot more...


Cheers

Andreas

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Out of Memory error

vsyamala
Thank you Sean and Andreas. I will look into those possibilities.

Vsyamala

Andreas Delmelle-2 wrote
On 12 Dec 2008, at 15:28, Griffin,Sean wrote:

Hi Sean, Vsyamala,

> I was assuming you were embedding FOP into your application and  
> calling directly through Java.  Since it appears you're launching  
> from the command-line, it's up to the FOP command-line program on  
> how it streams the output.  Since you're specifying file names as  
> your I/O, it stands to reason that you'd be reading from/writing to  
> File I/O streams, so the use of a ByteArrayOutputStream in the  
> PDFStream.add() method might be a red herring and perfectly normal.

Indeed. In general, a stack trace is next to useless for an OOMError.  
It will only tell you at what point in the process the error occurred,  
but it doesn't really say anything about the actual cause.
If the error were to occur during a StringBuffer.append(), chances  
would be very slim that it's actually the StringBuffer that is  
responsible. It just means that append() needed to allocate a few  
extra bytes of memory, but other objects/code already used up the  
available heap space.

In this case, the PDFStream.add() method is FOP-internal, and IIC, no  
cause for concern. With a FO file of 45 MB, I'm suspecting the issue  
is caused by the fact that the entire document consists of a single  
fo:page-sequence, which is a very well-known limitation at the moment.  
If that is the case, the only workaround would be to introduce more  
structure in the document, so that you get sequences of 10-15 pages  
max. Well, even 40-50 pages should work nicely, and with 2GB of heap,  
I guess this could even be a lot more...


Cheers

Andreas

---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
Loading...