[jira] [Updated] (FOP-2874) [PATCH] FOP conserve memory policy fails in multi-threaded environment

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (FOP-2874) [PATCH] FOP conserve memory policy fails in multi-threaded environment

simon steiner (Jira)

     [ https://issues.apache.org/jira/browse/FOP-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Piyush Khandelwal updated FOP-2874:
-----------------------------------
    Description:
When conserve memory policy is enabled 
 Code Snippet 1:
{code:java}
foUserAgent.setConserveMemoryPolicy(true);
{code}
in multi-threaded environment, it throws following exception
{code:java}
SEVERE: Error while serializing page 154. Reason: java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.createTempFile(ResourceResolverFactory.java:187)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.getOutputStream(ResourceResolverFactory.java:199)
        at org.apache.fop.apps.io.ResourceResolverFactory$TempAwareResourceResolver.getOutputStream(ResourceResolverFactory.java:159)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultResourceResolver.getOutputStream(ResourceResolverFactory.java:126)
        at org.apache.fop.apps.io.InternalResourceResolver.getOutputStream(InternalResourceResolver.java:103)
        at org.apache.fop.area.CachedRenderPagesModel.savePage(CachedRenderPagesModel.java:132)
        at org.apache.fop.area.CachedRenderPagesModel.checkPreparedPages(CachedRenderPagesModel.java:112)
        at org.apache.fop.area.RenderPagesModel.addPage(RenderPagesModel.java:146)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.finishPage(AbstractPageSequenceLayoutManager.java:316)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.finishPage(PageSequenceLayoutManager.java:243)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.makeNewPage(AbstractPageSequenceLayoutManager.java:287)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.makeNewPage(PageSequenceLayoutManager.java:192)
        at org.apache.fop.layoutmgr.PageBreaker.handleBreakTrait(PageBreaker.java:633)
        at org.apache.fop.layoutmgr.PageBreaker.startPart(PageBreaker.java:511)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:659)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:604)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:599)
        at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:338)
        at org.apache.fop.layoutmgr.AbstractBreaker.doLayout(AbstractBreaker.java:458)
        at org.apache.fop.layoutmgr.PageBreaker.doLayout(PageBreaker.java:112)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.activateLayout(PageSequenceLayoutManager.java:143)
        at org.apache.fop.area.AreaTreeHandler.endPageSequence(AreaTreeHandler.java:267)
        at org.apache.fop.fo.pagination.PageSequence.endOfNode(PageSequence.java:130)
        at org.apache.fop.fo.FOTreeBuilder$MainFOHandler.endElement(FOTreeBuilder.java:360)
        at org.apache.fop.fo.FOTreeBuilder.endElement(FOTreeBuilder.java:190)
        at org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1102)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:485)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}
*Cause:*
 CachedRenderPagesModel.java creates URIs using following code:

Code Snippet 2:
{code:java}
 String fname = "fop-page-" + page.getPageIndex() + ".ser";
 URI tempURI = tempBaseURI.resolve(fname);
{code}
CachedRenderPagesModel.java uses createTempFile method inside ResourceResolverFactory.java to create serialized PageViewport objects in the tmp directory.

Code Snippet 3:
{code:java}
 private File createTempFile(String path) throws IOException {
            File tempFile = File.createTempFile(path, ".fop.tmp");
            File oldFile = (File)this.tempFiles.put(path, tempFile);
            if (oldFile != null) {
                String errorMsg = oldFile.getAbsolutePath() + " has been already created for " + path;
                boolean newTempDeleted = tempFile.delete();
                if (!newTempDeleted) {
                    errorMsg = errorMsg + ". " + tempFile.getAbsolutePath() + " was not deleted.";
                }

                throw new IOException(errorMsg);
            } else {
                return tempFile;
            }
        }
{code}
Here the code is using a concurrent HashMap to map the URI path created in Code Snippet 2 with the temp file created and if it get replaced while storing, it throws an exception.

In multi threaded environment, different threads can create same path in code snippet 2 and will always replace the entry created in map in code snippet 3, and will throw the exception.

*Solution:*

There are multiple ways we can make the URI (being used as key in the map) unique.
 # We can use SecureRandom to generate  a radom number and append it to the URI.
 # Append thread name to the URI
 # Use combination of above two solutions i.e., append both in the URI
 
This will make sure that we get an unique path as key for the map across multiple threads and avoid this exception.
The patch uses a SecureRandom to make this path unique across different threads in code snippet 2.

{code:java}
 String fname = "fop-page-" + page.getPageIndex() + "-" + sr.nextLong() + ".ser";
 URI tempURI = tempBaseURI.resolve(fname);
{code}
Here sr is secure random object.


  was:
When conserve memory policy is enabled 
Code Snippet 1:
{code:java}
foUserAgent.setConserveMemoryPolicy(true);
{code}
in multi-threaded environment, it throws following exception
{code:java}
SEVERE: Error while serializing page 154. Reason: java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.createTempFile(ResourceResolverFactory.java:187)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.getOutputStream(ResourceResolverFactory.java:199)
        at org.apache.fop.apps.io.ResourceResolverFactory$TempAwareResourceResolver.getOutputStream(ResourceResolverFactory.java:159)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultResourceResolver.getOutputStream(ResourceResolverFactory.java:126)
        at org.apache.fop.apps.io.InternalResourceResolver.getOutputStream(InternalResourceResolver.java:103)
        at org.apache.fop.area.CachedRenderPagesModel.savePage(CachedRenderPagesModel.java:132)
        at org.apache.fop.area.CachedRenderPagesModel.checkPreparedPages(CachedRenderPagesModel.java:112)
        at org.apache.fop.area.RenderPagesModel.addPage(RenderPagesModel.java:146)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.finishPage(AbstractPageSequenceLayoutManager.java:316)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.finishPage(PageSequenceLayoutManager.java:243)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.makeNewPage(AbstractPageSequenceLayoutManager.java:287)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.makeNewPage(PageSequenceLayoutManager.java:192)
        at org.apache.fop.layoutmgr.PageBreaker.handleBreakTrait(PageBreaker.java:633)
        at org.apache.fop.layoutmgr.PageBreaker.startPart(PageBreaker.java:511)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:659)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:604)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:599)
        at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:338)
        at org.apache.fop.layoutmgr.AbstractBreaker.doLayout(AbstractBreaker.java:458)
        at org.apache.fop.layoutmgr.PageBreaker.doLayout(PageBreaker.java:112)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.activateLayout(PageSequenceLayoutManager.java:143)
        at org.apache.fop.area.AreaTreeHandler.endPageSequence(AreaTreeHandler.java:267)
        at org.apache.fop.fo.pagination.PageSequence.endOfNode(PageSequence.java:130)
        at org.apache.fop.fo.FOTreeBuilder$MainFOHandler.endElement(FOTreeBuilder.java:360)
        at org.apache.fop.fo.FOTreeBuilder.endElement(FOTreeBuilder.java:190)
        at org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1102)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:485)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}

*Cause:*
CachedRenderPagesModel.java creates URIs using following code:

Code Snippet 2:
{code:java}
 String fname = "fop-page-" + page.getPageIndex() + ".ser";
 URI tempURI = tempBaseURI.resolve(fname);
{code}

CachedRenderPagesModel.java uses createTempFile method inside ResourceResolverFactory.java to create serialized PageViewport objects in the tmp directory.

Code Snippet 3:
{code:java}
 private File createTempFile(String path) throws IOException {
            File tempFile = File.createTempFile(path, ".fop.tmp");
            File oldFile = (File)this.tempFiles.put(path, tempFile);
            if (oldFile != null) {
                String errorMsg = oldFile.getAbsolutePath() + " has been already created for " + path;
                boolean newTempDeleted = tempFile.delete();
                if (!newTempDeleted) {
                    errorMsg = errorMsg + ". " + tempFile.getAbsolutePath() + " was not deleted.";
                }

                throw new IOException(errorMsg);
            } else {
                return tempFile;
            }
        }
{code}
 
Here the code is using a concurrent HashMap to map the URI path created in Code Snippet 2 with the temp file created and if it get replaced while storing, it throws an exception.

In multi threaded environment, different threads can create same path in code snippet 2 and will always replace the entry created in map in code snippet 3, and will throw the exception.

*Solution:*
The solution is to make key for the map (in this case the path resolved in code snippet 2) unique across thread, so that it doesn't get replace by a different thread.
We can use SecureRandom to make this path unique across different threads in code snippet 2.
{code:java}
 String fname = "fop-page-" + page.getPageIndex() + "-" + sr.nextLong() + ".ser";
 URI tempURI = tempBaseURI.resolve(fname);
{code}
Here sr is secure random object.
This makes sure that we have a unique key for the map.


> [PATCH] FOP conserve memory policy fails in multi-threaded environment
> ----------------------------------------------------------------------
>
>                 Key: FOP-2874
>                 URL: https://issues.apache.org/jira/browse/FOP-2874
>             Project: FOP
>          Issue Type: Bug
>    Affects Versions: 2.2, 2.3
>            Reporter: Piyush Khandelwal
>            Priority: Critical
>         Attachments: cached-model.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When conserve memory policy is enabled 
>  Code Snippet 1:
> {code:java}
> foUserAgent.setConserveMemoryPolicy(true);
> {code}
> in multi-threaded environment, it throws following exception
> {code:java}
> SEVERE: Error while serializing page 154. Reason: java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
> java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
> at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.createTempFile(ResourceResolverFactory.java:187)
> at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.getOutputStream(ResourceResolverFactory.java:199)
> at org.apache.fop.apps.io.ResourceResolverFactory$TempAwareResourceResolver.getOutputStream(ResourceResolverFactory.java:159)
> at org.apache.fop.apps.io.ResourceResolverFactory$DefaultResourceResolver.getOutputStream(ResourceResolverFactory.java:126)
> at org.apache.fop.apps.io.InternalResourceResolver.getOutputStream(InternalResourceResolver.java:103)
> at org.apache.fop.area.CachedRenderPagesModel.savePage(CachedRenderPagesModel.java:132)
> at org.apache.fop.area.CachedRenderPagesModel.checkPreparedPages(CachedRenderPagesModel.java:112)
> at org.apache.fop.area.RenderPagesModel.addPage(RenderPagesModel.java:146)
> at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.finishPage(AbstractPageSequenceLayoutManager.java:316)
> at org.apache.fop.layoutmgr.PageSequenceLayoutManager.finishPage(PageSequenceLayoutManager.java:243)
> at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.makeNewPage(AbstractPageSequenceLayoutManager.java:287)
> at org.apache.fop.layoutmgr.PageSequenceLayoutManager.makeNewPage(PageSequenceLayoutManager.java:192)
> at org.apache.fop.layoutmgr.PageBreaker.handleBreakTrait(PageBreaker.java:633)
> at org.apache.fop.layoutmgr.PageBreaker.startPart(PageBreaker.java:511)
> at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:659)
> at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:604)
> at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:599)
> at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:338)
> at org.apache.fop.layoutmgr.AbstractBreaker.doLayout(AbstractBreaker.java:458)
> at org.apache.fop.layoutmgr.PageBreaker.doLayout(PageBreaker.java:112)
> at org.apache.fop.layoutmgr.PageSequenceLayoutManager.activateLayout(PageSequenceLayoutManager.java:143)
> at org.apache.fop.area.AreaTreeHandler.endPageSequence(AreaTreeHandler.java:267)
> at org.apache.fop.fo.pagination.PageSequence.endOfNode(PageSequence.java:130)
> at org.apache.fop.fo.FOTreeBuilder$MainFOHandler.endElement(FOTreeBuilder.java:360)
> at org.apache.fop.fo.FOTreeBuilder.endElement(FOTreeBuilder.java:190)
> at org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1102)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:485)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *Cause:*
>  CachedRenderPagesModel.java creates URIs using following code:
> Code Snippet 2:
> {code:java}
>  String fname = "fop-page-" + page.getPageIndex() + ".ser";
>  URI tempURI = tempBaseURI.resolve(fname);
> {code}
> CachedRenderPagesModel.java uses createTempFile method inside ResourceResolverFactory.java to create serialized PageViewport objects in the tmp directory.
> Code Snippet 3:
> {code:java}
>  private File createTempFile(String path) throws IOException {
>             File tempFile = File.createTempFile(path, ".fop.tmp");
>             File oldFile = (File)this.tempFiles.put(path, tempFile);
>             if (oldFile != null) {
>                 String errorMsg = oldFile.getAbsolutePath() + " has been already created for " + path;
>                 boolean newTempDeleted = tempFile.delete();
>                 if (!newTempDeleted) {
>                     errorMsg = errorMsg + ". " + tempFile.getAbsolutePath() + " was not deleted.";
>                 }
>                 throw new IOException(errorMsg);
>             } else {
>                 return tempFile;
>             }
>         }
> {code}
> Here the code is using a concurrent HashMap to map the URI path created in Code Snippet 2 with the temp file created and if it get replaced while storing, it throws an exception.
> In multi threaded environment, different threads can create same path in code snippet 2 and will always replace the entry created in map in code snippet 3, and will throw the exception.
> *Solution:*
> There are multiple ways we can make the URI (being used as key in the map) unique.
>  # We can use SecureRandom to generate  a radom number and append it to the URI.
>  # Append thread name to the URI
>  # Use combination of above two solutions i.e., append both in the URI
>  
> This will make sure that we get an unique path as key for the map across multiple threads and avoid this exception.
> The patch uses a SecureRandom to make this path unique across different threads in code snippet 2.
> {code:java}
>  String fname = "fop-page-" + page.getPageIndex() + "-" + sr.nextLong() + ".ser";
>  URI tempURI = tempBaseURI.resolve(fname);
> {code}
> Here sr is secure random object.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)