[jira] [Updated] (FOP-2874) [PATCH] FOP conserve memory policy fails in multi-threaded environment

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (FOP-2874) [PATCH] FOP conserve memory policy fails in multi-threaded environment

JIRA jira@apache.org

     [ https://issues.apache.org/jira/browse/FOP-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Piyush Khandelwal updated FOP-2874:
-----------------------------------
    Description:
When conserve memory policy is enabled 
Code Snippet 1:
{code:java}
foUserAgent.setConserveMemoryPolicy(true);
{code}
in multi-threaded environment, it throws following exception
{code:java}
SEVERE: Error while serializing page 154. Reason: java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.createTempFile(ResourceResolverFactory.java:187)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.getOutputStream(ResourceResolverFactory.java:199)
        at org.apache.fop.apps.io.ResourceResolverFactory$TempAwareResourceResolver.getOutputStream(ResourceResolverFactory.java:159)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultResourceResolver.getOutputStream(ResourceResolverFactory.java:126)
        at org.apache.fop.apps.io.InternalResourceResolver.getOutputStream(InternalResourceResolver.java:103)
        at org.apache.fop.area.CachedRenderPagesModel.savePage(CachedRenderPagesModel.java:132)
        at org.apache.fop.area.CachedRenderPagesModel.checkPreparedPages(CachedRenderPagesModel.java:112)
        at org.apache.fop.area.RenderPagesModel.addPage(RenderPagesModel.java:146)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.finishPage(AbstractPageSequenceLayoutManager.java:316)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.finishPage(PageSequenceLayoutManager.java:243)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.makeNewPage(AbstractPageSequenceLayoutManager.java:287)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.makeNewPage(PageSequenceLayoutManager.java:192)
        at org.apache.fop.layoutmgr.PageBreaker.handleBreakTrait(PageBreaker.java:633)
        at org.apache.fop.layoutmgr.PageBreaker.startPart(PageBreaker.java:511)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:659)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:604)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:599)
        at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:338)
        at org.apache.fop.layoutmgr.AbstractBreaker.doLayout(AbstractBreaker.java:458)
        at org.apache.fop.layoutmgr.PageBreaker.doLayout(PageBreaker.java:112)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.activateLayout(PageSequenceLayoutManager.java:143)
        at org.apache.fop.area.AreaTreeHandler.endPageSequence(AreaTreeHandler.java:267)
        at org.apache.fop.fo.pagination.PageSequence.endOfNode(PageSequence.java:130)
        at org.apache.fop.fo.FOTreeBuilder$MainFOHandler.endElement(FOTreeBuilder.java:360)
        at org.apache.fop.fo.FOTreeBuilder.endElement(FOTreeBuilder.java:190)
        at org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1102)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:485)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}

*Cause:*
CachedRenderPagesModel.java creates URIs using following code:

Code Snippet 2:
{code:java}
 String fname = "fop-page-" + page.getPageIndex() + ".ser";
 URI tempURI = tempBaseURI.resolve(fname);
{code}

CachedRenderPagesModel.java uses createTempFile method inside ResourceResolverFactory.java to create serialized PageViewport objects in the tmp directory.

Code Snippet 3:
{code:java}
 private File createTempFile(String path) throws IOException {
            File tempFile = File.createTempFile(path, ".fop.tmp");
            File oldFile = (File)this.tempFiles.put(path, tempFile);
            if (oldFile != null) {
                String errorMsg = oldFile.getAbsolutePath() + " has been already created for " + path;
                boolean newTempDeleted = tempFile.delete();
                if (!newTempDeleted) {
                    errorMsg = errorMsg + ". " + tempFile.getAbsolutePath() + " was not deleted.";
                }

                throw new IOException(errorMsg);
            } else {
                return tempFile;
            }
        }
{code}
 
Here the code is using a concurrent HashMap to map the URI path created in Code Snippet 2 with the temp file created and if it get replaced while storing, it throws an exception.

In multi threaded environment, different threads can create same path in code snippet 2 and will always replace the entry created in map in code snippet 3, and will throw the exception.

*Solution:*
The solution is to make key for the map (in this case the path resolved in code snippet 2) unique across thread, so that it doesn't get replace by a different thread.
We can use SecureRandom to make this path unique across different threads in code snippet 2.
{code:java}
 String fname = "fop-page-" + page.getPageIndex() + "-" + sr.nextLong() + ".ser";
 URI tempURI = tempBaseURI.resolve(fname);
{code}
Here sr is secure random object.
This makes sure that we have a unique key for the map.

  was:
When conserve memory policy is enabled 
Code Snippet 1:
{code:java}
foUserAgent.setConserveMemoryPolicy(true);
{code}
in multi-threaded environment, it throws following exception
{code:java}
SEVERE: Error while serializing page 154. Reason: java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.createTempFile(ResourceResolverFactory.java:187)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.getOutputStream(ResourceResolverFactory.java:199)
        at org.apache.fop.apps.io.ResourceResolverFactory$TempAwareResourceResolver.getOutputStream(ResourceResolverFactory.java:159)
        at org.apache.fop.apps.io.ResourceResolverFactory$DefaultResourceResolver.getOutputStream(ResourceResolverFactory.java:126)
        at org.apache.fop.apps.io.InternalResourceResolver.getOutputStream(InternalResourceResolver.java:103)
        at org.apache.fop.area.CachedRenderPagesModel.savePage(CachedRenderPagesModel.java:132)
        at org.apache.fop.area.CachedRenderPagesModel.checkPreparedPages(CachedRenderPagesModel.java:112)
        at org.apache.fop.area.RenderPagesModel.addPage(RenderPagesModel.java:146)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.finishPage(AbstractPageSequenceLayoutManager.java:316)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.finishPage(PageSequenceLayoutManager.java:243)
        at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.makeNewPage(AbstractPageSequenceLayoutManager.java:287)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.makeNewPage(PageSequenceLayoutManager.java:192)
        at org.apache.fop.layoutmgr.PageBreaker.handleBreakTrait(PageBreaker.java:633)
        at org.apache.fop.layoutmgr.PageBreaker.startPart(PageBreaker.java:511)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:659)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:604)
        at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:599)
        at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:338)
        at org.apache.fop.layoutmgr.AbstractBreaker.doLayout(AbstractBreaker.java:458)
        at org.apache.fop.layoutmgr.PageBreaker.doLayout(PageBreaker.java:112)
        at org.apache.fop.layoutmgr.PageSequenceLayoutManager.activateLayout(PageSequenceLayoutManager.java:143)
        at org.apache.fop.area.AreaTreeHandler.endPageSequence(AreaTreeHandler.java:267)
        at org.apache.fop.fo.pagination.PageSequence.endOfNode(PageSequence.java:130)
        at org.apache.fop.fo.FOTreeBuilder$MainFOHandler.endElement(FOTreeBuilder.java:360)
        at org.apache.fop.fo.FOTreeBuilder.endElement(FOTreeBuilder.java:190)
        at org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1102)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:485)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
{code}

*Cause:*
CachedRenderPagesModel.java creates URIs using following code:

Code Snippet 2:
{code:java}
 String fname = "fop-page-" + page.getPageIndex() + ".ser";
 URI tempURI = tempBaseURI.resolve(fname);
{code}

CachedRenderPagesModel.java uses createTempFile method inside ResourceResolverFactory.java to create serialized PageViewport objects in the tmp directory.

Code Snippet 3:
{code:java}
 private File createTempFile(String path) throws IOException {
            File tempFile = File.createTempFile(path, ".fop.tmp");
            File oldFile = (File)this.tempFiles.put(path, tempFile);
            if (oldFile != null) {
                String errorMsg = oldFile.getAbsolutePath() + " has been already created for " + path;
                boolean newTempDeleted = tempFile.delete();
                if (!newTempDeleted) {
                    errorMsg = errorMsg + ". " + tempFile.getAbsolutePath() + " was not deleted.";
                }

                throw new IOException(errorMsg);
            } else {
                return tempFile;
            }
        }
{code}
 
Here the code is using a concurrent HashMap to map the URI path created in Code Snippet 2 with the temp file created and if it get replaced while storing, it throws an exception.

In multi threaded environment, different threads can create same path in code snippet 2 and will always replace the entry created in map in code snippet 3, and will throw the exception.

*Solution:*
Use SecureRandom to make the URL unique across different threads, created in code snippet 2. See patch for fix


> [PATCH] FOP conserve memory policy fails in multi-threaded environment
> ----------------------------------------------------------------------
>
>                 Key: FOP-2874
>                 URL: https://issues.apache.org/jira/browse/FOP-2874
>             Project: FOP
>          Issue Type: Bug
>    Affects Versions: 2.2, 2.3
>            Reporter: Piyush Khandelwal
>            Priority: Critical
>         Attachments: cached-model.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When conserve memory policy is enabled 
> Code Snippet 1:
> {code:java}
> foUserAgent.setConserveMemoryPolicy(true);
> {code}
> in multi-threaded environment, it throws following exception
> {code:java}
> SEVERE: Error while serializing page 154. Reason: java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
> java.io.IOException: \tmp\fop-page-153.ser5056538250865196860.fop.tmp has been already created for /fop-page-153.ser
> at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.createTempFile(ResourceResolverFactory.java:187)
> at org.apache.fop.apps.io.ResourceResolverFactory$DefaultTempResourceResolver.getOutputStream(ResourceResolverFactory.java:199)
> at org.apache.fop.apps.io.ResourceResolverFactory$TempAwareResourceResolver.getOutputStream(ResourceResolverFactory.java:159)
> at org.apache.fop.apps.io.ResourceResolverFactory$DefaultResourceResolver.getOutputStream(ResourceResolverFactory.java:126)
> at org.apache.fop.apps.io.InternalResourceResolver.getOutputStream(InternalResourceResolver.java:103)
> at org.apache.fop.area.CachedRenderPagesModel.savePage(CachedRenderPagesModel.java:132)
> at org.apache.fop.area.CachedRenderPagesModel.checkPreparedPages(CachedRenderPagesModel.java:112)
> at org.apache.fop.area.RenderPagesModel.addPage(RenderPagesModel.java:146)
> at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.finishPage(AbstractPageSequenceLayoutManager.java:316)
> at org.apache.fop.layoutmgr.PageSequenceLayoutManager.finishPage(PageSequenceLayoutManager.java:243)
> at org.apache.fop.layoutmgr.AbstractPageSequenceLayoutManager.makeNewPage(AbstractPageSequenceLayoutManager.java:287)
> at org.apache.fop.layoutmgr.PageSequenceLayoutManager.makeNewPage(PageSequenceLayoutManager.java:192)
> at org.apache.fop.layoutmgr.PageBreaker.handleBreakTrait(PageBreaker.java:633)
> at org.apache.fop.layoutmgr.PageBreaker.startPart(PageBreaker.java:511)
> at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:659)
> at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:604)
> at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:599)
> at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:338)
> at org.apache.fop.layoutmgr.AbstractBreaker.doLayout(AbstractBreaker.java:458)
> at org.apache.fop.layoutmgr.PageBreaker.doLayout(PageBreaker.java:112)
> at org.apache.fop.layoutmgr.PageSequenceLayoutManager.activateLayout(PageSequenceLayoutManager.java:143)
> at org.apache.fop.area.AreaTreeHandler.endPageSequence(AreaTreeHandler.java:267)
> at org.apache.fop.fo.pagination.PageSequence.endOfNode(PageSequence.java:130)
> at org.apache.fop.fo.FOTreeBuilder$MainFOHandler.endElement(FOTreeBuilder.java:360)
> at org.apache.fop.fo.FOTreeBuilder.endElement(FOTreeBuilder.java:190)
> at org.apache.xalan.transformer.TransformerIdentityImpl.endElement(TransformerIdentityImpl.java:1102)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2967)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
> at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
> at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:841)
> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:770)
> at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:485)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> *Cause:*
> CachedRenderPagesModel.java creates URIs using following code:
> Code Snippet 2:
> {code:java}
>  String fname = "fop-page-" + page.getPageIndex() + ".ser";
>  URI tempURI = tempBaseURI.resolve(fname);
> {code}
> CachedRenderPagesModel.java uses createTempFile method inside ResourceResolverFactory.java to create serialized PageViewport objects in the tmp directory.
> Code Snippet 3:
> {code:java}
>  private File createTempFile(String path) throws IOException {
>             File tempFile = File.createTempFile(path, ".fop.tmp");
>             File oldFile = (File)this.tempFiles.put(path, tempFile);
>             if (oldFile != null) {
>                 String errorMsg = oldFile.getAbsolutePath() + " has been already created for " + path;
>                 boolean newTempDeleted = tempFile.delete();
>                 if (!newTempDeleted) {
>                     errorMsg = errorMsg + ". " + tempFile.getAbsolutePath() + " was not deleted.";
>                 }
>                 throw new IOException(errorMsg);
>             } else {
>                 return tempFile;
>             }
>         }
> {code}
>  
> Here the code is using a concurrent HashMap to map the URI path created in Code Snippet 2 with the temp file created and if it get replaced while storing, it throws an exception.
> In multi threaded environment, different threads can create same path in code snippet 2 and will always replace the entry created in map in code snippet 3, and will throw the exception.
> *Solution:*
> The solution is to make key for the map (in this case the path resolved in code snippet 2) unique across thread, so that it doesn't get replace by a different thread.
> We can use SecureRandom to make this path unique across different threads in code snippet 2.
> {code:java}
>  String fname = "fop-page-" + page.getPageIndex() + "-" + sr.nextLong() + ".ser";
>  URI tempURI = tempBaseURI.resolve(fname);
> {code}
> Here sr is secure random object.
> This makes sure that we have a unique key for the map.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)