Test if Close(Report) is SLOW on your System? – Mine 100x slower than C6

CarlBarnes · September 22, 2021, 2:35pm

I noticed when large 1000+ page Reports closed Preview there was an obvious pause of 10+ seconds before focus returned to the calling window. I traced this down to CLOSE(Report) which deletes the WMF files in the Temp folder. I added code before Close(Report) to delete files myself and the result was Close(Report) dropped from 10 to 1 second.

Attached project will let you time Report(close) on your system.

ReportCloseSlowTest.zip (9.1 KB)

Please enter your results in the poll and reply with screen captures of your tests. I’m curious how long it takes other to delete 1000 files? Note the Clarion version is in the window caption.

How much faster is API Delete before Close(Report)

2x to 9x faster
10x to 25x faster
26x to 50x faster
51x to 75x faster
76x faster or more
Same Speed
API Delete is Slower

0 voters

CarlBarnes · September 22, 2021, 2:48pm

I did show this on the Wednesday Clarion Live Open Webinar 9/22/21 at 4:30.

Test window looks like below. You can see on my system API delete is about 90x faster at 1000 pages.

Click “1. RTL Normal” to test time to Close(Report) which shows in the Normal column.

Click “2. API Delete” and it will run code like this that deletes all preview files before Close.

ReportCloseTempDeleteAPI(PreviewQ)
CLOSE(Report) 

ReportCloseTempDeleteAPI   PROCEDURE(QUEUE RptPreviewQ)
QX    LONG,AUTO 
cName CSTRING(261),AUTO   
  CODE
  LOOP QX=1 TO RECORDS(RptPreviewQ)
       GET(RptPreviewQ,QX)
       cName=CLIP(RptPreviewQ) 
       DeleteFile(cName)
  END
  RETURN
  !FYI DeleteFile(*CSTRING lpFileName ),BOOL,PROC,RAW,PASCAL,DLL(1),name('DeleteFileA')

“No Preview” skips displaying the Preview Window so the testing process is faster.

“Use RTL Remove” cause test #2 to use Remove() instead of API Delete. This proves that the RTL Remove() function is dreadfully slow. IMO that occured in C8 when it was changed to use SHFileOperation(). Below I highlighted one test where you can see Remove() takes 12.7 seconds versus 13.3 for Close(Report) so same speed and no where near as fast as API Delete.

noyantis · September 22, 2021, 4:04pm

The savings do speak for themselves…

Thanks Carl

Bostjan_Laba · September 22, 2021, 4:10pm

100x times faster (but without RTL Remove()).

CarlBarnes · September 22, 2021, 4:13pm

The “Use RTL Remove” box is to prove that the RTL Remove() is the slow thing. With that box checked you should see times similar to the #1 button Normal Close(Report).

Thanks for posting!

Bostjan_Laba · September 22, 2021, 4:25pm

This is huge. I wish there was some global template that would use the faster way of deleting files instead of “classic” with rtl remove().

CarlBarnes · September 22, 2021, 5:11pm

I use Legacy so it was easy for me to modify the shipping GReport.TPW.

#EMBED(%BeforeClosingReport,'Before Closing Report')

Inserted call to ReportPreviewPurge(PrintPreviewQueue) only #IF(%EnablePrintPreview) 

CLOSE(%Report)
#EMBED(%AfterClosingReport,'After Closing Report')
#IF(%EnablePrintPreview)
FREE(PrintPreviewQueue)
#ENDIF

I wanted my delete code on the line right before Close(Report) so I modified the shipping TPW. I did not want to risk using an #AT(%BeforeClosingReport),LAST in case there was other template code there trying to grab the WMFs. The #AT would probably be fine and allow a single Global Extension to apply to all reports.

In ABC the Close(Report) is buried in ReportManager.TakeCloseEvent(). It looks easy to modify ABReport.CLW and very nice it applies to ALL ABC reports. You just have to do it each new release, but it is worth it for the speed improvement.

TakeCloseEvent() was the wrong place, thank you Bruce, because it calls.AskPreview() which always does Free( PreviewQ ). So the files must be deleted in ReportManager.AskPreview before the Free(Q). I do that calling my new method DeleteTempFilesFast_Carl()

! ---------------------- ABReport.CLW ------------------

ReportManager.AskPreview PROCEDURE()
RetValue   BYTE,AUTO
  CODE
  RetValue = SELF.EndReport()
  IF SELF.Report &= NULL OR SELF.Response <> RequestCompleted OR SELF.OpenFailed

    !Report cancelled by user so there are files to delete if SELF.Response  <> RequestCompleted 
    SELF.DeleteTempFilesFast_Carl()  !<-- ADD -- Fast Close new code

    RETURN
  END
  IF RetValue = Level:Benign
    IF SELF.SkipPreview OR SELF.Preview &= NULL OR SELF.Preview.Display (SELF.Zoom)
      SELF.PrintReport()
    ELSE
      SELF.CancelPrintReport()
    END
  END

  SELF.DeleteTempFilesFast_Carl()  !<-- ADD -- Fast Close new code
  IF NOT SELF.Preview &= NULL
    FREE (SELF.Preview.ImageQueue)
  ELSE
    FREE (SELF.PreviewQueue)
  END
  RETURN

You need to add this new method to ABReport .INC and .CLW that deletes the files:

ReportManager.DeleteTempFilesFast_Carl PROCEDURE()!,VIRTUAL !<-- ADD -- new method
PreviewQ &PreviewQueue
Idx   LONG,AUTO
cName CSTRING(261),AUTO   
  CODE
  IF SELF.Report &= NULL OR SELF.OpenFailed THEN RETURN.  !Never got started 
  IF NOT SELF.Preview &= NULL
    PreviewQ &= SELF.Preview.ImageQueue
  ELSE
    PreviewQ &= SELF.PreviewQueue
  END
  IF PreviewQ &= NULL THEN RETURN. 
  LOOP Idx=1 TO RECORDS(PreviewQ)
     GET(PreviewQ,Idx)
     cName=CLIP(PreviewQ) 
     DeleteFile_Carl(cName)
  END
  RETURN
  
!TODO: Report.CLW Add to MODULE('WinApi') --> DeleteFile_Carl(*CSTRING lpFileName ),BOOL,PROC,RAW,PASCAL,DLL(1),NAME('DeleteFileA')  
!TODO: Report.INC Add to ReportManager    --> DeleteTempFilesFast_Carl PROCEDURE(),VIRTUAL
!Simpler: instead add DeleteTempFilesFast_Carl(ReportManager SELF) to the CLW Map so Private

This code compiles but has had minimal testing. I have been running similar code in Legacy for a year on many systems with no issues.

JaredDavis · September 22, 2021, 6:41pm

Hi,

Do you see the same time difference on remove vs api delete? Does the issue only show up on the close(report) usage?

CarlBarnes · September 22, 2021, 7:29pm

Yes REMOVE() is 100x slower than API DeleteFile. No its not just a Close(Report) issue. If you are deleting 100’s of files using REMOVE() will be slow.

You can test times for Remove() vs Delete file using the View Temp button. The PURGE button deletes the files on that window. Uncheck API Delete to use REMOVE(). Once you press Purge the files are gone so you’ll need generate another Preview to compare checked vs unchecked:

Alla · September 22, 2021, 8:25pm

I COME FROM integrating the API function into my APPs
The result is very good
I also thought of copying my previewqueue in PDF format
Is there a function to copy Image.wmf to PDF

Thank you very much for your initiative and your very precious help

Alla

JaredDavis · September 22, 2021, 8:52pm

Wow. Thanks.

I’ve had anti virus scanners block file deletes.
create file -> do something -> delete file
The antivirus was scanning it and blocked the delete.

Have you looked at the remove() operation api calls to see what it is doing?

Bruce · September 23, 2021, 7:31am

there are a number of tools available which will convert a queue of WMF files into a PDF document. The most obvious is the ReportToPDF functionality built into Clarion (EE?) itself - but there’s also wPDF and the now-not-available TrackerPDF. There are others as well.

Cheers
Bruce

Bruce · September 23, 2021, 2:14pm

Thanks for the whole thread Carl - I’ve implemented a PremiereDeleteFilesInPrintPreviewQueue function in Premiere (build 3.36) . The template will automatically (by default, you can override at global or local level) make the call for you for ABC and Legacy reports. You can also call the function in hand-code if you like.

cheers
Bruce

Bostjan_Laba · September 23, 2021, 2:51pm

Now this I call FAST RESPONSE

Thanks Carl and Bruce

CarlBarnes · September 23, 2021, 8:29pm

I stepped it in the Debugger and confirmed REMOVE( String ) calls SHFileOperation . That function has a zillion features that make it slow. For a single file the function DeleteFileA() should be used as it was prior to C8 adding a second parameter to REMOVE(, Options).

JaredDavis · September 23, 2021, 8:49pm

Thanks. Do the deleted files end up in the Recycle bin with SHFileOperationA?

I can’t check right now.

Rick_UpperPark · September 23, 2021, 9:30pm

That’s an option of the API, but I don’t believe Clarion is calling it that way.

CarlBarnes · September 23, 2021, 9:38pm

I checked and Close(Report) does not put files in Recycle bin.

The API has flag FOF_ALLOWUNDO to recycle files but I don’t see that as an option flag for REMOVE():

REMOVE:FILESONLY         EQUATE(01h)   ! Remove normal files only if name has wildcards
REMOVE:RECURSIVE         EQUATE(02h)   ! Process nested directories recursively
REMOVE:PROGRESS          EQUATE(04h)   ! Show progress indicator
REMOVE:CONFIRM           EQUATE(08h)   ! Ask confirmations

The enhanced REMOVE deletes files using wildcards and there is no option to turn it off.

JaredDavis · September 23, 2021, 10:47pm

Thanks. I was fishing to see if this was the source of another issue I have where the exe pauses for a couple of seconds. It freezes long enough for a watchdog to trigger. Then it gets ugly.

Oggy · September 26, 2021, 8:35am

Tested on the M2 “speedy” disk… Differences is massive… testing