Win10 C6 Directory() 8min 40 secs, C11 Directory() 31mins 47 seconds

C6 app on Win10, can scan the hard drive for all subfolders in 8min 40 seconds using logout/commit.
Same source code compiled in C11 on Win10, takes 31mins 47 seconds.

Any ideas why C11 takes more time?

TIA

Perhaps running on a small folder tree and ProcMon would tell you where the bottleneck is between the 2 versions.

I’d test a small folder so that your comparison can actually see something.

for example, Carl discovered this stuff about REMOVE()

Show or attach the source.

What does Logout/Commit have to do with using Directory() ?

1 Like

DeleteFile does not support wildcards in the parameter string. Clarion’s REMOVE - supports. If parameter of REMOVE is a name of directory, some extra checkings are required. In future Clarion REMOVE as well as some other file-oriented functions is also resolves mapped and SUBSTed drives, virtual folders, directory injections, etc. because this is related to access rights.

Obviously

LOGOUT(1, dirName)
DIRECTORY(dirQ, '*.*', ff_:DIRECTORY)
COMMIT

sorry, but I don’t understand the use of LOGOUT/COMMIT in that case.
Can you explaint it. thanks

1 Like

This is a topic starter’s lifehack.

Using logout/Commit to dump the Directory file queue into a tps file, there are lots of ways I’ve been trying this, like on a 100 second logout, and then if the commit code is encountered within 95 to 100 seconds, it automatically commits and things like that.

ie
Directory(File:Queue,Path,etc)
Counter = 0
Logout(100,TPSFile)
Loop Records(File:Queue) times
Counter +=1
Get(File:Queue,Counter)
TPSFile.Name = File:Queue.Name
Access:TPSFile.Insert
Commit

One thing I think is an overriding factor, is on XP theres a few thousand directories, on Win10 I have 160,000 subfolders, so I’m trying the
FindFirstFileA function (fileapi.h) - Win32 apps | Microsoft Learn
FindNextFileA function (fileapi.h) - Win32 apps | Microsoft Learn
api’s to see which is quicker, I dont know if the transacted api might be faster than the above or not, but I need to log all the files on various media, and hash their contents to spot when any changes occur.

I basically need to log every change that occurs on my PC’s to spot when stuff is being tampered with, like templates changed back to earlier versions and embed clw code changed, which is a form of resource burning me, a security services/military technique but also a criminal one. This is one of the reasons why I have been writing so many of my own templates, because a change in clw code can be explained as human error, templates that work in multiple apps, are harder to explain as human error where the template is used in multiple apps, and the generated clw code is different, highlighting the template may have been tampered with or the clw tampered with between the small window of opportunity that exists between generation and compilation.

A classic example of tampering with code which can alter the behaviour is this post


so if everything is being logged and hashed like using techniques similar to that seen in cryptocurrency to authenticate & log changes, it gets harder to insert backdoors as bugs. I know subversion is supposed to partly address this problem but its got its problems.

With Directory, I have to run through the directory file queue and strip out the old dos directory single and double dot and also the files because despite only using the attrib flag for directory, its still pulling in files, and there is a chance, havent counted that the avg number of files per folder have also gone up in Win10 compared to XP.

Hi Richard

first thing to determine is whether the time difference is due to the directory() command (which is what I thought from the original post) or if it is due to writing out to the tps file.

so time each part separately and compare

I will know shortly if its the difference between api’s. I suspect the Clarion Directory() is using FindFirstFile and not FindFirstFileExA

Namely
FindFirstFileExA function (fileapi.h) - Win32 apps | Microsoft Learn
has these options
FINDEX_INFO_LEVELS (minwinbase.h) - Win32 apps | Microsoft Learn

This one caught my eye.

FindExInfoBasic
The FindFirstFileEx function does not query the short file name, improving overall enumeration speed. The data is returned in a
WIN32_FIND_DATA structure, and the cAlternateFileName
member is always a NULL string.

Windows Server 2008, Windows Vista, Windows Server 2003 and Windows XP: This value is not supported until Windows Server 2008 R2 and Windows 7.

You see even though the clarion Directory() function help doc suggests I should be able to use the ff_:Directory equate to filter the results, that doesnt work. I have to specify a wildcard at the end of the path and include ff_:Directory to get it to include the subfolders. I then have to loop through the queue and ignore all entries ie files which are not subfolders.

Anyway just finishing off a class to use the various apis related to FindFirstFileA and FindFirstFileExA etc as I can get more info from these api’s which will be useful.

RichClaCode,
Maybe accessing the files on NTFS drives via the MFT data structures is what you need here?
From your other posts, I’m assuming you might be able to utilize the following technical article:
Super Fast File Enumeration with NTFS Structures

One of my hard disks is of Ext4 file system. Windows accesses it using the Paragon Linux FS driver. I have 2 letters to mapped drives on the home NAS. How suggested way can help with these drives? Incomplete solution = no solution.

I’ve also been looking at FindFirstFileTransactedA function (winbase.h) - Win32 apps | Microsoft Learn to see what the speed is like. Still testing, but I’ll check out your link thanks it looks interesting.

I had that Paragon Linux Ext Driver, I’m surprised you got it to work on windows, I got a refund because it didnt work properly on my machine, it was easier to just setup a Linux NAS with a variety of file systems and plonk Samba - opening windows to a wider world on the linux box so files could be accessed via SMB.

I understand what you are saying though, if you cant access a drive, how can you audit it. Market share is what I’m going after for now.

According to the linked article the process needs to be “Elevated”. That’s ok for something I run on my PC but not for end users.

To access the MFT structures, you must have elevated privileges – a normal user won’t be able to access it

The help clearly states that is NOT how it works… It ALWAYS has Files…

If you add ff_:DIRECTORY to ff_:NORMAL, you will get files AND sub-directories from the path . Since ff_:NORMAL is an equate for zero (0), you will always get files.

https://clarion.help/doku.php?id=directory_get_file_directory_.htm

The DIRECTORY procedure returns a directory listing of all files in the path with the specified attributes into the specified queue .

The help doc clearly states it returns a directory listing of all files in the path with the specified attributes into the specified queue.

But its not working like the help docs clearly states.

If you copy an app using api’s which need to be elevated onto a windows OS which uses elevation, the app’s application icon is automatically modified to include the elevation shield. This is without having to run the app, which suggests windows is analyzing all files copied onto a running OS.

Makes a mockery of the ransomware sideshow doesnt it!

Edit.

Its interesting the very first folder created during installation of windows on xp is
c:\windows
then
c:\windows\system32
c:\windows\system32\config
c:\windows\system32\ras
c:\windows\system32\spool
c:\windows\system32\spool\drivers
and so on. It gives an insight it how windows installs itself and what its installing first and why.