Win10 C6 Directory() 8min 40 secs, C11 Directory() 31mins 47 seconds

I will know shortly if its the difference between api’s. I suspect the Clarion Directory() is using FindFirstFile and not FindFirstFileExA

Namely
FindFirstFileExA function (fileapi.h) - Win32 apps | Microsoft Learn
has these options
FINDEX_INFO_LEVELS (minwinbase.h) - Win32 apps | Microsoft Learn

This one caught my eye.

FindExInfoBasic
The FindFirstFileEx function does not query the short file name, improving overall enumeration speed. The data is returned in a
WIN32_FIND_DATA structure, and the cAlternateFileName
member is always a NULL string.

Windows Server 2008, Windows Vista, Windows Server 2003 and Windows XP: This value is not supported until Windows Server 2008 R2 and Windows 7.

You see even though the clarion Directory() function help doc suggests I should be able to use the ff_:Directory equate to filter the results, that doesnt work. I have to specify a wildcard at the end of the path and include ff_:Directory to get it to include the subfolders. I then have to loop through the queue and ignore all entries ie files which are not subfolders.

Anyway just finishing off a class to use the various apis related to FindFirstFileA and FindFirstFileExA etc as I can get more info from these api’s which will be useful.

RichClaCode,
Maybe accessing the files on NTFS drives via the MFT data structures is what you need here?
From your other posts, I’m assuming you might be able to utilize the following technical article:
Super Fast File Enumeration with NTFS Structures

One of my hard disks is of Ext4 file system. Windows accesses it using the Paragon Linux FS driver. I have 2 letters to mapped drives on the home NAS. How suggested way can help with these drives? Incomplete solution = no solution.

I’ve also been looking at FindFirstFileTransactedA function (winbase.h) - Win32 apps | Microsoft Learn to see what the speed is like. Still testing, but I’ll check out your link thanks it looks interesting.

I had that Paragon Linux Ext Driver, I’m surprised you got it to work on windows, I got a refund because it didnt work properly on my machine, it was easier to just setup a Linux NAS with a variety of file systems and plonk Samba - opening windows to a wider world on the linux box so files could be accessed via SMB.

I understand what you are saying though, if you cant access a drive, how can you audit it. Market share is what I’m going after for now.

According to the linked article the process needs to be “Elevated”. That’s ok for something I run on my PC but not for end users.

To access the MFT structures, you must have elevated privileges – a normal user won’t be able to access it

The help clearly states that is NOT how it works… It ALWAYS has Files…

If you add ff_:DIRECTORY to ff_:NORMAL, you will get files AND sub-directories from the path . Since ff_:NORMAL is an equate for zero (0), you will always get files.

https://clarion.help/doku.php?id=directory_get_file_directory_.htm

The DIRECTORY procedure returns a directory listing of all files in the path with the specified attributes into the specified queue .

The help doc clearly states it returns a directory listing of all files in the path with the specified attributes into the specified queue.

But its not working like the help docs clearly states.

If you copy an app using api’s which need to be elevated onto a windows OS which uses elevation, the app’s application icon is automatically modified to include the elevation shield. This is without having to run the app, which suggests windows is analyzing all files copied onto a running OS.

Makes a mockery of the ransomware sideshow doesnt it!

Edit.

Its interesting the very first folder created during installation of windows on xp is
c:\windows
then
c:\windows\system32
c:\windows\system32\config
c:\windows\system32\ras
c:\windows\system32\spool
c:\windows\system32\spool\drivers
and so on. It gives an insight it how windows installs itself and what its installing first and why.

Or it could be the filename. Certain magic words will get you the shield, such as “Update”, “Setup”, and others.

Its a good point so I renamed the app from PCSetup to PCtest, copied from my XP virtual machine onto Win10 and it still added the shield. Maybe its looking at the embedded manifest file or something. Dont know, but using the API FindFirstFileA has got it down to 3mins something which is better.

C:\Users appears to be the first folder created on Win10, and there are some interesting attributes attached to folders, even in XP.

If you have an embedded manifest, and it says requireAdministrator, then yes. Windows is looking at that. You should be able to confirm by looking at the resources of the exe with a resource viewer. If a manifest is present, there should be a folder called “Manifest”.

If your going to limit your understanding to the FIRST sentence only … ahhh … well it’s going to be hard to make your code work. Read more.

All the FF:_ attributes ADD to Normal. If you specify FF:_Hidden you also get Normal non-hidden files.

I dont think I have eg

attributes An integer constant, variable, or expression that specifies the attributes of the files to place in the queue .

The attributes bitmap is an OR filter: if you add the equates, you get files with any of the attributes you specify. This means that, when you just set the attributes to ff_:NORMAL, you only get files (no sub-directories) without the hidden, system, or archive bits set. If you add ff_:DIRECTORY to ff_:NORMAL, you will get files AND sub-directories from the path . Since ff_:NORMAL is an equate for zero (0), you will always get files.

Yes it was the manifest, I had it set to requireAdministrator or HighestAvailable, but changing it to asInvoker has removed the elevation shield now when I copy it from XP to Win10. Oh well Red Herring there, still its interesting to know its scanning the manifest when copied. I wonder if it also scans the versioninfo embeded into an exe or dll GetFileVersionInfoA function (winver.h) - Win32 apps | Microsoft Learn.

I’ve have to think of test for that one…

Might be because better virus defender …w11 is scanning each file for virusses when you move, enumerate or copy, so that slows down.

Even comparing signatures on the ms cloud over internet while file manegement takes place

Especially with html extensions

You can turn it off.

But i might have misunderstood the problem/question

Most antivirus will do this, its not just restricted to Windows Defender.

I’ve stopped using the MS Cloud app because it was ignoring the settings changes I made and did its own thing so it filled up with data that triggered the charges. So I dont use it and have it switched off when the OS is installed.

So what I have found, is despite using the flags in the EX version to limit to directories the api’s fetch everything in the folder or path location. I still have to filter out everything that is not a directory, which is the same behaviour I see when I use the built in Clarion Directory function. The extra time I see on Win10 I am putting down to more files in each folder location which then needs to be filtered out.

I havent tried the Transacted api yet, still on the todo list, but in order to solve another problem, I’ve had to go off onto something else, namely modifying some templates to get them to work because the problem of the template group names having to be unique in the template registry and not just the template chain is possibly why some of my templates have failed when I go to use them later on.

So I’m having to build an app and function to scan all the template files and extract all the group names so that I can see whether they are unique at the registry level as per the problem detailed here How can I force the Registry to recognise the template code has changed inside a file that is loaded in the Registry?

Yes, we turn it off as well.

We wrote low level MFT records enumeration.

So in 2 minutes the entire folder file about 4 million fioes index with all atrributes are in a db.

Like voidtools.com. …the everthing (which does enumeration with use of the journal)

I’ll check that out. So far I’ve been able to get the EX api to list every file and folder, and then filter out the files or folders into a Class Q and that takes about 55 seconds, which is better, and its about 1.5times writing from the class Q to a tps file with logout, but theres a bug somewhere which causes it to hang when a bitlocker drive is not decrypted which I need to fix.

Aha.

Yes, you are right, it takes usally longer to insert in a q or db, which files you have found.

Its nice to see a fast thinker in programming land again :slight_smile: