Dynamic Data structures in Clarion using DynaLib

Ok up till now we havnt had time or the knowledge of runtime data structures in the clarion runtime.

With a quite winter and summer we have had a little time to look at the work of the russian in the early days of clarion 5 and 6 and re compile in clarion 11.

We have a need for this stuff in an up and coming project and i thought let publish a little of this as we go along in the next part of this project.

This is the cleaned up version of creating a queue at runtime using the russian code with some rewriting.

As we get know this code we will post tid bits of hopefully creating file and views and queues…

The statements sent to this method in the library are just simple text declarations of a queue.

This is where the queue is created and you can see the Self.Q is simple an address to a string with some data types of long , pretty simple really…we have put a window in to show the data created just for this example.

RTLQueueCreateClass.QConstruct PROCEDURE

QueHdr               LIKE(TQueueHeader),AUTO
DeclareMem           LONG,AUTO
QueueDeclareBuffer   &STRING
QueueTQueueHeader    &STRING
fld                 any  


Window                          WINDOW('test Runtime Queue - from deep inside russia.'),AT(,,395,224),GRAY,FONT('Segoe UI',9)
                                    BUTTON('&OK'),AT(291,201,41,14),USE(?OkButton),DEFAULT
                                    BUTTON('&Cancel'),AT(340,201,42,14),USE(?CancelButton),STD(STD:Close)
                                    LIST,AT(15,19,357,157),USE(?LIST1),FROM(''),FORMAT('47L(2)|M20L' & |
                                        '(2)|M20L(2)|M')
                                END

  Code
  Clear(QueHdr)
 
  if ~Self.LinkMode = true 
     Self.Q &= Null

      ! Is A Queue created here?   
         
     gSize# = Self.FSize(0,True); if gSize# < 0 then Return;end 
     if ~gSize# then Size# = 1 else Size# = gSize#;end
     DeclareMem = RTL::NewMem(Size#); if ~DeclareMem then Return;end
     QueHdr.BufferPtr = DeclareMem
     QueHdr.BufferSize = gSize#

    ! Is GenerateDeclare   
        
     QueueDeclareBuffer &= Self.GenerateDeclare()
     if QueueDeclareBuffer &= Null then 
        RTL::FreeMem(DeclareMem);
        Return;
     end
        
     QueHdr.FieldsDefPtr = Address(QueueDeclareBuffer)
   End
   if ~Self.LinkMode >= true OR ~(Self.LinkG &= Null)
     if Self.LinkMode >= true
        RefGrp.GRef &= Self.LinkG
        QueHdr.BufferPtr = OvrGrp.GPtr.BufferPtr
        QueHdr.BufferSize = OvrGrp.GPtr.BufferSize
        QueHdr.FieldsDefPtr = OvrGrp.GPtr.FieldsDefPtr
        
        ! set up queue header addresses
                
     End
     QueueTQueueHeader &= New(STRING(Size(TQueueHeader)))
     if QueueTQueueHeader &= Null ! memory allocation failed
        RTL::FreeMem(DeclareMem)
        Dispose(QueueTQueueHeader)
     else ! Set up Queue address 
        QueueTQueueHeader = QueHdr
        Self.Q &= Address(QueueTQueueHeader)
            
        if ~(Self.Q &= Null)
             
            fld &= what(self.q,1)
            fld = 'data'
            add(self.Q)
             
            open(WINDOW)
            ?List1{prop:from} = self.Q
            
            ACCEPT
            END
            
            close(WINDOW)
            
           Free(Self.Q); Clear(Self.Q)
           if Self.Thread then Self._Thread(Self.Thread);end
  ;end  ;end  ;end

  QueueDeclareBuffer &= Null; QueueTQueueHeader &= Null
  if ~(Self.Q &= Null)
     if ~Self.Props.BStrPresent = true
        Self.Props.GrpClear &= NEW(STRING(SIZE(Self.Q)))
        if ~Self.LinkMode = TRUE
           Self.Props.GrpClear = Self.Q
        else
           QueueDeclareBuffer &= _NewStr(Self.Q)
           Clear(Self.Q)
           Self.Props.GrpClear = Self.Q
           Self.Q = QueueDeclareBuffer
           Dispose(QueueDeclareBuffer)
      end  ;end
     Self.Props.Created = True
   end

  Return

Looks like Dynalib code.

We are going through and rewriting sections of the Russian DLIB libraries from the russians. What we found interesting was that once you have the locally declared typed groups or structures clarion database, queues and views become dynamic.

it is if that is the russian code base. We wont be taking credit for it but the code is being re worked as some of it is very very hard to read and repeats itself.

There is for a long time also a free version of Dynalib and you can contact Oleg. See https://www.clarionlife.net/novaya-versiya-dynalib-4-0-4/

1 Like

Oops, didn’t know, I delete my message then.

From the help:

Freeware Restrictions

Freeware-version of the given library has some restrictions of functionality:

The maximal size of structure up to 128 bytes
A maximal quantity of fields in a group up to 15
A maximal quantiy of keys in a file up to 2
A maximal quantity of MEMOs and BLOBs in a file up to 1
A maximal quantity of JOIN-blocks in a VIEW up to 2
A maximal quantity of levels (nests of JOIN-blocks) up to 1

These restrictions are valid both for created, and for “linked” structures. Nevertheless, I think, that not looking at these restrictions, freeware-version have sufficient functionality for their use in more - less complex projects. But if you need more flexibility with data manipulating you can purchase Professional version of DynaLib on the ClarionShop.

I dont think you will find that the product is supported any longer by the authors. Much of the code needs reworking as it not really very clean.

As we uncover it structures we are rewriting the parts we are using into a clearer to read format.

It is a very extensive library of code with a lot of functionality.

The Ukraine and Russian appear to have found and exposed all the clarion data structures and come up with a solution to runtime create all data types without using any special code except standard clarion code.

Since no work has been done on this product for what appear to about a decade we decided it was a matter of diving in and hoping the internal clarion data structures had not changed.

Much to our surprise it appear to run in clarion 11 and if clarion goes 64 we will try to support that also.

If the product is still sold on clarion shop i doubt you will get any support from the authors although we stand to be corrected on that.

As we connect the code to runtime macros i will update this thread and prehaps post some example with calc2 runtime scripts on github later in the year for clarions developers to try.

Glad all of the Russians were in on this. :slight_smile:

Could you put some examples on GitHub? That work within the free limitation of 128 bytes.

I usually make LIST / QUEUE examples with DIRECTORY() of the Temp or Windows folder. You could take the FILE:QUEUE and copy it to a DynaLib Queue created at runtime with a STRING(32) file name. This Repo has the code to show the Temp folder in a List

1 Like

That’s pretty neat. Would be cool if there was a textbox to add our own filter criteria for the temp files for our apps that do a lot of temp files. One filter per line. And perhaps an “ignore” textbox that has a list of file masks to never make available for deletion…

e.g.

MySpecialPrefix*.tmp
OtherPrefix*.xxx
Yada*.*

Not something I’d use every day, but it would be useful when I needed it.

This seems to be the last web scrape by the wayback machine.
Aug 2020
Download (archive.org)

dLib Doc file
Wayback Machine (archive.org)

dynalib.narod.ru/freeware/dLib.zip is not in the wayback machine
dynalib.narod.ru/freeware/fScan.zip is also not in the wayback machine.

There is an extended evaluate link on the wayback machine site (last wayback machine webscrape)
“Utilities” (archive.org)

This link mentions dynalib and the TDynaFileClass class
Wayback Machine (archive.org)

The only copy of ExtEval.zip on the wayback machine
Wayback Machine (archive.org)

Found the Russian forum which looks very active still, like posts still being made today.
forum.clarionlife.net - Main page

Example of Creating a Dynamic Queue using a Proxy class.

include(‘Dynaproxycontainer.inc’),ONCE

MAP

   module('ddynartl.clw')
      CreateQueuestruct(string DataStructure,Dynaproxycontainer q),long 
  END
    
END

qproxy &Dynaproxycontainer

Qflds ANY
qaddress long

CODE
Qflds = ‘Stockprices QUEUE,PRE(TST);’ & |
‘Name STRING(@S100),ALIGN©,Name(’‘Name’’);’ & |
‘Code SHORT(@N_5),ALIGN®;’ & |
‘Price DECIMAL(@N13`2),ALIGN®;’ & |
‘Input DATE(@D6.B),ALIGN®;’ & |
‘Closed DATE(@D6.B),ALIGN®;’ & |
‘END;’

qproxy  &=  new(Dynaproxycontainer)    
qaddress = CreateQueuestruct(Qflds,qproxy)
if qaddress > FALSE
     
      qproxy.IDSet.DSet('Name','META STOCK')
     qproxy.IDSet.DSet('TST:Code','META')
     qproxy.IDSet.DSet('TST:Price',157)
      add(qproxy.q)
      
     message(qproxy.IDGet.DGet('TST:Price'))
      
     
END

Dynamically Creating a Queue and Tables from text files.

 module('ddynartl.clw')
      RegisterDrivers
      CreateQueuestruct(string DataStructure,Dynaproxycontainer q),long 
      CreateFilestruct(string DataStructure,Dynaproxycontainer q),long 
      CreateQueuestructDeclare(string QueueName,string DataStructureFile,Dynaproxycontainer q),long 
      CreateFilestructDeclare(string DBFileName,string DataStructureFile,Dynaproxycontainer q),long 

    END
    
END

qproxy &Dynaproxycontainer
Fproxy &Dynaproxycontainer
Qflds ANY
qaddress long

CODE
Qflds = ‘Stockprices QUEUE,PRE(TST);’ & |
‘Name STRING(@S100),ALIGN©,Name(’‘Name’’);’ & |
‘Code SHORT(@N_5),ALIGN®;’ & |
‘Price DECIMAL(@N13`2),ALIGN®;’ & |
‘Input DATE(@D6.B),ALIGN®;’ & |
‘Closed DATE(@D6.B),ALIGN®;’ & |
‘END;’

qproxy  &=  new(Dynaproxycontainer)    
qaddress = CreateQueuestruct(Qflds,qproxy)
if qaddress > FALSE
     
     qproxy.IDSet.DSet('Name','META STOCK')
     qproxy.IDSet.DSet('TST:Code','META')
     qproxy.IDSet.DSet('TST:Price',157)
      add(qproxy.q)
      
     message(qproxy.IDGet.DGet('TST:Price'))
END   
RegisterDrivers    

qflds = 'C:\win32clacode\Ddyna\DynaTableDatastructures.txt'    
Fproxy  &=  new(Dynaproxycontainer)    
qaddress = CreateFilestructDeclare('TPSFile',Qflds, Fproxy)   

Files can be stored in text files for runtime declarations.

#FILE(TPSFile)

TPSFile FILE,DRIVER(‘TOPSPEED’),PRE(TPS),CREATE,Bindable
Name_Key KEY(+TPS:Name,+TPS:Desc),DUP,OPT,NOCASE
Code_Key KEY(+TPS:Code),OPT,NOCASE
Ndx_Key KEY(+TPS:Index),OPT,NOCASE,PRIMARY
Record RECORD
Name STRING(30)
Desc STRING(110)
Code LONG
Index LONG
END
END

#ENDFILE

Ok we are going to set up a BLOG for this topic because we are about to test Dynamic File, Queue, SQL and VIEW using runtime scripting in the next few weeks.

Scripting engine bindings are in final testing and we have repackaged the runtime code for the clarion structures above that came with DLIB.

The source code from the rus was full of omits to drive a coder spare to lunatic…

We set up whole new project and added Derived to the class methods to stop the clarion compiler doing a GPF where ever such methods were called.

Another thing was that the Original developer had gone ballistic (sorry about that pun) on using overloading to the point that prohibited the extension of the classes with interfaces.

In short we have copied / cut and pasted the code base to new project and separated what could be into new classes but due to the overloading manic’s of the developer not all classes were able to be separated out and renamed…

we are just trying to decide on the best scripting binding names for coding.

This we can do once people have a go to using it…

The scripting engine has gone through about 4 years of testing and design on linux…and has been designed to allow multiple languages to bind classes and data for what you might call a universal adaptor language.

The original file access was using Lotus 123 for Clarion dat files and we will post those examples up on GIT thing for you all to wonder through. Those example we created from the original technical doc published by Clarion and show C file structures for clarion dat files in the day when such information was available for developers.

The structures for runtime File, SQL, QUEUE and View are from the original RUS developer and for copy right reasons we wont publish those as for those of you who brought access to the DLIB code you will have them already.

Cheers.

So is this written in Clarion and if so how do you get it to run on Linux, or is this using something like Wine on Linux, or some other language?

No this is a C++ engine.

It is a project crafted from a C++ expression engine that was extended with Invokable bindables.

Clarion just happens to be what a lot of our software was written in and plays well with CPP.

For example clarion supports Interface,com

Example.

IAccountTrialBalances Interface,com
GetMember Procedure(SIFC Ifcref,cstring membername)
SetMember Procedure(SIFC Ifcref,cstring membername)
InvokeMember Procedure(SIFC Ifcref,cstring membername)
END

! Clarion Class

TrialBalanceClass Class,type|
,Implements(IAccountTrialBalances)|
,Implements(ISumTrialBalances)|

and away you go with your clarion class now bound to the scripting language.

Hence it can run on Linux, ok I understand, its not written in Clarion. Can you compile it with Topspeed C or is this compiled in some of other version of C?

Refactoring The Dynamic Data Structure Library in Clarion Code.

We have continued to refactor the code base that provides services for dynamic data structures written in clarion code. The code known as DLIB.

The code base has been moved into a new project format and the supporting classes are being moved one by one into separate modules for easier management.

One of the issues with the code was the overloading of methods and there being no derived statements. As each class has been moved to each own module the loading methods are being renamed. While this may cause a parent mismatch or failed virtual call these are carefully checked after the renaming. You may ask why bother but in fact the compiler could get confused if any alternations were made to the class such as an interface added to its structure.

A testing confirms that the queue structure is performing as expected and we will post Views and SQL as they tested.

Renaming of the methods now also leads to an element of self documentation.

Such as instead of Putref we have …

PutrefVar… PutrefGroup, putrefqueue, putrefblob, putrefFile… ect…

Once the classes are all seperated into modules and each method has a unique name they can be more easy coupled to a scripting binding engine for full stand alone runtime access for legacy clarion databases.

A test of the queue structure and 1000 records added…

We have promised we would try Dynamic SQL next and with a great deal of luck the refactoring of the code appears to have worked as we have renamed a lot of base class methods in the base group class and to reconnect them when one does not have full knowledge of a very complex code base is just that “LUCK”