Queue with nested queues (with nested queues) --> is it a good/efficient design?

questions
1

Hi,
In template code you can declare multi-valued symbols, where each value contains other multi-valued symbols, and, in that way, you can create rather complex structures. You can do, for example:

#DECLARE (%pParentList),UNIQUE
#DECLARE (%pParentProperty,%pParentList)   
#DECLARE (%pChildList,%pParentList),MULTI
#DECLARE (%pChildProperty,%pChildList)
#DECLARE(%pGrandChildList,%pChildList),MULTI
#DECLARE(%pGrandChildProperty,%pGrandChildList)
etc....

We could read such a structure into a clarion queue of queues of queues:

GrandChildList        QUEUE,TYPE
GrandChildProperty      STRING(50)
                      END
ChildList             QUEUE,TYPE
ChildProperty           STRING(50)
GrandChildren           &GrandChildList
                      END
ParentList            QUEUE,TYPE
ParentProperty          STRING(50)
Children                &ChildList
                      END

I have done things like this, and it does work, but I really wonder if at this level of complexity these structures are no longer efficient.

Do any of you know? Is this the better way to build complex structures in Clarion?

Thanks!

2

In the past i would have said these sorts of structures run the risk of memory leaks, but recent evolutions, like the Reflection library (free) make them safe.

Equally these nested structures best match nested text structures, like JSON and XML. Libraries like jFiles and xFiles, (not free) can really leverage them making creating and consuming JSON and XML much easier.

So generally I’d say if these structures best mirror the data then yes they are good structures. They’re efficient, fast, and when done properly don’t leak ram.

1 Like
3

like the Reflection library (free)

Um maybe, but Reflection requires StringTheory 3 which is paid for…

https://www.capesoft.com/accessories/reflectionsp.htm

so saying it’s free is a bit of a stretch.
Also worth pointing out that the Clarionshop link doesn’t have the requirement warning.

1 Like
4

Hi Graham,

Yeah, I’d forgotten about that requirement. I don’t really think about it because I always have it. It’s only my opinion of course, but if you program without StringTheory, then you’re doing it wrong. The cost/benefit ratio of StringTheory is sooo high. If you’re doing nested queues like this then the DisposeQueue feature alone in Reflection will save you a lot of programming time, not to mention removing the possibility of memory leaks.

Also worth pointing out that the Clarionshop link doesn’t have the requirement warning.

Which link are you seeing mate? I went to fix it, but everywhere I can see it mentions the requirement.

1 Like
5

Hi Bruce,

Which link are you seeing mate? I went to fix it, but everywhere I can see it mentions the requirement.

It’s when you click on the Clarionshop link and that opens the Clarionshop purchase page for Reflection…

https://www.clarionshop.com/secure/checkout.cfm?pid=1639&q=1

To be fair none of the items at the Clarionshop purchase stage mention pre-requisits.
So it’s expected developers take note of the requirements before that, which is reasonable.

Graham

6

The core concept of nested queues is an excellent way to implement complex structures.

If you know in advance how many children will be present you may prefer arrays.

Instead of direct references to queues I find it’s much simpler to write those structures by wrapping the queues in classes.

It’s simpler because the class is derived from a base class designed to wrap queues. Which takes care of things like freeing the queue when disposing. In other words dispose (or leave scope on non-reference instances) calls .destruct() which calls self. Free() and self.Free calls self.del() for each row. Which gives you a simple place to dispose of properties like .children and .grandchildren in the original post

Note, I have written hundreds of these queue wrappers. So many that I even have editor snippets written to aid in writing more.

1 Like
7

Thanks Bruce!
I was worried because a LONG time ago, I used a very complex queue of queues of queues to store a lot of information, and it was using up a rather disproportionate amount of memory…… Windows eventually freed the memory (it wasn’t a memory leak), but I must have kept a sense that it wasn’t the best method for complex+large-amounts-of-data….

8

Thanks Mark,
Yes, that’s good advice.

9

Hi Mark
Would you be happy to describe how that sort of functionality is achieved ?

10

CwUnit/Libsrc/ctQueue.inc at master · MarkGoldberg/CwUnit (github.com)

( Note: someday, I’ll create a repo for just ctQueue )

For your example, I threw the following together, I’d likely rename some things, and I’d also normally derive .ToString which is called by .Dump, which is very useful for debugging the contents of queues via OutputDebugString

Parent List --v

      INCLUDE('ctQueue.inc'),ONCE

gtParentList GROUP,TYPE
ParentProperty  STRING(50)
Children        &ctQ_ChildList
             END 

qtParentList QUEUE(gtParentList),TYPE
             END 
              

ctQ_ParentList   CLASS(ctQueue),TYPE,MODULE('ctQ_ParentList.clw'),LINK('ctQ_ParentList.clw')
Q                 &qtParentList
!------------------------
CONSTRUCT         PROCEDURE()
Del               PROCEDURE(),DERIVED
Add               PROCEDURE(STRING xProperty),LONG,PROC
                 END




  !for the 'ctQ_ParentList.clw'

   MEMBER()
   INCLUDE('ctQ_ParentList.inc'),ONCE
   MAP
   END

!=======================================
ctQ_ParentList.CONSTRUCT      PROCEDURE()
   CODE
   SELF.Q     &= NEW qtParentList
   CLEAR(SELF.Q)
   SELF.BaseQ &= SELF.Q

!=======================================
ctQ_ParentList.Del            PROCEDURE()!,DERIVED
   CODE
   DISPOSE(SELF.Q.Children)
   PARENT.Del()

!=======================================
ctQ_ParentList.Add            PROCEDURE(STRING xProperty)
   CODE 
   SELF.Q.ParentProperty = xProperty
   SELF.Q.Children      &= new ctQ_Childlist 
   ADD(SELF.Q)
   RETURN ErrorCode()

ChildList --v

      INCLUDE('ctQueue.inc'),ONCE

gtChildList  GROUP,TYPE
ChildPropery   STRING(50)
GrandChildren  &ctQ_GrandChildren
             END 

qtChildList  QUEUE(gtChildList),TYPE
             END 
              

ctQ_ChildList   CLASS(ctQueue),TYPE,MODULE('ctQ_ChildList.clw'),LINK('ctQ_ChildList.clw')
Q                 &qtChildList
!------------------------
CONSTRUCT         PROCEDURE()
Del               PROCEDURE(),DERIVED
Add               PROCEDURE(STRING xProperty),LONG,PROC
                END




  !for the 'ctQ_ChildList.clw'


   MEMBER()
   INCLUDE('ctQ_ChildList.inc'),ONCE
   MAP
   END

!=======================================
ctQ_ChildList.CONSTRUCT      PROCEDURE()
   CODE
   SELF.Q     &= NEW qtChildList
   CLEAR(SELF.Q)
   SELF.BaseQ &= SELF.Q

!=======================================
ctQ_ChildList.Del            PROCEDURE()!,DERIVED
   CODE
   DISPOSE(SELF.Q.GrandChildren)
   PARENT.Del()


!=======================================
ctQ_ChildList.Add            PROCEDURE(STRING xProperty)
   CODE 
   SELF.Q.ChildProperty  = xProperty
   SELF.Q.GrandChildren &= new ctQ_GrandChildList
   ADD(SELF.Q)
   RETURN ErrorCode()

GrandChild --v

      INCLUDE('ctQueue.inc'),ONCE

gtGrandChildren  GROUP,TYPE
GrandChildrenProperty STRING(50)
                 END 

qtGrandChildren  QUEUE(gtGrandChildren),TYPE
                 END 
              

ctQ_GrandChildren CLASS(ctQueue),TYPE,MODULE('ctQ_GrandChildren.clw'),LINK('ctQ_GrandChildren.clw')
Q                   &qtGrandChildren
!------------------------
CONSTRUCT           PROCEDURE()
Add                 PROCEDURE(STRING xProperty),LONG,PROC
                  END



  !for the 'ctQ_GrandChildren.clw'

   MEMBER()
   INCLUDE('ctQ_GrandChildren.inc'),ONCE
   MAP
   END

!=======================================
ctQ_GrandChildren.CONSTRUCT      PROCEDURE()
   CODE
   SELF.Q     &= NEW qtGrandChildren
   CLEAR(SELF.Q)
   SELF.BaseQ &= SELF.Q

!=======================================
ctQ_GrandChildren.Add            PROCEDURE(STRING xProperty)
   CODE 
   SELF.Q.GrandChildrenProperty  = xProperty
   ADD(SELF.Q)
   RETURN ErrorCode()

Here’s a very simple use of the classes
note: i’ll usually write a .Find method
I added the Grands reference as an example of using a short hand reference once the nesting gets a bit deep

 MAP
 END
 INCLUDE('ctQ_ParentList.inc'),ONCE 
Parents   ctQ_ParentList
Grands    &ctQ_GrandChildren
  CODE   
  Parents.Add('Beatrice')

  Parents.Q.Children.Add('Grace')
  Parents.Q.Children.Q.GrandChildren.Add('Lora')

  Parents.Q.Children.Add('Howard')
  Grands &= Parents.Q.Children.Q.GrandChildren
  Grands.Add('Julie')
  Grands.Add('Mark')
  Grands.Add('Deborah')
11

StringTheory has become such an essential part coding for me that I can’t imagine not having it. It’s well worth the $97.

If someone new to Clarion were to ask here, “What’s the first 3rd party tool I need to purchase?”

I would tell that person to buy StringTheory. Without a doubt.

3 Likes
12

that seems like a lot of code Mark. Reflection does the same with
reflection.DisposeQueue(queue)

:slight_smile:

13

But the class structure gives you the ability to add helper methods that are tailored to the particular situation. Such as the .add methods that I chose to add above. Which greatly improve use of the structures. Furthermore my code does NOT have the performance hit of runtime discovery which I’m assuming is done by something called a Reflection library

1 Like
14

sure, there’s obviously reasons to make classes around queues if you want to manipulate them. I guess disposing them though is not a sufficient reason anymore to require it.

There’s likely a small performance hit with the Reflection, but that code is pretty fast, and I don’t think that would add significantly to the time taken.

15

Dear Mark
That’s great, thanks for that, something to work with.
I take Bruce’s comment onboard regarding Reflection but examples like this help us gain knowledge and hence improve our expertise.

16

Mark your method the Class structure is in CODE and checked by the Compiler everywhere. If you get it wrong the compiler will error so you will know its wrong.

I don’t use the Reflection Class method but it appears to me to rely on Name('Text') e.g. NAME('Pets |Queue |...' in the example below?

Pets &PetsQueueType,NAME('Pets | Queue | RowName(Pet) | Rename(xPets)')

Get that NAME() wrong … I think you will not know until runtime or never? E.g. If you change the label “Pets” to “PetsQue” and fail to change the NAME() will it work? Forget to specify ' | Queue | ', or spell it wrong it wrong, won’t work right and you will not get an error.

Does Reflection class have some kind of Lint checker? Capesoft has some very helpful web utilities to format Queues with NAME() for JSON. One that would Parse a Group/Queue with NAME()'s and do some problem checking would be good, i.e. Name() Lint.

SV should Not have started to use NAME() for additional attributes, they should have added ATTRIBUTE() or METADATA() to designate that additional stuff.

I have the same issue with too much use of PROP:SQL that is text not checked with the compiler. Unless there would be considerable performance gains I would prefer Clarion code that is checked by the compiler.

The way Queues work had major changes in 5.5 (I think) to make them much more efficient, they are stored as a form of Array. I think this was done as they were being used much more in ABC so were improved. I would have no issue with your Queues of Queues code, go for it!

1 Like
17

You’re completely right Carl - if you make errors in your code then you’ll have errors in your program.
Regarding the first, it’s just a name, so yes it’ll work. The name is only important in code that references the name (like say exporting to json or xml.)

Regarding the second error (forget to add Queue, or spell it wrong) and you’ll get a GPF at runtime.

no

To be fair it was Topspeed that started this, not SV as I recall. And I agree, there should have been an alternate mechanism for this - but there isn’t - so we have what we have.

I think you’ll find it was in Clarion 5 - the work was done because the Wizatron code used them extensively - and that was the Topspeed era, not the CV era (which started with 5.5). Which, apart from anything else, shows the value of using the thing you’re making. Topspeed used Queues in Wizatrons and found them to be slow and memory hungry. Fixing the root of the problem, we all get fast, more memory effecient, queues.

1 Like
18

I’m 100% with you, and I was criticized by many when I suggested that the NAME is not intended for that. The should be a property like the PROP:EXTEND in the data declaration as it is on the REPORT (maybe one day).

Also for many years wanted SV to change the compiler to support nested QUEUE with out the reference, like it support GROUPs, no need to NEW/DISPOSE, but unfortunately that was never done (maybe one day).

1 Like
19

There should be. But there isn’t. When they add it ill be the first to change my code. Until then I’ll just have to make do with what we do have.

1 Like
20

The docs, which you can look into, even when you don’t purchase the template, do show the prerequisites.