Dynamically get Queue/File Prefix and use it in assigned statement

Johan_de_Klerk · November 24, 2022, 12:44pm

Hi,

Sorry for the long post.

I need some help again please.
Please tell me if you think my idea is stupid.

I have ± 23 Queues, could be even more or less sometimes.
The Queues are identical but each Queue contains a different set of data.

MyQueue1 Queue,PRE(MyQue1)
MyID		Long
MyField1	STRING(255)
MyField2	STRING(255)
MyField3	STRING(255)
END
The other Queues is the same with the same fields just:
MyQueue2 Queue,PRE(MyQue2)
MyQueue3 Queue,PRE(MyQue3)
MyQueue4 Queue,PRE(MyQue4)
MyQueue5 Queue,PRE(MyQue5)

Etc

I loop throught each Queue to create a CSV file.
I use StringTheory

IF RECORDS(MyQueue1) > 0
    LOOP X# = 1 TO RECORDS(MyQueue1)
        GET(MyQueue1,X#)
        ST.SetValue('"'&CLIP(MyQue1:MyID)&'","'&CLIP(MyQue1:MyField1)&'","'&CLIP(MyQue1:MyField2)&'","'&CLIP(MyQue1:MyField3)&'"'&'<13,10>')
        ST.SaveFile('MyFirstQ.csv',TRUE)
    END
END
IF RECORDS(MyQueue2) > 0
    LOOP X# = 1 TO RECORDS(MyQueue2)
        GET(MyQueue2,X#)
        ST.SetValue('"'&CLIP(MyQue2:MyID)&'","'&CLIP(MyQue2:MyField1)&'","'&CLIP(MyQue2:MyField2)&'","'&CLIP(MyQue2:MyField3)&'"'&'<13,10>')
        ST.SaveFile('MySecondQ.csv',TRUE)
    END
END

I need to do this for each of the Queues

If I add or remove fields from the Queues I need to go and add or remove the fields from each of the StringTheory SetValue where I assign them.

Is there a way that I can pick up the PreFix of the Queue that I am currently working with and assign it or get it to be something generic?
For instance:

Loc:QPreFix      STRING(50)
Loc:QPreFix = GetQueuePrefix
ST.SetValue('"'&CLIP(Loc:QPreFix:MyID)&'","'&CLIP(Loc:QPreFix:MyField1)&'","'&CLIP(Loc:QPreFix:MyField2)&'","'&CLIP(Loc:QPreFix:MyField3)&'"'&'<13,10>')

I can then put the ST.SetValue in a routine.
I then only have to change the ST.SetValue in one place and it will then work for all the Queues.

Something like:

IF RECORDS(MyQueue1) > 0
    LOOP X# = 1 TO RECORDS(MyQueue1)
        GET(MyQueue1,X#)
        Loc:QPreFix = GetQueuePrefix
        DO FillSTValues
        ST.SaveFile('MyFirstQ.csv',TRUE)
    END
END
IF RECORDS(MyQueue2) > 0
    LOOP X# = 1 TO RECORDS(MyQueue2)
        GET(MyQueue2,X#)
        Loc:QPreFix = GetQueuePrefix
        DO FillSTValues
        ST.SaveFile('MySecondQ.csv',TRUE)
    END
END


FillSTValues	ROUTINE
    ST.SetValue('"'&CLIP(Loc:QPreFix:MyID)&'","'&CLIP(Loc:QPreFix:MyField1)&'","'&CLIP(Loc:QPreFix:MyField2)&'","'&CLIP(Loc:QPreFix:MyField3)&'"'&'<13,10>')

donridley · November 24, 2022, 2:52pm

Hello Johan,

Just my 2 cents…

I would do one TYPE’d QUEUE and instantiate it as needed. Instead of a bunch of individual QUEUES. Then you can modify your fields as the changes affect all the queues at once.

YourQType QUEUE,TYPE
Fields…
End

As far as the prefix is concerned, there is a {PROP:Prefix}.

CarlBarnes · November 24, 2022, 4:23pm

As Don said a TYPE Queue is the way, also know as a “Named Queue”.

Define your TYPE then the Queues 1-5 are (derive) from it:

MyQueueTYPE Queue,TYPE   !TYPE Queue
MyID		Long
MyField1	STRING(255)
MyField2	STRING(255)
MyField3	STRING(255)
END
!The Queues are all (MyQueueTYPE) so are the same with the same fields just:
MyQueue1 Queue(MyQueueTYPE),PRE(MyQue1) !Based on TYPE so this is complete delcare
         END
MyQueue2 Queue(MyQueueTYPE),PRE(MyQue2).  !<-- can put period at end for 1 line declare
MyQueue3 Queue(MyQueueTYPE),PRE(MyQue3).
MyQueue4 Queue(MyQueueTYPE),PRE(MyQue4).
MyQueue5 Queue(MyQueueTYPE),PRE(MyQue5).

You can write generic code to access the Q’s by passing them to a Procedure that takes (MyQueueTYPE) as a parameter. The inside does not use a PreFix: it uses Dot Syntax with the Queue Name of the Parameters as shown below:

    MAP                   
              !Note takes "MyQueueTYPE so knows the fields inside"
MyQueue2CsvFile  PROCEDURE(MyQueueTYPE MyQueX, STRING CsvFileName)
    END
    CODE
    MyQueue2CsvFile(MyQueue1,'MyFirstQ.csv')
    MyQueue2CsvFile(MyQueue2,'MySecondQ.csv')
    MyQueue2CsvFile(MyQueue3,'MyThirdQ.csv')
    ...

MyQueue2CsvFile  PROCEDURE(MyQueueTYPE MyQueX, STRING CsvFileName)
QX LONG
ST StringTheory
    CODE     
    IF RECORDS(MyQueX) > 0
        LOOP QX = 1 TO RECORDS(MyQueX)
            GET(MyQueX,QX)
            !Note: Must use Dot Syntax so MyQueX. not MyQueX:
            ST.SetValue('"'&CLIP(MyQueX.MyID) & |
                      '","'&CLIP(MyQueX.MyField1) & |
                      '","'&CLIP(MyQueX.MyField2) & |
                      '","'&CLIP(MyQueX.MyField3) & |
                      '"'&'<13,10>')
        END
        ST.SaveFile(CsvFileName,TRUE)
    END

Gold Star for providing excellent code with your question and formatting it as code! It made it easy to understand your intent.

CarlBarnes · November 24, 2022, 6:31pm

You can also use a Reference of your MyQueueType you assign to an instance and access all the fields. That’s like your Loc:QPreFix code to ref any of those queues.

Loc:QRef      &MyQueueTYPE   !Note: &Reference
   CODE
   Loc:QRef &= MyQueue2     !Note &= Referance assignment to Q 2
   ST.SetValue('"'&CLIP(Loc:QRef.MyID)&'","'&CLIP(Loc:QRef.MyField1)&'","'&CLIP(Loc:QRef.MyField2)&'","'&CLIP(Loc:QRef.MyField3)&'"'&'<13,10>')
!Above will use values of MyQueue2

A bit more of your Loc:QPreFix code using &Ref.

Loc:QRef &= MyQueue1     !Note &= Referanceassign to Q #1
IF RECORDS(Loc:QRef) > 0   !was RECORDS(MyQueue1)
    LOOP X# = 1 TO RECORDS(Loc:QRef)
        GET(Loc:QRef,X#)
        DO FillSTValues
        ST.SaveFile('MyFirstQ.csv',TRUE)
    END
END

FillSTValues	ROUTINE
    ST.SetValue('"'&CLIP(Loc:QRef.MyID) & |
              '","'&CLIP(Loc:QRef.MyField1) & |
              '","'&CLIP(Loc:QRef.MyField2) & |
              '","'&CLIP(Loc:QRef.MyField3) & |
              '"'&'<13,10>')

anon23294430 · November 24, 2022, 11:59pm

Using a Queue,Type lets you create as many queues as you like.

You obviously have some criteria which creates these different data sets so can you isolate them and then pass them as parameters to a local procedure?

Local procedures makes it easier to create generic code.

With these local procedures you can pass parameters which you cant do with routines, and then the parameters passed to the local procedure can help make the code more generic, and then in your main procedure create the Q’s, pass the parameter to the local procedure which fills the Q.

vitesse · November 25, 2022, 12:23am

Hi Johan

I don’t disagree with what others already said about a TYPEd queue, however I think you are making more work for yourself than necessary.

look up SerializeQueue in the docs:

https://capesoft.com/docs/StringTheory3/StringTheory.htm#SerializeQueue

something like:

st.SerializeQueue(MyQueue1,’<13,10>’, ‘,’, ‘"’)
st.SaveFile(‘MyFirstQ.csv’)

st.SerializeQueue(MyQueue2,’<13,10>’, ‘,’, ‘"’)
st.SaveFile(‘MySecondQ.csv’)

give it a try and yell out if you have any problems.

as a side note, you were appending to your physical file with SaveFile for every record in your queue. You would be better to Append using ‘<13,10>’ as the delimiter and then do ONE SaveFile after the loop. This would be much faster - the only time you should consider writing it out one line at a time is if memory was really constrained. (But to be clear - just use SerializeQueue and then all that code can disappear anyway).

hth

vitesse · November 25, 2022, 3:04am

I know I said to try SerializeQueue instead but just to be clear on this for future reference.

where you had:

IF RECORDS(MyQueue1) > 0
    LOOP X# = 1 TO RECORDS(MyQueue1)
        GET(MyQueue1,X#)
        ST.SetValue('"'&CLIP(MyQue1:MyID)&'","'&CLIP(MyQue1:MyField1)&'","'&CLIP(MyQue1:MyField2)&'","'&CLIP(MyQue1:MyField3)&'"'&'<13,10>')
        ST.SaveFile('MyFirstQ.csv',TRUE)
    END
END

instead you would do this:

IF RECORDS(MyQueue1) > 0
    ST.free()
    LOOP X# = 1 TO RECORDS(MyQueue1)
        GET(MyQueue1,X#)
        ST.Append('"'&CLIP(MyQue1:MyID)&'","'&CLIP(MyQue1:MyField1)&'","'&CLIP(MyQue1:MyField2)&'","'&CLIP(MyQue1:MyField3)&'"'&'<13,10>')
    END
    ST.SaveFile('MyFirstQ.csv')
END

or you could have the CRLF as a parameter on the append for the line endings:

IF RECORDS(MyQueue1) > 0
    ST.free()
    LOOP X# = 1 TO RECORDS(MyQueue1)
        GET(MyQueue1,X#)
        ST.Append('"'&CLIP(MyQue1:MyID)&'","'&CLIP(MyQue1:MyField1)&'","'&CLIP(MyQue1:MyField2)&'","'&CLIP(MyQue1:MyField3)&'"',,'<13,10>')
    END
    ST.SaveFile('MyFirstQ.csv')
END

the difference is that the first way would have a CRLF on the end of the last line whereas the second way would not.

Bruce · November 25, 2022, 3:34am

All of what Geoff said.

However serializeQueue lacks field names. If those are important I’d recommend switching to json or xml. Preferably json.

Saving a queue to a json file is 1 to 3 lines of code with jfiles.

Johan_de_Klerk · November 25, 2022, 7:09am

Thank you very much for everyone’s suggestions.

I am making a small test program to test everyone’s suggestion so that I have all the possibilities in one small app for future reference.

Johan_de_Klerk · November 25, 2022, 7:15am

Hi Bruce,
I am first testing serializeQueue as it seems quicker.

I have it working with: ST.SerializeQueue(FirstQueue, , , , ,1,TRUE)
1,Johan,de Klerk,542
2,Piet,Pompies,678

However I also need it with " surrounding the field values.
“1”,“Johan”,“de Klerk”,“542”
“2”,“Piet”,“Pompies”,“678”

I tried: ST.SerializeQueue(FirstQueue, , ‘,’ , ‘"’ , ‘"’ ,1,TRUE)
but it does not put the " around the fields.

Regards
Johan de Klerk

Johan_de_Klerk · November 25, 2022, 11:23am

Hi Carl,
I am trying the passing of the TYPE Queue thing but I am missing something somewhere.
I am feeling so stupid.

I have attached a C10 app.
Please help and take away this stupidity.
TestCSVExport.zip (21.2 KB)

Regards
Johan de Klerk

vitesse · November 25, 2022, 12:54pm

OK I checked and that is correct - it only would put quotes around a field if it contains a comma (that you are using to separate the fields). Indeed the documentation says this: A start-quote string to surround fields which may contain the boundary character(s).

st.join has an “alwaysQuote” option and maybe Bruce might consider adding the same to SerializeGroup and SerializeQueue. It would need to be added onto the end of the parameters list for backwards compatibility and default to false.

if you wanted to temporarily test this out, then in stringTheory.clw change

elsif q > 0 and instring(boundary.value[1 : boundary._DataEnd],an,1,1)

to be

elsif q > 0 ! and instring(boundary.value[1 : boundary._DataEnd],an,1,1)

note I commented out the instring test which looks to see if the boundary (comma in this case) was in the value.

this is obviously not a long term fix to your requirement - so don’t leave it like that!

so I think for the moment you will need to do some additional steps:

IF RECORDS(FirstQueue) > 0
    st.SerializeQueue(FirstQueue) ! no quoting
    st.replace(',','","')
    st.replace('<13,10>','"<13,10>"')
    st.prepend('"')  ! quote at start
    st.append('"')   ! quote at end
    ST.SaveFile(CLIP(LONGPATH())&'\CSVExportedFiles\FirstQueue.txt')
    RUN(CLIP(LONGPATH())&'\CSVExportedFiles\FirstQueue.txt')
END

Note: I am assuming your fields do not contain the delimiters (commas or CRLF) as that would require some fancier code to allow for that.

you could put this in a separate function or routine if you liked.

hth and cheers

Geoff R

vitesse · November 26, 2022, 2:11am

like this:

SerializeQueueQuoted PROCEDURE  (StringTheory pSt,*Queue pQueue) 
  CODE
  pSt.SerializeQueue(pQueue) ! no quoting
  pSt.replace(',','","')
  pSt.replace('<13,10>','"<13,10>"')
  pSt.prepend('"')  ! add quote at start
  pSt.append('"')   ! add quote at end

there are two ways to call this now - the first way passes your StringTheory object as the first parameter of SerializeQueueQuoted:

IF RECORDS(FirstQueue) > 0
    SerializeQueueQuoted(st,FirstQueue)
    ST.SaveFile(CLIP(LONGPATH())&'\CSVExportedFiles\FirstQueue.txt')
    RUN(CLIP(LONGPATH())&'\CSVExportedFiles\FirstQueue.txt')
END 
IF RECORDS(SecondQueue) > 0
    SerializeQueueQuoted(st,SecondQueue)
    ST.SaveFile(CLIP(LONGPATH())&'\CSVExportedFiles\SecondQueue.txt')
    RUN(CLIP(LONGPATH())&'\CSVExportedFiles\SecondQueue.txt')
END
IF RECORDS(ThirdQueue) > 0
    SerializeQueueQuoted(st,ThirdQueue)
    ST.SaveFile(CLIP(LONGPATH())&'\CSVExportedFiles\ThirdQueue.txt')
    RUN(CLIP(LONGPATH())&'\CSVExportedFiles\ThirdQueue.txt')
END

and the second way uses the dot notation. The StringTheory class is automatically passed for you as the first parameter:

IF RECORDS(FirstQueue) > 0
    ST.SerializeQueueQuoted(FirstQueue)
    ST.SaveFile(CLIP(LONGPATH())&'\CSVExportedFiles\FirstQueue.txt')
    RUN(CLIP(LONGPATH())&'\CSVExportedFiles\FirstQueue.txt')
END 
IF RECORDS(SecondQueue) > 0
    ST.SerializeQueueQuoted(SecondQueue)
    ST.SaveFile(CLIP(LONGPATH())&'\CSVExportedFiles\SecondQueue.txt')
    RUN(CLIP(LONGPATH())&'\CSVExportedFiles\SecondQueue.txt')
END
IF RECORDS(ThirdQueue) > 0
    ST.SerializeQueueQuoted(ThirdQueue)
    ST.SaveFile(CLIP(LONGPATH())&'\CSVExportedFiles\ThirdQueue.txt')
    RUN(CLIP(LONGPATH())&'\CSVExportedFiles\ThirdQueue.txt')
END

cheers

Johan_de_Klerk · November 26, 2022, 5:35am

Hi Geoff,

Thank you very much.

I have the SerializeQueue Quoted and Unquoted working perfect now.

I just realized that I will also have to get Carl’s Queue passing way working.
In some cases I have to do some formatting on the field values before putting them into the CSV.
For instance to put the first name and surname together, format a field as @P####P, etc.
Carl’s way would then give me generic way to achieve this.
I am getting compile errors when I put Carl’s code in, somewhere I stuffed up.

Regards
Johan de Klerk

vitesse · November 27, 2022, 12:28am

good to hear, thanks.

looking at that example app you posted earlier, you have not defined MyQueueTYPE anywhere that I can see. Hence the compile errors. Make it global in scope so that it can be seen everywhere.

I can see you have MainTQueue defined as a TYPE so maybe you should be using that not MyQueueTYPE

Johan_de_Klerk · November 27, 2022, 8:08am

Hi Carl,
I have been rapped many times on the knuckles for not providing enough context.
So I learned and now rather provide to much than to little, my knuckles can’t take more rapping.

Hi Geoff,
Thanks for the code, it works perfect.

Hi Geoff,
Thanks for correcting me.
I did not realize that I need to put it into the Global Data.

Thanks to everyone’s help and suggestions I have it now working in different ways to accommodate my possible uses of it.

I am attaching my latest example, maybe it can help someone else as well.
TestCSVExport.zip (40.3 KB)

The Clarion community rocks.

Regards
Johan de Klerk

vitesse · February 27, 2023, 9:48am

Just in case anyone is following this, Bruce has released a new version of ST (3.54) which adds an option to always quote - st:AlwaysQuote. See:

https://www.capesoft.com/docs/StringTheory3/StringTheory.htm#SerializeQueue

https://www.capesoft.com/docs/StringTheory3/StringTheory.htm#SerializeGroup

Johan_de_Klerk · February 27, 2023, 10:10am

Great, will make my code that much shorter.