Flat Serializer: A class for serializing and deserializing Clarion structures to CSV (or TSV)

code
, ,
4

Hi Carlos,

I added a routine on TestApp to load a 28MB CSV that Jeff included in his project:

The code is:

TestCSV ROUTINE
DATA
fs FlatSerializer
TestResult ANY
!Queue definition by CSVReader:
MyQueue QUEUE,PRE(My)
id STRING(5)
first_name STRING(15)
last_name STRING(25)
email STRING(41)
ip_address STRING(15)
favorite_animal STRING(40)
favorite_color STRING(10)
favorite_guid STRING(36)
password STRING(12)
favorite_words STRING(584)
bitcoin_address STRING(34)
credit_card_type STRING(25)
currency STRING(13)
favorite_stock STRING(62)
balance STRING(6)
ticker STRING(14)
industry STRING(62)
favorite_plant STRING(40)
favorite_company STRING(41)
avatar STRING(82)
auto STRING(23)
buzzword STRING(24)
END
CODE
DebugView(TestsResult)
fs.Init
TestResult = FORMAT(TODAY(),@D10)&’ ‘&FORMAT(CLOCK(),@T04)&’ Init done’&RECORDS(MyQueue)
DebugView(TestResult)
fs.LoadTextFile(‘E:\test\CSVParseClass\SampleData\CSVDemo50K.Comma.CRLF.csv’)
TestResult = FORMAT(TODAY(),@D10)&’ ‘&FORMAT(CLOCK(),@T04)&’ LoadTextFile done’&RECORDS(MyQueue)
DebugView(TestResult)
fs.DeSerializeToQueue(MyQueue)
TestResult = FORMAT(TODAY(),@D10)&’ ‘&FORMAT(CLOCK(),@T04)&’ DeSerializeToQueue done. Records '&RECORDS(MyQueue)
DebugView(TestResult)
AssertEqual(50000,RECORDS(MyQueue),‘TestCSV: Records loaded’)

Originally it took about 1 minute to load, profiling some internal steps there were aproximately 20 seconds to load the CSV on a STRING (function FlatSerializer.StringFromTextFile), 30 seconds to process the STRING an build the pointers structure (FlatSerializer.LoadString) and 9 seconds to load the Queue (FlatSerializer.DeSerializeToQueue) on an intel i7.

Optimizing StringFromTextFile with String Slicing (and driver BUFFERS) it reduced from 20 to less than 1 sec. Both are techniques used on SystemStringClass.FromFile
Optimizing LoadString with precomputed LENs it reduced from 30 to 13 secs aprox.

About Excel formulas I thought about a flag, but found it is not necessary, default behavior should handle it right (it is “=…”, not =“…”).

I saw testApp is coded like unit tests, it is interesting.

Thanks to you

Federico

1 Like