CONTACTS 

SUBJECTS
Subject, 022A09
Supersort, 910306
Concepts Reports
Pointer Data Base, Duplicates

0806 -
0806 -    ..
0807 - Summary/Objective
0808 -
080801 - Follow up
080802 -
080803 - x0001 Further work on doing global record selection with Supersort.
080805 -  ..
080806 - Began calling to hire someone to do this job, but then we have a
080807 - problem of giving away our idea.  It is not good to tell others about
080808 - the design of SDS.
080810 -  ..
080811 - Completed the SDS code to process the large pointer file and it is
080812 - much faster for doing yearly and total DB record selection.  Our
080813 - objective is to get this job done without creating a separate large
080814 - pointer file.
080815 -
080817 -  ..
0809 -
0810 -
0811 - Analysis
0812 -   ..
081201 - Will evaluate using Norton to select just the pointers.  This
081202 - did not work because it will not output the stuff unless it gets a
081203 - bunch of other stuff I do not want.
081205 -  ..
081206 - Not sure this is actually true.  I have a recollection that this came
081207 - up once before and it turned out we can get just what is needed,
081208 - although maybe that is not true.  I checked the Norton manual and
081209 - found nothing that indicates we can do this.  There is no switch
081210 - indicating it can output just the lines that satisfy the specifica-
081211 - tion.  This means to get the filenames themselves, we need to operate
081212 - on the 02 data base and that took over 3 minutes to build the output
081213 - file.
081215 -  ..
081216 - Pointer Maintenance
081217 -
081218 - How can we overcome the delays of maintaining a large pointer file?
081219 -
081220 - It could be done just once a week.  Or everyday with the interim
081221 - backup operation.  But this overlooks things like modifying particular
081222 - records which then require updating the large pointer file. Similarly,
081223 - when a record is created for a prior date, things get delayed.
081224 -
081226 -  ..
081227 - Duplicate Pointer System
081228 -
081229 - This makes the choice one of having duplicate pointers.  One set that
081230 - can be quickly updated and another that can be quickly analysed once
081231 - it is updated.  Duplicate pointers greatly increases the disk space
081232 - devoted to overhead.  It also means duplicate procedures.
081234 -  ..
081235 - What we need is a method to use something like supersort that accepts
081236 - global path elements and filenames as input, and is not limited by the
081237 - number of files it handles.  It is not clear why such a limit is
081238 - necessary.
081240 -  ..
081241 - I thought that we might be able to save the large pointer file I made
081242 - last night, then just update each day with the interim backup
081243 - operation, but that really will not work because when records are
081244 - changed and records added for prior dates, it makes it necessary to
081245 - rebuild the entire thing in order to make sure nothing is overlooked.
081247 -  ..
081248 - Might avoid doing the entire file each day by getting a list of just
081249 - the changed files and doing a sort with a no duplicate files between
081250 - the changed files and the new ones.  This would still be a consider-
081251 - able operation, requiring several minutes to 5 minutes depending on
081252 - how big the data base is.
081253 -
081255 -  ..
081256 - Sorting Pointers
081257 -
081258 - I wonder how much good it does to sort the pointers, or whether it
081259 - would be just as fast to simply copy them into a single large file,
081260 - concatenating them with DOS copy command.  When we do the data
081261 - extraction, the thing still looks in the entire file, although
081262 - theoretically, once it passes the first char that does not match, it
081263 - moves to the next rec.
081265 -  ..
081266 - When the data base is sorted, the record selection should be able to
081267 - stop as soon as it gets a miss after getting a hit.
081269 -  ..
081270 - Since we do not have this capability, it does not make sense to build
081271 - the large pointer file by sort/merge.  Copy them together is better.
081273 -  ..
0813 -
0814 -
0815 - Getting a Better Processor
081601 -  ..
081602 - Supersort
081603 - Called Pacific Data Systems
081604 -
081605 - Called them to see if they have updated the program to handle more
081606 - files and/or can do globals with DOS directories.
081607 -
081608 - Kathleen gave me Rich Mangel's phone number.  I called and left
081609 - message for him to call back.
081610 -
081612 -  ..
081613 - Called Other Programmers
081614 -
081615 - Requested price to build separate utility to do this work.
081616 -
081617 - Also called Joel Budgor at 415 465 1426.
081618 -
081620 -  ..
081621 - It would be nice if this product can support more than 32 files,
081622 - and particularly if it permits global file names.  We need to make
081623 - something that can read filenames as numbers so we could enter a
081624 - specification like:
081625 -
081626 -              s d:\sd\08\*\01\25-43
081627 - or
081628 -              s d:\sd\08\*\01\*
081629 -
081630 -
081632 -  ..
0817 -
0818 - 000007
0819 -
081901 - Think we will just for now create some alternate language in 00007 to
081902 - process the entire data base.
081903 -
081904 - If we have the other pointer file should it be on d:?   That just
081905 - reduces the amount of space available for the data base records
081906 - themselves.
081908 -  ..
081909 - We could put it on e: where nothing ever changes.
081911 -  ..
081912 - It can be:  e:\sd\08\00000\06\total
081914 -  ..
081915 - This string length is the same as the existing code structure.
081916 -
081918 -  ..
081919 - Added at line 917 routine to apply the above new pointer file.  It
081920 - uses the flag set in 06112 under function Alt F1.
081922 -  ..
081923 - This routine seems to be okay.
081925 -  ..
0820 -
0821 -
0822 - 06112
0823 -
082301 - This is just a simple switch to call the special routine within macro
082302 - 120 called by Alt F1.
082304 -  ..
0824 -
0825 - 25.bat
0826 -
082601 - How to build the total data base pointer file.
082602 -
082603 - It should be part of interim backup because that is already a long
082604 - affair, that is done outside regular hours.  Adding to it is not a big
082605 - deal.
082607 -  ..
082608 - I made a routine that copies all of files into a single large pointer
082609 - file under
082610 -
082611 -                 d: sd 08 00101 06 total
082613 -  ..
0827 -
0828 - 0812
0829 -
082901 - This is called under 25.bat to get a list of all 01 pointers for a
082902 - particular user and copy them into a single large file.  It takes too
082903 - long though.
082904 -
082906 -  ..
082907 -  ..
0830 -
0831 - Copy command
0832 -
083201 - Tried DOS copy command.  When used without a filename spec, it will
083202 - concantenate all files in a dir into a single file, but could not get
083203 - it to avoid entering the eof marks.  This prevented making a single
083204 - large file.
083205 -
083206 -      [See problems of copy adding an EOF mark that messes things up,
083207 -      ref SDS 3 ref SDS 4 line 118, line 54; ref SDS 5 line 66.]
083208 -
083209 -
083210 -
083211 -
083212 -
0833 -