Click here to comment!
1...Future improvements -- make search function accessed through the menu
CONTACTS
SUBJECTS
Full Report Assemble with Start Command and Construct with Chained Se
Chain Segmented Subject Indexes Solve Memory Problem with Indexes Seg
Subject Full Report Assemble with Start Command and Construct with Ch
Chain Segmented Subject Indexes Solve Memory Problem with Indexes Seg
0506 -
0506 - ..
0507 - Summary/Objective
0508 -
050801 - Follow up ref SDS 1 0000.
050802 -
050803 - Improved the SDS full subject report to apply a system of "chained
050804 - segments" and uses a secondary Windows Start session to extract
050805 - relevant segments for a report. These changes improve memory
050806 - management and expand usefulness of the record for analysis,
050807 - understanding and follow up.
050808 -
050809 -
050810 -
050811 -
050813 - ..
0509 -
0510 -
0511 - Evaluation
0512 -
051201 - Full Subject Report Makes Progress on Virtual Memory for SDS
051202 -
051203 - This is a fairly siginficant improvement illustrated by the current
051204 - ability to assemble reports of any size, rather than truncate output
051205 - at 1500 lines. Several reports have been run so far with 30,000
051206 - lines, 20 times the prior limit. Since record segment extraction for
051207 - reports is now done in a separate memory space, there have been no
051208 - program failures. The report segments are all under 50K. This reduces
051209 - the problem of program failure when opening an original record from
051210 - within a report.
051212 - ..
051213 - Previously to assemble a report of 30K lines would require about 20
051214 - separate runs, and the time involved to enter separate specifications
051215 - and then to manage the output separately prevented doing such
051216 - research. Even though something like a 30K line report is not a
051217 - common requirement, the ability to assemble 2K to 3K line reports may
051218 - be even more significant by placing greater reliance on the record
051219 - for pattern and trend recognition.
051221 - ..
051222 - Scrolling through the reports is fast and easy, with no noticable
051223 - delay in swapping report segments.
051225 - ..
051226 - There is nothing new or different for the User to learn in calling and
051227 - using a full subject report.
051229 - ..
051230 - Future improvements -- make search function accessed through the menu
051231 - scroll through chained segments using Shift F4.
051232 -
051233 -
051234 -
051236 - ..
0513 -
0514 -
0515 - Planning
0516 -
051601 - Full Reports can be Assembled and Segmented for Memory Management
051602 -
051603 - Full reports on SDS subjects quickly overwhelm memory. Code developed
051604 - on 030902 to run Subject Index in separate memory helps but additional
051605 - steps are needed for a stronger solution. Since the data base grows
051606 - every day, the passage of time necessarily increases occassions when
051607 - the size of the reports exceeds DOS support for 1MB of RAM.
051609 - ..
051610 - This limitation was evident from system messages that saying reports
051611 - are truncated to 1500 lines or so. Even when a report is not
051612 - aribitrarily cut off to fit DOS memory parameters, "out of memory"
051613 - messages still occurred often, because opening a large file to extract
051614 - a small piece needed for a report, exceeds DOS memory, and the progam
051615 - fails.
051617 - ..
051618 - The system developed earlier on 030810 for the Subject Index to chain
051619 - large files into segments might provide a virtual form of memory by
051620 - creating the report in segments and reading them into the viewable
051621 - screen only as needed, ref SDS 1 QG7I, rather than holding everything
051622 - in memory.
051624 - ..
051625 - Currently, a full report is assembled by reading in succeeding pieces
051626 - from records on the disk. The part that fits the report specification
051627 - is retained and the other parts are removed.
051629 - ..
051630 - This process discloses the number of lines in the file, and so when
051631 - the number exceeds an established criteria, that record could be
051632 - placed in another segment. These are only temporary files, and a way
051633 - would be needed to run multiple reports, so the temporary filenames
051634 - would need a fixed part and a floating part to differentiate the
051635 - segments for each run. We could put them in sd td along with the
051636 - transfer files for opening and managing the SI.
051638 - ..
051639 - A problem in the design is figuring out in advance the size of the
051640 - next file to be read in with the get command. The current file may
051641 - only contain 500 lines. If the next record has only 1000, lines that
051642 - is a total of 1500 lines, so this can be processed to figure out the
051643 - number of lines to keep and delete the rest. If the record has only
051644 - 50 lines to keep, that makes 550 lines, which fits in memory.
051645 - However, if the record is 2500 lines, then to wind up with 550 lines
051646 - requires processing 3000 lines, and that can crash the system.
051648 - ..
051649 - A soluton might be developed using Start to process each record in a
051650 - separate memory segment so that the report file only gets the segment
051651 - needed for the report and does not have to allocate memory for the
051652 - entire SDS record of 200K when it only needs 3K - 10K within the
051653 - record.
051655 - ..
051656 - This analysis suggests two improvements...
051657 -
051658 - 1. Chained segments for long reports over 1500 lines. ref SDS 0
051659 - TJ8L
051661 - ..
051662 - 2. Process records for building the report in a separate Start
051663 - session to avoid reading in the entire file to the report file
051664 - on each pass, but only reading in the part that meets the
051665 - report specification. ref SDS 0 9K6B
051666 -
051668 - ..
051669 - Summary Report Modify to Open Record Segment of Specified Subject
051670 -
051671 - The summary subject report currently only opens the record. Since
051672 - pointers are developed, it should be possible to modify the Enter
051673 - feature so that the record is opened to the first instance of the
051674 - subject, and to list in the Summary each of the record segments.
051675 - This would significantly increase the length of the Summary listing,
051676 - and so presents a memory issue, but it would add a lot of value to
051677 - the report by making it fast easy to find the place in the record
051678 - that is of interest.
051680 - ..
051681 - Another idea that might be better is to load the search specification
051682 - into the Shift F4 locate command so that the user can scroll through
051683 - the report to find all instances where the subject occurs. This
051684 - would take less memory.
051685 -
051686 - [On 031101 applied this idea to improve the summary subject
051687 - report. ref SDS 4 PV6I
051688 -
051689 -
051690 -
051691 -
051692 -
0517 -
SUBJECTS
Start Command Assemble Full Report and Construct with Chained Segment
Full Report Assemble with Start Command and Construct with Chained Se
Chained Segments Subject Full Report
080501 - ..
080502 - 06112 Subject Report Process Records in Separate Memory
080503 -
080504 - The first thing to create is to extract the data in a separate session
080505 - in order to reduce the impact on memory. Typically, a report will
080506 - extract 3K - 10K from a record that may be from 100K - 300K. Since
080507 - the report file grows as segments are added, it can be 100K at a time
080508 - when it needs to process a file that is 200K - 300K, and this causes
080509 - memory to fail. Therefore, extracting relevant information separately
080510 - avoids this part of the memory problem. Below, develop system of
080511 - chained files that solves another part of the memory problem for large
080512 - full reports. ref SDS 0 TJ8L
080514 - ..
080515 - Line 330, ref OF 1 6612, -label xyz8 in 06112
080516 -
080517 - macro 120
080518 -
080519 - This is main engine in 000007 that does all the work, but
080520 - uses code in 06112 for various pieces.
080521 -
080523 - ..
080524 - Line 1030, ref OF 1 6384, -label cOlpO1 in 06112
080525 -
080526 - e 1y
080527 - top
080528 -
080529 - This is called from withing macro 120 after extracting the
080530 - pointers for building the report, and is called below.
080531 - ref SDS 0 S669
080533 - ..
080534 - Line 1240, ref OF 1 MU5K, -label cdrcS in 06112
080535 -
080536 - -label cdrcS
080537 - rel_cur 0 1
080538 - getgbl 126 126
080539 - -if @126 = 1 ins_text !c:\sd\03\0611251!
080540 - -if @126 = 2 ins_text !c:\sd\03\0611251 #2!
080541 - -if @126 = 3 ins_text !c:\sd\03\0611251 #3!
080542 -
080543 - After enter the file spec to process, move the cursor forward
080544 - and enter a parameter to process the record based on the file
080545 - type set in 000007. ref SDS 0 913O
080546 -
080547 -
080549 - ..
080550 - Pointers in 1y moved to 1y9a Process in Separate Memory by 0611251
080551 -
080552 -
080553 - Line 1280, ref OF 1 HV8O, -label lpst8 in 06112
080554 -
080555 - e 1y
080556 - zonereps
080557 -
080558 - Return to 1y list of pointers. zonereps captures record spec
080559 - to determine when all of the pointers for this record have been
080560 - moved to the transfer file for dose processing under macro file
080561 - 0611251, per below. ref SDS 0 4Q6K
080563 - ..
080564 - -label lpst8
080565 - setcnt 86 1
080566 - loc_cur 0 1
080567 - macro 1182
080568 - setcnt 0 0
080569 - immed d
080570 - -if @0 > 0 -goto plpd
080571 -
080572 - Do a loop setcnt 86 1 sets char read op to start with counter
080573 - 1; loc_cur 0 1 puts cursor on col 1 to read from the beginning
080574 - of the record ID string. macro 1182 reads the string, and
080575 - string there two spaces between the end of the record ID and
080576 - the line number for the pointer subject in the control field,
080578 - ..
080579 - e 1y9a
080580 - macro 1181
080581 - immed a
080582 - loc_cur 0 1
080583 - up
080584 -
080585 - Return to pointer listing and see if all of the pointers
080586 - for this record have been moved to transfer file.
080588 - ..
080589 - e 1y
080591 - ..
080592 - repscnt 1 0
080593 - -if @1 = 0 -goto lpst8
080594 -
080595 -
080596 - -label plpd
080597 - -if @0 > 0 setgbl 1 999
080598 - e 1y9a
080599 - linecnt 2 0
080600 - getgbl 1 1
080601 - -if @1 = 999 -if @2 = 1 -goto cOEND
080602 - save
080603 -
080604 -
080606 - ..
080607 - SDS Record Opened in Separate Memory for Procssing Under Start
080608 -
080609 -
080610 - Line 1320, ref OF 1 FR3L, -label plpd in 06112
080611 -
080612 - ecur 97
080613 - *%gc
080614 - dose /c c:\sd\01\e d:\sd\08\uuuuu\00\nnnn
080615 - bot
080616 - immed /
080617 - get stufq
080618 -
080619 - ecur 97 returns to q_out; dose... runs the secondary process
080620 - that opens the SDS record and calls new macro file 0611251 to
080621 - extract the segments from the record, per below. ref SDS 0 4Q6K
080622 - After dose op runs... control returns to the q_out file, bot
080623 - goes to the bot and immed / moves the last line to to tos and
080624 - get stufq reads in the content created by dose op.
080625 -
080626 -
080627 -
080628 -
0807 -
SUBJECTS
Chained Segments Start Command Assemble Full Report
Chained Segments Full Report to Minimize impact on Memory Using Segme
120401 - ..
120402 - Chain Full Subject Reports into Segments Minimize Impact on Memory
120403 -
120404 - Follow up ref SDS 1 0000.
120405 -
120406 - On 030810 developed design that segments large Subject Indexes into
120407 - smaller files that do not exceed memory of Medit. ref SDS 1 QG7I This
120408 - work implements planning above. ref SDS 0 JQ86
120410 - ..
120411 - Line 1340, ref OF 1 UK5J, -label plpd in 06112 about 50 lines below
120412 -
120413 - linecnt 1 0
120414 - -if @1 < 1000 -goto gstq
120415 - e Q1
120416 - line && *%5s
120417 - loc_cur 4 32
120418 - addcnt 231 1
120419 - inscnt 231 0
120420 - ic 32
120421 - e q_out
120422 - savx c:\sd\td\$$hhmmss\1
120423 - addcnt 231 1
120424 - immed a
120425 - up
120426 - ins_text !Chained report segment....... $!
120427 - eol
120428 - inscnt 231 0
120429 -
120430 - *%5s
120431 - -if @231 = 1 save c:\sd\td\qout1
120432 -
120433 - Decided to save the chained segmenets in separate directories
120434 - for each report based on time. This speeds the assembly of
120435 - reports when switching from one segment to the next using F9
120436 - and F10 and PgDn and PgUp to scroll through the report, per
120437 - below. ref SDS 0 RR4H These directories will be deleted once
120438 - a day with backup.bat.
120439 -
120440 -
120441 -
120443 - ..
120444 - Scroll Seemlessly through Virtual Memory in Full Subject Report
120445 -
120447 - ..
120448 - Page Up F10 and PgUp Scroll Up to Prior Chained Segment
120449 -
120450 -
120451 - Line 1540, ref OF 1 CS4K, -label pfup in 06112
120452 -
120453 - -label pfup
120454 - setcnt 1 98
120455 - setcura 2 0
120456 - -label pf10dn
120457 - setcnt 5 1
120458 - pushc
120459 - loc_cur 3 1
120460 - linecnt 5 0
120461 - -if @5 > 1 -goto scrlb
120462 - bot
120463 - chrcnt 1 0
120464 - -if @1 = 67 -goto nrseg
120465 - top
120466 - -goto ppce
120467 - -label scrlb
120468 - down 10
120469 - -if @3 < 14 -if @1 = 98 rel_cur 10 0
120470 - -if @3 > 13 -if @1 = 98 loc_cur 24 0
120471 - setcnt 3 0
120472 - -goto ppce
120473 -
120474 - F10 is assigned in the report profile to call -label fp10dn,
120475 - and the Page Up key is assigned to call -label pfup. setcnt 5
120476 - 1 is a flag for the common code
120477 -
120478 -
120480 - ..
120481 - Page Down F9 and PgDn Scroll Down to Next Chained Segment
120482 -
120483 -
120484 - Line 1520, ref OF 1 ST3J, -label pfdn in 06112
120485 -
120486 - -label pfdn
120487 - setcnt 4 99
120488 - setcura 2 0
120489 - -label pf9up
120490 - setcnt 5 2
120491 - -if @1 != 99 pushc
120492 - up 10
120493 - -if @3 > 12 -if @1 = 99 rel_cur -10 0
120494 - -if @3 < 13 -if @1 = 99 loc_cur 3 0
120495 - setcnt 3 0
120496 - -if @1 = 99 pushc
120497 - loc_cur 3 1
120498 - chrcnt 1 0
120499 - -if @1 = 67 -goto nrseg
120500 - popc
120501 - -exit
120502 -
120503 -
120505 - ..
120506 - Line
120507 -
120508 -
120509 -
120510 -
120511 -
120512 -
120513 -
120514 -
120515 -
120516 -
120517 -
120518 -
120519 -
120520 -
120521 -
120522 -
120523 -
120524 -
120525 -
120526 -
120527 -
1206 -
SUBJECTS
Start Command Assemble Full Report and Construct with Chained Segment
Full Report Assemble with Start Command and Construct with Chained Se
Extract Relevant Segments from SDS Records 0611251 for Full Report
160501 - ..
160502 - 06112510 Extracts Records Segments for Full Subject Report
160503 -
160504 - Line 60, ref OF 2 QO8K, -label fulls
160505 -
160506 - This is called to construct the standard full report without
160507 - references.
160509 - ..
160510 - Read list of control field pointers in the file 1y9a constructed
160511 - under 06112, per above, ref SDS 0 9Z8G, and find each one in the
160512 - SDS record. When a pointer is found, save this record segment, and
160513 - delete others, so that segments that do meet the subject
160514 - specification are removed from the record.
160515 -
160516 -
160518 - ..
160519 - Error Message Eliminated for Record Not Found
160520 -
160521 -
160522 - Line 110, ref OF 2 II6J, -label fulls in 0611251
160523 -
160524 - chrcnt 1 0
160525 - -if @1 > 64 -if @1 < 91 -goto gtre
160526 -
160527 - The SDS record identified by 06112 is open, per above.
160528 - ref SDS 0 913G chrcnt 1 0 reads the first char, and if it is a
160529 - capital letter, this means an SDS record was found and opened
160530 - for the pointer found in the selection process that lists
160531 - pointers in file 1y, per above. ref SDS 0 S651
160533 - ..
160534 - immed a
160535 - up
160536 - ins_text !Empty!
160537 - save stufq
160538 - -goto nfetr
160539 -
160540 - This means that a record does not exist for the pointer in 1y,
160541 - and so the pointer system is out of sync and does not align
160542 - with the actual state of the record.
160544 - ..
160545 - Decided to try not stopping the report to issue and error
160546 - message that a record is missing which forces the user to
160547 - invest time to investigate and correct the problem with the SDS
160548 - data base, rather than focus on the task at hand to work with
160549 - information that is available to address an immediate problem.
160551 - ..
160552 - immed a simply adds a line in the empty file and enters a
160553 - string "Empty" to notify the code in 06112 not to process this
160554 - line.
160556 - ..
160557 - The new design saves time and frustration in the moment. Not
160558 - sure if the time saved to investigate and correct the record is
160559 - vindicated. Extraneous pointers will accumulate in the record
160560 - over time for which no SDS record exists. In other words,
160561 - these are unnecessary strings that consume resources that will
160562 - never be used. It is wasteful on the one hand, and also
160563 - increases the time required to search the record, thus reducing
160564 - efficiency and productivity.
160566 - ..
160567 - The scheme adopted today of ignoring mismatch between pointers
160568 - and records has the potential of causing people to overlook
160569 - important information that is needed in the moment for an
160570 - accurate understanding of a subject.
160572 - ..
160573 - Could we develop a system of maintenance that does not stop the
160574 - report process, but allows people to investigate later if they
160575 - wish? If maintenance is not performed, would this create a
160576 - problem of many pointers accumulating for which there are no
160577 - records?
160579 - ..
160580 - Gary Johnson has proposed creating a utility to maintain the
160581 - data base by listing pointers where no records exist. This
160582 - could be like the report that maintains the Schedule, enabling
160583 - people to investigate, and then delete extraneous pointer lines
160584 - or attempt to recover a record that seems important. There
160585 - are four pointer data bases to investigate. The control field
160586 - data base which is 01 does not impart enough information to
160587 - signify importance. If a scheme could be devised to open the
160588 - corresponding 04 record description pointer that would give
160589 - people a strong clue about the importance of investing time to
160590 - recover a record.
160591 -
160592 -
160593 -
160594 -
160595 -
160597 - ..
160598 - Line 240, ref OF 2 YL4O, -label eldcf in 0611251
160599 -
160600 - -label eldcf
160601 - up 2
160602 - -gosub cOS_01
160603 -
160604 - Code has found the top of the Control Field; up 2 places next
160605 - Control Field line below line that has been found for
160606 - evaluation to compare with target lines in 1y9a list.
160607 -
160608 -
160609 - -label lpdcfl
160610 - loc_cur 0 1
160611 - chrcnt 1 0
160612 - rel_cur 0 1
160613 - chrcnt 2 0
160614 - rel_cur 0 1
160615 - chrcnt 3 0
160616 - -if @3 > 32 -goto feots
160617 - -if @1 = @61 -if @2 = @62 -goto eldcf
160618 - immed d
160619 - -goto lpdcfl
160620 -
160621 -
160623 - ..
160624 - Action Item Report Formatted with Macro 1203
160625 -
160626 -
160627 - Line 490, ref OF 2 PTWY, -label cOstop in 0611251 about 30 lines
160628 - below
160629 -
160630 - -if @126 != 3 -goto nAI
160631 - top
160632 - z 1 3
160633 - l /02 /
160634 - up
160635 - immed dd
160636 - z 3 3
160637 - l / /
160638 - z 8 10
160639 - l /==/
160640 - immed dd
160641 - down
160642 - macro 1203
160643 -
160644 - Action Item reports occur when 126 = 3, set in 06112 and passed
160645 - into 0611251 based on the line number of the call. Delete the
160646 - lines from below the primary contact line, 02, to the bottom of
160647 - the subject descriptions for the first control field. Then
160648 - call macro macro 1203 to format the entire record, since by all
160649 - of the segments that do meet the report subject specification
160650 - have been removed by processing in advance of formatting by
160651 - macro 1203 in 000007, per below. ref SDS 0 MT8N
160652 -
160653 -
160655 - ..
160656 - Save Record Segments for Report and Quit Return to 06112
160657 -
160658 -
160659 - Line 560, ref OF 2 QN7K, -label nAI in 0611251
160660 -
160661 - -label nAI
160662 - save stufq
160663 - gfname
160664 - macro 1180
160665 - e c:\sd\03\0611251
160666 - line && *%dy
160667 - loc_cur 4 7
160668 - setcnt 86 1
160669 - macro 1181
160670 - purge 1y9a
160671 - *%dy
160672 - purge
160673 - quit
160674 -
160675 - After all record segments have been processed based on the
160676 - control field pointers in 1y9a (which is the portion of 1y
160677 - created in 06112 for the current record, per above, ref SDS 0
160678 - 9Z8G,
160679 -
160680 -
160682 - ..
160683 - 000007 Macro 120 Modify to Use Start Command
160684 -
160685 - Line 1340, ref OF 5 3436, -entry 120 in 000007 about 100 lines below
160686 -
160687 - Starting debug here called from 06112 in order to study the
160688 - process to determine where to call Start.
160690 - ..
160691 - Seems to turn out that no changes were required to macro 120 in
160692 - 000007, as shown below.
160693 -
160695 - ..
160696 - Line 2190, ref OF 5 JV5K, -label GEXt in 000007 about 40 lines below
160697 -
160698 - -if @39 > 0 -goto cO_rcds
160699 -
160700 - Report has been run and this determins if there are records to
160701 - process.
160702 -
160704 - ..
160705 - Line 2310, ref OF 5 5E6M, -label nIntrnt in 000007 about 80 lines
160706 - below
160707 -
160708 - getgbl 147 147
160709 - getgbl 126 93
160710 - -if @147 = 12 getgbl 127 127
160711 - -if @127 > 0 setcnt 126 3
160712 - setgbl 126 @126
160713 -
160714 - counter 93 is set in macro 1202 when the report type is set by
160715 - the user. ref SDS 0 917G Get flags identifying report type to
160716 - format output. Since formatting is now done in part with new
160717 - sub-macro files, flag 126 needs to be read again in 06112, per
160718 - above. ref SDS 0 LT7L
160719 -
160720 -
160722 - ..
160723 - Line 2390, ref OF 5 5U4F, -label nIntrnt in 000007 about 150 lines
160724 - below
160725 -
160726 - -if @126 = 0 -goto refcnS
160727 - @q1 #33
160728 - -goto cO_msG
160729 -
160730 - If 126 is > 0 it is a full report and so processing occurs in
160731 - 06112 at line 33. ref SDS 0 S651
160732 -
160733 -
160735 - ..
160736 - Subject Report Type - Summary, Full w/o or with References
160737 -
160738 -
160739 - Line 2520, ref OF 5 6815, -entry 1202 in 000007
160740 -
160741 - addcnt 125 1
160742 - setcnt 93 @125
160743 - -if @147 = 10 modcnt 93 2
160744 - -if @147 != 10 modcnt 93 3
160745 - modcnt 93 3
160746 - ...
160747 - ...
160748 - ...
160749 - setgbl 93 @93
160750 -
160751 - Counter 93 establishes the report type selected by the user,
160752 - and analysed in macro 120 in 000007 to set counter 126,
160753 - ref SDS 0 913O, that controls processing in 06112 for selecting
160754 - the sub-macro to assemble the records for a report.
160755 -
160756 -
160758 - ..
160759 - Macro 1203 Action Items Formatted
160760 -
160761 -
160762 - Line 2550, ref OF 5 I38G, -entry 1203 in 000007
160763 -
160764 - Deletes all the lines except those flagged as action items; called
160765 - by macro file 0611251, per above. ref SDS 0 MS7N
160766 -
160767 -
160768 -
160769 -
1608 -
SUBJECTS
Backup Interim Run from the File Menu in Schedule, SDS Reco s or Edi
Remove Directories for Assembling Chained Segments in Transfer Direct
Transfer Directory Maintain from Local Disk to Server by Deleting Tra
Environment DOS Variables Windows 2000 Windows 98 SDS Batch Files and
Environment Variables Dir Structure Configure Different Versions of S
200701 - ..
200702 - Backup.bat Modify to Remove Temporary Transfer Files and Directories
200703 -
200704 - Follow up ref SDS 3 OS71, ref SDS 2 KR4G.
200705 -
200706 - Line 20, ref OF 3 TT4L, in backup.bat
200707 -
200708 - %STR% rd c:\sd\td /s /q
200709 -
200710 - Removed the transfer directory that now includes subdirectories
200711 - for full subject report chained segment files, per above.
200712 - ref SDS 0 TJ8L
200714 - ..
200715 - %STR% runs the environment variable in sds.bat that calls the
200716 - cmd.exe start command. ref OF 4 LV4K The system of environment
200717 - variables was applied to SDS beginning on 030920, ref SDS 2
200718 - KR4G, and later explained on 031020. ref SDS 3 OS71
200719 -
200720 -
200721 -
200722 -
200723 -
200724 -
200725 -
200726 -
200727 -
200728 -
200729 -
200730 -
200731 -
2008 -