CONTACTS 

SUBJECTS
Problem Gary Documents Analyse in SDS Lines Longer than 360    04
Documents Analyse in SDS Lines Longer than 360 Recl of SDS Record ar
SDS Analyse Documents Truncated Lines Longer than 360 Recl of SDS Re
Truncated Documents Analyse in SDS Lines Longer than 360 Recl of SDS

1106 -
1106 -    ..
1107 - Summary/Objective
1108 -
110801 - Follow up ref SDS 8 0000.
110802 -
110803 - Increased the range for flexibile structure of SDS to analyse
110804 - documents that have long lines.  This helps bridge the gap between the
110805 - two (2) world of information and knowledge tools.  Solution required a
110806 - long search testing to determine when the problem occurs, ref SDS 0
110807 - MU9J, and ended in a simple solution, ref SDS 0 8R3P, which later
110808 - proved to be complex.  Initially, seemed like we could set a system
110809 - recl 3000, because testing indicated this did not impact memory usage.
110810 - ref SDS 0 8Q4O  Subsequent experience seems to show that recl of over
110811 - 500 causes accumulation of error that crashes Medit.  This has severe
110812 - impacts on scrolling sorted indexes, ref SDS 0 686L, and on justifying
110813 - paragraphs with many lines. ref SDS 0 UL8G  Therefore wound up with
110814 - compromise solution that unfortunately requires giving attention to
110815 - system mechanics, per below. ref SDS 0 NF4K
110816 -
110817 -            [On 040910 Morris fixed problem with get command.
110818 -            ref SDS 11 0001
110819 -
110820 -
110821 -
110822 -
110824 -  ..
1109 -
1110 -
1111 - Problem
1112 -
111201 - Gary sent a letter this morning saying there is a problem reading a
111202 - letter into an SDS record for anlaysis, when the letter has lines
111203 - longer than the record length (recl) of SDS records.  The default line
111204 - length of SDS records is 360 columns.  Lines in files that are longer
111205 - than 360 columns are truncated, which frustrates the purpose of the
111206 - function to Analyse Documents.
111208 -  ..
111209 - Investigation shows the longest line in the sample file Gary
111210 - submitted to test the problem is about 900 columns.
111212 -  ..
111213 - Solution turns out to change the recl command from 360 to 2000, per
111214 - below. ref SDS 0 OD98  Considered using a conditional that would avoid
111215 - the recl command command in macro 157, but this didn't seem to be
111216 - effective. ref SDS 0 8R4S
111218 -  ..
111219 - Experience later showed that setting recl 2000 caused memory problem
111220 - that cause the program to fail scrolling through sorted subject index
111221 - accounts, per below, ref SDS 0 686L; and using Alt F9 to justify a
111222 - paragraph with a lot of lines, also below. ref SDS 0 UL8G
111223 -
111224 -
111225 -
111226 -
111227 -
111229 -  ..
1113 -
1114 -
1115 - Background
1116 -
111601 - On 890523 requirements were reviewed for flexible structure to support
111602 - a wide range of data use cases. ref SDS 1 P13O  In fact, Medit has
111603 - severe limits, but the design has proven to accomodate all of the
111604 - information people encounter on the job each day from movies, books,
111605 - pictures, articles, contracts, reports and correspondence.  Recently,
111606 - on 040513 requirements for flexible structure were summarized for a
111607 - proposal to the Aerospace company Architecture Committee. ref SDS 7 B56I  These
111608 - requirements bridge the gap between the two (2) worlds of information
111609 - technology (IT), like Microsoft Word, Internet Explorer, Lotus Notes,
111610 - Wikis, weblogs, Powerpoint, etc., and knowledge tools, like SDS, that
111611 - add intelligence to connect the dots for understanding cause and
111612 - effect within a framework of Knowledge Space essential for performing
111613 - Communication Metrics to ensure accurate understanding and timely
111614 - follow up, cited in the proposal to Aerospace company, reviewed on 040426.
111615 - ref SDS 6 6W6I
111617 -  ..
111618 - On 030823 developed feature using macro file 00702, ref SDS 2 5L6J, to
111619 - make analysing documents, created with other programs, fast and easy,
111620 - by automatically formatting in SDS. ref SDS 2 0001
111622 -  ..
111623 - On 031109 automatic formatting was enhanced to support outline and
111624 - conventional textual structures, ref SDS 3 1H6N, and to read any file
111625 - into an SDS record by entering a specification into a template menu.
111626 - ref SDS 3 0001  At that time, support was expanded to enable people to
111627 - select a file from a list created with the Directory Management System
111628 - (DMS), rather than having to remember a path and filename to specify.
111629 - ref SDS 3 JO67
111631 -  ..
111632 - The problem Gary reports today requires expanding support by SDS for
111633 - adding intelligence to a broader range of information technologies,
111634 - specifically files created with long lines, typically associated word
111635 - processing and sometimes in email.
111636 -
111637 -
111638 -
111639 -
111641 -  ..
1117 -
1118 -
1119 - Testing
1120 -
112001 - Tried reading the sample file into an Medit file with the get command
112002 - and this worked successfully, indicating the problem is with the code
112003 - in 00702 originally developed on 030823. ref SDS 2 5L6J
112005 -  ..
112006 - Line 340, ref OF 1 414J, -label good in 00702 about 40 lines below
112007 -
112008 -    This is where the code begins to setup for reading the target file
112009 -    into the SDS record.
112010 -
112011 -
112013 -  ..
112014 - Line 420, ref OF 1 OF5N, -label lpdwN in 00702 about 50 lines below
112015 -
112016 -    errorcnt 0 0
112017 -     setcnt 94 2288
112018 -     recl 2000
112019 -    g x.txt
112021 -         ..
112022 -        Set temp recl 2000, adjusting default recl 400, set in macro
112023 -        153, per below, ref SDS 0 QM5I, to read in files with long
112024 -        lines that sometimes occur in email and other wordprocessing
112025 -        files, per discussion below on using Alt F9 to call macro 49
112026 -        that justifies para. ref SDS 0 UL8G  The target file is read
112027 -        into the record with the get command. Testing shows that after
112028 -        get command executes, lines are read into the record with the
112029 -        correct lengths, nothing has been truncated yet.
112031 -         ..
112032 -        Temporary recl 2000 needs to be reset by the User following the
112033 -        steps listed below. ref SDS 0 NF4K
112035 -         ..
112036 -        Actually, testing has not established that recl impacts macro
112037 -        49 to justify and causes the program to fail. ref SDS 0 AS9K
112038 -
112039 -
112041 -  ..
112042 - Line 500, ref OF 1 U99I, -label 17s in 00702
112043 -
112044 -    -label 17s
112045 -    split
112046 -    -if @174 = 3 ins_text!000001 -      x!
112047 -    -if @174 = 1 ins_text!000001 -    x!
112048 -    -if @174 = 3 loc_cur 0 15
112049 -    -if @174 = 1 loc_cur 0 13
112050 -    join
112051 -    ic 32
112052 -    loc_cur 0 1
112053 -    -return
112054 -
112055 -        Subroutine that splits the line number off, constructs a new
112056 -        structure and then joins the new content, seems to be working
112057 -        correctly.  The lines longer than the nominal length of the
112058 -        record are not truncated by this process.
112060 -  ..
112061 - Line 520, ref OF 1 OO4F, -label end in 00702
112062 -
112063 -    macro 142
112064 -
112065 -        Everthing up until macro 142 is okay.  Macro 142 updates the
112066 -        outline structure and calls macro file 0702051 to add
112067 -        anchors,then saves the record.  After this op runs, the lines
112068 -        are truncated.
112069 -
112070 -
112071 -
112073 -  ..
112074 - 000008  Recl 400 Default Medit and SDS records to Support Justify
112075 -
112076 - Line 260, ref OF 3 1247, -entry 153 in 000008
112077 -
112078 -      recl 3000...................... disconnected
112079 -      recl 2000...................... disconnected
112080 -      -if @94 != 2288 recl 512....... disconnected
112082 -         ..
112083 -        Set recl to 3000 for reasons reported below for macro 1571 in
112084 -        SDS. ref SDS 0 OD98  This recl command controls the editor, the
112085 -        Subject Index and other features, Contacts and Directory
112086 -        Maintenance.  After working with recl 3000 for about 3 weeks
112087 -        found that this causes scrolling Subject Index to crash, per
112088 -        below. ref SDS 0 686L, so changed to 2000 as a precaution in
112089 -        everything except for Subject Index, set recl 500 in 03501.
112090 -
112091 -            [On 040910 Morris fixed problem with get command.
112092 -            ref SDS 11 0001
112094 -         ..
112095 -        More experience showed that recl 2000 seems to cause the SDS
112096 -        program to fail after multiple cycles using Alt F9 to
112097 -        justifying paragraphs with many lines for continuous editing,
112098 -        per below. ref SDS 0 UL8G  Therefore, changed recl in the main
112099 -        profile to 400, and will try setting a long recl when the code
112100 -        in 00703 is run to anlayse a document, per above. ref SDS 0
112101 -        OC6G  Finally decided to eliminate recl in macro 153 for
112102 -        ordinary editing, and use Medit default recl 264, and set a
112103 -        custom recl in SDS records under macro 157.
112105 -         ..
112106 -        This change will require some adjustment to work practice,
112107 -        listed below. ref SDS 0 NF4K
112108 -
112110 -  ..
112111 - Line 2740, ref OF 3 K56G, -entry 142 in 000008
112112 -
112114 -  ..
112115 - Line 2880, ref OF 3 1937, -label Srkg in 000008 about 130 lines below
112116 -
112117 -    save stufq
112118 -    immed 9999d
112119 -    dose /c c:\sd\01\0702051.bat
112120 -    g stufq
112121 -    top
112122 -
112123 -        Record is saved to stufq, and has the correct line lengths.
112124 -        0702051 is called to update pointers, add anchors, check format
112125 -        etc., and then is read back as stufq.  The stufq file coming
112126 -        back from processing by 0702051, also, seems to have the
112127 -        correct line lengths, which means nothing so far is causing the
112128 -        problem.
112129 -
112130 -
112132 -  ..
112133 - Line 3010, ref OF 3 68E4, -label 8y2w in 000008
112134 -
112135 -    macro 153
112136 -    macro 157
112137 -
112138 -        This seems to be the problem, restoring the standard profile
112139 -        after 0702051 has run.  Macro 153 is not a problem, so that
112140 -        leaves the SDS profile in macro 157 as the culprit.
112141 -
112142 -
112144 -  ..
112145 - Line 2540, ref OF 3 8O5G, -entry 1571 in 000008
112146 -
112147 -      recl 360
112148 -      recl 2000
112149 -      recl 3000
112150 -      recl 2000
112151 -    -if @94 != 2288 recl 512
112152 -
112153 -        Saving an SDS record, eventually gets around to calling the
112154 -        standard profile for SDS records in macro 157.  This has been
112155 -        setting a default record length of 360 columns to support doc
112156 -        log processing.  Lines longer than 360 are truncated.  Not sure
112157 -        if we can set a record length of 1000 or 2000 to solve the
112158 -        problem.  Will try this.
112160 -         ..
112161 -        Changed recl 360 to 3000 and compiled the program.  Then opened
112162 -        the record on 040102, ref SDS 5 0001, which is larger than
112163 -        most SDS records in the system.
112165 -         ..
112166 -        Medit free command shows for...
112167 -
112168 -             recl 360.......................... 18972
112169 -
112170 -             recl 3000......................... 18972
112172 -         ..
112173 -        This suggests that recl, only applies to an individual line,
112174 -        and does not increase memory demands.  It tells how long a line
112175 -        be, but does not allocate memory uniformly for every line.
112176 -        This seems promising for a solution.
112178 -         ..
112179 -        Testing seems to show that the time required for F2 to save a
112180 -        record varies slightly....
112181 -
112182 -             recl 360.......................... 3 seconds
112183 -             recl 2000......................... 3+ seconds, maybe
112184 -             recl 3000......................... 4 seconds
112186 -         ..
112187 -        Based on this work, tried setting recl at 2000, which is over
112188 -        twice the longest line in Gary's letter today.  Experience will
112189 -        show, if other use cases require further adjustment.  Finally,
112190 -        changed to 3000, but then changed back to 2000, because another
112191 -        type of performance issued has occurred with Subject Index.
112193 -         ..
112194 -        Tried making recl command a conditional in macro 157 so that
112195 -        when F2 is called, whatever record length has been created by
112196 -        using the get command remains.  This should avoid the
112197 -        truncating problem.  Tests showed that for some reason a string
112198 -        of text gets created on every line in the file beginning on
112199 -        column 360, which is a mess.  Not even sure where the text is
112200 -        captured, so it suggests a major memory issue.
112202 -         ..
112203 -        As a result, will stick with a fixed solution, which is not
112204 -        very elegant, but may be adequate.
112206 -         ..
112207 -        After making this change, testing shows the test file Gary
112208 -        submitted is handled correctly, without an loss of line length
112209 -        due to truncating.
112211 -         ..
112212 -        Further experience over the next several weeks revealed impacts
112213 -        setting a high value for recl command.  When recl 3000 or recl
112214 -        2000 were set, SDS failed when scrolling a sorted Subject
112215 -        Index, ref SDS 0 686L, which is the same problem reported on
112216 -        031114. ref SDS 4 0001
112218 -         ..
112219 -        Justifying a para with many lines when Alt F9 calls macro 49,
112220 -        also, causes system failure ref SDS 0 UL8G  Initially, this
112221 -        seemed to also correlate with the recl command.  Further
112222 -        testing of failure when Alt F9 calls macro 49 to justify a
112223 -        paragraph did not establish a correlation with the recl
112224 -        command.  The record on 040610 was tested. ref SDS 9 F566  This
112225 -        para has 24 lines.  System failure occurs consistently when
112226 -        macro 49 is called 34 times.  However, this occurs whether recl
112227 -        is set at the default value of 264 or at 3000. ref SDS 0 AS9K
112229 -         ..
112230 -        So, wound up removing the recl command in macro 153 that is the
112231 -        primary profile for the editor.  This makes the default recl
112232 -        264, set in the Medit code. ref SDS 0 QM5I; and, developed a
112233 -        conditional that sets recl 512 for SDS records. ref SDS 0 OD98
112234 -        Also, set recl 512 in macro file 03501 for the Subject Index
112235 -        profile, because this seems to support scrolling through large
112236 -        indexes at least once. ref SDS 0 686L  Added temporary recl
112237 -        2000 to 00703 to try supportig reading in files with long lines
112238 -        for analysis. ref SDS 0 OC6G
112239 -
112240 -
112241 -
112242 -
112243 -
112244 -
1123 -

SUBJECTS
Recl 3000 Subject Index Fails Scroll through Sorted Index with Over
Get Command Medit Fails after 386 Cycles May Indicate Cause of Medit
Recl 3000 Subject Index Fails Scroll through Sorted Index with Over

1305 -
130601 -  ..
130602 - 03501  Recl 500 Command Added to Subject Index Profile
130603 - Subject Index Crashing When Scrolling Sorted Lists
130604 -
130605 - Over the next several weeks began to experience problem of not being
130606 - able to scroll through sorted indexes beyond line 1177.  Any sorted
130607 - index that had over 1177 lines was crashing consistently, similar to
130608 - the original problem.
130610 -  ..
130611 - This conflicted with prior experience where we could scroll through
130612 - 3,860 lines using 386 cycles of get command for 10 lines each pass,
130613 - reported on 031114. ref SDS 4 0001
130615 -  ..
130616 - Recalled that recl was changed to 3000 in this record, ref SDS 0 QM5I,
130617 - to support the function for analysying documents with long lines, per
130618 - above. ref SDS 0 PP6Y  On a hunch, tried setting recl 300 in the
130619 - Subject Index.  When the index that was failing to scroll, was then
130620 - sorted and scrolled again, the code supported scrolling through the
130621 - entire index.
130622 -
130623 -        The recl command might provide a clue to Morris for finding the
130624 -        cause of get command failure that occurs with 3,860 lines, now
130625 -        that we have found a way to cause a variable.
130626 -
130627 -            [On 040910 Morris fixed problem with get command by
130628 -            eliminating automatic increment to recl. ref SDS 11 0001
130630 -             ..
130631 -            [On 040929 Morris fixed problem with join that improves
130632 -            memory management. ref SDS 12 0001
130634 -  ..
130635 - Therefore, decided to change recl in macro 153 back to 2000, since it
130636 - appears this affects performance some way or another. ref SDS 0 OD98
130637 -
130638 -        Later, on 040706 changed recl to 400, ref SDS 0 QM5I, because
130639 -        subsequent experience indicated SDS fails on the justify op,
130640 -        and it was initially thought this was due to recl 2000;
130641 -        however, further testing does not seem to support this
130642 -        conclusion.
130644 -  ..
130645 - This still accomodates reading into an SDS record or a document file
130646 - an 800 char line that Gary reported having problems processing today,
130647 - per above. ref SDS 0 PP6Y
130649 -  ..
130650 - Line 3560, ref OF 2 UE6K, -label bHpf081 in 03501
130651 -
130652 -    macro 153
130653 -    recl 500
130654 -
130655 -        Added recl 500 command after calling primary profile
130656 -        environment in macro 153, which uses recl 2000, per above.
130657 -        ref SDS 0 QM5I
130659 -         ..
130660 -        Experimenting showed that recl 500 enables scrolling through
130661 -        an entire sorted index at least once now, and so solves a very
130662 -        big problem.
130663 -
130664 -
130665 -
130666 -
1307 -

SUBJECTS
Recl 3000 Subject Index Fails Scroll through Sorted Index with Over
Recl 3000 Justify Para Fails on Very Few Passes Similar to Failure o
49 Macro Justify Double Space between End of Last Sentence on a Line
Justify Fails Recl 3000 on Very Few Passes Similar to Failure on 031

1706 -
170701 -  ..
170702 - 000008  Justify Macro 49 Crashes More Frequently with Recl 2000
170703 - Recl 2000 Causes SDS to Fail When Justifying Para With Many Lines
170704 - Justify Para With Many Lines Causes SDS to Fail When Recl 2000
170705 -
170706 - Experience over the next month or so, shows that the recl command
170707 - seems to cause Medit failure when macro 49 is called to justify a
170708 - paragraph with a lot of lines processed by Medit split and joint
170709 - commands.  Typically, this is not noticable.  Alt F9 can call macro 49
170710 - to justify a para as often as needed, because normally this is only a
170711 - few cycles.  When constructing a big paragraph, however, Alt F9 might
170712 - be used 20 - 100 times calling macro 49, and this seems to accumulate
170713 - small problems to critical mass that causes system failure, reported
170714 - on 031114, when first identified, as an Medit memory issue for
170715 - scrolling a sorted subject index. ref SDS 4 0001
170717 -      ..
170718 -     [On 040610 example of SDS record where program failed repeatedly
170719 -     and consistently with recl at both 2000 and 400 on 34 cycles using
170720 -     Alt F9 calling macro 49 to justify a paragraph "Description," with
170721 -     many lines. ref SDS 9 RU6F  This suggests recl does not impact
170722 -     failure problem for justify, which seems to conflict with the
170723 -     report above on scrolling a sorted index. ref SDS 0 686L
170725 -      ..
170726 -     [On 040929 Morris fixed problem with join that improves memory
170727 -     management. ref SDS 12 0001
170729 -  ..
170730 - Tested an medit file with one paragraph margins 1 79 and 40 lines.  It
170731 - failed on 57 cycles with the default Medit recl 264, i.e., no recl
170732 - command is set by SDS code.  Actually, things are more complicated.
170733 - With the standard profile set called with macro 150, failure occurs on
170734 - 57 cycles.  However, if the standard profile is not set, failure
170735 - begins on the first cycle, but the macro continues to run until cycle
170736 - 60.  Tried setting recl 4000, and this does not seem to impact the
170737 - number of cycles to failure.
170739 -  ..
170740 - Test macro code is....
170741 -
170742 -       * debug 100
170743 -       top
170744 -       relc 4000
170745 -       z 1 1
170746 -       l /</
170747 -       down 10
170748 -       setcnt 110 0
170749 -       m 1 79
170750 -       -label lpa
170751 -       addcnt 110 1
170752 -       -if @110 > 50 debug 100
170753 -       loc_cur 9 20
170754 -       inscnt 110 0
170755 -       rel_cur 8 0
170756 -       macro 49
170757 -       -if @110 > 100 -goto exit
170758 -       -goto lpa
170759 -
170760 -       -label exit
170761 -       -exit
170762 -
170763 -
170765 -  ..
170766 - Work Practice for Using Function to Analyse Docs with Long Lines
170767 -
170768 - Today, try solution of adding temporary recl 2000 when macro file
170769 - 00703 reads in a file for anlaysis, per above. ref SDS 0 OC6G  This is
170770 - not a perfect solution, because we likely need Medit adjustments, per
170771 - reporting on 031114. ref SDS 4 0001  This method can be effective
170772 - using the following practice....
170773 -
170774 -          [On 040706 notified Gary and Wayne of this change.
170775 -          ref SDS 10 0001
170776 -
170777 -    1.  Call function from menu to read in a file for analysis.  This
170778 -        automatically sets recl 1200, which should be adequate for
170779 -        most situations.
170781 -         ..
170782 -    2.  Split the lines with Alt F5 to the left margin, which makes all
170783 -        lines 79 columns.
170785 -         ..
170786 -    3.  Scroll through the profile, F1 F1 F1, to reset default recl of
170787 -        400, because under step 2, all lines are less than 80 columns.
170789 -         ..
170790 -    4.  F2 to save record, and then do normal editing.  Alt F9 to
170791 -        justify text should now (hopefully) not crash because of the
170792 -        recl is 1200, since it is reset in step 3.
170794 -         ..
170795 -        When the record is closed and the SDS session ended, then when
170796 -        the record is next open in a new session, Alt F9 to justify
170797 -        text should work normally, which means it will take longer to
170798 -        fail, that with recl 1200.
170800 -  ..
170801 - Notes....
170802 -
170803 -    This practice is likely needed for analysing files that do not have
170804 -    long lines, because the recl is set to 2000 by the code without
170805 -    regard to the file being read.  The basic point is that after
170806 -    reading in a document, there is probably a limit on how many times
170807 -    Alt F9 can be used, espcially on para with many lines, because that
170808 -    process consumes memory toward failure using split and join
170809 -    commands, under the record on 031114. ref SDS 4 0001
170811 -     ..
170812 -    If Alt F9 is not used much, there is likely minimal danger of the
170813 -    session crashing and losing work.  If it is need for heavy editing,
170814 -    then the four (4) steps listed above should likely be followed.
170815 -    ref SDS 0 NF4K
170816 -
170817 -
170818 -
170819 -
170820 -
170821 -
170822 -
170823 -
170824 -
170825 -
170826 -
170827 -
1709 -