More weirdness...

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Marjolein Katsma
    Expert
    • Oct 2007
    • 98

    More weirdness...

    OK, so I survived my adventure with the endless loop, and Craig helpfully explained some of the process behind Cirrus' handling of File Formats. So I happily go back to working on my Apache File Format...

    The first action of course is to replace that brilliant regex with one that does not match an empty string, but actually matches a path. To my utter surprise it still looks as though it's looping - but it's not: it just takes an extraordinary much longer time to "build" the grammar than without that Grammar item. I try to refine it a bit, but that doesn't help.

    So, I take it out, and go on to add other Grammar items I'll need. Building times go back to what is normal on my admittedly old machine. Following Craig's hint, I've now turned "off" my Apache FF, so loading a text compare will initially use <default>; then I switch to Apache from the "guy-with-a-cap" button to see the actual building time. Looking at Task Manager during the process, you can sort of follow what's happening; Attachment 1 shows a typical sequence:
    1. Situation just after opening Cirrus: Mem usage and Peak mem usage are identical, and it takes a small amount of Virtual memory as well.
    2. Now I've started a (Folder compare) session
    3. After loading an Apache config file for Text comparison - with the <default> FF; then
      I switch to my Apache FF, and I gradually see Mem usage and Virtual mem usage rise in parallel, until just over 230,000K (no screenshot)
    4. Finally, when the building and comparison is done, the file shows up with color coding, and the Mem usage and Virtual mem usage both drop down ... only Peak mem usage shows the level it's been to. Total time: 136.19 seconds

    While loading time is a bit longer (as expected) than before I added my new collection of Grammar items, but not excessively so. The FF used for this is in the second Attachment (Apache_1.3_20071021_1.zip), and a small test file (just in case you don't have any Apache config files handy) is in Attachment 3 (htaccess.zip). (I was comparing identical files in different directories, but you could just compare it to itself.)

    Now, when I add my "Path" Grammar element back (at the end of the FF), suddenly loading time is almost an order of magnitude larger... it took me quite a while to figure out what was happening, though I still don't understand why. Here's what's happening (see fourth attachment for the screenshots, and the fifth (Apache_1.3_20071021_2.zip) for the FF):
    1. Situation just after starting a new comparison, having switched to Apache again: mem use and Virtual mem use are slightly up, Peak Memory use still shows what it was after the previous comparison; you can also see it's taking a good deal of CPU time (almost as much as it can take)
    2. Now a series of shots while it's building; in this first one you can see how it's already overshot the previous Peak by quite a bit, but Mem usage and Virtual mem usage are still climbing in parallel, and CPU is quite high;
    3. But then as Virtual memory usage continues to climb, Mem usage drops and then starts going up and down, while CPU gets quite low: it's now using almost as much Virtual memory as it can get I may actually get warnings about low Virtual memory at this stage): it's thrashing now, so most of its time is spent waiting for new memory blocks to become available; Virtual memory usage actually crept up to over 480,000K
    4. This is what I saw when building was finally complete, and the comparison became visible: as with the first run (without the "Path" Grammar item), both memory and virtual memory usage suddenly drop - but Virtual memory usage is more than four times as high, and final Peak memory (max Mem it has used) is also quite a bit higher than in the previous run. And loading time was 817.38 seconds.


    continued in next message - running out of attachment slots...
  • Marjolein Katsma
    Expert
    • Oct 2007
    • 98

    #2
    [continued]

    My final series of screenshots shows a similar run (with the "Path" Grammar item) as seen with Process Explorer and is typical: it starts out with gradually and regularly increasing memory use, and high CPU - only occasionally interrupted by a background process that takes some CPU away from it, and flattens out the memory-growth curve for a bit. Then (Virtual) memory hits a maximum and you see the curve become totally flat - it's at this point that the Virtual memory warnings pop up - and CPU visually dropping down. Finally, it's done building, and memory usage quickly drops down again (but to a much higher level than it started out with).

    It doesn't always hit my maximum Virtual memory level, but in this case it did - and the result was a whopping 1035.87 seconds loading time! (And that series was shot before adding a bunch of other Grammar items.)

    Now what is going on here?

    I don't have an issue with "normal" loading times - my machine is slow. I also understand that extra memory usage can lead to thrashing, and it's the thrashing that causes the order-of-magnitude longer processing times.

    But what is causing the exceptionally larger memory needs caused by adding this particular Grammar item while adding other ones doesn't make much difference? I just cannot explain why this particular regex seems to cause it to need more than twice the amount of memory (about 480,000K vs. 230,000K) while adding other items don't have such an influence. It's not an endless loop, and it works, too. It's not the number of Grammar items either, since I first took it out and added a whole bunch of others, and things were still normal. It just that this particular Grammar item behaves (apparently) very, very differently than any of the other Grammar items.

    Why?

    Comment

    • Aaron
      Team Scooter
      • Oct 2007
      • 16005

      #3
      Still investigating

      I haven't found the exact cause yet, but it doesn't seem to be that particular rule. Deleting a number of the other rules also greatly reduced the amount of time it takes to open (the example file for me is about 50 seconds with version 2 of your grammar, down to 10 or so if I delete the top half of the grammar definition). It is possible that that particular line, in combination with other lines greatly increases the length.

      I'll continue to look into it. I just thought you may want to know what progress I've made.
      Aaron P Scooter Software

      Comment

      • Marjolein Katsma
        Expert
        • Oct 2007
        • 98

        #4
        Thanks for the progress report, Aaron - I know time increases with just more rules, but usually it does so in proportion (or so it has seemed so far); I've never seen such a big jump caused by one rule though (while adding others in-between hardly made a difference, or at least not out of proportion...)

        Comment

        • Aaron
          Team Scooter
          • Oct 2007
          • 16005

          #5
          Question

          Question for you:

          If you leave the program open, switch to the rule that takes awhile, then switch to another rule, and within the same instance switch back to the rule that takes awhile, does it take as long to load the 2nd time?

          (Let the rules complete loading each time.)
          Aaron P Scooter Software

          Comment

          Working...