Is this possible using DataViewer?

[Guest]

I could do with some help on this one...

Our systems generate various text based logs where the format is always "yyyy-mm-dd hh:mm:ss.xxx" (where xxx is msec.) and then followed by a null terminated message.

I think it should be possible to use BC to 'compare' two log files and make BC align them so that both logs are sorted in chronological order.
Or: only events that occured at exactly the same time would be aligned.

Successful alignments would be rather unlikely, but the comparison would still be very useful to us as it enables us to unfold the sequence of events.

Using the DataViewer plug-in, I was partly successful in the chronological alignment by using the 'Fixed Column Fields', creating two fields with the first field ending at column 23 (end of timestamp) and the second column - because the plug-in leaves no alternative - I gave a width of 255.
The first column (timestamp) was declared 'key', the second (message) was declared 'unimportant'.
With the option to 'sort tables by key columns' checked, BC had no problems getting the timestamps nicely aligned and sorted in chronological order

However, the second column was rendered illegible due to the fixed width of 255 chars.

Is there any other way of getting timestamped lines aligned in chronological order without garbling the message?
Perhaps even by using a different plug-in?

Iwar

[Chris]

Re: Is this possible using DataViewer?

The Fixed Width format will not work unless the entire record is fixed length. If the messages are variable width, it will keep reading the 255 characters into the next message. The Fixed Width format reads files as a stream of bytes and doesn't treat line terminators in a special way. It does not stop at the end of a line.

The easiest way to compare these in the data viewer might be to use a white space delimiter so you end up with an yyyy-mm-dd column and a hh:mm:ss.xx column. Mark them both as key columns and sort on key columns.

If you want to use a fixed width format, another option is to pre-process the files and pad the message portion to a fixed width.

One other option for sorting is to use the Sort Rule from our Additional File Viewer Rules page.

[Guest]

Re: Is this possible using DataViewer?

Chris,

Thanks for the swift reply.

My mistake, but I should not have used the word 'sort' in my query.
Each log is already sorted in chronological order (because messages are written to the file in that way), so there is really no need to sort each file.
I just want to align two files in chronological order.

Reading your reply, I am slowly coming to the conclusion that BC cannot assist me on this one. Shame, but all is not lost.

[Erik]

Re: Is this possible using DataViewer?

To use only the first 23 columns for aligment:
<ul type="square">[*]In the File Viewer, select Tools -> Pick Rules -> New Rules....[*]On the General tab, enter appropriate Name and Associated with values.[*]On the Importance tab:
- Click New in the Unimportant Text section.
- Change the Category to Columns.
- Enter '24' as the Start Column and check stopping at end of line.
- Click OK.[*]On the Alignment tab, check Never align mismatches.[*]Click OK.[/list]

We will consider supporting this in a future Data Viewer release.

[Guest]

Re: Is this possible using DataViewer?

Erik,

Thanks for the info, but this doesn't quite align the two files in chronological order.
Instead it just generates two massive differences; one block being fileA in its entirety and the other block being fileB in its entirety.

I reckon what I thought to be possible just cannot be done with BC.
It would be unreasonable to assume that BC2 would be the answer to all my problems. It already does so much for me.

Setbacks make life interesting...

Iwar

[Chris]

Re: Is this possible using DataViewer?

Sorry that our suggestions did not help. If you have some example log files that can go outside the company, you're welcome to email them to [email protected] and we will try to come up with a better suggestion.

[Guest]

Re: Is this possible using DataViewer?

Have you considered using a text editor? My editor of choice (Multiedit) allows you to sort a file on selected columns. What I would do in your situation is edit the two logs you want to compare to indicate which system they come from (add "A" or "B" to the beginning of each line, for example), combine the two logs into one file, then sort the merged file on the columns with the timestamp.