Showing posts with label Code Libraries. Show all posts
Showing posts with label Code Libraries. Show all posts

Tuesday, March 8, 2011

FileHelpers is my new best friend

I have gained countless tools and information over the years from Scott Hanselman’s blog . My new favorite tool is now Marcos Meli's FileHelpers . It is great tool for dealing with delimited or fixed length field messages. Despite the name, it is extremely useful in dealing with messages that are memory-based as well. I wish I had this tool years ago, it would have saved my countless hours of development! In particular, dealing with problems in data or record length changes is a teeth-gnashing, prototypically “bejabbers” experience.

As a longtime C++ developer, the only feature I missed in .NET is the ability to have structures/union of character arrays to deal with fixed length field messages. In C++, this feature relies on the ability to directly access memory. FileHelpers gives me a similar capability using reflection and attributes to accomplish this task. You adorn a class and fields with FileHelpers specific attributes and it knows how to do the rest. The FileHelpers documentation is superb, unlike many open source project, so I’ll just include a brief excerpt of the documentation here to whet your appetite.

[FixedLengthRecord()]   
 public class Customer   
 {   
     [FieldFixedLength(5)]   
     public int CustId;   
        
     [FieldFixedLength(20)]   
     [FieldTrim(TrimMode.Right)]   
     public string Name;   
}

As cool as reflection is, performance is always a potential concern. FileHelpers is a great illustration of how .NET can create highly-performant code. Many times developers, including myself, only think of reflection as a way to access methods, properties, and fields dynamically. Using Reflection.Emit, as FileHelpers does, is a mechanism to allow code to be generated at runtime. Combined with caching, dynamic code generation allows excellent performance.

To work with FileHelpers I create a wrapper around usage of the fixed length record.


using System;
using System.Collections.Generic;
using System.Text;
using FileHelpers;
    /// <summary>
    /// This generic class provides convenience functions for the FileHelpers library.
    /// This class implements the Facade design pattern by encapsulating the FileHelpers
    /// engine and providing a simple interface to that functionality.
    /// </summary>
    /// <typeparam name="T">the class type representing a message. This type must use the 
    /// FixedLengthRecord attribute</typeparam>
    public class FixedLengthMessageParser2<T>  where T:class
    {
        FileHelperEngine engine;
        /// <summary>
        /// Initializes a new instance of the <see cref="FixedLengthMessageParser&lt;T&gt;"/> class.
        /// It creates the file helper object.
        /// </summary>
        public FixedLengthMessageParser2()
        {
            engine = new FileHelperEngine(typeof(T));
        }
        /// <summary>
        /// Parses the specified aggregate data.
        /// </summary>
        /// <param name="aggregateData">The aggregate data.</param>
        /// <param name="records">The records found to be contained in the data</param>
        /// <param name="resultText">Results of parsing. Empty string if 
        /// successful, or exception details otherwise</param>
        /// <returns>True if able to parse, False otherwise.</returns>
        public bool Parse(string aggregateData, out T[] records, out string resultText)
        {
            records = null;
            resultText = String.Empty;
            try
            {
                records = (T[])engine.ReadString(aggregateData);
            }
            catch (Exception ex)
            {
                resultText = "entry " + typeof(T) + " [" + aggregateData
                    + "]couldn't be read due to error " + ex.ToString();
                return false;
            }
            return true;
        }
 
        /// <summary>
        /// Provides a string representation of a record.
        /// </summary>
        /// <param name="msg">The MSG received as input.</param>
        /// <returns>String representation of a record.</returns>
        public string ReturnToString(T msg)
        {
            string retVal;
            retVal = engine.WriteString(msg.AsEnumerable());
            retVal = retVal.Substring(0, retVal.Length - 2); //trim end of line characters
            return retVal;
        }
 
 
        /// <summary>
        /// Parses a single record
        /// </summary>
        /// <param name="singleData">A single data record.</param>
        /// <param name="fixedLengthRecord">The fixed length record.</param>
        /// <param name="resultText">Results of parsing. Empty string if 
        /// successful, or exception details otherwise</param>
        /// <returns></returns>
        public bool ParseSingle(string singleData, out T fixedLengthRecord, out string resultText)
        {
            T [] records = null;
            fixedLengthRecord = null;
            resultText = String.Empty;
            try
            {
                records = (T[])engine.ReadString(singleData);
                if (records.Length == 0)
                {
                    resultText  = "entry " + typeof(T) + " [" + singleData
                       + "]couldn't be read due to empty data set";
                    return false;
 
                }
                fixedLengthRecord = records[0];
            }
            catch (Exception ex)
            {
                resultText = "entry " + typeof(T) + " [" + singleData
                    + "]couldn't be read due to error " + ex.ToString();
                return false;
            }
            return true;
        }
    }

Wrasslin with Log4Net

I have a love-hate relationship with Log4Net. It is powerful, capable of doing everything I have ever wanted, yet configuration is always a challenge for me to get right. I wish there was a good, free GUI configuration tool available for it. In this post I will share the current Log4Net configuration that I am using. It is a moderately sophisticated configuration, relative to many of the posted samples. This example shows use of console appenders, event log appenders, file appenders, and multiple loggers. I also illustrate how to call Log4Net in a way that allows logging configuration changes to take effect immediately while a process is running.

Here are some points of interest. As I stumble through a configuration, the log4net.Internal.Debug AppSettings key is very helpful in showing me where my mistake lies. In order to take advantage of multiple loggers, it took me a while to figure out the secret sauce that made things work. First, loggers are hierarchical in nature. I found that I needed to specify all appenders used at the root logger level. Then each logger name I used specified the ones I used it for it. The key here was setting Additivity=”FALSE” otherwise I got an error indicating that the appender was closed.
I also found the concept of a single log message being able to be processed by multiple appenders to be powerful. One use was that I use a ConsoleAppender while I am testing to see the Log4Net output without having to pull up a log file. The message is still written to logs too, so I can verify that I log precisely what I intend. At the other end of the logging severity, I log all errors (both ERROR and FATAL) to a file but also log fatal errors to the Windows Event log for easy processing.
A couple of items on my to-do list are exploring buffered file appending and rolling over logs after they reach a maximum size. The BufferingForwardingAppender allows you to utilize the Unit of Work pattern to by aggregating a bunch of messages for the expensive, synchronous act of logging. The drawback to using this approach is that if you are concerned about troubleshooting program abends, you might lose the last few entries in this scenario. Also, although Log4Net is generally threadsafe, this appender is not.

<?xml version="1.0"?>
<configuration>
  <configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
  </configSections>
  <appSettings>
    <!--if having trouble getting log4net to work turn its detailed internal logging on-->
    <!--<add key="log4net.Internal.Debug" value="true"/>-->
  </appSettings>
  <log4net>
    <root>
      <level value="ALL" />
      <appender-ref ref="FatalFile" />
      <appender-ref ref="ErrorFile" />
      <appender-ref ref="WarningFile" />
      <appender-ref ref="DebugFile" />
      <appender-ref ref="SomeInfoFile" />
      <appender-ref ref="SomeExtraFile" />
    </root>

    <logger name="PrimaryLogs" additivity="FALSE">
      <level value="ALL" />
      <appender-ref ref="FatalFile" />
      <appender-ref ref="ErrorFile" />
      <appender-ref ref="WarningFile" />
      <appender-ref ref="DebugFile" />
      <appender-ref ref="SomeInfoFile" />
    </logger>

    <logger name="ExtraLogs" additivity="FALSE">
      <level value="ALL" />
      <appender-ref ref="SomeExtraFile" />
    </logger>
    <appender name="FatalFile" type="log4net.Appender.EventLogAppender">
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%5thread] %type{1}.%method() - %message%newline%exception" />
      </layout>
      <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="FATAL" />
        <levelMax value="FATAL" />
      </filter>
    </appender>
    <appender name="SomeExtraFile" type="log4net.Appender.RollingFileAppender">
      <file value="C:\logs\ AppLateMessage.txt" />
      <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
      <appendToFile value="true" />
      <staticLogFileName value="false" />
      <rollingStyle value="Date" />
      <datePattern value="'.'yyyyMMdd-HH'.log'" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%5thread] %type{1}.%method() - %message%newline%exception" />
      </layout>
      <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="WARN" />
        <levelMax value="WARN" />
      </filter>
    </appender>
    <appender name="ErrorFile" type="log4net.Appender.RollingFileAppender">
      <file value="C:\logs\ AppError.txt" />
      <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
      <appendToFile value="true" />
      <staticLogFileName value="false" />
      <rollingStyle value="Date" />
      <datePattern value="'.'yyyyMMdd-HH'.log'" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%5thread] %type{1}.%method() - %message%newline%exception" />
      </layout>
      <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="ERROR" />
        <levelMax value="FATAL" />
      </filter>
    </appender>
    <appender name="SomeInfoFile" type="log4net.Appender.RollingFileAppender">
      <file value="C:\logs\ AppSomeInfoFile.txt" />
      <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
      <appendToFile value="true" />
      <staticLogFileName value="false" />
      <rollingStyle value="Date" />
      <datePattern value="'.'yyyyMMdd-HH'.log'" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%5thread] %type{1}.%method() - %message%newline%exception" />
      </layout>
      <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="INFO" />
        <levelMax value="INFO" />
      </filter>
    </appender>
    <appender name="WarningFile" type="log4net.Appender.RollingFileAppender">
      <file value="C:\logs\AppWarning.txt" />
      <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
      <appendToFile value="true" />
      <staticLogFileName value="false" />
      <rollingStyle value="Date" />
      <datePattern value="'.'yyyyMMdd-HH'.log'" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%5thread] %-5level %type{1}.%method() - %message%newline%exception" />
      </layout>
      <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="WARN" />
        <levelMax value="FATAL" />
      </filter>

    </appender>
    <appender name="DebugFile" type="log4net.Appender.ConsoleAppender">
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%-5level %type{1}.%method() - %message%newline%exception" />
      </layout>
      <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="DEBUG" />
        <levelMax value="ERROR" />
      </filter>
    </appender>
  </log4net>
</configuration>