If you can think of a good solution feel free to post your ideas, today I will be writing about my solution and things that worked and did not work, and things that still do not work.

The situation I am referring to here is parsing a call data file known as a BAF file.  The file structure and definitions for these files are described in detail in a document known as the GR-1100 which is published by Telcordia.

To parse this file your program must know and recognize about  1000 different field types and 200 different record types.  Fields are organized in records, you can probably say that records are defined by their fields.  Oh, also, there are about 100-200 defined modules that theoretically can be appended to any record.  Each module is again a set of fields.  So if you were to summarize this file it would be laid out like this.

Record Start
Fields
Record End
Any number of modules
New Record

I have already talked about pluggable factories, which is the main design element I chose when first designing bafprp.  I decided to create a maker factory for fields and records, and define each field and record as a class.  Now this was BEFORE I found out exactly how many different types of fields and records there were in total.  All I had to build off of was an open source project called bafview which contained a very small subset of the actual fields and records that are defined in the GR-1100.  I only recently learned about the inadequacies of this subset when I was notified about missing records and fields that were not in bafview, and thus, not in bafprp.

When I released version 1.0 of bafprp it contained only about 100 different fields and records.  A number small enough that the maker factory pattern made sense.  Obviously however this needed to be changed if bafprp was to become a complete GR-1100 compliant program.  So now we must free ourselves from the basic design and fall back to the drawing board.  Something that many designers, myself included, are very bad at doing, and so I did not.  Instead I though about another solution, which extended the concept of pluggable factories into what I will very loosely call run time programming.

Now do not jump to any conclusions about a self correctly, self building, automated executable.  This is NOT run time programming in the strict sense of the word.  What I did not was allow for the extendability of certain classes at run time.  Meaning that a basic class is interpreted hundreds of different ways depending on run time data. Using this method I created a handful of basic classes that could be extended by strings passed by a user to define the specifics of the field.  To define the default fields the default data is hard coded into the program, but the user has complete control over any field in the program through two specific command line options.

There are some similarities to basic scripting, if that is what you were thinking.  There is one major difference however, bafprp remains in complete control over the program.  With scripting languages the program surrenders part of itself so the user can change default behavior.  In bafprp the basic classes remain in complete control of the operation, but the user has complete control over the data.  Something that I have not seen done in any other program yet.

So by now if you are still reading you are probably itching for some source code.  The design is pretty simple, we have eight or so basic types using the pluggable factory design.  These basic types operate on string data passed to them, how they operate and the string syntax is completely up to them, however for these elements I had them conform to property name, property value pair of strings for operating data.

Here is a simple example of my Date field type

std::string DateField::getString() const
	{
		LOG_TRACE( "DateField::getString" );

		std::string ret;
		if( !_converted )
		{
			LOG_WARN( "Tried to get string before field was converted" );
			ret = "";
		}
		else
		{
			char year[5] = "";
			time_t ltime;
			struct tm* mytm;
			ltime = time( NULL );
			mytm = localtime( &ltime );
			strftime( year, sizeof( year ), "%Y", mytm );
			year[3] = _return[0];

			property_map::const_iterator itr = _properties.find( "format" );
			if( itr != _properties.end() && itr->second.find( "Y" ) != std::string::npos && itr->second.find( "M" ) != std::string::npos && itr->second.find( "D" ) != std::string::npos )
			{
				ret = itr->second;
				ret.replace( ret.find("Y"), strlen( year ), year );
				ret.replace( ret.find("M"), 1, _return.substr(1,2) );
				ret.replace( ret.find("D"), 1, _return.substr(3,2) );
			}
			else
			{
				std::ostringstream os;
				os << year << "-" << _return[1] << _return[2] << "-" << _return[3] << _return[4];
				ret = os.str();
			}
		}

		LOG_TRACE( "/DateField::getString" );
		return ret;
	}

The date field will take a format property and change the data it converted appropriately. This is an example of a basic property that each field can have. For a more complex operation here is the same switch function

std::string SwitchField::getString() const
	{
		LOG_TRACE( "SwitchField::getString" );

		std::string ret = "";
		if( !_converted )
		{
			LOG_WARN( "Tried to get string before field was converted" );
			ret = "";
		}
		else
		{
			props_pair switches = _properties.equal_range( "switch" );
			std::string sw;
			if( switches.first == switches.second )
			{
				// No "switch" property so assume we switch on character 0
				sw = "0" + _return;
				sw.resize( 2 );
				property_map::const_iterator string = _properties.find( sw );
				if( string != _properties.end() )
					ret = string->second + " ";
				else
					ret = "Unknown: " + sw.substr(1);
			}
			else
			{
				for( property_map::const_iterator pos = switches.first; pos != switches.second; pos++ )
				{
					sw = pos->second + (char*)&_return[ atoi( pos->second.c_str() ) ];
					sw.resize(2);
					property_map::const_iterator string = _properties.find( sw );
					property_map::const_iterator desc = _properties.find( pos->second );
					if( string != _properties.end() )
					{
						if( desc != _properties.end() )
							ret += desc->second + " = " + string->second + " : ";
						else
							ret += string->second + " : ";
					}
					else
						ret += "Unknown: " + sw.substr(1) + " : ";
				}
				ret.resize( ret.length() - 3 ); // remove last ':'
			}
		}
		LOG_TRACE( "/SwitchField::getString" );
		return ret;
	}
}

Now here is my general disclaimer, if you see problems with this code feel free to let me know I know it is not 100% efficient and perfected and this is because I have never done this before so I was making up design as I wrote the code.  Eventually I will do yet another rewrite and fix the code with a more complete understanding of the design.

But the switch function needs some explaining.  I think it would be best to see an example switch field

		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "datatype:switch" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "size:1" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "desc:Called Party Answer Indicator" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "switch:0" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "00:Called Party Answer Detected" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "01:Called Party Answer not Detected" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "02:Answered Attempt" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "03:Simulated Called Party Off-Hook Indicator" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "04:NCD, CAS, Blocked After Answer" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "05:NCD, CAS, Blocked Before Answer" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "07:Service Features Not Provided, Call Answered" );
		FieldMaker::setFieldProperty( "calledpartyanswerindicator", "08:Service Features Not Provided, Call Unanswered" );

Ok so first we set the basic data that defines a field: type, size, and desc. This is all that is required to make a field. We then setup a switch on character zero by setting the switch property to zero. The properties after that list the values for the switch. If the switch character is zero print “Called Party Answer Detected,” and so on. The syntax for these are to have the property name before the value, separated by a colon. This of course is completely arbitrary and can be anything depending on how you want to assign properties. The property name syntax is the switch number first, then the switch value. Only two characters are required because I only allow one character switches.

The trick here is how the switch class uses its properties.  For the date object, all it did was change the format of the date, however the switch completely depends on the properties to be set correctly to function properly.  The properties of a switch class define the switch, whereas the properties of the date class hardly do anything.  It is this range of effective designs that make the application successful and indicate proper orthogonality, which is the number one most important design philosophy.

I am finding that increasingly complex posts are getting difficult to write and even more difficult to read so from now on I am going to try and keep posts small and exercise a specific element instead of the whole design. I left a lot out of this post simply because this is a blog not a research paper, so I will try and get some more data posted soon.
In the meantime, you can always check out the source at bafprp’s web site.

DSN-Less ODBC Connection String

I found it really annoying that in order to connect through ODBC you had to setup a data source. In windows this process involves going to the control panel, administrative tools, and data sources. In Linux, this involves setting up the ODBC driver, setting up unixODBC to recognize the driver, and finally setting up the server information and data source in both your driver and unixODBC.

I spent some time looking online and found a nice and easy way to connect without an external data source in windows. Basically it involves supplying all the information in one string like so:

std::string dsn = “DRIVER=sql server;DATABASE=” + _database + “;SERVER=” + _server + “;Uid=” + _user + “;Pwd=” + _password + “;”;

You must also use SQLDriverConnect instead of the usual SQLConnect function since the former will accept a dsn string, instead of just a dsn name and login info.

In linux its still a little annoying. You must still define your driver in unixODBC for starters but you do not need to setup a dsn or give server type information. Also, the support the DSN-less connections depends on the driver you choose. Since my primary use for bafprp was to connect to a ms sql database, I used FreeTDS which as of a recent update does have dsn-less capability. If you are using something else, please consult the documentation about this as well.

In FreeTDS the basic connection string goes like this

std::string dsn = “DRIVER=FreeTDS;SERVER=” + _server + “;Uid=” + _user + “;Pwd=” + _password + “;DATABASE=” + _database + “;TDS_VERSION=8.0;Port=1433;”;

TDS_VERSION and Port are FreeTDS specific settings, but the basic idea remains the same. Also it should be noted that different version of ms sql require a different TDS_VERSION. If you are using FreeTDS its important to know the correct version.

As a final side note about connection strings, make sure to include the terminating semi-colon. If you find your string unable to connect no matter what you do, this is probably the problem.

File Output

Sometimes when your program terminates unexpectedly, text being written to a file can be lost. If you are using fprintf or writef or any of the stdio functions for logging you will probably come across this problem. The only solution I found to guarantee that the text gets written is to use std::fstream and call the flush() method when you are done writing. Flush will make sure the text gets written before returning so it will be a bit slower, but for something as important as logging this is important.

Duplicate Removal

I remember reading about a similar situation in Programming Perls. It involved making a hash of your data and comparing collisions I believe. The situation I was in was that I had thousands of records that could be byte for byte duplicates with any other record in the original binary file. Like Programming Perls states, comparing each record against every other record is a joke. Hashing is definitely a better solution, but you need not create such a complex hash table for something like this. I ended up pulling a crc32 method to calculate the crc for the originals bytes in the record. After the record parsing was completed I sorted the array of records by their crc value. It was then a very easy procedure to remove any duplicates since they would be sitting right next to one another with the same crc.

One thing to note however, std::unique in algorithm.h seems like a wonderful function, but I could not get it to work for the life of me. It is supposed to sort the array, and place any duplicates at the end of the array, returning an iterator pointing at the start of the duplicates. Theoretically you can then use std::remove to remove all elements after that point to erase the duplicates. I managed to get the list sorted and std::unique did identify the correct number of duplicates ( the number of elements after the returned iterator matched the number of duplicates I later removed by said method ), but it did not seem to place the real duplicates at the end of the array. I ended up removing valid records that were unlucky enough to have a high crc and thus were at the end of the array.

So in the end I went through the entire list and removed neighbors with the same crc, which worked quite nicely.

Static Factories

I do not believe I have covered this concept here before so I will do a brief summary. This subject requires a much more detailed post but here is the cut and dry. If you are familiar with abstract factories you might know its a bit of a pain to add a new object. If you are using some kind of enumeration you need to add the id to that list, and add the correct new object code in the create method of your factory. Eventually you end up with enumerations of 100+ elements and a very scary switch statement. Fear not however, there is a better way!

Imagine a system where all you need to do to add a new object in your factory is compile a cpp. Thanks to static factories this is not just a dream. The trick involves a very natural side effect of static objects. The basic idea is simple. You have a main ‘maker’ class with a static registry variable that stores the names and pointers to other maker classes. When you want an object to be built through this factory you need to create a simple maker class for your object with a method called make, which is defined in the parent maker as pure virtual. The child maker defines an instance of itself as static and thus when the program starts it is created.

When the child gets created it calls the parent constructor with the name, or some other form of identification, of the object it creates. The parent maker then adds the information to its static database. When the programmer needs an instance of that object it simply calls the parent’s make method which looks at the database, pulls up the correct child maker and has the child make the object.

This technique is quite powerful if used correctly. It is absolutely necessary for data driven applications in my opinion, and very handy when working with any kind of file data. Using this method you can seperate file structure from logic in a very effective and pretty design.

My first logging framework was log4cplus, which is a very nice and complete logging framework for C++. I used it for one linux application I wrote and it works perfectly despite being several years old. Unfortunetly I was uncomfortable with how much work it took to write a message each new function ( two lines instead of one ) and it would not compile in windows so for my next application, bafprp, I set out to write my own.

When I completed the class I was left with one macro to print a log message depending on the level you wanted, ie, LOG_TRACE( string ), LOG_DEBUG( string ), etc. I then made sure I was adding trace messages to each and every function as I made them, instead of later on in the design. For example, an empty function would look like


void BafRecord::getType()
{
LOG_TRACE( "BafRecord::getType" );
LOG_TRACE( "/BafRecord::getType" );
return;
}

Each function would have a start and stop trace message.

Now this might raise an eyebrow or two but I assure you it definitely helps when your program crashes and you do not have a nice debugger on hand. Like say, if a non-technical user is using it.

This is how my linux application was programmed and I went the extra step in bafprp to work these in while I was working. Enough about this though, a short while ago, after adding one of the major structure types to the program my application slowed down from parsing a 12 meg file in 2 seconds to 2 minutes and I was greatly concerned over the well being of my design.

I tried many things, first I greatly reduced the number of memory copies my program executed, then I changed my file input so that it would read the entire file at the start and reference a data bank instead of reading the file each cycle. However none of these things put a dent in the processing time. So then I went online to try and find a nice and easy code profiler. Code profilers will basically watch your program execution and tell you which function your program is spending the most amount of time in. I ended up finding LTProf which allowed me to profile my program without recompiling or changing my program at all. I am actually very surprised at how well it works with compiled binaries. After running a release version of my program it was still able to accuratly determine function names and operate like it had a window into the source code.

I found that my time stamp function, NowTime, which returns the current time as a string, was taking a noticably large amount of my program’s time. Thinking back to what code uses this function I discovered the flaw.

When I wrote the log program I wrote the log level exception into the log class. This way if the program tried to log a trace message it would get sent to the output class, which would then pass it on to the logs if the level was at or below the log level type. However this was not enough. as it turns out simply creating the log message twice in each function had a substantial effect on the processing speed of my program.

Needless to say as soon as I moved the log level check to the macro the process speed dropped from 2 minutes to 30 seconds. Now some might say that trace logs this detailed are excessive, however I believe they can help greatly when dealing with a malfunctioning program. So as a final statement, be careful when you design systems, and beaware of how to use your tools. And if your program runs 3 times longer then a similar program, there is probably something horribly wrong.

I also created a second project of a more work related function. This project deals with reading and parsing Bellcore BAF files created by various soft switches in use around the world. These records contain call records in a highly coded and formated to Telcordia specifications in their GR-1100 document. This document is not very cheap to come by so I have started work on improving one of the best free parser’s available today, bafview. You can find this project here http://code.google.com/p/bafprp/