When working on projects its helpful to know in which direction you are headed. Its good to place goals, do a bit of planning, and get things right to save time in the future. Unfortunately however, there is little to no information online on how to successfully manage and keep a big software project going. Sure there are management books on how to motivate people, or how to lead coworkers, or even how to plan every step of the project with careful diagrams, flow charts, and process graphs. As far as I could find a year ago there was nothing on how to get down and dirty and manage things like a build process, or an automatic unit test suite, or how to deal with different platforms in your build system. For the past year I have been working on a big project that required all these things and whenever I fell into a trap, or discovered a new idea, or came up with a new operation I wrote down little snip-its to be eventually published in this article. I hope others looking for information on how to manage a project effectively and efficiently will find some of these ideas very helpful.
You are currently browsing the archive for the Programming category.
Tags: advice, dry, gmake, kiss, linux, Programming, project management, series, visual studio
I ran into an interesting problem the other day that I think is worth sharing. Fans of the pluggable factory design pattern, inversion of control, etc will be interested to know that these methods do not work very well when compiled into static libraries. Sounds obvious but what may not be so obvious is why.
I stumbled onto this problem when I changed one of my exe projects to a static library so I could setup two exe’s, one for unit tests, the other for releasing. (Which by the way is a wonderful way to unit test an exe project) Most of my projects these days use some form of inversion of control so I had static variables creating maker classes which would register themselves with the factory. As explained in one of my previous articles here.
When I changed my project to a static library and linked it to a separate exe project my maker classes stopped registering themselves. Why does this happen? Well after a bit of a research I found out that when you link in a static library, the linker only links code into your project that you actually use. All extra code is left uninitialized and put in a corner to sit while the big boss code runs. A very interesting side effect of static linking.
Anyway, to fix this problem in visual studio you need to set “Use Library Dependency Inputs” in Linker options on your exe project. This will make the linker link in all the object files themselves, instead of selectively linking them as needed. I am researching some way to do this per library, because as of right now if you set this it will link ALL libraries like that.
In linux it would seem the option is -z allextract, but I have not tested this one.
If you need your static objects from a static library to be initiated like normal, this should do the trick.
Tags: factory, Help, pluggable factories, Programming
User defined types in Microsoft SQL are really handy. Using these objects you can construct queries like
select duration.minutes from cdrs; select duration.seconds from cdrs;
You can also program them to read weird string inputs that other data types can not handle. Recently I wrote a custom data type to read a duration field of the format minutes:seconds.tenths; so standard input went something like ’002:34.2.’
Writing a custom data type is actually very simple. You just open a new sql data type project in Visual Studio and VS will integrate nicely with your sql server; even going so far as to upload the data type for you and run tests. There are numerous tutorials online for programming a data type however so I will not cover that aspect.
Like I said I recently created a data type for my data parsing. I put it into my table but soon found that assigning my specific data processing user permissions to the object was harder than you would expect. I kept getting the error
[Microsoft][ODBC SQL Server Driver][SQL Server]EXECUTE permission denied on object 'data_type', database 'db', schema 'dbo'. (SQL-42000)
Trying
grant execute on [data_type] to cdr;
returned ‘object not found.’ Ms sql’s security assignment tool did not help either, it never did ‘find’ the object I created.
I eventually found my way to this msdn page which at least revealed the secret missing keyword. To set the permission on the object use this query
grant execute on TYPE::[dbo].[data_type] to cdr;
Tags: Help, microsoft, msdn, Programming, sql
Unique character names in MMOs have long been a pet peeve of mine. I am sure most of you have experienced the frustration when your favorite character name ends up taken by some else, and if you are like me it might take as many as 20 tries to get an available name. Well, here is a question for all you MMO programmers, why the hell are you still using character names as a unique identifier? There are so many better options to identify players in the world, every name does not need to be unique. In fact I frequently find that running across players entitled ‘FancyPants12′ or ‘Moooo4me’ does more damage to game play than a simple change would. When I am in the process of being adsorbed into an MMO through very nicely executed immersion techniques, stumbling onto these players breaks flow, 
disrupts my concentration and I find it harder to follow the story line (or even care about it).
I bring this up because this last weekend I started playing Aion. A very nice game that I have high hopes for, but its probably the first MMO that I am actually interested in a bit of the lore; I even went so far as to read some of the important quest text. Its shocking yes, but they really did do a good job making the player feel like they are not just a drop of water in an ocean; which is worthy of recognition in itself. Unfortunately, Aion suffers from the unique name dilemma so while I am trying to rescue the damsel from a horde of black winged bandits I am constantly running into ‘XXSlayerXX’ or ‘RoGeGardian’ and that is just terrible from an immersion perspective.
But what can be done about it? After all, a lot depends on character names. PMs, brokers, mail, friend lists, etc etc. Obviously a new system has to enhance game play, not restrict it. I propose a system where each character is assigned a unique number, not a name, which can be used for all these means of communication with just a little extra effort on the programmers side.
Tags: design, Games, mmo, Programming
Here is a problem for all the theoretical computer scientists out there. Say you want to write a program, this program needs to parse a certain file structure and extract useful data. A single file is setup as a bunch of independent records built on a handful of fields in said record. There are almost 1000 different kinds of fields, and close to 200 different kinds of records. The requirements for output is standard duplicate checking and removal, and programmable output, meaning you can change how the output looks or is handled without changing the program.
If you can think of a good solution feel free to post your ideas, today I will be writing about my solution and things that worked and did not work, and things that still do not work.
Tags: bafprp, design, pluggable factories
So in my last article I touched on the subject of what I termed static factories. I did a little searching online and found the original article that first gave me the idea. Apparently this type of design is called pluggable factories. A term I had forgotten until just now.
Here is the basic idea, you have x number of objects your program can create. These objects are all children of a base interface. You have cleverly created an abstract factory to create the various instances of your objects, but there is a problem, you are asked to add several new objects and/or remove several objects. So you jump back into your code and make a new object or remove an object. You compile your code and try to use the new object, OOPS, you forgot to add it to the abstract factory! Or, after removing the object you run some old data and OOPS, your program crashes as it tried to return an instance to a class that does not exist.
To be fair, if your program was written correctly it would not compile in the latter case, but that is besides the point. Sometimes abstract factories can be a pain in the ass. So, here is where pluggable factories step up to the plate.
Read the rest of this entry »
Tags: design, factory, pluggable, Programming, static
I setup the release of version 1.0 for bafprp the other day. It has been a fun project that I hope is useful to others like myself who needed an improvement over bafview without going to the big software companies. For today’s post I want to list some of the helpful tips I picked up while writing the program.
DSN-Less ODBC Connection String
I found it really annoying that in order to connect through ODBC you had to setup a data source. In windows this process involves going to the control panel, administrative tools, and data sources. In Linux, this involves setting up the ODBC driver, setting up unixODBC to recognize the driver, and finally setting up the server information and data source in both your driver and unixODBC.
I spent some time looking online and found a nice and easy way to connect without an external data source in windows. Basically it involves supplying all the information in one string like so:
std::string dsn = “DRIVER=sql server;DATABASE=” + _database + “;SERVER=” + _server + “;Uid=” + _user + “;Pwd=” + _password + “;”;
You must also use SQLDriverConnect instead of the usual SQLConnect function since the former will accept a dsn string, instead of just a dsn name and login info.
In linux its still a little annoying. You must still define your driver in unixODBC for starters but you do not need to setup a dsn or give server type information. Also, the support the DSN-less connections depends on the driver you choose. Since my primary use for bafprp was to connect to a ms sql database, I used FreeTDS which as of a recent update does have dsn-less capability. If you are using something else, please consult the documentation about this as well.
In FreeTDS the basic connection string goes like this
std::string dsn = “DRIVER=FreeTDS;SERVER=” + _server + “;Uid=” + _user + “;Pwd=” + _password + “;DATABASE=” + _database + “;TDS_VERSION=8.0;Port=1433;”;
TDS_VERSION and Port are FreeTDS specific settings, but the basic idea remains the same. Also it should be noted that different version of ms sql require a different TDS_VERSION. If you are using FreeTDS its important to know the correct version.
As a final side note about connection strings, make sure to include the terminating semi-colon. If you find your string unable to connect no matter what you do, this is probably the problem.
File Output
Sometimes when your program terminates unexpectedly, text being written to a file can be lost. If you are using fprintf or writef or any of the stdio functions for logging you will probably come across this problem. The only solution I found to guarantee that the text gets written is to use std::fstream and call the flush() method when you are done writing. Flush will make sure the text gets written before returning so it will be a bit slower, but for something as important as logging this is important.
Duplicate Removal
I remember reading about a similar situation in Programming Perls. It involved making a hash of your data and comparing collisions I believe. The situation I was in was that I had thousands of records that could be byte for byte duplicates with any other record in the original binary file. Like Programming Perls states, comparing each record against every other record is a joke. Hashing is definitely a better solution, but you need not create such a complex hash table for something like this. I ended up pulling a crc32 method to calculate the crc for the originals bytes in the record. After the record parsing was completed I sorted the array of records by their crc value. It was then a very easy procedure to remove any duplicates since they would be sitting right next to one another with the same crc.
One thing to note however, std::unique in algorithm.h seems like a wonderful function, but I could not get it to work for the life of me. It is supposed to sort the array, and place any duplicates at the end of the array, returning an iterator pointing at the start of the duplicates. Theoretically you can then use std::remove to remove all elements after that point to erase the duplicates. I managed to get the list sorted and std::unique did identify the correct number of duplicates ( the number of elements after the returned iterator matched the number of duplicates I later removed by said method ), but it did not seem to place the real duplicates at the end of the array. I ended up removing valid records that were unlucky enough to have a high crc and thus were at the end of the array.
So in the end I went through the entire list and removed neighbors with the same crc, which worked quite nicely.
Static Factories
I do not believe I have covered this concept here before so I will do a brief summary. This subject requires a much more detailed post but here is the cut and dry. If you are familiar with abstract factories you might know its a bit of a pain to add a new object. If you are using some kind of enumeration you need to add the id to that list, and add the correct new object code in the create method of your factory. Eventually you end up with enumerations of 100+ elements and a very scary switch statement. Fear not however, there is a better way!
Imagine a system where all you need to do to add a new object in your factory is compile a cpp. Thanks to static factories this is not just a dream. The trick involves a very natural side effect of static objects. The basic idea is simple. You have a main ‘maker’ class with a static registry variable that stores the names and pointers to other maker classes. When you want an object to be built through this factory you need to create a simple maker class for your object with a method called make, which is defined in the parent maker as pure virtual. The child maker defines an instance of itself as static and thus when the program starts it is created.
When the child gets created it calls the parent constructor with the name, or some other form of identification, of the object it creates. The parent maker then adds the information to its static database. When the programmer needs an instance of that object it simply calls the parent’s make method which looks at the database, pulls up the correct child maker and has the child make the object.
This technique is quite powerful if used correctly. It is absolutely necessary for data driven applications in my opinion, and very handy when working with any kind of file data. Using this method you can seperate file structure from logic in a very effective and pretty design.
While working on the BAF file project I learned a very interesting life lesson. See like most new computer science graduates I have worked on few actual projects, and quite frankly the few I have finished could not be considered corporation ready. While writing a few console applications, file parsers, corba interfaces, etc I learned a valuable lesson in documentation and log files. Log files can save your life if done correctly, and if you are really up to speed they can save many long hours of debuging too. In my case I fell in love with trace log messages. Trace logs are the log files that trace a programs execution so you can get a general idea of where your software is failing or producing an error. Now, most trace logs I have had the pleasure to work with are not quite where I wanted to be when I approached trace logs in my program. Most of the time they are a little more helpful then debug messages when trying to pinpoint an application’s problem. So when I faced the decision to add trace messages I decided to go all out.
My first logging framework was log4cplus, which is a very nice and complete logging framework for C++. I used it for one linux application I wrote and it works perfectly despite being several years old. Unfortunetly I was uncomfortable with how much work it took to write a message each new function ( two lines instead of one ) and it would not compile in windows so for my next application, bafprp, I set out to write my own.
When I completed the class I was left with one macro to print a log message depending on the level you wanted, ie, LOG_TRACE( string ), LOG_DEBUG( string ), etc. I then made sure I was adding trace messages to each and every function as I made them, instead of later on in the design. For example, an empty function would look like
void BafRecord::getType()
{
LOG_TRACE( "BafRecord::getType" );
LOG_TRACE( "/BafRecord::getType" );
return;
}
Each function would have a start and stop trace message.
Now this might raise an eyebrow or two but I assure you it definitely helps when your program crashes and you do not have a nice debugger on hand. Like say, if a non-technical user is using it.
This is how my linux application was programmed and I went the extra step in bafprp to work these in while I was working. Enough about this though, a short while ago, after adding one of the major structure types to the program my application slowed down from parsing a 12 meg file in 2 seconds to 2 minutes and I was greatly concerned over the well being of my design.
I tried many things, first I greatly reduced the number of memory copies my program executed, then I changed my file input so that it would read the entire file at the start and reference a data bank instead of reading the file each cycle. However none of these things put a dent in the processing time. So then I went online to try and find a nice and easy code profiler. Code profilers will basically watch your program execution and tell you which function your program is spending the most amount of time in. I ended up finding LTProf which allowed me to profile my program without recompiling or changing my program at all. I am actually very surprised at how well it works with compiled binaries. After running a release version of my program it was still able to accuratly determine function names and operate like it had a window into the source code.
I found that my time stamp function, NowTime, which returns the current time as a string, was taking a noticably large amount of my program’s time. Thinking back to what code uses this function I discovered the flaw.
When I wrote the log program I wrote the log level exception into the log class. This way if the program tried to log a trace message it would get sent to the output class, which would then pass it on to the logs if the level was at or below the log level type. However this was not enough. as it turns out simply creating the log message twice in each function had a substantial effect on the processing speed of my program.
Needless to say as soon as I moved the log level check to the macro the process speed dropped from 2 minutes to 30 seconds. Now some might say that trace logs this detailed are excessive, however I believe they can help greatly when dealing with a malfunctioning program. So as a final statement, be careful when you design systems, and beaware of how to use your tools. And if your program runs 3 times longer then a similar program, there is probably something horribly wrong.
I finally got around to setting up svn for my project. Thanks in part to some new tools I recently came across the transition was fairly painless and after about 2 hours of cleaning up my old project and trimming the fat I uploaded what I will be working on to http://code.google.com/p/anaa/
I also created a second project of a more work related function. This project deals with reading and parsing Bellcore BAF files created by various soft switches in use around the world. These records contain call records in a highly coded and formated to Telcordia specifications in their GR-1100 document. This document is not very cheap to come by so I have started work on improving one of the best free parser’s available today, bafview. You can find this project here http://code.google.com/p/bafprp/
Tags: bafprp, computer science, linux, open source, Programming, update
Ok ok, I am writing again! If you read my previous posts on dynamic npc actions then this is an update on my progress. If you have not, this is about how to create good file formats for your applications.
As some might have expected this project is taking more time then expected, partially because I am currently downing bosses in WoW, and partially because its very hard. So lets just cover what has been going on since my last post.
I spent a lot of time thinking of the best structure for my idea in terms of objects and file formats. I do not know if I covered this before because I am to lazy to view my past posts so heres a recap of the basic composition of what I have been calling a ‘Puppet.’ A puppet is a game object with x number of movable joints. These joints are built in a tree structure and move based on messages passed to his parent. These messages are generated from a actor class who learns actions and movements as I previously discussed. When I was designing the puppet class I hit a big fat road block. I wanted the ability for a puppet to be independent of his underlying joint structure. That way I could have a puppet control everything from a human to grass. So you might think, ‘No problem, do some tricky interface work and have derived classes for each type of puppet.’ WRONG!
See I myself have read many articles about component game object design, and I do believe I wrote an article on this in the past. However do not get the wrong idea, what I did is not EXACTLY component objects, but kind of close. Because when I thought about components I also thought how to best describe them in a file format. And after thinking about that I decided that I wanted a global file format to describe not only each type of puppet but the entire world. And that the entire game will be built out of components in a tree structure built at load time from this file format. I am getting a bit ahead of myself here so lets get some examples here.
<world>
<name type=string>Main</name>
<description type=string>
Big world for which to run tests in
</description>
<floor type=plane>
<p1 type=vector>-1000 -1000 -1000</p1>
<p2 type=vector>1000 1000 1000</p2>
<material>dirt01.mat</material>
</floor>
<joint>
<position type=vector>0 0 0</position>
<orientation type=vector>0 0 0</orientation>
<puppet>arm.puppet</puppet>
</joint>
</world>
This is my toy world I am working in. The file format is xml and I use tinyXML to load and parse these files. For those familiar with ogre, I made a resource of type XMLFile to load these. Now here comes the clincher, this XMLFile resource will load every type of game object I desire, regardless of type or purpose. Not only that, but I only added 2 methods to the class.
So let me explain a bit about these methods, they are named ‘build’ and they are called usually immediately after loading the file. Basically what they do is look at an element in the xml, find a builder for it, have the builder build it, and attach the new object to the world tree. In this example, the world has no parent and all elements inside the world tags are children of the world. I use the type descriptors so that I would not have to write many classes that do the exact same thing. Elements that load another file, like the puppet tags, will load that file in their own world and all objects created from that are children of the puppet. After the build the main program is not aware of any object other then the world. Messages are passed to the world when need be, however most thought processing is done in the world object, not by the game engine.
Lets get an example puppet up here:
<puppet>
<name type=string>arm</name>
<description type=string>
Generic arm
</description>
<joint>
<name type=string>shoulder</name>
<gib>upperarm.gib</gib>
<joint>
<name type=string>elbow</name>
<gib>forearm.gib</gib>
<joint>
<name type=string>wrist</name>
<gib>hand.gib</gib>
<joint>
<name type=string>pinky</name>
<gib>pinky.gib</gib>
</joint>
<joint>
<name type=string>ring</name>
<gib>ring.gib</gib>
</joint>
<joint>
<name type=string>middle</name>
<gib>middle.gib</gib>
</joint>
<joint>
<name type=string>index</name>
<gib>index.gib</gib>
</joint>
<joint>
<name type=string>thumb</name>
<gib>thumb.gib</gib>
</joint>
</joint>
</joint>
</joint>
</puppet>
As you can see this file format is very flexible, the same structure that built the world can build a puppet and also a gib:
<gib>
<name type=string>forearm</name>
<notes type=string>
Fore arm gib
</notes>
<mesh>forearm.mesh</mesh>
<input>
<name type=string>shoulder</name>
<position type=vector>0 0 0</position>
</input>
<output>
<name type=string>elbow</name>
<position type=vector>7 0 0</position>
</output>
</gib>
I want to clarify one thing before I go on. Elements called position or orientation, these attributes are not given any attention by the builders. Instead those objects simply exist on the tree for someone else to read them and make the correct adjustments. For example, the puppet’s name element simply sits as an element on the tree until a debugger opens and the entire tree structure is displayed. At that time the debugger will go through each element and find name tags that it will use to give more information on the node. And the description works the same way.
They are both created by the string builder, as denoted by the ‘type=string’ attribute I give them. The value is assigned and the name of the new object is set, but the type is always a string.
So what exactly is required to declare a new type of object? Just a builder and an actual object. Take this example of the string object.
class ScktString : public ISocketable
{
friend class StringBuilder;
public:
void setValue( std::string val ) { _string = val; }
std::string getString() { return _string; }
private:
ScktString( std::string propName ) : ISocketable( propName ) {}std::string _string;
};class StringBuilder : public IBuilder
{
public:
StringBuilder() : IBuilder() {}
Socket* build( std::string element, Socket* parent );
const std::string& getType() const;
static std::string BUILDER_TYPE_NAME;
private:
static const StringBuilder registerThis;
};
As you can see its relatively easy to create a new object. Lets start at the builder, when the xmlfile is building it will search for a builder based on its type name, which is assigned by you. A builder list is created at the start of the program, as each builder has a static ‘registerThis’ variable that adds it to the builder list. Those familar with static factories should be familiar with this. The builder’s build method gets called with the name of the element and its parent’s socket. A socket is an object that accepts links from children. The builder then builds the object passing its name. The sockable base class is for objects that can be socketed or linked with a parent.
So the string object is created and the string builder passes back the parent object. This is done in case the new object has a socket himself. In that case he would pass back his socket and the builder would build his children and attach them to him. In string’s case however, there is no socket and a little while later the builder assigns the string object a value by calling the setValue method in the parent object with the objects name and value as parameters.
After all the work we are left with one tree that contains all objects in the game. Kind of like a scene manager, but for ALL objects, not just renderable ones. And oh, don’t forget that the entire tree was created by one instance of XMLFile. As programmers we should always strive to achieve the cleanest and most powerful format we can muster. This format is not only really easy to implement, but also holds unlimited power. You want to load a texture? Create a texture socketable object, and implement file specific loading inside it. You want to script AI? Either create an AI socketable object that reads children objects that decribe a strategy, or create a python socketable object that will load your ai scripts and parse them. In this way our file format can act a proxy for another format or as the direct format. The most important thing however is that none of our in game objects need to give file formats a single thought.
Tags: computer science, design, factory, Programming, research
Recent Comments