utcDateTime and Timezone Issues with Dynamics AX 2012

The Insanity of AX Timezones and DateTimeUtil

I am working on an implementation that imports employee data. Some of the data is employment related dates. Many of these dates use the utcDateTime type. When importing with a date I found dates to be off by one day. I figured it was related to the timezone so I have done some experimentation and found some shocking results.

The first issue is the msdn documentation seems to be misleading if not incorrect. This document states: 

When you are programming with dates, Best Practices are ... Use DateTimeUtil::getSystemDateTime instead of systemDateGet or today... Only DateTimeUtil::getSystemDateTime compensates for the time zone of the user.

However, when I execute the following code on November 2nd at 3:26 PM:

utcDateTime testDateTime;        
testDateTime = DateTimeUtil::getSystemDateTime();
info(
    strFmt(
        'testDateTime is %1 using timezone %2.', 
        testDateTime, 
        enum2str(
            DateTimeUtil::getOriginatingTimeZone(
                testDateTime)
            )
        )
    );

I am surprised to get this:

testDateTime is 11/2/2012 08:26:01 pm using timezone (GMT) Casablanca, Monrovia, Reykjavik.

The dateTimeUtils::getSystemDateTime did in fact compensate for the time zone of the user but not as I expected. I expected it to return a utcDateTime with the time system time in my timezone. I can live with this by simply applying my timezone. (I'll cover how to do this and why it doesn't work a little later.)

However, my issue was with importing times in a CSV file. So let's try this:

testDateTime = str2datetime('11/2/2012 15:26:00',213);

What I get is predictable:

testDateTime is 11/2/2012 03:26:00 pm using timezone (GMT) Casablanca, Monrovia, Reykjavik.

Since I didn't specify a timezone, GMT is assumed for the time I created. So let's apply a timezone offset to this time. My timezone is CST which is GMT minus 6 hours. The timezone enum for this is Timezone::GMTMINUS0600CENTRALTIME.

testDateTime =  DateTimeUtil::applyTimeZoneOffset(testDateTime,
    DateTimeUtil::getUserPreferredTimeZone());

I can't figure out what how I got this:

testDateTime is 11/2/2012 10:26:00 am using timezone (GMT) Casablanca, Monrovia, Reykjavik. 

There are two serious things wrong with this. First,  the time zone for the utcDateTime is still GMT. How can I possibly pass that data around and know if need to apply the timezone or not. Second, if you do the math you will see this is 5 hours different not 6.

There is another way to create a utcDateTime and apply a time zone offset so I thought I would try that. The DateTimeUtil class has a static method that will create a utcDateTime for you. The DateTimeUtil::newDateTime function takes the parameters. The third (optional) parameter is a Timezone enum value. The other two are a date and a time. I figure I can read the date string from the import, add a constant time value and apply the timezone and get a utcDateTime that is correct.

Here is the code:

DateTimeUtil::newDateTime(
    str2Date('11/2/2012', 213),
    str2time('15:26:00'),
    Timezone::GMTMINUS0600CENTRALTIME);

Maybe it was unreasonable for me to expect that I could create a utcDateTime value of November 2nd, 3:26 PM CST with that code, but that is what I expected. Instead I got this:

testDateTime is 11/2/2012 10:26:00 am using timezone (GMT) Casablanca, Monrovia, Reykjavik.

I didn't initially look at the documentation for that method because  simply saw the parameter in the intellisense and used it. With the unexpected result, I went to the documentation to see it names the third parameter "tzOffsetToRemove". Well there's my problem. I takes the time I want to use and removes the timezone offset. Now we have another problem. It actually added the timezone offset instead of removing it. Remember the timezone offset is minus 6 hours. It actually subtracted the (incorrect) 5 hours. So instead of having a GMT time of 9:26 PM I have ... oh who knows what I have.

To show how insane this is, let's re-apply the timezone offset. Here's the code:

testDateTime = DateTimeUtil::applyTimeZoneOffset(testDateTime,
    Timezone::GMTMINUS0600CENTRALTIME);

What do you think this will give me? Here it is:

testDateTime is 11/2/2012 05:26:00 am using timezone (GMT) Casablanca, Monrovia, Reykjavik.

Of course, it removed the (incorrect) 5 hours again. 

The final straw is this. If I create a GMT utcDateTime and then "remove" the timezone offset it actually sets the timezone to the one "removed" while actually removing the offset. Here is the code:

testDateTime = str2datetime('11/2/2012 15:26:00',213);
testDateTime = DateTimeUtil::removeTimeZoneOffset(testDateTime,
    DateTimeUtil::getUserPreferredTimeZone());

You will recognize the first line of code from my first attempt. This takes a string and converts it to a GMT utcDateTime. Then, I call DateTimeUtil::removeTimeZoneOffset telling it to remove the CST timezone offset, hoping to end up with a GMT date/time. Here is what I get:

testDateTime is 11/2/2012 08:26:00 pm using timezone (GMT-06:00) Central Time (US & Canada). 

I finally have a utcDateTime for CST but it contains the GMT time. Go figure.

So, the system attempts to provide time localization but instead manages to simply screw everything up. I don't know why they didn't just simply use the functionality built into the Windows OS since the client and server only run on Windows. I suspect the one hour difference is related to DST but I have not yet figured out how to take that into consideration.

Besides simply having buggy code, I think Microsoft has outdone themselves this time in complicating obfuscating their API. I would suggest they start by thinking how someone would want to use these utilities. Maybe someone would actually like to simply create a data and time for a specific timezone and have the value actually carry the timezone they specify.

Tory

Update:
I had to quit yesterday before I could test my last idea. I tested it this morning and found that by reapplying the removed timezone offset I finally ended up with the November 2nd 3:26 PM CST I was looking for. Here is the code and the result for the final (work around) solution:

testDateTime = str2datetime('11/2/2012 15:26:00',213);
testDateTime = DateTimeUtil::removeTimeZoneOffset(testDateTime,
    DateTimeUtil::getUserPreferredTimeZone());
testDateTime = DateTimeUtil::applyTimeZoneOffset(testDateTime, 
    DateTimeUtil::getUserPreferredTimeZone());

Result:

testDateTime is 11/2/2012 03:26:00 pm using timezone (GMT-06:00) Central Time (US & Canada).

Wow! Has it really been eight and a half months since my last post?

This summer I started working on an ERP implementation project that contains a fair amount of custom development. When I came to the project it was in iteration #2. This iteration was to focus on product definition. The project team was struggling to define the product requirements to obtain a design that could be implemented. They had been thrashing a little because so much of the product design was dependent on how the product would be used on the sales order.

When I arrived on-site, the sprint was already supposed to be complete. In fact, I was involved in a kickoff meeting intended to start capturing requirements for the Customer / Contact iteration (#3). However, the product design had been "completed" and data had been loaded but everyone knew it was not correct. They just didn't know how to make it correct yet because many of the design details would be discovered during the "Sales Order" sprint.

The project team was a little reluctant to define the customer requirements for the current sprint because they didn't want to end up in the same position with customer design that they were in with the product design. So a decision was made to start the "Sales Order" sprint right away and then use some of that discovery to "inform" the work in the "Customer Contact" and "Product" sprints so they can complete.

I'm sure you have guessed what happened. The attempt to capture "Sales Order" requirements dissolved into an attempt to understand "invoicing" requirements without starting that sprint also.

Did Agile Renege?

What happened to the "Agile" promise? You know, "early and continuous delivery of valuable software." One may argue that you can't expect an Agile methodology to deliver on a promise if core principles of the methodology are violated. For example, one of the principles is "Welcome changing requirements, even late in development." It is clear the team was not willing to live with unknown requirements, much less "changing" requirements.

Before you write off this organization as ignorant of agile methods and assume the solution is simply found in training let me tell you that this is a business is a mature development organization that has been successfully delivering software using agile methodologies for three years. There may some missteps involved but most of the issue revolves around trying to stick the "square peg" of development methodology into the "round hole" of integrated ERP system deployment and implementation.

Microsoft (the ERP vendor) provides project guidance for using an "Agile" implementation methodology. If you google "agile implementation methodology" the pages found will fall into one of two categories:

• information on how to implement an agile methodology for software development 
• Sales materials on how consultants and resellers will help you implement software using an "agile implementation methodology"

The Mythical Agile Implementation Methodology

What you won't find is any credible source that has actually defined an agile implementation methodology. There is a reason for this. Agile methodologies are applied to the domain of software development not general IT project management. The methodologies have one objective, build working software.

Software development and system implementation are two completely different animals. As a small example of where the disciplines diverge simply take on of the basic processes in agile methodologies - refactoring. "Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior." (Martin Fowler)

The purpose for refactoring is to preserve the current software functionality while changing the design to support the addition of new functionality. Since agile projects to not capture requirements and do "big design" at the beginning of a project, they need to continuously modify their design to support newly discovered requirements.

With modern development environments and tools it is often very easy to refactor code and with automated testing strategies it can be relatively risk free. This is not true with implementations. How do you refactor an implementation simply and with little risk? I have seen attempts but have not seen a cost effective solution.

Another aspect that does not transfer from software development to implementation is "deliver working software frequently." How does the business community define "working software?" I define working as "using the software for the business purpose for which it was purchased." This excludes a "conference room pilot" from "working software." Implementations often require so many concurrently "working" pieces that it becomes difficult to draw any line in the sand for a single sprint. My conclusion is that wide-breadth, integrated systems (like ERP) by their very nature do not allow for "early and continuous delivery of valuable software." The breadth and integration creates so many dependencies that you can not "deliver working software frequently." 

Let me say this a different way. If there is "go-live" date, you have a single point in time at which the software will be "valuable" and "working." That is not early and often. That is not continuous. That is "big bang."

Phased Implementation

My conclusion does not exclude iterative implementations. I simply contend that implementation iterations does not an agile methodology make. My recommendation is to iterate over the business processess defining requirements and possible solutions. These iterations can even produce useful (but not "valuable") conference room pilots. These pilots are often used to discover more requirements with changes in the current configuration. Much of this is messing and at times more closely resembles sausage making than software development.

Often these iterations may include actual development efforts that may be managed in an agile manner. Great! Scrum away, or use whatever agile methodology floats your boat. Just don't call your implementation agile.

Serious or Semantics

Is this really a serious topic or am I getting my shorts in a bunch over semantics. What difference does it make if we call it "agile" or "phased?"

My main concern is with successful implementations. To be successful an implementation has to meet business needs and it must do it in an affordable manner. A realistic project plan is mandatory to a successful implementation.

It is my opinion that the project approach that is being referred to as "agile" is neither workable nor agile. I want any project plan in which I have a stake to have a known path to success. If that path has a "go-live" date then I want to know the plan has a realistic management methodology that admits some sort of waterfall approach. 

To have a minimal level of comfort in an implementation plan I need to see an integrated approach to managing the dependencies. Managing dependencies often means that all the related requirements are discovered and all the related designs are defined so all the related configurations can be determined and entered and all the related reference data can be defined and entered and all the dependent master data can be defined and entered / imported etc.

My next blog (hopefully less than 8 1/2 months from now) will include more detail on what this implementation methodology may look like.

Stating what the User 'Really' Wants

In my previous post I stated my opinion that the "biggest problem" in software engineering is "bridging the 'gap' between understanding what the user truly wants to accomplish and how to best leverage technology to help accomplish that." For some reason, I seem to be wired to perform this particular task intuitively. What is missing for me is a way to communicate my discoveries and understandings. When I have successfully discovered what the user "truly wants" I have failed to accurately communicate that anyone else.


This article contains some of my thoughts on how I intend to address that failure moving forward. Let me forewarn the (unlikely) reader that I have not yet successfully validated this approach. It is simply my intent to test its validity.


User Stories


User stories are intended to be a tool used to capture "what the user wants to achieve." If you look at examples you will find they provide real promise in accomplishing that objective. One of the common structures recommended is in the form: "As a <role>, I want <goal/desire> so that <benefit>."  The benefit portion is sometimes considered optional.


There is an aspect of what the user wants that is missing in user stories. User stories don't talk about what the user doesn't want. I propos to add a special category of user story. I will call this a 'User Gripe'. In a user grip, we address what the user doesn't want to have happen, what the user hates about the current system or what activity the user wants to avoid. If we use the same format to describe a user gripe it may look like this: "As a <role>, I don't want < obstacle/undesired behavior> because <pain caused>." 


I like user stories. I think they have value, especially if we add the concept of a user gripe. There is something I don't like about user stories however. Who's story is it? If we use the recommended format we find ourselves having to record the same story for different roles with potentially different benefits. I could just say "As a <role> or a <role>, I want ..." but that can get really ugly.


I think I want to use a connection to associate actors to user stories. If I define the benefit on the connection then I can associate multiple actors to the same story and define the benefit the actor gets from the story. The definition of the benefit could still be optional but the connection is not.


Related User Stories


What should be done with two different stories that are simply two ways to accomplish the same thing? If each individual piece of functionality needs to be developed they both must be documented.


Some people would advocate making a generic user story and defining the details in the design phase of the sprint. In many cases this will work. However there are times when you want one method to be developed in one sprint and another method in another sprint. In these cases a single story will not allow you to plan your sprints appropriately.


Another issue is related to 'epics'. If you start with an epic and then refine the epic to multiple user stories, is there any value in keeping the epic? If so, how do you establish (track) the relationship between the epic and the story?


These and other scenarios drive me to the concept of modeling relationships between user stories. It appears to me there are two relationships:

• Aggregation or containment - One story includes others. This is the case with epics. 
• Specialization or inheritance - One story may be a specific means of accomplishing a more general story.

As an example, take the scenario with which I am currently working. Imagine a help desk system that helps manage support incidents for a machinery manufacturing company. A service technician and a service administrator are interviewed and they both want to create a customer interaction and relate it to a specific incident and a specific piece of machinery. Since they often interact with the customer via e-mail, they also want to create an interaction using e-mail.


With a traditional approach of creating user stories on a 3x5 card you would likely end up with about 8 cards. Four would start with "As a service technician" and four with "As a service administrator." All the stories would be about creating a customer interaction but two (two copies) would have the same content about a specific way to create an interaction, while four (two copies of two) would be about some information that may optionally be included during the creation of an interaction. With a modeling technique you could have four stories with their relationship to each other and their actors. Here is what it may look like.

By putting information on the connections I can describe the benefit portion of the user story from each actor's perspective. If I have the ability to generate document text from the diagram, I could assemble user stories for review with the user. Because of the connections I can focus on all stories for a specific user or all stories related to a specific story. If I can search text associated with the connections I can find all stories that provide a certain benefit.


I wanted to cover more ground but this post is long enough already. I will pick it up in another post with the relationship between user stories and use cases.


Tory

The Need for a Better Tool

I am currently in the process of implementing and ERP system (Microsoft Dynamics AX 2012) for my current employer  (The Aagard Group). One of our overarching implementation principles is Requirements First. What we mean by that is our business requirements will drive the implementation decisions. This requires us to understand our requirements (a topic for another day) and understand the out-of-the-box functionality of the system (another topic to be tackled) and document the gap between those. Once we have establish the gap we can design a solution to meet the requirement.

It has always been my view that requirements are part of the design process. If you start with a list of "musts" it is very likely you will find the software that needs those musts will not adequately meet the users needs. Sometimes the users needs are met but the software is still less than it could be.

During decades of software development and system implementations, it has been my experience that after the software engineer has designed the system to meet the requirements he understands what the user actually wants to accomplish and would do it differently if time allowed. I have had many experiences where we have developed to a spec, received high praise from the users and with my newly acquired knowledge of the problem domain, asked the user a few "what if"s only to find we could have provided a significantly better system because the user had never thought of doing the task in a different way. (Here is a great article by Mike Cohn about this phenomenon.)

In his book "Software Project Survival Guide", Steve McConnell identifies three intellectual phases of a software development project. He calls those phases Discovery, Invention and Implementation.McConnell lists "discovering the user's real requirements", technical investigation and building prototypes as activities in the Discovery phase.

In my experience the best solutions use the most skilled people in this phase. It is the most likely predictor of success or failure. Failure in discovery makes is very difficult to succeed in implementation.

Yet, when you think of the tools we use to aid in the discovery phase, what comes to mind? There are use cases / user stories, UI prototypes and requirements documents.How can one capture intent? This is my current struggle. I believe this to be one of the glaring holes in the tools available to aid with the software development lifecycle.

Of course, this is not just my opinion. Others have said this and even offered some ideas to address it. Sriram K. Rajamani states "the biggest problem in software engineering continues to be the bridging of the `gap' between the intent captured in requirements and expressed at a high level, and the detailed encoding of this intent in the code. There are no good tools, either mental or mechanical, that allow comprehension of large programs, and provide a mapping between how different parts of the code work together to satisfy the requirements.Thus, looking for any high-level requirements within a million line program is analogous to running around New York City looking for a lost cat, without a map, and without any street signs."

I would move the "biggest problem" up stream slightly to bridging the 'gap' between understanding what the user truly wants to accomplish and how to best leverage technology to help accomplish that. However, we agree on the issue. There are no tools to allow the business analyst, the software architect and the software engineer to capture what they understand. The only options we have is to capture what we have concluded is the solution.

For example, we may have a list of requirements that have been tediously recorded. What we won't have is the information that was discovered that lead to those requirements. Why is this important? It isn't important if you only want someone to design a system to meet the captured requirements. However, if you want to invent something that will actually add value to the job at hand it may be very important.

My proposal is to combine disciplines and tools. I propose that software projects should team business analysts and architects in the discovery phase. I propose we build a set of tools that combine a good narrative , use cases / story cards and technical diagrams into an interactive, connected modeling environment that allows an analyst to link multiple types of content and define the types of links. Text elements could be linked to other design artifacts like UI prototypes, UML static and dynamic diagrams. I can imagine a narrative explaining why a user currently performs a particular task being linked to a use case, a number of derived requirements and even an activity, sequence or state diagram.The derived requirements could also be linked to UI prototypes and UML classes that realize those requirements.

Here is a list of the attributes that a system of this sort may need to have.

• Objective: To provide a means of capturing the intent of a software development initiative including the problem domain and desired outcomes.
• The Target Audience: Outcome Stakeholders 
• Content Properties
• Multilevel - Captures the highest level of intent to the lowest level of detailed requirement or design - Captures highest level of abstraction to the lowest level of concrete implementation
• Connected - Associative / Relational - identify and document the associations and dependencies and use them for tractability - able to trace from the lowest level of detailed design to the highest level of intent
• Multifaceted - multiple perspectives or aspects
• Process view
• Requirements view
• Data view
• Workflow view
• User Role
• Security
• Accessible - Available and understandable to all stakeholders
• Maintainable
• discrete elements that can easily be modified without major impact to other information 
• Revisionable - able to fit any sane view of revision / change control
• Comparable - able to compare iterations
• Interactive - electronic interactive document with the ability to produce static, printable documents for all or portions.

Over the next several articles, I intend to flesh-out the essential components of such a system and give a few examples.

Tory

Fixing a bad XPO File for Microsoft Dynamics 2012

Ok, this my first topic entry and I can't imagine a more boring topic. Yet - it has taken some time to find a way to solve this problem so in keeping with my intent for this blog I'm posing it.

I am currently in the process of implementing Microsoft Dynamics AX 2012 (DAX) for my company (The Aagard Group). We are doing extensive customization within the system. One of the great features of DAX is that it integrates with Microsoft Team Foundation Server (TFS). To do this it exports (and imports) customization in the form of formatted text file (similar to an XML file). This allows synchronization between the individual developer environments. The text file has an extension of ".XPO".

Well, there appears to be a bug in the code that generates the .XPO file. Occasionally DAX will create a bad Project XPO file – either when explicitly exporting or when checking in changes. The problem is manifested when importing the project XPO file. It will cause a Critical Stop error with a message that says “Error in line ###: expected BEGINNODE but found FILETYPE." (see the image below)

This error unfortunately may send you on a wild goose chase looking for missing “BEGINENODE” lines. In fact, the problem is a missing  PROJECTCLASS identifier. A correct project XPO will have the project class type after the keyword PROJECTCLASS. For example a typical project will have the following line:


PROJECTCLASS ProjectNode


While a test project will have:


PROJECTCLASS SysTestProject


But the bad XPO will simply have:


PROJECTCLASS 


To fix the XPO file simply put ProjectNode or SysTestProject (depending on the type of project the XPO represents) after the PROJECTCLASS keyword.


The screen capture below shows the file prior to the fix compared to the file after the edit.

Interestingly, not every project XPO that is missing the PROJECTCLASS identifier will create the critical stop. If the nodes in the project are all contained in groups apparently the parser can recover from the missing PROJECTCLASS identifier. The screen capture below shows a project file with and without the PROJECTCLASS identifier. Both versions will import properly.

I have encountered this error seemingly randomly. I don't know what causes it I haven't found any information about anyone else complaining about it. So, hopefully this information will be helpful in future. Here's to also hoping the defect is fixed soon so this information becomes obsolete.

Tory

Welcome

I have been meaning to start a blog for sometime. Not because I think anyone would actually be interested in what I have to say (if you are reading this you are probably lost) but because my memory is getting so bad I have trouble remembering the conclusions I have reached after the hard work of reaching one.

My objectives for this blog are to provide a place for me to record the significant and insignificant "take-aways" from my day-to-day life. It is a intended to be a reference site for me. Anyone else who may benefit from it is welcome to consume as much content as they can stomach.

Since I am making this blog public I suppose I should address any potential readers and provide a brief summary of what they may expect.

The topics I expect to cover revolve around the areas of my life that require any significant thought. The topic may not itself be significant, only the thought I invested in the topic.You may be amazed at how much significant thought I can put into insignificant topics. Since (according to my teenage daughter) I am "older than dirt" I have some experience with thinking and am familiar with the topics that have or will likely cause significant thought on my part. So I can provide the reader with a list. I do however, reserve the right to think about and therefore blog about any topic that suits my fancy.

Software Development
I am a software architect by trade and have a passion for software development. Naturally I have and will continue to think about software development. So I expect to write extensively (relatively speaking) about software development. This topic is large and could include any area of the software development life cycle (SDLC), software development best practices (SDBP) or even the penchant for the technology community to use and/or invent new abbreviations and/or acronyms. Our industry is, of course rivaled only by the military.

Philosophy and Worldview
I am very interested in philosophical topics. I especially enjoy thinking about reality, our view or reality and how our view influences our behavior. Sometimes these topics bleed into some areas of science, but I don't spend much time researching science so I can't say I will have much to say about it. One area of science that I have thought some about is the theory of relativity. Specifically the theory of special relativity. Our view of time and space has very interesting ramifications on our philosophical views.

The topics of Philosophy and worldview have lead me down many paths including family, education, vocation and of course religion. I have thought quite a bit about these topics. It also has triggered thoughts about culture and society. I expect I will continue to think about and therefore blog about these topics.

I hope I am disciplined enough to consistently record my thoughts on this site for myself, my posterity and anyone else who may stumble upon them. If you enjoy any of my thoughts feel free to comment, even if you disagree. If you don't enjoy my thoughts don't read them any longer and don't bother with a comment - I can't imagine I would enjoy your comment.

Tory