How I nearly wet myself.

It's almost as if an external consultant was providing measured commentary on our team's management imposed attempt to go agile. We using Mingle as a time sheet application, amongst other things, with the sweetener that the alternative is either Microsoft Project and/or a dedicated time sheet application. So far I've not yet found where I'm able to record getting up between 3am and 4am to check the ETL process which fails on account of a fragile FTP interface to Bloomberg. Still, it's early days and the good news is that we're hurtling towards automated builds.

For all it's usefulness XQuery never really took off the way it seems it should have done. I mean we have a query language from the W3C for XML and since XML use is pervasive... and it's kinda like SQL which is wildly successful... and there the story appears to end without the desired/expected conclusion. All the same it's becoming increasingly invaluable for us as we're using XML datatype input/output stored procedure parameters.

To demonstrate its value by example. Here's one way to generate XML using LINQ and hash table to validate against a given schema:

using System.Collections.Generic;
using System.Linq;
using System.Xml.Linq;

namespace Wimiro.Data.Examples {
  public static class TransformXmlImperatively {
    public static string GenerateCdsSpreadOutput(this string xml, string requestXml) {
      var rootNode = xml.GetXmlDocument().FirstChild;
      var xmlWithNewDocumentRoot = rootNode.GetProcessingInstruction();

      xmlWithNewDocumentRoot += "<" + requestXml.GetResponseRootElementName()
      + " xsi:noNamespaceSchemaLocation=\"static_analysis.xsd\" "
      + " xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">";

      xmlWithNewDocumentRoot += xml.GetCurvesPointsFromXml();

      xmlWithNewDocumentRoot += "</" + requestXml.GetResponseRootElementName() + ">";

      return xmlWithNewDocumentRoot;
    }

    public static string GetCurvesPointsFromXml(this string xml) {
      var curveDictionary = new Dictionary<string, XElement>();
      var rawDoc = XDocument.Parse(xml);
      var rawElements = from elements
                 in rawDoc.Elements("root").Elements("row")
               select elements;

      var curveRoot = new XDocument(new XElement("root"));

      foreach (var rawElement in rawElements) {
        if (!ContainsCurve(curveDictionary, rawElement)) {
          AddCurve(curveDictionary, rawElement, curveRoot);
        }

        FindCurve(curveDictionary, rawElement).Add(GetCurvePoint(rawElement));
      }

      return curveRoot.ToXml().StripRoot();
    }

    private static XElement FindCurve(IDictionary<string, XElement> curveDictionary, 
                    XElement rawElement) {
      return curveDictionary[rawElement.GetKey()];
    }

    private static XElement GetCurvePoint(XElement rawElement) {
      XElement curvePoint;
      curvePoint = new XElement("Point");
      curvePoint.SetAttributeValue("Spread", rawElement.Attribute("Spread").Value);
      curvePoint.SetAttributeValue("MaturityYears", rawElement.Attribute("MaturityYears").Value);
      return curvePoint;
    }

    private static void AddCurve(Dictionary<string, XElement> curveDictionary, 
                    XElement rawElement, 
                    XDocument curveRoot) {
      XElement curveElement;
      curveElement = new XElement("Curve");
      curveRoot.Element("root").Add(curveElement);

      curveDictionary.Add(rawElement.GetKey(), curveElement);

      foreach (var curveAttribute in rawElement.Attributes())
        curveElement.SetAttributeValue(curveAttribute.Name, curveAttribute.Value);

      curveElement.SetAttributeValue("Spread", null);
      curveElement.SetAttributeValue("MaturityYears", null);
    }

    private static bool ContainsCurve(IDictionary<string, XElement> curveDictionary, 
                    XElement rawElement) {
      return curveDictionary.ContainsKey(rawElement.GetKey());
    }
  }
}

And here's the way it should have been done using XQuery:

select @xml_out.query
('    for $Ticker in distinct-values(/root/row/@Ticker)
    let $Tickers := /root/row[@Ticker=$Ticker]
        for $Currency in distinct-values($Tickers/@Currency)
        let $Currencies := $Tickers[@Currency=$Currency]
            for $Date in distinct-values($Currencies/@Date)
            let $Dates := $Currencies[@Date=$Date]
    return
            <Curve    Ticker="{$Ticker}"
                    Currency="{$Currency}"
                    Date="{$Date}">{
                        for $node in $Dates
                        return 
                        <Point Maturity="{data($node/@MaturityYears)}"
                              Spread="{data($node/@Spread)}" />
                    }
            </Curve>')

StackOverflow! For all your hardware needs.

The Vanilla BlogEngine.NET allows comments to be posted via an HTTP post which is great in terms of enabling an AJAX implementation for blog post comments. However, it's great too for spam bots, almost exclusively, offering pay day loans throughout the comment sections of this site. One new year's resolution was to implement a solution using ReCaptcha. In this case the solution requires writing no code.

StackOverflow: How would one integrate ReCaptcha in to BlogEngine.net (ASP.net C#)?

I can't think why anybody would think a pay day loan a good idea when there's Zopa, for example.

Having tremendous fun playing find the missing ThoughtWorks.CruiseControl.MSBuild.dll.
At times it feels as if some twisted soul is curating the Internet simply to thwart my efforts.

During our three day Agile Training course with too many examples contrived to maintain audience engagement through cute caveman cartoons and engineering attempts familiar to all (house-building), one colleague questioned how suitable agile might be in model driven development.

The Agile view was presented in one instance as making use of Zeno's Paradox in reverse. The paradox says, essentially, that motion is illusory because to travel any distance there is a point half the way between start and finish (let's call this half-way) and there is also a point half the way between start and half-way (let's call this a quarter of the way) and so on. Because there are an infinite number of these half-way points it's impossible ever to get anywhere. This being the case the Agile take is that perhaps we're able to make better progress by considering how to only get half-way as opposed to considering in too much detail the end-game (or the whole journey).

If Agile's Raison (Scrum in this example) primarily is to produce some complete functionality periodically (frequently) in tight iterations then the question in the case of model development is "how much value does half an algorithm provide, if any?" If it's not possible to go to market with half a model then shooting for half-way appears only to help as a strategy for maintaining motion rather than for more frequent delivery.

Stated another way: Because the Quant team who are building complex mathematical models are unsure what the finished product will look like they almost have no choice but to work iteratively. The question is then whether their iterations include the development team and so far it looks as if they've not done so sufficiently that Agile's value here probably isn't more frequent delivery of complete vertical slices but helping to ensure that the direction traveled is more likely to be correct by facilitating conversation.

If more frequent contact between the Quant and Development team then mean fewer wasted cycles and fewer trips down blind alleys which might have resulted from more isolated efforts then it's another tick in the Adds Value column - this scenario leverages the Wisdom of Crowds. However, design by committee might just as easily be a problem instead. We'll see.

Returning to the initial question of how well suited the Agile Methodology might be for Model Development, Scott Ambler provides one possible answer: Agile Model Driven Development (AMDD): The Key to Scaling Agile Software Development.

Meanwhile I'll be discovering how well Continuous Integration works on a development team of one and whether the overhead can be justified.

Differences

It was more than a little odd to see tests pass using the ReSharper Test Runner but fail when using either NUnit or the as yet mystery default option (whichever is used when Test With > Debugger is selected from the context menu).

Reason being that ReSharper is kind enough to run the tests in the order they appear whilst the others do so in alphabetic order. Dependencies between tests become apparent. Slaps to the forehead are delivered. We resolve to steer clear of using statics for unit testing.

Mouse

With only one monitor at work - everyone has just the one monitor so we're not too keen on standing out for being flash by having two monitors! - it would be handy having test results appear in the status bar using an ALT-T shortcut and TestDriven.NET. However, licensing! So it's tabbed Unit Test Sessions and CTRL-TAB until we're in the promise land.

Keyboard Shortcuts

Also, it turns out the the advertisement for going mouseless is true. Keeping the hands on the keyboard does indeed speed up coding. The next problem then was trying to access the Debug Tooltips (available ordinarily by hovering the mouse over variables). The solution is AutoHotkey and the script is courtesy of Rob Henry.

Bloomberg connectivity is a riot. We submit (by FTP) a text file with an expected format using carriage return and line feed to separate data entity requests.

From C# this means shelling out to a batch file (remember DOS) which executes some Java code responsible for handling request/response. If the request is valid then one minute later a zip file is returned that contains a text file. We parse and we have a database ready data file. If the request is badly formed, however, then an error file is created on the remote FTP directory but that is never returned!

Five years ago someone asked on WILMOTT (serving the Quantitative Finance Community) how to connect a Java application with Bloomberg to send and receive data. Dominic Connor was kind enough to reply.

At another time, at another place we used the DLLs on the Bloomberg terminal to return the data we needed. It was fast, efficient, predictable and illegal. Good times.

You may be here looking for something on bbdlftp or BBDL FTP. Good luck.

It's been almost one week since we started doing this daily code Kata: Roy Osherove's TDD Kata 1 - String Calculator. Thirty minutes every day in October so far. Six days. Three hours. Writing the same functionality over and over. What happens is that, naturally, each time we get a little closer to the end (we've not yet completed it in thirty minutes) and we optimise by anticipating functional requirements and code to allow for simple changes ahead. We'll decide how much of this is cheating once we're actually done a few times within thirty minutes before moving onto the next Kata.

We're using just the one code snippet, TDDtestmethod, which generates a test method. We abandoned the calculator variable name, opting for c.Add instead.

Is there a prize for the longest [Test] case method name?

Calculator_ 
MultipleDelimeterVaryingLengthSpecifiedDelimitedMultipleStringDecimalValues_
ShouldReturnSummedValues()

Using underscores would only make matters worse. And when we looked at some of our tests recently we discovered that we were unit testing parts of the framework (guess we're still not sure about Entity Framework) and using unit tests for spikes too. The words "Gone Too Far" appeared suddenly.

Today, a courier cyclist arrived bearing the gift of a new contract for new employment. If everything goes as expected we'll be working in the City within two weeks.

It's perhaps better not to mention the employer by name but they're a solutions provider working within an interesting area of finance and our role's likely to involve helping to ensure the reliability of systems processing terrabytes of data with issues of cleanliness and performance being most important. We're thinking optimised data querying, concurrency (scale) and transformation/calculation.

Without much recent UI experience all the cool Winforms & Web developer roles seemed out of reach so this is where our spare time needs to be invested over the next couple of years whilst the day job is all about the data, touching possibly on F# and elements of grid computing and much more of the modelling languages: UML & XML.

In terms of development approach the experience from the last consultancy role at Man Investments should prove invaluable in terms of transferring agile methodology and process where appropriate but because this is a smaller company, enterprise systems integration complexity ought not to present the same kind of challenges. I'm hoping for a more contained, bounded, intra-application kinda complexity. Those woods and trees managed to get quite blurred in that last assignment.

The new mission is to help provide Market Transparency. After five years in Hedge Funds it seems fitting.

I should add that Man Investments was a fantastic place to work.