In our last post, we saw how to use F# to read historical stock quotes from Yahoo. Today we’ll take the raw response, which is a big block of text, and break it up into a list of individual quotes.

Breaking up the response into lines

The function we wrote last time, GetResponse, receives one chunk of text from the web service, formatted like this:

Date,Open,High,Low,Close,Volume,Adj Close
2010-03-08,28.52,28.93,28.50,28.63,39414500,28.50
2010-03-05,28.66,28.68,28.42,28.59,56001800,28.46
2010-03-04,28.46,28.65,28.27,28.63,42890600,28.50

What we need to do now is break up this into individual lines of text, and parse them to read individual quotes. The first part is straightforward: the function BreakIntoLines calls String.Split(), using char(10), the code for line break, as a delimiter, and returns an Array of strings.

let BreakIntoLines (response:string) =
  response.Split((char)10)

Note the type annotation on the function argument: without context, F# cannot infer the type of “response”, and we need to specify that this function expects a string argument.

Parsing valid lines into Quote records

The second part is a bit more complex. We need to break each line into 7 components (date, open, etc…), deal with lines that are not valid, like the header, and store the result in an appropriate structure.

We will store individual quotes into records. A Record is a data type somewhat similar to the C# struct. It has named fields, which makes it more expressive than Tuples, and is less involved than a class. Here is the declaration for our Quote record – concise, and pretty self-explanatory:

type Quote={
  Symbol:string;
  Date:DateTime;
  Open:double;
  Close:double;
  Low:double;
  High:double;
  Volume:int64}

More...

Lately I have spent time on a pet project, which requires access to historical financial data. Mads Kristensen has a nice post where he shows how to read  stock quotes from Yahoo finance using C#, which was very helpful to get started. I figured it would be interesting to try out a conversion to F# and see what the result looked like.

Mads focus is on getting quasi real-time updates of a quote; my interest is in an easier problem: retrieving historical data. Fortunately, Yahoo provides a free service for that, too. Given a quote symbol and two dates, it returns a comma-separated file list of all the values for the quote between these 2 dates.

So what do we need to do? Given a valid symbol and 2 dates, we want to create the WebRequest to send to Yahoo, retrieve the response, break it into lines, and parse each line into a quote, which will be added to a list. The core of the resulting program will be the ReadQuotes function, which will look like this:

let ReadQuotes symbol date1 date2 = 
  CreateRequest symbol date1 date2 
  |> GetResponse 
  |> BreakIntoLines
  |> CreateQuotes symbol

Creating the WebRequest

The web request required to obtain historical data from Yahoo follows this pattern:

http://ichart.finance.yahoo.com/table.csv?s=S&a=A&b=B&c=C&d=D&e=E&f=F&g=d&ignore=.csv

where:

• S is the symbol (ex: MSFT)
• A, B, C are the start month, day and year, the month being coded in base 0 (i.e. January is 0)
• D, E, F are the end month, day and year, the month being coded in base 0 (i.e. January is 0)

For instance, replacing S with MSFT, A with 0, B with 1, C with 2010, D with 1, E with 15, F with 2010, will return all the available quotes for Microsoft between January 1 and February 15, 2010.

Let’s start by creating a Console application, by selecting new F# project > F# Application, and typing in the following code:

open System;
open System.Net
open System.IO
open System.Text

let RetrieveDateInfo (date:DateTime) =
  (date.Day, date.Month-1, date.Year)

let CreateRequest symbol startDate endDate =

  let startDay, startMonth, startYear = RetrieveDateInfo startDate
  let endDay, endMonth, endYear = RetrieveDateInfo endDate

  let query = String.Format("&a={0}&b={1}&c={2}&d={3}&e={4}&f={5}&g=d&ignore=.csv", startMonth, startDay, startYear, endMonth, endDay, endYear)
  let url = "http://ichart.finance.yahoo.com/table.csv?s=" + symbol + query
  WebRequest.Create(url)

More...

One of the immediate benefits I saw in digging into F# is that it gave me a much better understanding of LINQ and lambdas in C#. Until recently, my usage of LINQ was largely limited to returning IEnumerable instead of List<T> and writing simpler queries, but I have avoided the more “esoteric” features. I realize that now that F# is becoming familiar to my brain, whenever I see a statement in C# which contains a foreach:

foreach (var item in items)
{
   // do something with item.
}

… I ask myself if this could be re-written in a more functional way. Sometimes it works, sometimes not. Just like classic OO Design Patterns, functional programming has its own patterns, and I find that having a larger toolkit of patterns in the back of my mind helps criticizing my own code and think about alternatives and possible improvements.

I encountered one such case a few days ago, with the following snippet:

public bool IsValid()
{
    foreach (var rule in this.rules)
    {
        if (!rule.IsSatisfied())
        {
            return false;
        }
    }

    return true;
}

There is nothing really wrong with this code. However, seeing the foreach statement, and an if statement with a return and no else branch made me wonder how I would have done this in F# – and my immediate thought was “I’d use a Fold”.

More...

One of my recent posts looked into reading the VBA code attached to a workbook, and lead to a discussion on analyzing the differences between the macros of two workbooks – what is commonly called a “diff”.

This got me curious as to how diffs are generated. A quick search lead to the Longest Common Subsequence problem: once (one of) the longest common sub-sequence (abbreviated LCS from now on) of characters between two texts has been identified, it is straightforward to determine what has been added and removed from the original text to get the second text.

Example

Original: this is my great code
Modified: that is my awesome code

LCS: th is my  code

Changes to original: this is my great code

The idea behind the algorithm used to identify such a longest common subsequence (LCS) is a nice example of dynamic programming, and goes along these lines. If I have two sequences,

• if they start with the same character (the head), then their LCS is the head + the LCS of the right-hand remainder of each string (the tail),
• if they don’t start with the same character, then their LCS could start either with the head of the first sequence, or of the second one. Their LCS is the longest of the LCS of the first sequence and the tail of the second, and of the LCS of the second sequence and the tail of the first one.

More...

After spending a few years working mostly with C#, it has become a natural, comfortable way to think about programming problems. I won’t complain about it -- comfort is nice. At the same time, I strongly believe in questioning what you do, especially the un-stated assumptions. When you start doing things a certain way without asking yourself if this is indeed the way to go, you are on a dangerous path, especially in a fast-evolving field like software engineering. So when I read the advice to “learn one new language every year”, it resonated, and I decided to give a shot at F#. I purchased “Programming F#”, and I am working my way through the Project Euler problems as an exercise.

This is my first exposure to functional languages, and it has proven a very stimulating mental exercise so far. One particular aspect I have struggled with is if … then statements. Chris Smith says that “if expressions work just as you would expect”. That’s sort of true, except for the fact that an if … then statement with no else clause can’t return a value.

This made me realize how much I use single-pronged if statements in C#, guard clauses being a prime example.

More...