Mathias Brandewinder on .NET, F#, VSTO and Excel development, and quantitative analysis / machine learning.
3. May 2015 16:34

I got curious the other day about how to measure the F# community growth, and thought it could be interesting to take a look at this through StackOverflow. As it turns out, it’s not too hard to get some data, because StackExchange exposes a nice API, which allows you to make all sorts of queries and get a JSON response back.

As a starting point, I figured I would just try to get the number of questions asked per month. The API allows you to retrieve questions on any site, by tag, between arbitrary dates. Responses are paged: you can get up to 100 items per page, and keep asking for next pages until there is nothing left to receive. That sounds like a perfect job for the FSharp.Data JSON Type Provider.

First things first, we create a type, Questions, by pointing the JSON Type Provider to a url that returns questions; based on the structure of the JSON document it receives, the Type Provider creates a type, which we will then be able to use to make queries:

#I"../packages"
#r @"FSharp.Data.2.2.0\lib\net40\FSharp.Data.dll"
open FSharp.Data
open System

[<Literal>]
let sampleUrl = "https://api.stackexchange.com/2.2/questions?site=stackoverflow"
type Questions = JsonProvider<sampleUrl>

Next, we’ll need to grab all the questions tagged F# between 2 given dates. As an example, the following would return the second page (questions 101 to 200) from all F# questions asked between January 1, 2014 and January 31, 2015:

https://api.stackexchange.com/2.2/questions?page=2&pagesize=100&fromdate=1420070400&todate=1422662400&tagged=F%23&site=stackoverflow

There are a couple of quirks here. First, the dates are in UNIX standard, that is, the number of seconds elapsed from January 1, 1970. Then, we need to keep pulling pages, until the response indicates that there are no more questions to receive, which is indicated by the HasMore property. That’s not too hard: let’s create a couple of functions, first to convert a .NET date to a UNIX date, and then to build up a proper query, appending the page and dates we are interested in to our base query – and finally, let’s build a request that recursively calls the API and appends results, until there is nothing left:

let fsharpQuery = "https://api.stackexchange.com/2.2/questions?site=stackoverflow&tagged=F%23&pagesize=100"

let unixEpoch = DateTime(1970,1,1)
let unixTime (date:DateTime) =
(date - unixEpoch).TotalSeconds |> int64

let page (page:int) (query:string) =
sprintf "%s&page=%i" query page
let between (from:DateTime) (to:DateTime) (query:string) =
sprintf "%s&&fromdate=%i&todate=%i" query (unixTime from) (unixTime to)

let questionsBetween (from:DateTime) (to:DateTime) =
let baseQuery = fsharpQuery |> between from to
let rec pull results p =
let nextPage = Questions.Load (baseQuery |> page p)
let results = results |> Array.append nextPage.Items
if (nextPage.HasMore)
then pull results (p+1)
else results
pull Array.empty 1

And we are pretty much done. At that point, we can for instance ask for all the questions asked in January 2015, and check what percentage were answered:

let january2015 = questionsBetween (DateTime(2015,1,1)) (DateTime(2015,1,31))

january2015
|> Seq.averageBy (fun x -> if x.IsAnswered then 1. else 0.)
|> printfn "Average answer rate: %.3f"

… which produces a fairly solid 78%.

If you play a bit more with this, and perhaps try to pull down more data, you might experience (as I did) the Big StackExchange BanHammer. As it turns out, the API has usage limits (which is totally fair). In particular, if you ask for too much data, too fast, you will get banned from making requests, for a dozen hours or so.

This is not pleasant. However, in their great kindness, the API designers have provided a way to avoid it. When you are making too many requests, the response you receive will include a field named “backoff”, which indicates for how many seconds you should back off until you make your next call.

This got me stumped for a bit, because that field doesn’t show up by default on the response – only when you are hitting the limit. As a result, I wasn’t sure how to pass that information to the JSON Type Provider, until Max Malook helped me out (thanks so much, Max!). The trick here is to supply not one sample response to the type provider, but a list of samples, in that case, one without the backoff field, and one with it.

I carved out an artisanal, hand-crafted sample for the occasion, along these lines:

[<Literal>]
let sample = """
[{"items":[
{"tags":["f#","units-of-measurement"],//SNIPPED FOR BREVITY}],
"has_more":false,
"quota_max":300,
"quota_remaining":294},
{"items":[
{"tags":["f#","units-of-measurement"],//SNIPPED FOR BREVITY}],
"has_more":false,
"quota_max":300,
"quota_remaining":294,
"backoff":10}]"""

type Questions = JsonProvider<sample,SampleIsList=true>

… and everything is back in order – we can now modify the recursive request, causing it to sleep for a bit when it encounters a backoff. Not the cleanest solution ever, but hey, I just want to get data here:

let questionsBetween (from:DateTime) (to:DateTime) =
let baseQuery = fsharpQuery |> between from to
let rec pull results p =
let nextPage = Questions.Load (baseQuery |> page p)
let results = results |> Array.append nextPage.Items
if (nextPage.HasMore)
then
match nextPage.Backoff with
| None -> ignore ()
pull results (p+1)
else results
pull Array.empty 1

So what were the results? I decided, quite arbitrarily, to count questions month by month since January 2010. Here is how the results looks like:

Clearly, the trend is up – it doesn’t take an advanced degree in statistics to see that. It’s interesting also to see the slump around 2012-2013; I can see a similar pattern in the Meetup registration numbers in San Francisco. My sense is that after a spike in interest in 2010, when F# launched with Visual Studio, there hasn’t been much marketing push for the language, and interest eroded a bit, until serious community-driven efforts took place. However, I don’t really have data to back that up – this is speculation.

How this correlates to overall F# adoption is another question: while I think this curves indicates growth, the number of questions on StackOverflow is clearly a very indirect measurement of how many people actually use it, and StackOverflow itself is a distorted sample of the overall population. Would be interesting to take a similar look at GitHub, perhaps…

2. February 2014 08:08

tl/dr: Community for F# has a brand-new page at www.c4fsharp.net – with links to a ton of recorded F# presentations, as well as F# hands-on Dojos and material. Check it out, and let us know on Twitter what you think, and what you want us to do next… and spread the word!

If you are into F# and don’t know Community for F#, you are missing out! Community for F#, aka C4FSharp, is the brainchild of Ryan Riley. Ryan has been running C4FSharp tirelessly for years, making great content available online for the F# community.

The idea of C4FSharp is particularly appealing to me, because in my opinion, it serves a very important role. The F# community is amazingly active and friendly, but has an interesting challenge: it is highly geographically dispersed. As a result, it is often difficult to attend presentations locally, or, if you organize Meetups, to find speakers.

Ryan has been doing a phenomenal job addressing that issue, by regularly organizing online live presentations, and making them available offline as well, so that no matter where you are, you can access all that great content. The most visible result is an amazing treasure trove of F# videos on Vimeo, going back all the way to 2010. While I am giving credit where credit is due, special hats off to Rick Minerich, who has been recording the NYC meetings since forever, and making them available on Vimeo as well – and also has been lending a helping hand when C4FSharp needed assistance. Long story short, Rick is just an all-around fantastic guy, so… thanks, Rick!

In any case, the question of how to help grow the F# community has been on my mind quite a bit recently, so I was very excited when Ryan accepted to try working on this as a team, and put our ideas together. The direction I am particularly interested in is to provide support for local groups to grow. Online is great, but nothing comes close to having a good old fashioned meeting with like-minded friends to discuss and learn. So one thing I would like to see happen is for C4FSharp to become a place where you can find resources to help you guys start and run your own local group. While running a Meetup group does take a bit of effort, it’s not nearly as complicated as what people think it is, and it is very fun and rewarding. So if you want to see F# meetings in your area, just start a Meetup group!

In that frame, Ryan has put together a brand-new web page at www.c4fsharp.net, where we started listing resources. The existing videos, of course, but also a repository of hands-on Dojos and presentation/workshop material. The hands-on Dojos is something we started doing in San Francisco last year with www.sfsharp.org, and  has been working really well. Instead of a classic presentation, the idea is to create a fun coding problem, sit down in groups and work on it, learn from each other, and share. It’s extremely fun, and, from a practical standpoint, it’s also very convenient, because you don’t need to fly in a speaker to present. Just grab the repository from GitHub, look at the instructions, and run with it!

Just to whet your appetite, here is a small selection of the amazing images that came out of running the Fractal Forest Dojo in Nashville and San Francisco this month:

… and special mention goes to @Luketopia, for his FunScript Fractal Generator!

What’s next? We have a ton of ideas on what we could do. We will obviously add more resources as we go – but we would really like to hear from you guys. So help us make Community for F# the resource you would like to have! Here is what we would like from you:

• Contact us on Twitter at @c4fsharp, and let us know what you like and don’t like, and want to see!
• Take a look at the Dojos, and let us know how to make them better! Pull requests are highly appreciated. We have more Dojos and presentation material coming up, stay tuned! And if you have Dojo ideas you want to contribute, we’d love to hear about it.
• If you are organizing a Presentation, talking at a user group or a conference, ping us on Twitter, and we’ll let the Community know about your event!
• If you want to broadcast a presentation live, contact us, we would love to help out and make it available to the broader community.
• If you like what we are doing, please spread the word!

In short – we intend to make C4FSharp the best resource we can make it for local F# communities, and we would love your input and help on how to make that happen!

4. January 2014 12:00

Tis’ still the Season for yearly retrospectives, and making foolish predictions or commitments; here is a very incomplete and disorganized review of my year 2013 with F#, and some of my take-aways for the year ahead.

2013 has been a CRAZY year for me. I used to be proud of myself when I gave one talk per quarter – this is the map of places where I gave F# presentations / Dojos this year (note that I spoke multiple times in some of these places, and some online talks are not listed…):

So yes, it’s been a crazy year. One thing I find interesting here is that most of these talks were direct requests from the community. Not so long ago, I had to knock on doors and “sell” F# talks to user groups – recently, my worry is that I won’t be able to keep up with the requests for F# presentations. In my opinion, the interest in F# is currently totally under-estimated; I was stunned at how many developers showed up for some of these, in unexpected places.

What 2013 has taught me is that the picture “the F# community is all finance in London and NYC” is rather incorrect; while they have the largest concentrations of F# developers, there are also incredibly passionate developers all over the place, and the interest for the language is widespread. The problem here is that the community is pretty scattered; there are many trees, but it’s easy not to notice the sparse forest (yes I am looking at you, Microsoft :)).

I believe one of the main reasons for the current surge in interest is the fantastic work happening around the F# Foundation. It’s been instrumental in shaping a consistent message around the language, providing resources, and getting the community to coalesce around the idea “F# is our language, let’s make it what we want to be”. Huge props and thanks to Don Syme, Tomas Petricek and Philip Trelford for getting the ball rolling there – what has happened in very little time is amazing. And if you want to get involved in a language with an amazing community, where you can make a difference and help shape an ecosystem, GO TO FSHARP.ORG. NOW. We want you!

It’s been a crazy amount of fun, but I have been flirting with burnout quite a bit, too, hence my new year resolution:

As much as I would love to keep going everywhere talking about F# to anyone who expresses interest (my 2013 policy in a nutshell…), I’ll have to focus in 2014 and calm down a bit. Instead of talking everywhere (I’ll probably still come if you ask nicely and offer a couch to crash on ;) ), I want to work on scaling.

One format that has worked extremely well is hands-on Dojos: instead of a formal presentation, just get people to code together on an interesting problem, in a fun and friendly atmosphere. It’s great for bootstrapping people, and has the added benefit of being more centered on the community itself, and less on a speaker. So one of my goals this year is to begin building a library of ready-to-use Dojos, which groups can simply grab and run, without the hassle of finding speakers, something which is always a bottleneck for Meetups / user groups. I plan on doing this via Community for F# (@c4fsharp), the brain child of functional cow-boy Ryan Riley. If you are interested in that project, and in general in questions around growing a local community, I’d love to hear from you!

In a similar vein, I want to spend more time in my own backyard, San Francisco and the Bay Area, and help grow a stronger, inclusive, economically viable F# community there. Lots of reasons for optimism: we already have a strong, passionate, and growing community (hello, @FoxyJackFox!), we have stable hosting for sfsharp.org at ThoughtWorks (thanks for your support and enthusiasm, Logan!), and seeing exciting companies like GitHub, Xamarin or Kaggle embrace F# is awesome. The goal for 2014 is simple: crank up the level with Dojos and talks in SF, and start an outpost in the Silicon Valley (I hear there are some developers there, too).

As an aside, I wanted to tip my hat off Bryan Hunter, whose ideas on community building have been very inspirational. Nashville is slowly becoming a hotbed of functional programmers, and seems to be the place to be for F#ers lately; and I am sure this is in no small part due to Bryan’s focus on building a community that emphasizes cross-language, inclusion, empowerment, and dare I say, happiness.

What else? A big part of my year has been focused on my brand-new hashtag (vocation?) #OpenSourceMom ©. If you follow this blog, it shouldn’t come as a surprise to hear that I am very, very interested in Machine Learning and Data Science. I have been busy doing my best helping F# gain the recognition it deserves in that space (in part for selfish reasons: I think it’s a fantastic language for the job, and I want to be able to use it as much as possible), and that has lead me to try and help the community work better together. It never ceases to amaze me how much high-quality code the community has produced already; at the same time, writing code alone is only fun for that long, and because we are so dispersed, good ideas go unfinished or unnoticed, which is a shame. So while I prefer writing my own code (and not write any documentation for it), I was very excited when the F# foundation began launching working groups, and did my best to take a backseat and just try to facilitate communication and cooperation in the area of data science. It has been a fantastic experience, and I am incredibly happy with the results, and the opportunity this has given me to get to know better, and learn a lot, from all you guys (you know who you are). Also, a tip of the hat to Keith and his “Up for Grabs” initiative, which I hope we’ll get to leverage more this year – it’s IMO a great way to channel help, as well as provide easy entry points for beginners who are interested in getting started with a new language. Oh, and this was also the year of my first pull request ever :)

Finally, one highpoint of the year was the month of December, which gave me the chance to get to know the European community better. I had the pleasure to speak at BuildStuff in Vilnius, which was a fantastic conference. Greg, Neringa and Laura put together the kind of event you know you won’t forget – great speakers, of course, but also, and perhaps more importantly, an event with a soul. So thank you guys, and everyone I had the pleasure to talk to there! Oh, and by the way registration is open for 2014, and the price is unbelievable. Go there, buy your ticket now, you’ll thank me later.

While in Europe, I figured I might as well travel around a bit; isn’t that what people do while on vacation? So I went to visit the F# communities in Paris, London and Minsk, which was a blast. Having no organized F# community in France, a country with a strong OCaml history and my place of origin, was a thorn in my side for a long time; that problem has since been solved, the Paris meetup is in very good hands, and I was thrilled to speak there. Similarly, taking the trip to Minsk to speak at that user group was awesome. It was so great to finally meet Natallie and Serguey in person, after years of online contact! And I don’t know what they put in the water in Minsk (Vitamin F, maybe?) but the talent level there is just unbelievable. And I capped that year with London, which I expected to be great, and totally delivered.

So yes, this has been a pretty crazy year of F# for me. At the same time, this has been one of my most fun and rewarding years – all because of you, the F# community. I don’t know how to say it better, but this community just completely, utterly, massively kicks ass. Which makes me even more grateful and humbled that I got nominated F# MVP of the year for 2013. So from the bottom of my heart, thank you – even if it was a grueling year at times, you made it all worth it, and I can’t wait to see what we’ll do together this year. Happy 2014 – the Year of F#!

1. September 2013 13:39

I have been back for about a week now, after nearly three weeks on the road, talking about F# all over the US. The first day I woke up in my own bed, my first thought was “where am I again? And where am I speaking tonight?”, now life is slowly getting back to normal, and I thought it would be a good time to share some impressions from the trip.

• I am very proud to have inaugurated two new F# meetup groups during that trip! The Washington DC F# meetup, organized by @devshorts, is off to a great start, we had a full house at B-Line Medical that evening, with a great crowd mixing F# fans, C# developers, as well as OCaml and Python people, it was great. My favorite moment there was with Sam. Sam, a solid C# developer, looked very worried about writing F# code for the first time. Two hours later, he was so proud (and legitimately so) of having a nice classifier working, all in F#, that he couldn’t resist, and presented his code to the entire group. Nice job! Detroit was my final stop on the road, and didn’t disappoint: the Detroit F# meetup was awesome. It was hosted at the Grand Trunk Pub; while the location had minor logistics drawbacks, it was amply compensated by having food and drinks right there, as well as a great crowd. Thanks to  @OldDutchCap and @JohnBFair for making this happen, this was a suitable grand finale for this trip!
• In general, August seems to be the blossoming period for F# meetups – two other groups popped up in the same month, one in Minsk, thanks to the efforts of @lu_a_jalla and @sergey_tihon, and one in Paris, spearheaded by @tjaskula, @robertpi and @thinkb4coding, this is very exciting, and I am looking forward to meeting some F#ers next time I stop back home!
• A lesson I learnt the hard way is that San Francisco is most definitely not a good benchmark for what to wear in August in the US. My first stops were all in the south – Houston, Nashville, Charlotte and Raleigh, and boy was I not ready for the crazy heat and humidity! On the other hand, I can confirm the rumor, the South knows how to make a guest welcome. For that matter, I am extremely grateful to everyone who hosted me during this trip – you know who you are, thank you for all the help.
• One surprise during this trip was the general level of interest in F#. I regularly hear nonsense sentences like “F# is a niche language”, so I expected smaller crowds in general .NET groups. Well, apparently someone forgot to tell the .NET developers, because I got pretty solid audiences in these groups as well, with an amazing 100 people showing up in Raleigh. Trinug rocked!
• In general, I was a bit stressed out by running a hands-on machine learning lab with F# novices; for an experienced F# user, it’s not incredibly complex, but for someone who hasn’t used the language before, it’s a bit of a “here is the deep-end of the swimming pool, now go see if you can swim” moment. I was very impressed by how people did in these groups, everyone either finished or ended up very close. Amusingly, in one of the groups, the first person who completed the exercise, in very short time, was… a DBA, who explained that he immediately went for a set-oriented style. Bingo! The lesson for me is that F# is not complicated, but you have to embrace its flow, and largely forget about C#. One trick which seemed to help was to ask the question “how would you write it if you were using only LINQ”. Otherwise, C# developers seemed to often over-think and build code blocks too large for their own good, whereas F# works best by creating very small and simple functions, and then assembling them in larger workflows.
• Another fun moment was in Boston, where I ran the Machine Learning dojo at Hack/Reduce, language agnostic (thanks @JonnyBoats for making the introductions!). Pretty much every language under the sun was represented (C#, Java, F#, Scala, Python, Matlab, Octave, R, Clojure, Ruby) – but one of the participants still managed to pull “something special”, and tried to implement a classifier entirely in PostgreSQL. It didn’t quite work out, but hats off nevertheless, that was a valiant experiment!
• As a Frenchman, I take food seriously. As a scientist, I want to see the data. Therefore, I was very excited to have the opportunity to investigate whether Northern Carolina style BBQ is indeed an heresy, first hand. I got the chance to try out BBQ in Houston and Raleigh, and I have to give it to Texas, hands down.

• Lesson learnt the hard way: do not ever depend on the internet for a presentation. Some of my material was on a Gist on GitHub, and a couple of hours before a presentation, I realized that they were under a DOS attack. Not happy times.
• I am more and more of a fan of the hands-on, write code in groups format. It has its limitations – you can’t really do it with a very large crowd, and it requires more time than a traditional talk – but it’s a very different experience. One thing I really enjoyed when starting with F# was its interactivity; the “write code and see what happens” experience rekindled the joy of coding for me. The hands-on format captures some of that “happy hacking” spirit, and gets people really engaged. Once someone start writing code, they own it – and working in groups is a great way to accelerate the learning process, and build a community.

• I have been complacent with the story “it works on environments other than Windows/Visual Studio”. It does, but the best moment to figure out how to make it work exactly is not during a group coding exercise. In these situations, fsharp.org is your friend – and since I came back, I started actually trying all that out, because “I heard it should work” is just not good enough.
• I saw probably somewhere between 500 and 1,000 developers during this trip, and while this was completely exhausting, I don’t regret any of it. One of the highpoints of the whole experience was to just get some time to hang out with old or new friends from the F#/functional community – @panesofglass in Houston, @bryan_hunter and the FireFly Logic & @NashFP crew in Nashville, @rickasaurus, @tomaspetricek, @pblasucci, @mitekm and @hmansell in New York City, and @plepilov, @kbattocchi and @talbott in Boston (sorry if I forgot anyone!). If this trip taught me one thing, it’s that there is actually a lot of interest for F# in the .NET community, and beyond – but we, the F# community, are very scattered, and from our smaller local groups, it’s often hard to get a sense for that. Having a chance to talk to all of you guys who have been holding the fort and spreading F# around, discussing what we do, what works and what doesn’t, and simply having a good time, was fantastic. We need more of this – I am incredibly invigorated, and very hopeful that 2014 will be a great year for F#!
21. May 2013 12:55

Last week, we had our first Coding Dojo at SFSharp.org, the San Francisco F# group – and it was great! A few people in the group had mentioned that at that point they were already convinced F# was a great language, and that what they wanted was help getting started writing actual code, so I figured this would be a good format to try out.

What I wanted was something fun, something cool people could realistically achieve under 2 hours. I settled for one of the Kaggle introduction problems, a classic of Machine Learning, where the goal is to automatically recognize hand-written digits. I didn’t think it would be fair to just throw people in the shark tank without any guidance, especially for F# beginners, so I prepared a minimal slide deck to explain the problem and data set, and a “guided script”, with hints and language syntax examples.

And… it worked! The attendees were absolutely awesome. We had people from Kaggle, Rdio, and two people who drove all the way from Sacramento; we had beginners and experienced FSharpers – and everybody managed to get a classifier working, from scratch. Having some beers available definitely helped, too.

My favorite part is this one attendee, a F# beginner, who kept going at it after the meeting was over, and posted an algorithm improvement in the comments section of the Meetup a couple days after. Way to go! And given the positive response, we’ll definitely have more of these.

Also wanted to say a huge thanks to Matt Harrington, first for starting this user group back then, and then for still being an incredible supporter of the F# community in SF, in spite of a crazy work schedule. Thanks, Matt!

Introduction slide deck

“Guided script”

#### Need help with F#?

The premier team for
F# training & consulting.