Roslyn Scripting

We recently needed to support local functions in runtime C# code. This runtime code is to allow us or the Customer to customise how a HTTP POST JSON document is constructed. At the time we were using CSScript to add runtime scripting to the application, however the default Evaluator was Mono which did not support local functions.

Fortunately Roslyn did. Due to that and some other requirements, we decided that we’d step away from CSScript and try and use Roslyn Scripting directly.

So here is the core of the code:

using System;
using System.Threading.Tasks;
using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Scripting;

// ...

public static async Task<object> GenerateAsync(string codeAsString, GlobalData data)
{
    var options = ScriptOptions.Default
        .AddReferences("System", "System.Collections.Generic")
        .AddImports("System", "System.Collections.Generic");

    var script = CSharpScript.Create(codeAsString, options: options, globalsType: typeof(GlobalData));

    var state = await script.RunAsync(data);

    return (state.Exception == null) ? state.ReturnValue : throw new Exception("Runtime code had issues.", state.Exception);
}

Some comments on the code:

  • GlobalData type is the class that contains the variables that you want to expose to the runtime code. It needs to marked as public class. All its public properties are exposed to the script as global variables.
  • AddReferences adds namespaces/assemblies to the runtime code.
  • AddImports is effectively adding “using …” to the runtime code.

You can find a complete sample at roslyn-scripting-demo on GitHub.

Build ASP.NET Core with Jenkins on Windows

A post on how I configured Jenkins v2 on Windows for ASP.NET Core applications using Pipeline as code.

Before we start

  • Why Windows when ASP.NET Core can build and run on linux? This Jenkins instance was going to be used for other specific Windows builds as well. At least one of the ASP.NET Core applications needs to be built against the .NET Full Framework.
  • Why Jenkins? It’s what I’ve been using personally or on other teams for many years now. I’m sure Teamcity or VSTS are more than capable.
  • I use rake to run my build tasks (choco install ruby)
  • I’m using git (choco install git)

Versions as of writing

  • Windows 10 Pro (Build 15063.138)
  • Jenkins v2.55.
  • git 2.12

Install Jenkins

I’m using Jenkins v2 with the new Blue Ocean UI (currently installed as a plugin).

  1. Go to https://jenkins.io/download/
  2. Download either LTS or Weekly installer for Windows.
  3. Run the installer. I usually install Jenkins to an empty volume (e.g. D:\).
  4. Go into Manage Jenkins -> Manage Plugins. Install “Blue Ocean”. Restart Jenkins when finished.

Install VSTools

This installs MSBuild and related files. I’ve based it on this answer on StackOverflow.

  1. Go to https://www.visualstudio.com/downloads/.
  2. Scroll down to the expandable section and click on “Other Tools and Frameworks”.
  3. Select “Download” button for “Build Tools for Visual Studio 2017”.
  4. Open up Command Prompt and execute the following: vs_buildtools.exe --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Workload.WebBuildTools --quiet

You can find more information about those component identifiers at https://<docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools>

A couple of notes about MSBuild and Visual Studio 2017:

  • MSBuild has now been moved into the Visual Studio 2017 installation folder.
  • You can install Visual Studio 2017 anywhere.
  • Visual Studio 2017 now supports side-by-side installation between the different sku’s (enterprise, professional community, buildtools).
  • There is no standard registry key or environment variable that indicates where Visual Studio, let alone MSBuild is installed.
  • Microsoft has some recommendations on how to find where Visual Studio 2017 is installed at. E.g. vswhere.
  • It is no longer as easy to determine where MSBuild is, unlike before.

To get around this, what I’ve done for my environments is add an MSBUILD15 environment variable that points to the folder of msbuild. Any build scripts I have bomb out with a helpful message if that variable isn’t set.

Note that you can add this variable through Jenkins so its presented to your build scripts without having to edit Windows settings. Go to Manage Jenkins -> Manage Nodes then edit the node that you are configuring.

Configure Your Project for Pipelines

As we will be using Pipelines in Jenkins (instructions to Jenskins on what to do with your project), we need to create a Jenkinsfile in the root of our repository. This will tell Jenkins what the steps are for our pipeline and how to execute them.

As this is a simple project and I’m using rake to execute the task, this is what my Jenkinsfile looks like:

node {
    stage 'Checkout'
        checkout scm

    stage 'Build'
        bat 'rake build'

    stage 'Test'
        bat 'rake test'
}

You can read all about Pipelines at https://jenkins.io/doc/book/pipeline/.

Create new Pipeline

As I stated before I’m using Blue Ocean so the steps below reflect that.

  1. In the normal Jenkins view, click on the “Open Blue Ocean” button in the nav-bar.
  2. Select “New Pipeline”.
  3. Select where your code is stored. For me this is “Git” (this project is on Bitbucket).
  4. Enter in the repository URL and create a new Credential for it. I use SSH Private Key which is configured as a read-only key for this particular repository in bitbucket.
  5. Select Create Pipeline.
  6. Jenkins should connect to the repository, find the Jenkinsfile and perform the defined tasks. If everything was run successfully, you should see something like this:

Poll Repository for Changes

As my Jenkins instance is running within a LAN and is not visible to bitbucket, we need to tell Jenkins to poll regularly for changes to the repository. To do that:

  1. Edit the project in Jenkins (Select the cog in the top-right hand corner).
  2. Scroll down to “Scan Multibranch Pipeline Triggers” and check the checkbox and select an appropriate Interval:
  3. Save

And that should be it.

Unable to start kestrel – An attempt was made to load a program with an incorrect format

Hopefully this saves someone some time in the future.

I was developing a throw away ASP.NET Core app on .NET Full framework in Visual Studio 2017. I noticed that the Platform target was set to x86 (the default), so I changed it to AnyCPU then redeployed.

I tried to run it from the console (I love how I can just run a webapp from the console now… so awesome), but got the following error:

crit: Microsoft.AspNetCore.Server.Kestrel[0]
      Unable to start Kestrel.
System.AggregateException: One or more errors occurred. ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B)

Turns out that even though I had changed the Platform target via the Project Properties, there was a setting within the csproj that indicated an x86 runtime.

To solve the issue, open up the csproj file for your project and change the x86 part of the value of the RuntimeIdentifier element to x64.

So instead of (for example):

<RuntimeIdentifier>win7-x86</RuntimeIdentifier>

You want:

<RuntimeIdentifier>win7-x64</RuntimeIdentifier>

Save that and rebuild or redeploy your project.

For more information about the above, I first found the error message described at:

https://docs.microsoft.com/en-us/aspnet/core/publishing/iis#platform-conflicts-with-rid.

Information about the RuntimeIdentifer property, see https://docs.microsoft.com/en-us/dotnet/articles/core/tools/csproj#runtimeidentifier. For information about the RID values, see https://docs.microsoft.com/en-us/dotnet/articles/core/rid-catalog.

REPL Development using F# Interactive in Visual Studio Code

Now that we have Ionide installed, we can do REPL development by using the F# Interactive console (FSI).

There are a couple of ways of getting FSI to run your code:

  1. Sending the selected code to FSI.
  2. Sending the current line to FSI.
  3. Sending the whole file to FSI.

For the first two options, you don’t need to be working in an existing F# file. You can just create a new tab, start typing, select the code and send to FSI.

Caveat: I found that if you wanted Intellisense to work you needed to be working in a saved .fsx or .fs file (see reported issue). You can’t just change the language mode to F#.

Start FSI

To get started with FSI, you need to start it. To do so, open the Command Palette (⇧⌘P) and typing in “FSI: Start”.

The output panel will display like so:

Run code in FSI

Type in some F#. Then select the F# code you want to run and then select “FSI: Send Selection” from the Command Palette or type in ⌥Enter. This will take the code you have selected and run it in FSI and give you the result in the Output panel.

If you made a mistake in your F# code, you’ll see the compiler or runtime output in the Output panel. E.g.:

You can also send the whole file to FSI, however for that you need to be working in an existing file. Once you are working in an F# file, you can send the whole file (including your current unsaved changes) by selecting “FSI: Send File” from the Command Palette. However I found it easier to just select the content of the file (⌘A) and then send the selection to FSI.

Getting started with F# in Visual Studio Code on macOS

Previously I showed how to install F# on OSX. In this post I’ll start showing you how you can start devloping F# in Visual Studio Code on OSX (macOS… whatever).

Assumptions

  • You are using OSX. These instructions may work on Windows, but I haven’t tested on Windows. Let me know if they do.
  • You have mono already installed. If not, follow the instructions.
  • You have Visual Studio Code (VSCode from now on) installed. If not, download it and install it.

Before we Get Started

These instructions were taken when using mono 4.4.2, VSCode 1.4, OSX 10.11.6, and Ionide extension v2.2.7. If any of the below does not work for you it may be due to differences in the versions of any of the above.

Installing Ionide Extension

The bulk of the F# support comes from the excellent Ionide project which I believe was originally developed for the Atom editor. There are a set of extensions for VS Code.

So as with all other extensions in VSCode, open up the Extensions view via one of these methods:

  • click on Extensions icon in View Bar, or
  • press ⇧⌘X, or
  • Open up the Command Palette (⇧⌘P) and type in Install Extensions and press enter.

Now search for Ionide. You should see at least the three extensions below:

Install each extension (click on the green install button) then restart VSCode (rather than having to enable each extension). I installed them in the following order, though I’m assuming the order doesn’t matter: ionide-fsharp, ionide-fake, ionide-paket.

Now that Ionide is installed, you have should everything needed to get into F# now. There are a couple of ways you can do that:

I’ll describe each in the next set of posts.

C# and F# Xamarin iOS Quick Start Projects

I’m starting to get into iOS development using Xamarin.

So to start out, I followed the Quick Start tutorials for Xamarin Forms but as a twist decided to also do it using F#. I didn’t just figure out how to do it in F# however, I used the following resources to help out:

You can find my implementations of the Quick Start on github at:

Xamarin Forms Quick Start in C# and F# on GitHub

Though I want to draw attention to one part of the project which was a puzzle for me given my current knowledge (or lack thereof) of F#.

Async Event Handlers in F#

In the Quick Start tutorial the Call button click is an async event handler. So this (taken from MainPage.xaml.cs):

async void OnCall(object sender, EventArgs e)
{
        if (await DisplayAlert(
                        "Dial a Number",
                        "Would you like to call " + translatedNumber + "?",
                        "Yes",
                        "No"))
        {
                var dialer = DependencyService.Get<IDialer>();
                if (dialer != null)
                        dialer.Dial(translatedNumber);
        }
}

So its an async event handler in C#, but how on earth do you do this in F#?

As F# has similar but different ways of doing async (that came before C# async) we need to code it differently. So here is the equivalent in F# (taken from MainPage.fs). Note that I’ve split it into two methods to make a bit clearer:

member this.makePhoneCall = async {
        let! ok = Async.AwaitTask(this.DisplayAlert("Dial a Number", sprintf "Would you like to call %s ?" translatedNumber, "Yes", "No"))
        if ok then
            let dialer = DependencyService.Get<IDialer>()
            if not(Object.ReferenceEquals(dialer, null))
            then dialer.Dial(translatedNumber) |> ignore
    }

member this.OnCall(sender : Object, e : EventArgs) =
    this.makePhoneCall |> Async.StartImmediate

One of the key differences between C# and F# async is that in F# the async block (async { ... } in makePhoneCall) is a specification; it does not execute the code. You need to specifically start it using Async.StartImmediate in OnCall. This method starts the async block immediately on the current thread which in this case is the UI thread (which is needed as we’re interacting with the UI).

Some resources I found really helpful:

Install F# on OS X

As the title indicates, this is how to install the various bits and pieces to be able to compile F# applications and run F# scripts on OS X.

There are two ways I ended up doing it, both involve installing Mono. I went with the first option initially until I ended up doing iOS development using Xamarin.

Option 1 – Install using Homebrew

Assumptions

You have homebrew already installed.

Install mono

Pretty straight forward:

  1. Open Terminal
  2. Install mono and nuget via homebrew: brew install mono nuget

And away you go.

Option 2 – Install Xamarin Studio

  1. Download Xamarin Studio.
  2. Run the installer and choose the options that suite you and then follow the prompts. I actually went with a ‘manual’ install of Xamarin Studio as I didn’t want or need Android development resources installed on the Mac I was using.

How to manually install Xamarin Studio components

I went with this option as even though I didn’t want to install Android, Xamarin Studio was still going to install Java and Android SDK.

So this is how I performed a manual install:

  1. Run the Xamarin Studio installer.
  2. Accept the terms and select Continue.
  3. Select the products to install then select Continue. I chose Xamarin.iOS. Note: This won’t actually start the installation.
  4. On the “Please configure the installation” page, Select the “Xamarin Installer” in the menu bar, and then choose “View manual install instructions”. This will then show a new dialog with instructions on how to install the various bits of Xamarin Studio. It basically boils down to downloading a couple of seperate installers (Mono SDK, Xamarin Studio, Xamarin iOS product) and installing them. Note that if you wish to you can completely ignore the Java and Android.SDK options as I did.

Regardless of how you installed Xamarin Studio, you should have the various mono executables in your Terminal path.

Lastly, you can apparently use homebrew mono and Xamarin together though for me atm, it was just easier to stick with what Xamarin installed.

Install Node on OS X

Instructions that I used to install node recently.

Assumptions

You have homebrew already installed.

Install nvm

We’re going to install nvm, the node version manager (similar to rbenv). At the time this installed version 0.31. To see the latest instructions of setting up nvm, type in brew info nvm.

  1. Open Terminal
  2. Install nvm via homebrew: brew install nvm
  3. We want our node versions to be installed to a user local path, so make the folder ~/.nvm. Then append export NVM_DIR=~/.nvm to your bash profile: mkdir ~/.nvm echo "export NVM_DIR=~/.nvm" >> ~/.bash_profile
  4. Make sure nvm is available from the terminal: echo ". \$(brew --prefix nvm)/nvm.sh" >> ~/.bash_profile

Reload the bash profile source ~/.bash_profile or restart the terminal.

Install Node

Using nvm let us install the latest version. At the time, this was 5.8.0.

nvm install v5

Node that nvm will also make this the default node version which will be available from your PATH.

To see what the latest node version is type in (there may be v6, v7, etc.):

nvm ls-remote

Upgrade to latest npm

Finally, let’s make sure we have the latest npm:

npm install -g npm

And that should be it.

Windows, Jenkins, Git and SSH

Having had to set up Git in Jenkins running on Windows recently, I thought I’d document what I did to get it to work.

Firstly, the software involved:

  • Windows 10
  • Jenkins (currently it is version 1.651)
  • Git (Git-2.7.2-64-bit.exe from https://git-scm.com/downloads). Note that this Git install comes with ssh (and some other linux utilities).

Assumptions

My requirements for Git are to pull a remote repository so Jenkins can perform a build, run tests, then deploy the artifacts somewhere. Thus in my case I only need read-only access to the remote repository.

So:

  1. You must have or know how to generate an SSH key with no password. Your remote repository must have been configured to use that key (e.g. Bitbucket – Use deployment keys). Note: Having no password definitely reduces the security of the key, so make sure your build server is secured (e.g. good password), and ideally that key only has readonly access to the remote repository.
  2. You are running Windows 10. Adjust the steps as necessary for other Windows versions.

Aside – Why SSH?

We use BitBucket and when configuring the Deployment Keys, you can only use SSH keys.

Step 1 – Install Jenkins

Pretty easy. Download the Jenkins Windows Installer from https://jenkins-ci.org (direct link – http://mirrors.jenkins-ci.org/windows/latest).

Run the installer.

Tip – I normally install Jenkins on its own drive as depending on how many builds you keep, jobs you have, it can consume quite a bit of space.

Step 2 – Run Jenkins as User

As Git will be using SSH and thus will be looking for the .ssh folder we’re going to run Jenkins as its own User rather than as the default “Local System Account”. If you want or need to run as “Local System Account” stop now and follow the instructions at http://opensourcetester.co.uk/2013/06/28/jenkins-windows-ssh/.

We first need to create a Windows User. I’m doing everything on a machine that is not connected to a domain. Adjust the steps necessary to create the user within a domain.

Create the User

Follow these steps to bypass all the

  1. Open up the Computer Management management console app (tip – search in Cortana for “Computer Management”):
  2. Go to “System Tools” -> “Local Users and Groups” -> “Users”:
  3. Select “Actions” main menu then “New User…”. The New User dialog should show. Enter in the details you want. Uncheck “User must change password at next login”, check “User cannot change password”, and “Password never expires”:
  4. Select “Create” to create the user.

Configure User to Run Jenkins Service

We now need to configure the user so that they can run the Jenkins Windows Service. Normal users by default cannot.

  1. Open up the “Local Security Policy” management console app (tip – search in Cortana for “Local Security”):
  2. Go to “Local Policies” -> “User Rights Assignment”:
  3. Find the Policy “Log on as a service”, then double-click on it. Add the user you created above. You should end up with something like:
  4. Now the Jenkins service needs to be configured to run as this new user, so to do so, go to the Services management console app (tip – search in Cortana for “Services”).
  5. Find the “Jenkins” service and stop the service. Now double-click on it. Under the “Log On” tab, select “This account:” option, enter in the newly created user and its password and select “OK”:
  6. Now re-start the “Jenkins” service. Open up a local web-browser and browse to “http://localhost:8080” to make sure that Jenkins is running. If not, there may be a configuration issue so re-check that everything was done correctly as above.

Step 2 – Install Git

Run the Git installer downloaded from https://git-scm.com. I chose the following options during the installation. Note that it is important that you select “Git and optional tools” otherwise pulling git repositories using SSH is not going to work too well as git will not be able to find ssh.

Tip – I usually install Git at D:\apps\git so that there is no likelyhood of spaces causing issues.

  1. As this is a build server, we don’t need any integration with Explorer.
  2. When it prompts for how to use git from the command line, make sure you choose the Git with optional tools as this will add the ssh utility to the %PATH% environment variable.
  3. Click next a couple more times and git should be installed (the other options are not relevant to this post).

Step 2 – Configure SSH

I’m using SSH to connect to the remote Git repository. This is really where most of the complexity is with this set up.

Again, just to re-iterate, you need to have an passwordless SSH key generated already and have configured your remote repository to accept that key.

Just to describe what is going to happen – When Jenkins polls the remote repository, it is going to run “git.exe” as the User that the Jenkins service runs as. When Git connects to the remote repository, it is going to use ssh.exe which in turn is going to look for the SSH Private Key at ~/.ssh/id_rsa. Note that ~/ maps to %HOME% which on Windows normally doesn’t exist, but we’ll map it to %userprofile% (e.g. C:\Users\<username>\).

I’m doing all of the set up while still logged in as an admin user. You could perform the same configuration by logging in as the Jenkins Windows User.

Configure .ssh folder

We’ll firstly set up the .ssh folder and known_hosts for the Jenkins Windows user.

  1. Open up Command Prompt. Then type in the following replacing <username> with the Jenkins Windows User username (e.g. jenkins) runas /profile /user:<username> cmd It will prompt you for the password of the user and if successful, open up a new Command Prompt window.
  2. In the new Command Prompt Window, go to the user’s profile: cd %HOME% And you should be taken to the user’s home directory (E.g. C:\Users\jenkins).
  3. Now create the .ssh directory in the user’s home directory: mkdir .ssh
  4. Copy your id_rsa private key file into the Jenkins user’s .ssh folder (note: you’ll have to manually browse to the directory in Windows Explorer as explorer . will not work in the Jenkins user command prompt).
  5. If you already have the known_hosts file for the remote repository readily available, copy it under .ssh now.

Close the jenkins user Command Prompt for now.

Configure HOME Environment Variable for Jenkins Windows User

We need to set the HOME environment variable for the jenkins user, so to do so we’ll need to edit the Registry for that user.

  1. We first need to find the SID of the Jenkins Windows User. Open up Command Prompt and type in the following, replacing <username> with the Jenkins Windows User username: wmic useraccount where name='<username>' get sid You should then see something like: C:\Users\admin>wmic useraccount where name='jenkins' get sid SID S-1-5-21-2036844044-1338580185-3091361466-1002
  2. Now open up regedit and go to HKEY_USERS and find the folder for the SID above. Then go to Environment.
  3. Now create an “Expandable String Value” (REG_EXPAND_SZ) with the Name of “HOME” and the value of ‘USERPROFLIE%’ (you have to modify it to set the value).
  4. Lastly restart the Jenkins Service so that it get the new Environment Variable.

Test that Git can connect

Now we’ll test that Git can connect to the remote repository. Going back the Jenkins user command prompt (e.g. runas /profile /user:<username> cmd), type in the following, replacing with the remote repository url.

git ls-remote <git_repo>

If you haven’t configured known_hosts, you will get prompted on the authenticity of the remote host. If ok, type in yes. This will populate the ~/.ssh/known_hosts file.

If everything is configured, you should see a list of the references in the remote repository (e.g. master, list of tags, etc.).

Step 3 – Configure Jenkins with Git

We need to install a plugin before being able to clone a repository from Jenkins.

To do so, open up Jenkins, go to “Manage Jenkins” then “Manage Plugins”. Select the Available tab, then search for Git plugin. You should find the following:

Select and install that. It will install Git client plugin as well which is a dependency.

Step 4 – Configure Jenkins Job

Finally, we need to configure a Jenkins Job to pull from the Git repository, so under Source Code Management set the Repository URL. For example:

Then configure to Poll the remote repository (assuming you’re polling it):

And that should be it.