Wednesday, September 14, 2016

Accessing an authenticated S3 file in Xamarin

Context

I have a S3 JSON file that I need to access from a Xamarin Forms mobile application.  My S3 resource is not a public resource, its protected and requires credentials to be able to access it.  There is another Windows Desktop application that will upload this file periodically, and the mobile app must only have read only access to the file.

Note: no special authentication is required to access public (unprotected) S3 files.

Setting up S3 and Credentials

The AWS portal offers a free plan to store up to 5GB of storage for 12 months.  After that it'll depend on number of requests and storage size. For me, for a 100Kb file accessed maybe 100 times a month, it'll be approximately 5 cents per month based on the calculator.

To access protected (non-public) data in S3 you'll need to set up users and access tokens.
Users are created with the AWS IAM service.  This is a pretty straight forward process, I needed two users, one with full access, and one with read only access.  To do this you need to attach an appropriate policy to the user, the read only policy is shown here:

Once this policy is attached, user will have read only access to everything stored in the S3 account.  
The next step is to generate an access token. This is done with the Security Credentials tab.


Managing Files in S3

Again this is very simple so not much detail here.  There are a couple of basics to be aware of.  You first choose where your data is stored geographically (ie Sydney, Oregon, Tokyo to name a few) and is not mirrored between regions (but this can be set up).  Data is stored inside "Buckets" - think folder.  Although you can't nest. Files can then be uploaded and accessed inside these buckets.
The red line is obscuring my chosen bucket name (because its top secret), and you can see the file name shown in the list.

Using the AWS S3 SDK

Writing a C# desktop application is pretty easy with the AWS SDK, certainly no challenges.

To be able to access the file created above here's all we need:
     private const string AwsBucketName = "XXXXXXXXXsummary";  
     private const string AwsBucketFileName = "MobileDataExport.json";  
     public async Task UploadDataFileAsync(string data, string storageKeyId, string storageSecret, string region)  
     {  
       var regionIdentifier = RegionEndpoint.GetBySystemName(region);  
       using (var client = new AmazonS3Client(storageKeyId, storageSecret, regionIdentifier))  
       {  
         try  
         {  
           var response = await client.GetObjectAsync(AwsBucketName, AwsBucketFileName);  
           using (var reader = new StreamReader(response.ResponseStream))  
           {  
             Debug.WriteLine(await reader.ReadToEndAsync());  
           }  
         }  
         catch (AmazonS3Exception amazonS3Exception)  
         {  
           if (amazonS3Exception.ErrorCode != null &&  
             (amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId") || amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))  
           {  
             throw new SecurityException("Invalid Amazon S3 credentials - data was not uploaded.", amazonS3Exception);  
           }  
           throw new HttpRequestException("Unspecified error attempting to upload data: " + amazonS3Exception.Message, amazonS3Exception);  
         }  
       }  

  • storageKeyId is the access token generated earlier. AWS IAM calls this the Access Key ID.
  • storageSecret is the access token secret generated ealier. AWS IAM calls this the Secret Access Key.
Easy, huh.

The bad news, is that this doesn't work in Xamarin, as the AWS SDK has an incomplete portable class library port.  If you try to use the SDK in a PCL or a Xamarin native project you will get NotImplementedExceptions.


AWS Mobile Recommended Approach

AWS recommend you use their Cognito service to authenticate for Mobile and for Xamarin.  There's a good set of instructions here:
http://docs.aws.amazon.com/mobile/sdkforxamarin/developerguide/setup.html

There's a couple of issues with this however.  At the time of writing the docs are a little out of date and don't match the Cognito console web site, which makes things a little difficult to follow.  (I have sent them feedback on this issue).
Also, more importantly, AWS Cognito is currently only available in the US East (N. Virginia), EU (Ireland), and Asia Pacific (Tokyo) regions. This doesn't work for me as I need to use AWS Sydney.

The Solution

Fortunately, AWS does have some comprehensive instructions how to craft and pass in the right HTTP headers so you can do everything yourself.  Also, its worth noting that Postman supports AWS authentication natively which is pretty handy for testing and to see example requests.

The AWS Roll Your Own Headers Instructions

This approach, I think, has a couple of redeeming attributes: Its lightweight, and no extra dependencies are required.  Its also easy to take the headers and use your favourite REST testing tool.

 using System;  
 using System.Diagnostics;  
 using System.Net.Http;  
 using System.Text;  
 using System.Threading.Tasks;  
 using BAXMobile.Model;  
 using JetBrains.Annotations;  
 using Newtonsoft.Json;  
   
 namespace BAXMobile.Service  
 {  
   public class AmazonS3BaxSummaryDataService : IBaxSummaryDataService  
   {  
     private readonly IHashingAlgorithm hashingAlgorithm;  
     private readonly string accessKey;  
     private readonly string secret;  
     private const string TargetAwsRegion = "ap-southeast-2";  
     private const string TargetAwsService = "s3";  
     private const string Aws4HasingAlgorithm = "AWS4-HMAC-SHA256";  
     private const string Aws4Request = "aws4_request";  
   
     private const string Host = "XXXXXXXXsummary.s3-ap-southeast-2.amazonaws.com";  
     private static string resourcePath = "/MobileDataExport.json";  
   
     public AmazonS3BaxSummaryDataService([NotNull] IHashingAlgorithm hashingAlgorithm, [NotNull] string accessKey, [NotNull] string secret)  
     {  
       if (hashingAlgorithm == null) throw new ArgumentNullException(nameof(hashingAlgorithm));  
       if (accessKey == null) throw new ArgumentNullException(nameof(accessKey));  
       if (secret == null) throw new ArgumentNullException(nameof(secret));  
       this.hashingAlgorithm = hashingAlgorithm;  
       this.accessKey = accessKey;  
       this.secret = secret;  
     }  
   
     public async Task<SummarisedLedgerMobileData> DownloadDataAsync()  
     {  
       string jsonData;  
       // following instructions from http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html  
       using (var client = new HttpClient())  
       using (var request = new HttpRequestMessage(HttpMethod.Get, "https://" + Host + resourcePath))  
       {  
         var dateTime = DateTime.UtcNow.ToString("yyyyMMddTHHmmssZ");  
         var date = DateTime.UtcNow.ToString("yyyyMMdd");  
   
         client.BaseAddress = new Uri("https://" + Host);  
   
         var requestPayloadHash = CalculateHash(string.Empty); // My request, as its a GET, will be empty, so hash an empty string. Yep thats correct.  
   
         var canonicalisedResource = GetCanonicalisedResource(Host, dateTime, requestPayloadHash);  
         var hashedCanonicalisedRequest = CalculateHash(canonicalisedResource);  
   
         var stringToSign =  
           $"{Aws4HasingAlgorithm}\n"  
           + $"{dateTime}\n"  
           + $"{date}/{TargetAwsRegion}/{TargetAwsService}/{Aws4Request}\n"  
           + hashedCanonicalisedRequest;  
   
         var signature = GetSignatureKey(this.secret, date, TargetAwsRegion, TargetAwsService, stringToSign);  
         var authHeaderString = $"{Aws4HasingAlgorithm} Credential={this.accessKey}/{date}/{TargetAwsRegion}/{TargetAwsService}/{Aws4Request}, SignedHeaders=host;x-amz-date, Signature={signature}";  
   
         request.Headers.Host = Host;  
         request.Headers.Add("X-Amz-Date", dateTime);  
         request.Headers.Add("X-Amz-Content-Sha256", requestPayloadHash);  
         request.Headers.TryAddWithoutValidation("Authorization", authHeaderString);  
         // Note that Content-Type cannot be added to GET request using HttpClient - using Content-Type does seem to be common in examples found on the web including GET requests.  
   
         var response = await client.SendAsync(request);  
         jsonData = await response.Content.ReadAsStringAsync();  
       }  
   
       return JsonConvert.DeserializeObject<SummarisedLedgerMobileData>(jsonData);  
     }  
   
     private string CalculateHash(string canonicalisedResource)  
     {  
       return ToHex(hashingAlgorithm.ComputeSha256Hash(Encoding.UTF8.GetBytes(canonicalisedResource)));  
     }  
   
     private string GetCanonicalisedResource(string host, string date, string requestPayloadHash)  
     {  
       var canonicalisedResource = "GET\n" // Http Request Method   
                     + $"{resourcePath}\n" // Url to file  
                     + "\n" // Query string - blank in this case  
                     + $"host:{host}\n"  
                     + $"x-amz-date:{date}\n\n"  
                     + "host;x-amz-date\n" // headers that will be included in the request and signature calc  
                     + requestPayloadHash; // Hashed payload of the request - in my case empty string  
       return canonicalisedResource;  
     }  
   
     private string ToHex(byte[] data)  
     {  
       string hex = BitConverter.ToString(data);  
       return hex.Replace("-", "").ToLower();  
     }  
   
     private byte[] HmacSha256(string data, byte[] key)  
     {  
       return this.hashingAlgorithm.ComputeHmacSha256Hash(data, key);  
     }  
   
     private string GetSignatureKey(string key, string dateStamp, string regionName, string serviceName, string stringToSign)  
     {  
       var secret = Encoding.UTF8.GetBytes(("AWS4" + key).ToCharArray());  
       var date = HmacSha256(dateStamp, secret);  
       var region = HmacSha256(regionName, date);  
       var service = HmacSha256(serviceName, region);  
       var derivedSigningKey = HmacSha256(Aws4Request, service);  
       var signature = ToHex(HmacSha256(stringToSign, derivedSigningKey));  
       return signature;  
     }  
   }  
 }  
   

The AWS Access Key and its Secret are passed into the function.
I also pass in a IHashingAlgorithm. While .NET does have the correct SHA hashing classes these classes are not available in PCL. However they are available in Mono, so using Dependency Injection, you can defer the hashing to the iOS or Android projects.

   //Note: System.Security.Cryptography is not PCL compliant and cannot be used in any PCL. But, it can be used from the Android project because a port of it is available in Mono.  
   public interface IHashingAlgorithm  
   {  
     byte[] ComputeSha256Hash(byte[] data);  
     byte[] ComputeHmacSha256Hash(string data, byte[] key);  
   }  

   public class AndroidHashingAlgorithm : IHashingAlgorithm  
   {  
     private static readonly SHA256 HashingFunction = SHA256.Create();  
   
     public byte[] ComputeSha256Hash(byte[] data)  
     {  
       return HashingFunction.ComputeHash(data);  
     }  
   
     public byte[] ComputeHmacSha256Hash(string data, byte[] key)  
     {  
       var algorithm = "HmacSHA256";  
       var kha = KeyedHashAlgorithm.Create(algorithm);  
       kha.Key = key;  
   
       return kha.ComputeHash(Encoding.UTF8.GetBytes(data));  
     }  
   }  

Hope that helps someone.

Monday, July 25, 2016

Android xbuild: 'OutputPath' property is not set

Background

My team and I are building a Xamarin Forms mobile application that includes iOS and Android projects. I've set up automated CI builds and I've created some custom build configurations for CI and Production (Store) builds. The idea is to use one solution level build configuration that configures both iOS and Android projects appropriately with one build script.

Here's the Ad-hoc (deploy to test devices) Build Config:
(The first project is the PCL which simply the App name with no suffix).

Here's the Production (Store) Build Config:


This all works great in Visual Studio. But not in Bitrise (my chosen online mobile build system).

The Error Message

Here's the command line used to build the Android project extracted from the build output in Bitrise:
xbuild /t:PackageForAndroid /p:Configuration"PlayStore" "./XXXXXXXX/XXXXXXXX.Droid/XXXXXXXX.Droid.csproj" /verbosity:minimal /nologo

Within a few seconds produces this error message.
/Library/Frameworks/Mono.framework/Versions/4.4.0/lib/mono/4.5/Microsoft.Common.targets: error : 'OutputPath' property is not set for this project. Usually this is caused by invalid Configuration/Platform combination. Original values: Configuration: PlayStore Platform: AnyCPU.

There's been reports of this error on the Xamarin Forums (and here) that include other build system like Jenkins.  It looks like there has been bugs in xbuild and Xamarin Studio that have been investigated and fixed.  So its no fault of Bitrise.

The build is given "AppStore" as the target build config, and "RealDevices" as the platform.  The solution file is successfully parsed and based on what we can see from the command line above, it has passed the right target and platform to xbuild (see Store Build Config screenshot above).

The iOS project builds fine.

The diagnostic process on Bitrise

It seems unlikely that the bugs discussed in the forum posts mentioned above are still a problem, the posts are dated some time ago. The suggestions center around App code problems rather than real bugs with Xamarin or Mono.

Bitrise only offers limited options to pass parameters to the Xamarin-Builder step. Basically only build configuration, platform, and solution path.  This is where their generic Script step comes in handy.  Delete the Xamarin-Builder step and replace it with the Script step and use the command line to invoke xbuild directly:

xbuild /t:PackageForAndroid /p:Configuration="PlayStore" /p:Platform="AnyCPU" "./XXXXXX/XXXXXXXX.Droid/XXXXXXXX.Droid.csproj" /verbosity:detailed /nologo /p:OutputPath="bin/PlayStore"

This produced the same results, with nothing new of value.
Running it locally on a Mac also produced the same results.

I then realised that because the build of the Android project is triggered with a build config of "PlayStore" it is passing this value to the its dependency the PCL project.  But its supposed to be built in "Store" config according to the solution file. It doesn't have a config in it's csproj file named "PlayStore" hence the error.

The Solution

xbuild is not building the solution in the same was as msbuild, Visual Studio or Xamarin Studio.  It is using the Android csproj file as its entry point not the solution file.  So the Build Config name provided to build the Android project has to be the same Config name provided to build the PCL.

I added a "PlayStore" Build Config to the PCL and bingo. 
The only downside is that I have two Build Configs to maintain for the PCL: PlayStore and AppStore.

Thursday, July 21, 2016

AndroidDeploymentException: [INSTALL_FAILED_UPDATE_INCOMPATIBLE]

Xamarin.AndroidTools.AndroidDeploymentException: InternalError ---> Mono.AndroidTools.InstallFailedException: Failure [INSTALL_FAILED_UPDATE_INCOMPATIBLE]

I've come across this problem several times with different physical Android devices.  This message is thrown when a real production Google Play App with the same App Id has been installed and then you try to install a debug version.  It will happen with any means of deploying a debug version including with Visual Studio, Xamarin Studio, and the command tools.

I think what is happening is that Android is not happy with a debug version being deployed over the top of a properly signed and deployed version of a Play Store App.  Android stores a flag to block further install attempts for that App Id.  When an App deployment attempt is made, the flag that Android created is not reset.

To fix this, you will need to reinstall the App from Google Play and then uninstall it normally. After that you'll be able to debug your App with Visual Studio.

Tuesday, July 19, 2016

How to create a signed and aligned APK using Visual Studio

This article guides you through creating an APK file for your Xamarin.Android App that is ready to publish on Google Play.

Creating an Android APK that is signed and aligned can mostly be achieved with Visual Studio.  To sign it and align it, command line tools must be used, but the APK can be created with Visual Studio.

You're App does need to be signed to be submitted to Google Play, but it doesn't necessarily need to be "aligned".  It is highly recommended however. Its basically a very cheap optimisation that will minimise the amount of RAM your App will need at runtime.

Step 1 - Prepare your App for Release

This is out of scope for this article. Check out this link:

Step 2 - Build a Release build of your App

Build your Android App in Release Mode.  Ensure the platform drop down is set to "Any CPU".

Step 3 - Export to APK

This does not automatically occur when you build a Release build. Creating the APK must be triggered manually by selecting a context menu item when you right click the Android Project in Solution Explorer.


The APK will be created in your bin\Release\ folder for the Android project.


Step 4 - Sign the APK

Ensure you have the JDK path in your environment Path variable.  My JDK is installed here:
C:\Program Files (x86)\Java\jdk1.7.0_55\bin

The "signed" file you see in the folder has been automatically signed with a Developer certificate.  This isn't suitable for publishing purposes, so you'll need a certificate that identifies you as a developer.

To sign your APK, you'll need your Java KeyStore (JKS) file.  These files usually have a JKS extension or a keystore extension.  If you don't have one you can create one using the instructions in the Google reference above. In summary the command line is: keytool -genkey -v -keystore my-release-key.keystore -alias alias_name -keyalg RSA -keysize 2048 -validity 10000

Once you have a JKS file you need to keep it in a safe place you must use the same file to sign your App everytime. This file uniquely identifies you as a developer.

jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore my-release-key.jks my_application.apk alias_name

There are three things you need to specify in this command:
Replace "my-release-key.jks" with the filename to your JKS file.
Replace "my_application.apk" with your app's APK filename.
Replace the "alias_name" with the alias name you used when creating your JKS file.
You'll also be prompted for the keystore password.

Step 5 - Align the APK

Finally use the Zipalign tool to optimise your APK file.
Zipalign isn't in the same folder as the JDK, for me its in the Android tools folder here:
C:\Program Files (x86)\Android\android-sdk\build-tools\22.0.1
This also should be added to your environment path if it isn't already.

zipalign -v 4 my_application.apk my_application_aligned.apk


That's it.