Accessing an authenticated S3 file in Xamarin

Context

I have a S3 JSON file that I need to access from a Xamarin Forms mobile application.  My S3 resource is not a public resource, its protected and requires credentials to be able to access it.  There is another Windows Desktop application that will upload this file periodically, and the mobile app must only have read only access to the file.

Note: no special authentication is required to access public (unprotected) S3 files.

Setting up S3 and Credentials

The AWS portal offers a free plan to store up to 5GB of storage for 12 months.  After that it'll depend on number of requests and storage size. For me, for a 100Kb file accessed maybe 100 times a month, it'll be approximately 5 cents per month based on the calculator.

To access protected (non-public) data in S3 you'll need to set up users and access tokens.

Users are created with the AWS IAM service.  This is a pretty straight forward process, I needed two users, one with full access, and one with read only access.  To do this you need to attach an appropriate policy to the user, the read only policy is shown here:

Once this policy is attached, user will have read only access to everything stored in the S3 account.  

The next step is to generate an access token. This is done with the Security Credentials tab.

Managing Files in S3

Again this is very simple so not much detail here.  There are a couple of basics to be aware of.  You first choose where your data is stored geographically (ie Sydney, Oregon, Tokyo to name a few) and is not mirrored between regions (but this can be set up).  Data is stored inside "Buckets" - think folder.  Although you can't nest. Files can then be uploaded and accessed inside these buckets.

The red line is obscuring my chosen bucket name (because its top secret), and you can see the file name shown in the list.

Using the AWS S3 SDK

Writing a C# desktop application is pretty easy with the AWS SDK, certainly no challenges.

To be able to access the file created above here's all we need:

     private const string AwsBucketName = "XXXXXXXXXsummary";  
     private const string AwsBucketFileName = "MobileDataExport.json";  
     public async Task UploadDataFileAsync(string data, string storageKeyId, string storageSecret, string region)  
     {  
       var regionIdentifier = RegionEndpoint.GetBySystemName(region);  
       using (var client = new AmazonS3Client(storageKeyId, storageSecret, regionIdentifier))  
       {  
         try  
         {  
           var response = await client.GetObjectAsync(AwsBucketName, AwsBucketFileName);  
           using (var reader = new StreamReader(response.ResponseStream))  
           {  
             Debug.WriteLine(await reader.ReadToEndAsync());  
           }  
         }  
         catch (AmazonS3Exception amazonS3Exception)  
         {  
           if (amazonS3Exception.ErrorCode != null &&  
             (amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId") || amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))  
           {  
             throw new SecurityException("Invalid Amazon S3 credentials - data was not uploaded.", amazonS3Exception);  
           }  
           throw new HttpRequestException("Unspecified error attempting to upload data: " + amazonS3Exception.Message, amazonS3Exception);  
         }  
       }  

• storageKeyId is the access token generated earlier. AWS IAM calls this the Access Key ID.
• storageSecret is the access token secret generated ealier. AWS IAM calls this the Secret Access Key.

Easy, huh.

The bad news, is that this doesn't work in Xamarin, as the AWS SDK has an incomplete portable class library port.  If you try to use the SDK in a PCL or a Xamarin native project you will get NotImplementedExceptions.

AWS Mobile Recommended Approach

AWS recommend you use their Cognito service to authenticate for Mobile and for Xamarin.  There's a good set of instructions here:
http://docs.aws.amazon.com/mobile/sdkforxamarin/developerguide/setup.html

There's a couple of issues with this however.  At the time of writing the docs are a little out of date and don't match the Cognito console web site, which makes things a little difficult to follow.  (I have sent them feedback on this issue).
Also, more importantly, AWS Cognito is currently only available in the US East (N. Virginia), EU (Ireland), and Asia Pacific (Tokyo) regions. This doesn't work for me as I need to use AWS Sydney.

The Solution

Fortunately, AWS does have some comprehensive instructions how to craft and pass in the right HTTP headers so you can do everything yourself.  Also, its worth noting that Postman supports AWS authentication natively which is pretty handy for testing and to see example requests.

The AWS Roll Your Own Headers Instructions

This approach, I think, has a couple of redeeming attributes: Its lightweight, and no extra dependencies are required.  Its also easy to take the headers and use your favourite REST testing tool.

 using System;  
 using System.Diagnostics;  
 using System.Net.Http;  
 using System.Text;  
 using System.Threading.Tasks;  
 using BAXMobile.Model;  
 using JetBrains.Annotations;  
 using Newtonsoft.Json;  
   
 namespace BAXMobile.Service  
 {  
   public class AmazonS3BaxSummaryDataService : IBaxSummaryDataService  
   {  
     private readonly IHashingAlgorithm hashingAlgorithm;  
     private readonly string accessKey;  
     private readonly string secret;  
     private const string TargetAwsRegion = "ap-southeast-2";  
     private const string TargetAwsService = "s3";  
     private const string Aws4HasingAlgorithm = "AWS4-HMAC-SHA256";  
     private const string Aws4Request = "aws4_request";  
   
     private const string Host = "XXXXXXXXsummary.s3-ap-southeast-2.amazonaws.com";  
     private static string resourcePath = "/MobileDataExport.json";  
   
     public AmazonS3BaxSummaryDataService([NotNull] IHashingAlgorithm hashingAlgorithm, [NotNull] string accessKey, [NotNull] string secret)  
     {  
       if (hashingAlgorithm == null) throw new ArgumentNullException(nameof(hashingAlgorithm));  
       if (accessKey == null) throw new ArgumentNullException(nameof(accessKey));  
       if (secret == null) throw new ArgumentNullException(nameof(secret));  
       this.hashingAlgorithm = hashingAlgorithm;  
       this.accessKey = accessKey;  
       this.secret = secret;  
     }  
   
     public async Task<SummarisedLedgerMobileData> DownloadDataAsync()  
     {  
       string jsonData;  
       // following instructions from http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html  
       using (var client = new HttpClient())  
       using (var request = new HttpRequestMessage(HttpMethod.Get, "https://" + Host + resourcePath))  
       {  
         var dateTime = DateTime.UtcNow.ToString("yyyyMMddTHHmmssZ");  
         var date = DateTime.UtcNow.ToString("yyyyMMdd");  
   
         client.BaseAddress = new Uri("https://" + Host);  
   
         var requestPayloadHash = CalculateHash(string.Empty); // My request, as its a GET, will be empty, so hash an empty string. Yep thats correct.  
   
         var canonicalisedResource = GetCanonicalisedResource(Host, dateTime, requestPayloadHash);  
         var hashedCanonicalisedRequest = CalculateHash(canonicalisedResource);  
   
         var stringToSign =  
           $"{Aws4HasingAlgorithm}\n"  
           + $"{dateTime}\n"  
           + $"{date}/{TargetAwsRegion}/{TargetAwsService}/{Aws4Request}\n"  
           + hashedCanonicalisedRequest;  
   
         var signature = GetSignatureKey(this.secret, date, TargetAwsRegion, TargetAwsService, stringToSign);  
         var authHeaderString = $"{Aws4HasingAlgorithm} Credential={this.accessKey}/{date}/{TargetAwsRegion}/{TargetAwsService}/{Aws4Request}, SignedHeaders=host;x-amz-date, Signature={signature}";  
   
         request.Headers.Host = Host;  
         request.Headers.Add("X-Amz-Date", dateTime);  
         request.Headers.Add("X-Amz-Content-Sha256", requestPayloadHash);  
         request.Headers.TryAddWithoutValidation("Authorization", authHeaderString);  
         // Note that Content-Type cannot be added to GET request using HttpClient - using Content-Type does seem to be common in examples found on the web including GET requests.  
   
         var response = await client.SendAsync(request);  
         jsonData = await response.Content.ReadAsStringAsync();  
       }  
   
       return JsonConvert.DeserializeObject<SummarisedLedgerMobileData>(jsonData);  
     }  
   
     private string CalculateHash(string canonicalisedResource)  
     {  
       return ToHex(hashingAlgorithm.ComputeSha256Hash(Encoding.UTF8.GetBytes(canonicalisedResource)));  
     }  
   
     private string GetCanonicalisedResource(string host, string date, string requestPayloadHash)  
     {  
       var canonicalisedResource = "GET\n" // Http Request Method   
                     + $"{resourcePath}\n" // Url to file  
                     + "\n" // Query string - blank in this case  
                     + $"host:{host}\n"  
                     + $"x-amz-date:{date}\n\n"  
                     + "host;x-amz-date\n" // headers that will be included in the request and signature calc  
                     + requestPayloadHash; // Hashed payload of the request - in my case empty string  
       return canonicalisedResource;  
     }  
   
     private string ToHex(byte[] data)  
     {  
       string hex = BitConverter.ToString(data);  
       return hex.Replace("-", "").ToLower();  
     }  
   
     private byte[] HmacSha256(string data, byte[] key)  
     {  
       return this.hashingAlgorithm.ComputeHmacSha256Hash(data, key);  
     }  
   
     private string GetSignatureKey(string key, string dateStamp, string regionName, string serviceName, string stringToSign)  
     {  
       var secret = Encoding.UTF8.GetBytes(("AWS4" + key).ToCharArray());  
       var date = HmacSha256(dateStamp, secret);  
       var region = HmacSha256(regionName, date);  
       var service = HmacSha256(serviceName, region);  
       var derivedSigningKey = HmacSha256(Aws4Request, service);  
       var signature = ToHex(HmacSha256(stringToSign, derivedSigningKey));  
       return signature;  
     }  
   }  
 }  
   

The AWS Access Key and its Secret are passed into the function.
I also pass in a IHashingAlgorithm. While .NET does have the correct SHA hashing classes these classes are not available in PCL. However they are available in Mono, so using Dependency Injection, you can defer the hashing to the iOS or Android projects.

   //Note: System.Security.Cryptography is not PCL compliant and cannot be used in any PCL. But, it can be used from the Android project because a port of it is available in Mono.  
   public interface IHashingAlgorithm  
   {  
     byte[] ComputeSha256Hash(byte[] data);  
     byte[] ComputeHmacSha256Hash(string data, byte[] key);  
   }  

   public class AndroidHashingAlgorithm : IHashingAlgorithm  
   {  
     private static readonly SHA256 HashingFunction = SHA256.Create();  
   
     public byte[] ComputeSha256Hash(byte[] data)  
     {  
       return HashingFunction.ComputeHash(data);  
     }  
   
     public byte[] ComputeHmacSha256Hash(string data, byte[] key)  
     {  
       var algorithm = "HmacSHA256";  
       var kha = KeyedHashAlgorithm.Create(algorithm);  
       kha.Key = key;  
   
       return kha.ComputeHash(Encoding.UTF8.GetBytes(data));  
     }  
   }  

Hope that helps someone.