SOAP UI (Free) (New Rest and Json support)
Send raw SOAP, XML, REST/JSON requests to services.
Soap Sonar (Free version available)
Offers a much better user experience than Soap UI.
PortSwigger (Free)
Send handcrafted HTTP requests + Proxy
Fiddler (Free)
Web Debugging Proxy
OpenSSL
Test supported SSL protocols.
Sunday, September 22, 2013
Tuesday, September 17, 2013
Documenting Technical Design - Ideas Smorgasbord
I get asked a lot what Technical Design is, sometimes what it isn't, and what to include in a technical document. As always the answer is it depends. It depends on your audience, their skill set and level, and the environment.
Here's a few ideas that I have found work. This isn't a comprehensive list that should always be included, again think about who your audience is and would they be interested (relevance) in this section. Consider the analogy of building a car: The person who will use it doesn't care about the tensile strength of the second and fourth piston. They care about how to drive it, maybe maintaining it, and what to do when something abnormal happens. But remember it depends on the audience, the mechanic will want different information, but he still won't necessarily care about the technology used to build the piston.
Technical Design should consider the following topics/sections: (In no particular order).
Tech Design is not:
A See Also Section.
List links to other relevant or related information.
Overview Section.
No more than two paragraphs. This is for business or non-techies who just want to know something about how the system works and how it fits into
the bigger picture with other systems. What is it used for? What problem does it solve?
System Context.
A single diagram showing this system as a single box and the other systems it integrates into.
Main Sequences and Flows.
This is the main or significant work flows thru the system. Use Abstract high level sequence diagrams. Each line should not represent a class but
a major component / library / namespace within the system. Do not get into class and object detail, it will get out of date too quickly.
Hosting and Deployment.
List all different options on how to host in a production environment. Also state the recommended or preferred hosting model. State why there are
multiple models. Use deployment diagrams not text. Try to show one box per deployed application, not one box per library. Draw lines only where
external calls are made between applications. Add text to each line describing http/tcp, authentication, authorisation, json/soap etc.
Could include details of configuration options and environment settings required, if necessary. This is not a installation guide.
How to integrate.
Link to a document (best to keep it separate as it will not be relevant to all readers and could be long) describing how a developer takes the
system as a framework and uses it. How do they install it, use it, and what do they need to do, how long is this getting up and running process
going to take. Obviously only applies if the system is a framework for developers rather than for consumers.
Multi-Tenancy
Is the system multi-tenanted. If so how have you acheived this, trade-offs made, limitations etc. How many tenants can it handle, how few?
Concurrency
Multi-threading? STA? UI Threading? ASP.Net async? Task Factory - how?
Code Organisation.
How is the code organised and why. Possibly code metrics too so another party can get a sense for the size and complexity of the application.
Service Contracts, Endpoints and Data Contracts
Describe all inbound services and their contracts. Describe all outbound service calls, their purpose, when they happen. Also define or link to a
data dictionary that describes all SOAP / Json payloads and fields.
Diagnostic Logging, Monitoring and Auditing
Database tables involved. What logging framework? Links to configure it. Default config (debug/release builds). What is auditing vs logging.
Why is auditing required. Performance counters and definition. Other standard recommended performance counters. How do the Ops team support
the system? Monitor it? Can they tell when it is failing before it fails? Windows Event Log. Health / Support service endpoints that might be available.
(Ping, heartbeat, health check services).
Resilience
Estimated usage stats (year 1,2,3). Estimate db size, requests per second, tenants, 24 hour (or other time period) load patterns.
Where are the fail points from a business perspective? Is 1000ms per request ok? 3000? 10,000ms? 2 hours to process the last item in a queue behind a
large batch?
Exceptions: Different scenarios - (db is down, external 3rd party service is down, one server in the farm is down, entry point service endpoint is down,
validation issues with inbound service data, inflight data cannot be updated back to db after reading it, system crashes during processing - how to
recover? etc) General exception strategy within the app. Where are they logged or not.
Security
What security is applied to all endpoints. User security. Pen testing considerations. OWASP. STRIDE. Known security concerns to address and how they
are addressed.
Database.
Trusted subsystem?
Required infrastructure security.
Deployment time required security config not covered by installer.
Configuration Options for Runtime System.
Are there runtime settings applied immediately without restart? App.config only? Other config? How are server farms updated?
All config options listed and described. Or links to other doc.
Database Schema
Data Dictionary or link to
Describe data access strategy - stored procs, ORM, both, etc
Backwards Compatibility
Are the existing devices out there with a version of the application. Ie are you upgrading backend services and there are existing UI's out there?
Forwards Compatibility: How will future versions be compatible with this?
Are your service contracts / Json transmissions versioned adequately?
Are you recommending to clients that the URL contain a version number?
Other important / architecturally significant items
Queueing / ServiceBus / Asynchronous design
Installation issues - not an install guide
Availability (do you need a draining shutdown, how much down time can you get / need)
Usability
Accessibility
Here's a few ideas that I have found work. This isn't a comprehensive list that should always be included, again think about who your audience is and would they be interested (relevance) in this section. Consider the analogy of building a car: The person who will use it doesn't care about the tensile strength of the second and fourth piston. They care about how to drive it, maybe maintaining it, and what to do when something abnormal happens. But remember it depends on the audience, the mechanic will want different information, but he still won't necessarily care about the technology used to build the piston.
Technical Design should consider the following topics/sections: (In no particular order).
Tech Design is not:
- A complete War And Peace of how every line works. (Tech Design should be as short as possible using as many diagrams as possible to avoid text).
- Sequence Diagrams based exactly on code or Class diagrams based exactly on code. (This will go stale too quickly and creates a maintenance nightmare)
- An install guide.
- Xml Code Comments/SandCastle.
- Any Documentation Generated from code.
- An API Guide.
- A Test Plan.
- How-to-use / user guide (this should be a separate document).
A See Also Section.
List links to other relevant or related information.
Overview Section.
No more than two paragraphs. This is for business or non-techies who just want to know something about how the system works and how it fits into
the bigger picture with other systems. What is it used for? What problem does it solve?
System Context.
A single diagram showing this system as a single box and the other systems it integrates into.
Main Sequences and Flows.
This is the main or significant work flows thru the system. Use Abstract high level sequence diagrams. Each line should not represent a class but
a major component / library / namespace within the system. Do not get into class and object detail, it will get out of date too quickly.
Hosting and Deployment.
List all different options on how to host in a production environment. Also state the recommended or preferred hosting model. State why there are
multiple models. Use deployment diagrams not text. Try to show one box per deployed application, not one box per library. Draw lines only where
external calls are made between applications. Add text to each line describing http/tcp, authentication, authorisation, json/soap etc.
Could include details of configuration options and environment settings required, if necessary. This is not a installation guide.
How to integrate.
Link to a document (best to keep it separate as it will not be relevant to all readers and could be long) describing how a developer takes the
system as a framework and uses it. How do they install it, use it, and what do they need to do, how long is this getting up and running process
going to take. Obviously only applies if the system is a framework for developers rather than for consumers.
Multi-Tenancy
Is the system multi-tenanted. If so how have you acheived this, trade-offs made, limitations etc. How many tenants can it handle, how few?
Concurrency
Multi-threading? STA? UI Threading? ASP.Net async? Task Factory - how?
Code Organisation.
How is the code organised and why. Possibly code metrics too so another party can get a sense for the size and complexity of the application.
Service Contracts, Endpoints and Data Contracts
Describe all inbound services and their contracts. Describe all outbound service calls, their purpose, when they happen. Also define or link to a
data dictionary that describes all SOAP / Json payloads and fields.
Diagnostic Logging, Monitoring and Auditing
Database tables involved. What logging framework? Links to configure it. Default config (debug/release builds). What is auditing vs logging.
Why is auditing required. Performance counters and definition. Other standard recommended performance counters. How do the Ops team support
the system? Monitor it? Can they tell when it is failing before it fails? Windows Event Log. Health / Support service endpoints that might be available.
(Ping, heartbeat, health check services).
Resilience
Estimated usage stats (year 1,2,3). Estimate db size, requests per second, tenants, 24 hour (or other time period) load patterns.
Where are the fail points from a business perspective? Is 1000ms per request ok? 3000? 10,000ms? 2 hours to process the last item in a queue behind a
large batch?
Exceptions: Different scenarios - (db is down, external 3rd party service is down, one server in the farm is down, entry point service endpoint is down,
validation issues with inbound service data, inflight data cannot be updated back to db after reading it, system crashes during processing - how to
recover? etc) General exception strategy within the app. Where are they logged or not.
Security
What security is applied to all endpoints. User security. Pen testing considerations. OWASP. STRIDE. Known security concerns to address and how they
are addressed.
Database.
Trusted subsystem?
Required infrastructure security.
Deployment time required security config not covered by installer.
Configuration Options for Runtime System.
Are there runtime settings applied immediately without restart? App.config only? Other config? How are server farms updated?
All config options listed and described. Or links to other doc.
Database Schema
Data Dictionary or link to
Describe data access strategy - stored procs, ORM, both, etc
Backwards Compatibility
Are the existing devices out there with a version of the application. Ie are you upgrading backend services and there are existing UI's out there?
Forwards Compatibility: How will future versions be compatible with this?
Are your service contracts / Json transmissions versioned adequately?
Are you recommending to clients that the URL contain a version number?
Other important / architecturally significant items
Queueing / ServiceBus / Asynchronous design
Installation issues - not an install guide
Availability (do you need a draining shutdown, how much down time can you get / need)
Usability
Accessibility
Sunday, September 15, 2013
Get out of the way I'm trying to work
A great concise post by Ayende on the Corporate development environments.
Part 1
Part 2
Take aways:
Part 1
Part 2
Take aways:
- Development teams need access to allocated discretionary budgets for the express purpose to remove impediments -
- Lack of tools
- Lack of specialised skills (contractors, coaches, trainers)
- Access to immediate replacement of broken hardware
- Its more damaging for the business in the long run to burn out staff and deliver substandard software on time than to deliver late.
Saturday, September 7, 2013
Web.Config Security Guidelines
http://www.iis.net/configreference/system.webserver/security
http://www.petefreitag.com/item/741.cfm
<configuration>
<system.webServer>
<httpProtocol>
<customHeaders>
<remove name="X-Powered-By"/>
</customHeaders>
</httpProtocol>
<security> <requestFiltering> <!-- block /CFIDE --> <denyUrlSequences> <add sequence="/CFIDE"/> </denyUrlSequences> <!-- block all file extensions except cfm,js,css,html --> <fileExtensions allowUnlisted="false" applyToWebDAV="true"> <add fileExtension=".aspx" allowed="true" /> <add fileExtension=".svc" allowed="true" /> <add fileExtension=".cfm" allowed="true" /> <add fileExtension=".js" allowed="true" /> <add fileExtension=".css" allowed="true" /> <add fileExtension=".html" allowed="true" /> </fileExtensions> <!-- hide configuration dir --> <hiddenSegments applyToWebDAV="true"> <add segment="configuration" /> </hiddenSegments> <!-- limit post size to 10mb, query string to 256 chars, url to 1024 chars --> <requestLimits maxQueryString="256" maxUrl="1024" maxAllowedContentLength="102400000" /> <!-- only allow GET,POST verbs --> <verbs allowUnlisted="false" applyToWebDAV="true"> <add verb="GET" allowed="true" /> <add verb="POST" allowed="true" /> </verbs> </requestFiltering> </security> </system.webServer> </configuration>
Or it could be scoped to a section of the web application.
<location path="Contoso"> <system.webServer> <security> <authentication> <windowsAuthentication enabled="true" /> <basicAuthentication enabled="false" /> <anonymousAuthentication enabled="false" /> </authentication> <access sslFlags="Ssl, SslNegotiateCert, Ssl128" /> <requestFiltering> <fileExtensions> <add fileExtension=".inc" allowed="false" /> </fileExtensions> <denyUrlSequences> <add sequence="_vti_bin" /> <add sequence="_vti_cnf" /> <add sequence="_vti_pvt" /> </denyUrlSequences> </requestFiltering> </security> </system.webServer> </location>
Additionally, also consider:- Applying an Http Module to remove the header "Server" as this cannot be removed by a web.config.
- Applying an Http Module to ensure that ASP.Net does not throw an error on a custom Asp.Net error page, if it does it may disclose sensitive information. It is cleaner to write another Http Module to catch these kind of errors and show a standard error page.
- Also consider prefering later versions of TLS if available. TLS 1.2 will not automatically be used and is configured off by default in windows server 2008. See here.
- Encrypt viewstate.
- Don't ever use session id in the Url.
- Rescope any cookies (always prefer session cookies if possible) to your site only.
- Preferably allow only one user session at a time. If the same user attempts to access the site twice, alert them, or worst case revoke their session.
Friday, September 6, 2013
REST/Web API and WCF
Recently I had an opportunity to ask some questions to Daniel Roth from the Microsoft Asp.Net Web API team directly. I found the answers quite interesting, so I thought I'd share. It almost seems like WCF-Http is falling out of favour: "...all future innovations around REST and HTTP services will be focused on WebApi..."
Daniel: While you can use WCF to develop RESTful services, WCF at its core was intended to be a SOAP stack and this bleeds through many of its abstractions. ASP.NET Web API was intended to be a pure HTTP stack from the its inception. It provides a first-class HTTP programming model (HttpRequestMessage, HttpResponseMessage) that is symmetrically supported on both client and server. It provides first class support for things like content-negotiation, JSON, XML, form URL encoded data, OData, custom formats, HTTP headers and status codes, etc. If your goal is to create front end HTTP services that can reach the broadest set of clients, then ASP.NET Web API is probably your best choice. ASP.NET Web API is also a great platform choice for following the REST architectural style. WCF is still a great choice for backend services where you have complete flexibility over the transport protocol and encoding.
Me: What is the future road map for Wcf and WebApi? Will WCF change into a non-http service platform and WebApi take over in the Http space?
Daniel: The WCF Web HTTP programming model will continue to be supported, but all future innovations around REST and HTTP services will be focused on ASP.NET Web API.
Me: Is it possible to use Transactions and/or Message Security over WebApi? If not what do you recommend?
Daniel: There is no standard way of flowing transaction scopes over HTTP outside of SOAP that I know of. We are seeing more folks move to a RESTful compensation model instead of trying to manage a global transaction. In general flowing transactions across the Web is probably not a good idea. Message Security is typically referred to in the context of SOAP and ASP.NET Web API is not a SOAP stack. You can of course manually encrypt the request or response payloads if you’d like or just use HTTPS.
Me: My current understanding of the WebApi from a security stand point is that it is best suited for simple services that don't have a need for strong security.
Daniel: Web APIs generally do need strong security as they often have broad exposure to a variety of clients including malicious ones. Standard HTTP security mechanisms apply (HTTPS, HTTP auth schemes, OAuth, etc) WebApi is best suited for RESTful Json and Pox in terms of performance and simplicity. JSON is very common for Web APIs, but ASP.NET Web API allows you to easily plug in support for any format that you can think of. There are a lot of MIME types out there!
Me: WCF has adequate support for Json, and would be a safe choice, if the service is intended to be exposed using other protocols as well as Json.
Daniel: Right, there is an argument to be made that WCF makes it really easy to add an HTTP endpoint along with your SOAP endpoints. However, we find that most customers quickly run into limitations with WCF when working with HTTP directly and start to feel like they are swimming upstream. ASP.NET Web API and WCF can cleanly sit side-by-side for these scenarios.
Me: WCF is still the best choice for Tcp, MsMq, and Named Pipes.
Daniel: Yup
Daniel: All standard HTTP security mechanisms apply: HTTPS+client cert, Basic, Forms, Windows, OAuth2, claims based auth, etc.
Me: WCF has better runtime flexibility for tweaking hosting and endpoint configuration.
Daniel: In general ASP.NET Web API has the same level of flexibility as WCF and it is much easier to configure. The main difference is that ASP.NET Web API is HTTP specific and WCF is a transport agnostic SOAP stack.
Daniel: For vanilla web APIs this is true. Most public web APIs rely on some form of public docs and SDKs to deal with this issue (like https://dev.twitter.com/). However, we are building out support for OData which does have a metadata and client code generation story.
Me: WCF4 has support for routing (both transparent and not).
Daniel: WCF has virtually no support for routing (no parameters, no constraints). The route template is a constant base address and then you use UriTemplates to dispatch requests. ASP.NET Web API has full support for ASP.NET Routing.
Daniel: Hope this helps! You can also send future questions to webapitalk.
After this some colleagues of mine on a private architecture forum discussed this at length.
Me: I am a little confused how Web API fits in with WCF. Basically, I believe WCF to be the full-featured choice for flexibility, extensibility and security. Web API seems to be a quick “time-to-market” approach that is simple but offers little in the form of extensibility and security.
Daniel: ASP.NET Web API is a first-class HTTP stack for building HTTP services that target a broad range of clients including browsers and mobile devices. ASP.NET Web API is very extensible and powerful although it is specifically targeted at HTTP scenarios only. ASP.NET Web API makes it very easy to treat HTTP as an application layer protocol, unlike WCF which in general treats HTTP as a transport. ASP.NET Web API leverages standard HTTP mechanism for security. We do plan to publish guidance about when to use WCF vs ASP.NET Web API later this month on MSDN.
Me: There's a lot of ambiguity out there on how to use WebApi, most blogs claim its a bolt-on to your existing MVC site to enable mobile App consumption. It doesn’t sound like good architecture to use a Web site (which is a client), as a service API? What factors have driven Ms to infer that business logic should be tied up in the client tier and not encapsulated in a service?
Daniel: ASP.NET Web API is not tied to a web site client. Web APIs can and generally are implemented as a loosely coupled services tier. Public examples include the Facebook and Twitter APIs. You can even host ASP.NET Web API in your own process if you want to like you can with WCF.
Me: My current thinking is that WCF is my best bet for my current design, but I’m wondering what is there to gain by switch to Web API service end point?
Daniel: While you can use WCF to develop RESTful services, WCF at its core was intended to be a SOAP stack and this bleeds through many of its abstractions. ASP.NET Web API was intended to be a pure HTTP stack from the its inception. It provides a first-class HTTP programming model (HttpRequestMessage, HttpResponseMessage) that is symmetrically supported on both client and server. It provides first class support for things like content-negotiation, JSON, XML, form URL encoded data, OData, custom formats, HTTP headers and status codes, etc. If your goal is to create front end HTTP services that can reach the broadest set of clients, then ASP.NET Web API is probably your best choice. ASP.NET Web API is also a great platform choice for following the REST architectural style. WCF is still a great choice for backend services where you have complete flexibility over the transport protocol and encoding.
Me: What is the future road map for Wcf and WebApi? Will WCF change into a non-http service platform and WebApi take over in the Http space?
Daniel: The WCF Web HTTP programming model will continue to be supported, but all future innovations around REST and HTTP services will be focused on ASP.NET Web API.
Me: Is it possible to use Transactions and/or Message Security over WebApi? If not what do you recommend?
Daniel: There is no standard way of flowing transaction scopes over HTTP outside of SOAP that I know of. We are seeing more folks move to a RESTful compensation model instead of trying to manage a global transaction. In general flowing transactions across the Web is probably not a good idea. Message Security is typically referred to in the context of SOAP and ASP.NET Web API is not a SOAP stack. You can of course manually encrypt the request or response payloads if you’d like or just use HTTPS.
Me: My current understanding of the WebApi from a security stand point is that it is best suited for simple services that don't have a need for strong security.
Daniel: Web APIs generally do need strong security as they often have broad exposure to a variety of clients including malicious ones. Standard HTTP security mechanisms apply (HTTPS, HTTP auth schemes, OAuth, etc) WebApi is best suited for RESTful Json and Pox in terms of performance and simplicity. JSON is very common for Web APIs, but ASP.NET Web API allows you to easily plug in support for any format that you can think of. There are a lot of MIME types out there!
Me: WCF has adequate support for Json, and would be a safe choice, if the service is intended to be exposed using other protocols as well as Json.
Daniel: Right, there is an argument to be made that WCF makes it really easy to add an HTTP endpoint along with your SOAP endpoints. However, we find that most customers quickly run into limitations with WCF when working with HTTP directly and start to feel like they are swimming upstream. ASP.NET Web API and WCF can cleanly sit side-by-side for these scenarios.
Me: WCF is still the best choice for Tcp, MsMq, and Named Pipes.
Daniel: Yup
Me: What forms of authentication does WebApi support?
Daniel: All standard HTTP security mechanisms apply: HTTPS+client cert, Basic, Forms, Windows, OAuth2, claims based auth, etc.
Me: WCF has better runtime flexibility for tweaking hosting and endpoint configuration.
Daniel: In general ASP.NET Web API has the same level of flexibility as WCF and it is much easier to configure. The main difference is that ASP.NET Web API is HTTP specific and WCF is a transport agnostic SOAP stack.
Me: WebApi doesn't have any built in mechanism for defining a message level contract and the exchange of this metadata (WSDL).
Daniel: For vanilla web APIs this is true. Most public web APIs rely on some form of public docs and SDKs to deal with this issue (like https://dev.twitter.com/). However, we are building out support for OData which does have a metadata and client code generation story.
Me: WCF4 has support for routing (both transparent and not).
Daniel: WCF has virtually no support for routing (no parameters, no constraints). The route template is a constant base address and then you use UriTemplates to dispatch requests. ASP.NET Web API has full support for ASP.NET Routing.
Daniel: Hope this helps! You can also send future questions to webapitalk.
After this some colleagues of mine on a private architecture forum discussed this at length.
Juval Lowy: This is a great exchange and it covers a lot, but none of it is new – this is a pretty good statement of where things are, and have been for a long time. Even before WebAPI it was a dead-end to use REST/WCF where you could use WCF, which is why I never covered it in my books (the books were about how to use WCF, not how best to do without it). The bottom line is and has always been: if you can use the full spectrum of WCF with reliability, transactions, security, discovery, encoding and the 20 other features - great. If you can't – choose carefully from a variety of options, WebApi being just one of them.
Anonymous: On your question about the lack of a WSDL... Check out the ApiExplorer.
Anonymous: Is there a connectivity technology more in need of WS-* than Mobile?
Michael "Monty" Montgomery: Ever notice how everyone’s JSON packets are looking more and more like SOAP envelopes?
I’m just sayin’…
Juval: Very much agreed on all counts. Inspired me to get on a soapbox (pun intended) and make a few bigger picture points on what the proper motives should be for using REST to connect applications.
There is some context that should be considered when making general (and true) statements such as “the web is REST.” The web is indeed REST, and if one watches a WCF client talk to a WCF service (using some wire sniffer) over one of the SOAP http bindings, you’ll see REST. The RESTful interactions are simply lower down in the stack. We all know this quite well. So taking this statement: “the web is REST and the web is HTTP“ allow me to refactor it in a more architecturally minded manner:
HTTP is REST.
The question is, do we just adopt a SOAP stack to go over HTTP (for the web in our current conversation) and have the option to use any number of its features which are already there and ready to go? Or do we skip SOAP and just form the RESTful interactions ourselves? So we skip supporting SOAP, it’s too difficult…Then what? Are we not just re-implementing the features of the WS-* standards in a non-standard way (as Monty alluded to)?
That’s not an ideal situation to be in… The only reason to use REST, in my opinion, is only after we’ve admitted the situation is indeed not ideal.
We can’t wait for the vendors to adopt a common set of standards tomorrow, so for today, we’ll do it ourselves over REST. Eventually, all the disparate groups who are trying to implement SOAP’s features in their own manner (without admitting to themselves that’s what they’re actually doing) over REST (because it’s so much “simpler”) will have to admit that all their effort must be shot in the head (like so many VB6 diehards have had to do). Our responsibility is to make smart decisions in how invested we really should get into implementing needed fanciness over REST, while we wait for opportunities to adopt something better (*COUGH* WS-* *cough*).
Me: Hey, many thanks to all. Especially Juval and Monty, I couldn't agree more. With my involvement in Mobile over the last two years coming from a purely .NET to .NET background, mobile is definitely screaming for better standards, like some of what is already in WS-*. I really don't see why it wouldn't be possible to write a fully WS-* Json binding. Imho - Mobile-philes are too caught up in what is fashionable rather than looking at robust solutions. Example: a great deal of mobile apps and their services have shocking levels of security and secret-data management. Favouring instead investigating cool new open source frameworks for UX and others. Famous last words: "It uses SSL so it's secure".
Ok, so my choices are keep it simple in the absence of any real standards support in mobile and go with HTTP/REST/JSON with hand crafted support for message security. Reliable sessions and indempotency would be nice, but I'll have to hand craft these on my own. Or stick with WCF - Http+Soap and the future support of the full WS-* and SOAP (and possible JSON maybe) in Mobile.
Anonymos: REST is something very different [to Soap]. "True" REST (as defined by the guy who coined the term, Roy Fielding) requires you to satisfy a number of architectural constraints (similar to SOA tenets) that include things like:
Being resource and representation focused, not remote procedure call focused
Using the URI for more than just how to get to the service, but to also include information about what resource you are accessing and how you want to manipulate that resource
Leveraging HTTP as more than just a transport protocol, including using Verbs, response codes, headers, content negotiation. This becomes important when you notice little things like GET requests can be cached by web infrastructure, POST cannot, and SOAP is all POST.
Being Hypermedia-driven (which involves embedding URIs in representations that guide the caller through the allowable interactions with the service)
In my experience, there is a very wide range of compliance with these constraints in service people claim to be REST services. Most I would say are doing the first three bullets to some degree. Almost none are doing the last. But if you give any credit to the guy who defined REST, he says it ain't REST without the last bullet:
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
But bottom line, SOAP service calls are not REST in any way shape or form because they do not do any of the above. Now I have in the past been one of the people saying "meh, close enough...", but my opinion on that has changed. We can't expect architects to communicate precisely about SOA concepts and then say it is OK to be loose and sloppy about the use of the term REST.
Anonymous: How come when you transfer money from your bank to another bank there's no distributed transaction? Why don't Expedia wrap a transaction around booking your airline tickets, hotel stay and transfers? Why don't the airlines even use a transaction around just the airline ticketing part? Where's the transaction around payment, inventory etc.. when you make a purchase on Amazon? Is the reason because they don't have WS-*?
Anonymous: I'm not saying never use transactions, just that you should keep the scope small with resources that are close geographically .. distributing them all over is never going to work at scale.
Here's an interesting article : http://blogs.msdn.com/b/clemensv/archive/2011/10/06/achieving-transactional-behavior-with-messaging.aspx
Anonymous: Is there a connectivity technology more in need of WS-* than Mobile?
Michael "Monty" Montgomery: Ever notice how everyone’s JSON packets are looking more and more like SOAP envelopes?
I’m just sayin’…
Juval: Very much agreed on all counts. Inspired me to get on a soapbox (pun intended) and make a few bigger picture points on what the proper motives should be for using REST to connect applications.
There is some context that should be considered when making general (and true) statements such as “the web is REST.” The web is indeed REST, and if one watches a WCF client talk to a WCF service (using some wire sniffer) over one of the SOAP http bindings, you’ll see REST. The RESTful interactions are simply lower down in the stack. We all know this quite well. So taking this statement: “the web is REST and the web is HTTP“ allow me to refactor it in a more architecturally minded manner:
HTTP is REST.
The question is, do we just adopt a SOAP stack to go over HTTP (for the web in our current conversation) and have the option to use any number of its features which are already there and ready to go? Or do we skip SOAP and just form the RESTful interactions ourselves? So we skip supporting SOAP, it’s too difficult…Then what? Are we not just re-implementing the features of the WS-* standards in a non-standard way (as Monty alluded to)?
That’s not an ideal situation to be in… The only reason to use REST, in my opinion, is only after we’ve admitted the situation is indeed not ideal.
We can’t wait for the vendors to adopt a common set of standards tomorrow, so for today, we’ll do it ourselves over REST. Eventually, all the disparate groups who are trying to implement SOAP’s features in their own manner (without admitting to themselves that’s what they’re actually doing) over REST (because it’s so much “simpler”) will have to admit that all their effort must be shot in the head (like so many VB6 diehards have had to do). Our responsibility is to make smart decisions in how invested we really should get into implementing needed fanciness over REST, while we wait for opportunities to adopt something better (*COUGH* WS-* *cough*).
Me: Hey, many thanks to all. Especially Juval and Monty, I couldn't agree more. With my involvement in Mobile over the last two years coming from a purely .NET to .NET background, mobile is definitely screaming for better standards, like some of what is already in WS-*. I really don't see why it wouldn't be possible to write a fully WS-* Json binding. Imho - Mobile-philes are too caught up in what is fashionable rather than looking at robust solutions. Example: a great deal of mobile apps and their services have shocking levels of security and secret-data management. Favouring instead investigating cool new open source frameworks for UX and others. Famous last words: "It uses SSL so it's secure".
Ok, so my choices are keep it simple in the absence of any real standards support in mobile and go with HTTP/REST/JSON with hand crafted support for message security. Reliable sessions and indempotency would be nice, but I'll have to hand craft these on my own. Or stick with WCF - Http+Soap and the future support of the full WS-* and SOAP (and possible JSON maybe) in Mobile.
Anonymos: REST is something very different [to Soap]. "True" REST (as defined by the guy who coined the term, Roy Fielding) requires you to satisfy a number of architectural constraints (similar to SOA tenets) that include things like:
Being resource and representation focused, not remote procedure call focused
Using the URI for more than just how to get to the service, but to also include information about what resource you are accessing and how you want to manipulate that resource
Leveraging HTTP as more than just a transport protocol, including using Verbs, response codes, headers, content negotiation. This becomes important when you notice little things like GET requests can be cached by web infrastructure, POST cannot, and SOAP is all POST.
Being Hypermedia-driven (which involves embedding URIs in representations that guide the caller through the allowable interactions with the service)
In my experience, there is a very wide range of compliance with these constraints in service people claim to be REST services. Most I would say are doing the first three bullets to some degree. Almost none are doing the last. But if you give any credit to the guy who defined REST, he says it ain't REST without the last bullet:
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
But bottom line, SOAP service calls are not REST in any way shape or form because they do not do any of the above. Now I have in the past been one of the people saying "meh, close enough...", but my opinion on that has changed. We can't expect architects to communicate precisely about SOA concepts and then say it is OK to be loose and sloppy about the use of the term REST.
Anonymous: How come when you transfer money from your bank to another bank there's no distributed transaction? Why don't Expedia wrap a transaction around booking your airline tickets, hotel stay and transfers? Why don't the airlines even use a transaction around just the airline ticketing part? Where's the transaction around payment, inventory etc.. when you make a purchase on Amazon? Is the reason because they don't have WS-*?
Anonymous: I'm not saying never use transactions, just that you should keep the scope small with resources that are close geographically .. distributing them all over is never going to work at scale.
Here's an interesting article : http://blogs.msdn.com/b/clemensv/archive/2011/10/06/achieving-transactional-behavior-with-messaging.aspx
The articles were enlightening though! :) Especially that second one by Roy himself. http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
"There are probably other rules that I am forgetting, but the above are the rules [he has 6 bullets] related to the hypertext constraint that are most often violated within so-called REST APIs. Please try to adhere to them or choose some other buzzword for your API."
Anonymous: Designing service API's in a purely RESTful manner does seem a little functional decomp to me.
Get, Post, Delete, Put == CRUD right?
CRUD services don't sound like they are encapsulating behaviour very well and will result in business logic in many different clients.
Its extremely oriented towards resource access, create me one of these... now update it so that this property has this value... ok now delete it. This doesn't appear to be encapsulating the business use case inside the service. When another client wants to use the same functionality it will need to duplicate this orchestration code.
Unless I'm missing something using full blown REST doesn't seem prudent to me?
Anonymous: Its easiest to think about RESTful design in terms of CRUD operations because of the natural mapping of those verbs onto CRUD. And really a major portion of most business services involves CRUD. But it is not limited to that and you don't have to do functional decomposition with REST services any more than you need to with SOAP RPC services. It is more a different way of thinking about things. Instead of thinking about invoking a manager method, you think about it in terms of manipulating the resource that that manager represents. Calling a service operation that kicks off a workflow is ultimately manipulating a resource or a set of resources on the back end. If a set, then you can define a resource that represents that aggregate so that the service still represents a single resource. In aggregate it is either creating new resources, modifying existing resources, removing those resources, or simply retrieving information about resources. There can be any degree of CRUD, business logic, workflows, activities and so on under the covers of a REST service call.
You do have to get into the intentions of HTTP for each verb (the idempotency of each) to figure out how to map them onto service operations. And in reality, most complex business operations map to a POST anyway because they are not idempotent (they cause side effects and changes to the back end and cannot be repeated with the same outcome). But as I mentioned one of the benefits is the fact that for large scale services with lots of consumers and a significant number of retrieval call chains, those can be done with GET and can be far more efficient and scalable due to caching at the web server, router, or CDN level.
So nothing about REST forces you into functional decomp, but I'll admit that it is even easier to slip into that with REST. And really the big benefit to me of REST over SOAP RPC (WCF specifically) for some scenarios is the simplicity for things that don't need to be complex and the ease with which these services can be consumed from dumber clients (Win 8 WinRT, iOS, Android specifically).
Like anything else, REST is not a silver bullet. You have to pick the right technology based on your requirements. And big enterprise services being consumed by .NET clients or other services it is still usually better to use WCF. It is only for those internet facing services where others are going to write the consuming applications and you want to make it as easy as possible for them to be able to consume your functionality or for targeting device consumption that REST becomes attractive to me, and even then I have yet to embrace the hypermedia constraint fully. The only time I really do that is if I am using OData because it handles a lot of that automatically for you. But then you are dealing with pure CRUD, so I use that with caution as well.
Anonymous: I read Roy Fielding's thesis some time ago and my understanding is that REST is an architecture style specifically for the Web (WWW). Not an architectural style for distributed applications such as SOA. The main goal (requirements) of the Web as summarised by Berners-Lee "...was to be a shared information space through which people and machines could communicate" (see section 4.1 p.66 for more details). As Roy states on page 82 "The REST interface is designed to be efficient for large grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction."
Therefore REST is best used for information sharing and not for service interactions (albeit its CRUD nature where REST targets especially the Read part). SOAP is best suited for service interactions. Unfortunately as the WS-* standards are not as ubiquitously implemented and easy as HTTP, developers resorted to HTTP for Service interactions and then mistakenly call their APIs as RESTful. It reminds me of a quote I read somewhere that went something like “Just because you code in C++ it doesn’t mean you are writing an object-oriented application”.
My conclusion is that if we are building a SOA and use HTTP and various Web APIs in place of WCF and SOAP (because of technology limitations etc) we mustn't try to adhere to REST but to SOA. However if that part of the application is for information sharing over the internet then we should adhere to REST.
Balance dogma with pragmatism
The true verification of good service encapsulation is 3 or more clients exposing the same functionality to different devices.
As it turns out the question concerning whether to use WCF or the new Web API depends on the context on how it will be used. If you are looking for a quick development time that doesn't require strict security (although security can be implemented) and ease of defining the data output (JSON, XML, etc.) then the Web API is the route that makes the most sense. A hybrid solution could be created where the API exists in the same middle layer as the WCF.
"There are probably other rules that I am forgetting, but the above are the rules [he has 6 bullets] related to the hypertext constraint that are most often violated within so-called REST APIs. Please try to adhere to them or choose some other buzzword for your API."
Anonymous: Designing service API's in a purely RESTful manner does seem a little functional decomp to me.
Get, Post, Delete, Put == CRUD right?
CRUD services don't sound like they are encapsulating behaviour very well and will result in business logic in many different clients.
Its extremely oriented towards resource access, create me one of these... now update it so that this property has this value... ok now delete it. This doesn't appear to be encapsulating the business use case inside the service. When another client wants to use the same functionality it will need to duplicate this orchestration code.
Unless I'm missing something using full blown REST doesn't seem prudent to me?
Anonymous: Its easiest to think about RESTful design in terms of CRUD operations because of the natural mapping of those verbs onto CRUD. And really a major portion of most business services involves CRUD. But it is not limited to that and you don't have to do functional decomposition with REST services any more than you need to with SOAP RPC services. It is more a different way of thinking about things. Instead of thinking about invoking a manager method, you think about it in terms of manipulating the resource that that manager represents. Calling a service operation that kicks off a workflow is ultimately manipulating a resource or a set of resources on the back end. If a set, then you can define a resource that represents that aggregate so that the service still represents a single resource. In aggregate it is either creating new resources, modifying existing resources, removing those resources, or simply retrieving information about resources. There can be any degree of CRUD, business logic, workflows, activities and so on under the covers of a REST service call.
You do have to get into the intentions of HTTP for each verb (the idempotency of each) to figure out how to map them onto service operations. And in reality, most complex business operations map to a POST anyway because they are not idempotent (they cause side effects and changes to the back end and cannot be repeated with the same outcome). But as I mentioned one of the benefits is the fact that for large scale services with lots of consumers and a significant number of retrieval call chains, those can be done with GET and can be far more efficient and scalable due to caching at the web server, router, or CDN level.
So nothing about REST forces you into functional decomp, but I'll admit that it is even easier to slip into that with REST. And really the big benefit to me of REST over SOAP RPC (WCF specifically) for some scenarios is the simplicity for things that don't need to be complex and the ease with which these services can be consumed from dumber clients (Win 8 WinRT, iOS, Android specifically).
Like anything else, REST is not a silver bullet. You have to pick the right technology based on your requirements. And big enterprise services being consumed by .NET clients or other services it is still usually better to use WCF. It is only for those internet facing services where others are going to write the consuming applications and you want to make it as easy as possible for them to be able to consume your functionality or for targeting device consumption that REST becomes attractive to me, and even then I have yet to embrace the hypermedia constraint fully. The only time I really do that is if I am using OData because it handles a lot of that automatically for you. But then you are dealing with pure CRUD, so I use that with caution as well.
Anonymous: I read Roy Fielding's thesis some time ago and my understanding is that REST is an architecture style specifically for the Web (WWW). Not an architectural style for distributed applications such as SOA. The main goal (requirements) of the Web as summarised by Berners-Lee "...was to be a shared information space through which people and machines could communicate" (see section 4.1 p.66 for more details). As Roy states on page 82 "The REST interface is designed to be efficient for large grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction."
Therefore REST is best used for information sharing and not for service interactions (albeit its CRUD nature where REST targets especially the Read part). SOAP is best suited for service interactions. Unfortunately as the WS-* standards are not as ubiquitously implemented and easy as HTTP, developers resorted to HTTP for Service interactions and then mistakenly call their APIs as RESTful. It reminds me of a quote I read somewhere that went something like “Just because you code in C++ it doesn’t mean you are writing an object-oriented application”.
My conclusion is that if we are building a SOA and use HTTP and various Web APIs in place of WCF and SOAP (because of technology limitations etc) we mustn't try to adhere to REST but to SOA. However if that part of the application is for information sharing over the internet then we should adhere to REST.
Balance dogma with pragmatism
The true verification of good service encapsulation is 3 or more clients exposing the same functionality to different devices.
As it turns out the question concerning whether to use WCF or the new Web API depends on the context on how it will be used. If you are looking for a quick development time that doesn't require strict security (although security can be implemented) and ease of defining the data output (JSON, XML, etc.) then the Web API is the route that makes the most sense. A hybrid solution could be created where the API exists in the same middle layer as the WCF.
Performance Tuning Resources
Old MSDN posts but a lot of this is still useful:
http://support.microsoft.com/kb/821268
- Chapter 17 — Tuning .NET Application Performance
- Chapter 15 — Measuring .NET Application Performance
http://support.microsoft.com/kb/821268
Practical uses for Secure String
Secure Strings (or sstrings for short) seem to be a seldom used class in .NET. There is quite a lot of misunderstanding of what it is used for. The basic idea is to not store passwords in memory in clear text. Its not going to help you transmit passwords over the wire or in a serialised format. The main vulnerability it protects against is someone being able to read memory, or memory dumps. Realistically this is a tiny fringe case, but may have more benefit on a device susceptible to being lost or stolen as opposed to servers.
To use it properly the string must be added to the SecureString object one character at a time. If you grab the password from the user / UI and put it into a string first, you have defeated the purpose and might as well not bother with secure strings. As soon as the string is in memory as a string the GC could make any number of copies of it and it could stick around for some time before the memory is actually overridden.
The secure string object is tagged so the GC does not make copies of it or move it.
See:
http://stackoverflow.com/questions/4502676/c-sharp-compare-two-securestrings-for-equality?lq=1
Also consider:
To use it properly the string must be added to the SecureString object one character at a time. If you grab the password from the user / UI and put it into a string first, you have defeated the purpose and might as well not bother with secure strings. As soon as the string is in memory as a string the GC could make any number of copies of it and it could stick around for some time before the memory is actually overridden.
The secure string object is tagged so the GC does not make copies of it or move it.
See:
http://stackoverflow.com/questions/4502676/c-sharp-compare-two-securestrings-for-equality?lq=1
Also consider:
- Secure Long aka slong
- Secure Int64 aka BigSlong
Using IoC for Runtime Plug-ins
If you are already using an IoC container in your code it can be
easier to leverage it to create a plug-in model. (Instead of using a
plug-in framework like MEF - which is still valid, but you may not want
to introduce another similar paradigm into your code).
You'll need to add the following to the app.config so that plug-in DLL's will be in the probing path.
This is where the plug in DLLs will be saved at runtime. In your composition root the plug-ins assemblies will need to be loaded into the AppDomain so the IoC container can find the plug in types.
public class AssemblyMetadata
{
public string FileName { get; set; }
public string Path { get; set; }
}
public class PlugInAssemblyLoader
{
public void DiscoverAssembliesInPlugInFolder(string path, Predicate<AssemblyMetadata> assemblyFilter)
{
if (!File.Exists(path))
{
return;
}
var assemblies = Directory.GetFiles(path).Where(f => Path.GetExtension(f) == ".dll").Select(f => new AssemblyMetadata {FileName = Path.GetFileName(f), Path = f});
assemblies = assemblies.Where(a => assemblyFilter(a));
foreach (var assemblyFile in assemblies)
{
Assembly.LoadFile(assemblyFile.Path);
}
LoadAssembliesIntoAppDomain(discoveredAssemblies);
}
}
public class CompositionRoot
{
public void MapTypes()
{
// Other type registrations with your IoC container...omitted...
// Load any (if any) plugin libraries into memory
string plugInLibraryLocation = AppDomain.CurrentDomain.BaseDirectory + @"\Plugin"; // Or look up from app.config etc
new PlugInAssemblyLoader().DiscoverAssembliesInPlugInFolder(plugInLibraryLocation, metadata => metadata.FileName.StartsWith("MyCompany"));
// Be sure to trigger loading app.config registrations for your IoC container. This is where your plugins will be registered.
// Plugin types could be replacement for standard types necessary in the application, or they could be optional plugins (ie additional filters e
}
}
You'll need to add the following to the app.config so that plug-in DLL's will be in the probing path.
<runtime> <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1"> <probing privatePath=".\Plugin" /> </assemblyBinding> </runtime>
This is where the plug in DLLs will be saved at runtime. In your composition root the plug-ins assemblies will need to be loaded into the AppDomain so the IoC container can find the plug in types.
public class AssemblyMetadata
{
public string FileName { get; set; }
public string Path { get; set; }
}
public class PlugInAssemblyLoader
{
public void DiscoverAssembliesInPlugInFolder(string path, Predicate<AssemblyMetadata> assemblyFilter)
{
if (!File.Exists(path))
{
return;
}
var assemblies = Directory.GetFiles(path).Where(f => Path.GetExtension(f) == ".dll").Select(f => new AssemblyMetadata {FileName = Path.GetFileName(f), Path = f});
assemblies = assemblies.Where(a => assemblyFilter(a));
foreach (var assemblyFile in assemblies)
{
Assembly.LoadFile(assemblyFile.Path);
}
LoadAssembliesIntoAppDomain(discoveredAssemblies);
}
}
public class CompositionRoot
{
public void MapTypes()
{
// Other type registrations with your IoC container...omitted...
// Load any (if any) plugin libraries into memory
string plugInLibraryLocation = AppDomain.CurrentDomain.BaseDirectory + @"\Plugin"; // Or look up from app.config etc
new PlugInAssemblyLoader().DiscoverAssembliesInPlugInFolder(plugInLibraryLocation, metadata => metadata.FileName.StartsWith("MyCompany"));
// Be sure to trigger loading app.config registrations for your IoC container. This is where your plugins will be registered.
// Plugin types could be replacement for standard types necessary in the application, or they could be optional plugins (ie additional filters e
}
}
Subscribe to:
Posts (Atom)