Bottlenecks and Micro-Performance

After my last blog, I got a response regarding functional programming and performance. This is actually a common theme when people talk about functional programming versus imperative programming. Before we move into the actual performance discussion, I will openly admit, there are often times when functional programming is slower, performance-wise, than imperative programming. I have never claimed otherwise, nor will I begin doing so today.

Now, let’s move away from the very specific case of functional versus imperative programming and take a look at application performance in general. It is common to look for performance bottlenecks in applications. An application slows down and we want to dig in and uncover the performance issue. This particular situation arose at my work just a few weeks ago.

We had a form which, when big enough, slowed the entire experience to a crawl. This was bad news for us as our customers were quite unhappy with the way our application was behaving. I couldn’t blame them. The experience was miserable.

My colleague and I started digging into the offending code. I discovered a few methods which were running at O(n^2) time and he discovered a, seemingly innocuous, call to perform an external validation. When we moved our search to the validation code, it became obvious this was the problem. The entire form was being revalidated multiple times for every single element on the screen.

I fixed the O(n^2) algorithm, reducing it to an O(n) time execution, which made a visible difference, but the real win was decoupling the localized validation logic from the form logic. We estimated that for each validation call that was made, validation work was being done in the neighborhood of 60,000 times.

This demonstrates the difference between micro-performance and macro-performance. My algorithm enhancement was a macro-performance fix when looking at execution times of single lines of code, but when looking at the application as a whole, it was actually just a micro-performance tuning. The real win came when a true macro-performance fix was implemented and our total iteration count was reduced from 60,000 to about 600. That kind of performance gain can be measured in orders of magnitude and saved the experience of our customers.

Jeff Atwood talks about micro-performance benchmarking as something that only matters when a bad choice is made. If a piece of code is not optimally performant, but it is only executed once, does it matter? Jeff and I agree, it doesn’t.

Let’s take a look at two different blocks of code:

Clearly, addEvensImpertaive and addEvensFunctional produce the same output. If we look at the behavior with regard to constants, addEvensImperative loops over the array once so we can say it has a behavior characteristic function something like 1n + c_0. Meanwhile addEvensFunctional actually loops through the entire list twice in a pathological (worst) case. This means we can estimate the characteristic function to look something like 2n + c_1. This means for each time the functional behavior is called, the pathological behavior will be half as fast as the imperative call.

Let’s take a look at this using big-O notation. In big-O notation, the efficiency of the algorithm is reduced to the highest-power term in the approximate function. This means, all constants are discarded as well as coefficients. When we annotate our functions the imperative function performance is O(n) and the functional function performance is O(n) as well.

What this means is both of these functions have a linear growth behavior. The more values we have in our list, the longer each of these take to complete. Typically what we will see is total execution time measured in microseconds or a few milliseconds. Even large arrays of numbers can be iterated over very very quickly, so, even though the functional behavior is half as fast, the overall performance characteristic loss is negligible.

This is the pitfall of micro-optimization. Even though the perceived performance doesn’t change drastically for the user, it is fussed over because it’s easy to see and easy to measure. The problem is, there is a large blind spot that opens up when thinking about managing efficiency.

There is a phrase “what is measured is managed.” When applied to things which are properly and fully measured and everything is measureable, this is valuable. The problem is by measuring micro-optimizations, we can encounter the problem of “not seeing the forest for the trees.” This means we have accurate measurements of parts of a system, but we haven’t actually accounted for the system.

It’s common to talk about early optimization both in positive and negative light (depending on where you stand). I tend to work on the task at hand, and measure when the work is done. This means, if one function is not fully optimized, but the overall performance is as good or better than business requirements dictate, there is no need to optimize further. If, on the other hand, the system is under-performing and slow, I will look for new optimizations.

The important thing to note, however, is I prefer to optimize from the top-down. If a system is slow, it is highly unlikely that the culprit is a single function which is slower than we could be. More likely, we have a hidden optimization problem which is taking an O(n) function and looping, creating an O(n^) behavior, then looping again creating an O(n^3) behavior, and so on.

This hidden exponential growth characteristic is precisely what bit us in our slow-validating application. The reason my O(n^2) to O(n) optimization only gave us a small win is because we had another set of functions which were creating an algorithm performing at O(n^5) or so. Once we converted that behavior to an O(n) algorithm, the app sped up significantly.

In the end, the difference between a function that is performing at 2n versus n is significantly less critical than a system of functions performing at n^5 instead of 5n. Micro-performance benchmarks are important in systems like robotics, neural nets and so on. Common application behaviors like working with small, finite sets of data, typically benefit most from macro-optimizations, so fix those problems first, then measure again. In the end it is far more important to fix the bottleneck and leave the non-micro-optimized code readable and simple.

Testing Notes

Running a test on my local computer there was no measurable difference in performance between the two functions until the list of numbers got to 10,000 values, at which point the difference appears to be less than 1ms (0ms vs 1ms). At 10,000,000 the difference had grown significantly to about 850ms. Please don’t make your users process 10,000,000 numbers in their browser.

Here is my test script:


This demonstrates that even a large number of arithmetic operations, my computer (an older Macbook Pro) can still push out better than 10,000,000 functional operations* a second. Arguably, in any software system processing fewer than 100,000 values at a time, the difference in performance when running each function once would be imperceptible to the end user.

* megaFLOPS so, blah blah, analogy, blah 10 megaFOPS (million functional operations per second)

Refactoring from Imperative to Functional

I was talking with a coworker one day pretty recently and I had a pretty sizeable revelation, people think I write code the way I do from the get-go. It’s actually really hard to look at the final output in a code block and understand how someone might have arrived at that solution. Even when the code is completely obvious, it might not have been obvious how the original problem was solved.

I tend to write in a very terse, functional style. I prefer declarative functions over long, tedious loops and conditionals. This style is not something you are born with and it is definitely not always the first thing to come out. I have been in situations where I KNOW I have the answer in my head, but I just can’t quite spit it out, so I take the long way around, then come back and do some cleanup.

My goal with this post is to expose what this kind of code surgery looks like on a problem which doesn’t seem to be reducible. Along the way I will introduce a variety of helper functions. These functions should not be cut and pasted. Instead they are intended for use in a library or shared module. Most (though not all) can actually be found in my JFP library, which saves the work of creating and refining these small abstractions for yourself. For the purpose of illuminating what is actually happening as I identify work, I will carry all added code forward. This is for your convenience as reference material. At the end, I will break apart the code and show which pieces are actually kept in the core source code.

Let’s start off with a function that takes an array and sums over all numbers which are either a multiple of 3 or 5. This can be done in about 15 lines including whitespace, so it’s a pretty tight little function. This is actually pretty representative of code I see in the wild.

This is pretty straightforward. We take in a list, prepare a variable for the total and one for the index, then we test that the list is actually an array. Finally, we iterate over each element, perform a modulus test and then add any matching values. It’s short, but there is a lot to digest on each line. One of the first things we can do to simplify the code is strip that top conditional out. Let’s create a helper function and sanitize our data instead of wrapping everything in a big conditional.

That’s already a lot easier to digest. Keep in mind, since this is refactoring, some steps show an immediate impact while others will make the code worse before it gets better. In this case, we actually made a nice impact. Now we have a sanitary array we can perform safe operations on each time. In a standard imperative language, we would probably stop here and call it a day. This, however, is Javascript. We can do better!

Let’s pull the modulus check up into helper functions. It is not going to be immediately obvious why we would want to do this bit of work, since it seems to just confuse the matter, but it will make sense all in due time.

Now our checks are wrapped up in functions which will make our lives much, much easier. For now, it ends up being a little more typing. That’s alright, though. When our checks are made with functions, it exposes repeated code we might not have seen before. We are actually making a check against 0 twice. This is something we can abstract away with pure functions. Let’s add a simple compose function and create a new function we can use more freely.

Now we have a nice little function we can pass our value and modulus base into and get a boolean result. This gives us a more declarative way of performing our check. Here’s where we are going to first introduce two functions many people associate with functional programming: map and reduce. The mapping and reduction we are going to do will be so small it’s almost a direct manipulation of single values, but it gives us a way to find new pieces to pull up and out.

Now we are actually performing a map across our modValues array and a reduce on the result. By performing this behavior inline, inside an if condition, it’s not very clear what we are doing or why. This kind of behavior is great for naming and pulling out of our core code. Really, this is a long, confusing way of asking a question: is the value a multiple of 3 or 5?

Let’s stop beating around the bush and just ask the question.

By pulling our check up and out, we can ask if a value is acceptable. Now we are speaking in a more natural language. We want to sum multiples of 3 and 5 so we only want to work with acceptable values. Now we have a new problem, we have to wade through a loop and a conditional structure to actually see what we are looking for. Fortunately, Javascript has a built-in function for handling this kind of operation: filter. Filtering can be read as “I want to take this list of items and only keep the acceptable ones.” Let’s filter our values.

Now we’re really cooking. we’ve eliminated the last conditional block around our data management. We have one ternary left for assigning the values variable, but we’ll deal with that later. For now, let’s get rid of our remaining loop. When we are performing an action like iterating over a list and summing the values, we can convert that to a reduction. Reduce is useful for lots of different purposes, but here we are going to use it at its most foundational. Let’s create an add function and reduce our list, adding all the values together.

Reduce helped us eliminate a few bits of waste from our function. First, we removed the loop structure and just declared the objective of our function directly. Second, we eliminated the need for an accumulator variable and an index variable. This reduces the cognitive footprint of our function since we no longer need to keep track of the variables we are using in our function. Now that we have done all of this work, let’s do a little final cleanup by adding a sanitizeArray function and chaining our filter and reduce operations. This will give us our final code sample.

As you can see, this code is actually significantly longer than our original example. It might make functional programming actually seem like a longer path to walk than simply writing everything inline. This would definitely be true, except that much of this code can, and should, be pulled into a library and abstracted away from our core code. Let’s pull the general-use code out into its own sample block.

Library Code – Should exist in one place

All of these functions are built for reuse. We should not have to rewrite or abstract these out again. This also means, as we come across other code which uses these behaviors, we can immediately perform a drop-in replacement and get a big mess moved out of our way. Now, let’s do a side-by-side comparison of our original function and what our final product look like.

Original Function

Final Refactored Product

This is what we arrive at when we really strip out the repeated code. There’s almost nothing left!

By cleaning up our code, and preferring functional abstractions, we can strip away a tremendous amount of clutter and leave the essence of our intent. We can practically read the last function as a sentence: “To sum multiples of 3 and 5, sanitize provided list, filter for multiples of 3 and 5, then add all values together, starting with 0.” This is how we would solve the problem if we were to talk it through, so we should write our code the same way. This is how I refactor to functional code. How do you do it?

REST Verbs and HTTP

If your journey to programming was anything like mine, you started by looking at HTML forms to understand data transfer across HTTP. There is a lot of talk about REST and what it means on the web. I have heard people talk about the formatting of URLs and the return behavior of documents. I have even heard people discuss REST and its relation to micro-services or hypermedia API behaviors.

I am not going to get into the deep philosophical debate about what properties emerge from REST, nor will I discuss how to break services down or any other oddball topics which diverge from what someone interacting with REST should understand to begin with. Instead, let’s discuss what the REST acronym means, what REST verbs are and how we would interact with them through XMLHttpRequest (XHR).

What REST Is

REST stands for Representational State Transfer. What this means is, REST provides a standardized way to take state (user interaction created data) and transmit it across the Internet. I say “the Internet” because there is no requirement that we interact only with the standard HTTP port, 80. We can, theoretically, make an HTTP request to any port listening for and accepting HTTP requests conforming to the REST protocol.

Blah blah blah, what this really means is using the HTTP standard, if a service accepts requests passed with GET, POST, PUT or DELETE calls, it is probably RESTful. The primary reason I say it is “probably” RESTful is each of those four verbs come with a set of standards which define how the service should behave when a request is made.

Before we go further, let’s create an XHR script so we can look at how we can make browser-side requests to RESTful services. I am offering this script with no explanation, but you are welcome to use it as you see fit. I am also not promising this script is particularly robust, reliable or works at all, so use it for anything other than learning at your own risk.

Now that we have that lined up, let’s have a look at making HTTP requests. There are four common methods we can use for sending HTTP requests, GET, POST, PUT and DELETE. Although there are other request types we’re going to stick the common ones for now.

When working with REST verbs, there are certain rules we must follow. Different server technologies may or may not allow violation of the core REST rules, but when working in a RESTful system, it is best to know the conventional uses and deviate only as much as necessary to match your company or team conventions.

Before we dive into REST requests, I would recommend reviewing my post on URLs for web developers. This post assumes you are comfortable with the way URLs and their structure impact HTTP requests.


GET requests are the most common and best known requests used on the web. GET requests are used all the time to view web sites, get static files and retrieve data from service endpoints. You used a GET request to view this page and all of the links on this page will initiate a new GET request to retrieve a different page.

GET requests can only use a request URL to capture data from a server. Effectively, a GET request will use all parts of the URL, though it is not allowed to use any other means to communicate information about the resource it requests. This means any body information sent to the server will likely be ignored.

Suppose we used our XHR script to make a request, let’s have a look at what the request would look like.

We can see our request uses everything except the fragment, but it passes our data as a query string. This is the kind of behavior we should always expect with a GET request. HTML requests allow bodies to be included, however, if our data were to be included in the body, however, our GET URL would simply be which would display the Google search page instead of the search results we wanted.

When a GET request is sent, the browser constructs an HTTP header which provides all of the information the server needs to resolve the request correctly. Here’s what a GET header looks like.

The new line character (\n) in these header examples is added to demonstrate the number of new lines that are actually part of the HTTP header. We can see the very first element in our request is the type of request, specifically GET. This is the key player in our REST methodologies.

The short version of this is, GET returns a document or resource. It always does. If something fails along the way, our browser will tell us with a different document. Ultimately, GET always gets something, it should never write, update or delete.


DELETE is the close cousin to the standard GET request. DELETE requests, much like GET requests, do not contain a body. In the standard REST behavior, any body in a GET or DELETE request should be ignored. As it turns out, it is entirely likely you could make a DELETE request against your favorite website and the server would respond with the HTML content you get with a GET request.

What makes DELETE special is the way the server responds to the request. If a DELETE request is made against a service which accepts DELETE, it is likely you will receive an error. This is due to the fact that DELETE is typically a privileged (authorization-restricted) command.

Let’s construct a valid DELETE request with our XHR service with a header. For the sake of this discussion, I am assuming the service we are calling is completely insecure and accepts credentials as clear text in the header.

This will send a request with extra header information. I am not promising anything about the way credentials should be sent. Every service will work differently. That said, this new request is still informative. We can see credentials being provided and an id is given to tell our service what to delete.

Our callback also makes an assumption which can be considered common. DELETE requests often do not provide any data in their response body. The response code may actually be 204 No Content. This tells us the request was successful, but there is no content to return, so the response body is empty. Let’s take a look at the request and response headers.

DELETE Request:

DELETE Response:

If we were to make a GET request after deleting record at id 17, we should get the following response:

The HTML may vary, but the header should contain a 404 status code to inform the browser or client application that the record does not exist at that address.


POST requests are primary request type for sending large amounts of data in a request body. Typically in a RESTful service, POST requests are used for creating a record or asset. Our xhr service does not support all the ways information could be sent in a request body. Instead we are set up, primarily, to send JSON blobs.

JSON has become one of the most common ways to transmit information to and from web services. Most languages support JSON natively or they have a well-used, stable library for managing and manipulating JSON data.

Let’s create a new POST request and create some data we can interact with. Much like DELETE, we are going to treat POST as a privileged endpoint and send our credential data.

Our post request will generate a rather different request structure. We will have the request header as we have seen before, but this will also pass a request body, which contains all of the information we added in the data property. Let’s take a look.

This is actually quite similar to the response message we received when we got our 404. The header contains all of the important HTTP request information and the body contains all of the representational state, i.e. user data. Our response message may vary, but, traditionally, a representation of the record id, or some other identifying information is returned with the success response.

The asset URL may or may not exist in the header. If a service provides this kind of header information is is referred to as an affordance. Essentially, an affordance in REST and hypermedia APIs is essentially a URL to retrieve the record the service just created. This makes it convenient for the program to capture the URL and immediately request the asset from the service via GET. The JSON blob in the body contains the record ID. This service is assuming, since all data sent to create the record was assembled by the client, the only information the client doesn’t currently have is the new record id.

We can use the provided ID to create a delete request if the user decides they don’t want to persist that data, or we can use it to update the information when the user changes their input options.


The final REST verb we are taking a look at is PUT. PUT is the update variation on POST. Some older, or less robust services may either not recognize PUT or they may treat PUT in the same way they handle POST requests. In either of these cases, they are not adhering to common modern expectations of REST endpoints.

It’s important to understand the original specification for REST does not necessarily cover or agree with what current development practices enforce. This means, what most places refer to as REST may not match what the originator intended, but understanding what most developers consider REST behaviors today is far more important to effective, new development than adhering to the original documentation.

PUT requests function almost identically to POST requests from the client perspective. We can actually copy our original request, make a couple small modifications and update the information stored on the server with any changes we like. Let’s take a look.

This xhr request would create the following HTTP request. Please note, we did not specify a parentId. We will take a look at how that works out at the service layer in a moment.

Let’s send a get request and see what we get back from the service.

The parentId property was persisted as it was originally defined. This behavior is common for services supporting PUT. Typically, the only properties in a record which get updated are the ones which are defined in the request. This provides security when sending data that, if the entire set of data is not known, a property isn’t deleted or modified accidentally.


This was a rather long post for what ends up being a relatively simple concept. The big take-away from this discussion is the interrelation between the kinds of requests that can be made and how they are interpreted and managed through HTTP and the service layer.

HTTP supports a variety of verbs which reach beyond the list of four we reviewed in this post, but these are the most common. The next time you make an asynchronous request to a server somewhere on the web, think about how that request is being processed and represented by the browser. When data comes back from the your service, take a look at the response message and see what kind of information you can uncover that you might not have known.

Similar posts in Coding, Foundation, Javascript

URLs for Web Developers

At this point it seems like everyone knows about URLs. People can rattle off and, but knowing domain names is not the same as really understanding URLs. URLs, AKA Uniform Resource Locators, actually adhere to a set of standards and behaviors which apply to the way your application interacts with the browser and services which provide data and resources.

First, let’s take a look at the anatomy of a URL and break it down piece by piece regarding how each part directs the browser and informs your application. Once we have a common language to discuss the elements, we can make sense of how the pieces impact a web application.


Commonly, the protocol is http or https, however it doesn’t stop there. Browsers can support a variety of protocols such as ftp, mailto and even gopher. If you opt for an alternate protocol, be aware there is no promise that all browsers will support it. Mailto, for instance, used to function for sending emails since mail servers were wide open, but browsers typically attempt to open a mail client on your computer instead, now. Some browsers may have dropped mailto support altogether. In short, your mileage may vary.

The protocol can be accessed directly via Javascript. The window object contains a location property. Location contains several properties including protocol. Knowing the protocol can be useful, especially when it is not clear whether your site will be communicating via standard http or through a secure channel. The protocol is typically returned with a trailing colon, so you may want to strip the colon. Let’s take a look at two different lines of code for interacting with the protocol.


Although the domain is broken into three parts in our diagram, domains are hard to actually split programmatically. Domain names, including the subdomain, domain name and top-level domain may be as simple as, domains can also look like Clearly co is not the domain name since is the top-level domain for companies in the United Kingdom. This means that quux is the domain name and is the subdomain.

Unfortunately, this means we can’t parse a fully-qualified domain without some sort of prior knowledge. However, if we make certain assumptions, we can safely work backwards through a fully-qualified domain, specifically, if we assume that all domains we are interacting with are .com, .net, .org, .gov or .edu top level domains, we can then parse out the pieces pretty easily.

This knowledge actually lines us up for beginning to understand an HTTP header as it is compiled by the browser for requests to a server. Let’s take a look at what a basic HTTP header would look like using a url like

That header information is a fully-formed acceptable HTTP request. If you were to issue this header through a telnet client you would get the front page of this site. This makes the relation between a URL and the request that is issued fairly transparent.


The port is precisely as it sounds: a number representing the port to communicate through. HTTP typically goes through port 80, however other ports can and have been used for HTTP communication like 8080 and 8000. Express (a Node web service framework) defaults to port 3000. We can specify a different port like the following.

This call will, of course, result in a refused connection. This website uses the default web port and does not respond on 8080 or any other non-standard ports. If, however, this site were to accept requests on port 8080, you could open a connection with a telnet connection in the following way.

This is effectively what the browser does when connecting to a website. Our URL model is actually getting pretty rich and we haven’t introduced any of the other variables. If we wanted to capture the port that was requested programmatically when your browser script loads up, we can do that with the port property on location.


The path in a URL is nearly as familiar as the .com in a big company URL. This path can represent a variety of different resources. It can be a path to a file resource like an HTML file or an image, or it could represent a path to a service resource with a specific data query associated. A good example of a non-physical resource is the page displaying this blog post.

We need to be able to break this down, so let’s take a look at how we can capture the path from the location. We can access the path from the location object in a similar fashion to the port. Window.location has a pathName property which will give you just the path without any other data attached to it.

When a request is sent through the browser with a path, our request header changes a little to manage the enhanced request data. Our original header actually included a path, but it simply pointed at the root directory. Our new header will reflect our correct path.

Query String

The query string is one of the last and most interesting parts of the URL. This is traditionally where information can be stored and passed along to the server. This ties in with REST, which we will discuss in another post. Let’s take a look at a representation of user state in a query string and how to retrieve it with Javascript.

Unfortunately, Javascript doesn’t provide any special facility for parsing or managing query strings, so we have to handle it ourselves. Fortunately parsing out the information is fairly simple. Of course, if we want to change the data, we need to reconstruct the original string with updated values. Let’s add a new key/value pair and change one of our values, then we’ll update and refresh the URL.

Most importantly, when we update the search string, the page is refreshed and a request is sent to the server for any data updates that might need to occur. This means, for standard page views, form data can be sent via query string to a service if an AJAX call is inappropriate.

Much like we did when we did with our previous request updates, let’s have a look at the update to our request header including a query string. You’ll note the change is merely an appended string on the request path. Let’s have a look.


The final part of a url is the fragment, otherwise known as the hash. The hash data is the only part of the URL which does not affect the request sent back to the server. Any fragment information that is included in the URL is only used for the browser to handle the document returned from the server. Originally the fragment information was used to move to the focus of the browser to a specific part of the document.

Now the fragment information has been claimed by single page applications (SPAa) to maintain state and route information. This new use stems from the very same fact that the fragment is in place for browser use alone and does not refresh the page, or send any new information to the server.

The first application I can recall using a fragment was Twitter with a #!/ to denote the SPA route. It seems they were borrowing against the Unix script shebang used to identify the executable to use when interpreting the script. Useless trivia. It won’t be on the test.

Fragments can contain anything including #, and will be made available to any client-side script running in the window. Let’s have a look at a URL containing a hash.

This setup is precisely what we would need to handle routing for a SPA. Without digging into the routing logic, let’s access the fragment and log it.

It might have been the long way around, but this is how URLs and web development tie together. Hopefully this filled in any information gaps you might have wondered about. It’s really important for someone working in web development to understand how requests and URLs are related and what each does for the other.

3 Rules to Improve Your App State Management

As SPAs become more common for web application development, the concept of state is moving almost completely out of the realm of servers and services and into the browser, and rightly so. Ideally services should only maintain state as persistent storage like databases, document stores and file systems. Application state like where the user is and what they are interacting with should be maintained as close to the user as we can keep it.

Systems like Angular have built in state machines as well as third-party solutions, but these solutions are often not enough, on their own, to truly manage all the state your application needs. By relying on just a finite-state machine and persistent storage, your application takes on a chatty characteristic and leaves a lot of data living in limbo before it is actually ready for consumption by the services which persist the data from one application session to the next.

The goal of this post is to fill in some of those gaps and smooth out the application development process. Three main ideas are going to be presented: keep state where you can get it, make state mutation obvious, keep state out of meddling hands. Each of these principles will guide our direction and create something that will serve us well as we work on integrating data transit into our systems both through the client-side application as well as moving data back and forth between our application and our data persistence system.

Keep State Where You Can Get It

The first main idea is, we actually do want our state to be accessible to our application. A common reaction I have seen from many developers I’ve worked with is to simply create a global variable and push data right into an object on the window scope. We will see later, this is a violation, not only of general best practice, but also of one of our three rules.

Nevertheless, we need some place where we can access our data, so let’s create a very simple state cache factory, which will give us the most fundamental piece of the puzzle. We will create an object, if one doesn’t already exist, and hand it out to components which request it. This kind of design keeps state off the global scope, while still making it readily accessible when we need it.

** Note: Our state factory makes use of the either function from my sanitary data post.

So far so good. We can easily reach our state, but it is kept out of the global scope. This makes our state a little more stable, but it also exposes a big problem. We are hand-editing an object. This means, each time we update state, we are actually modifying all state for everyone. What’s worse, it’s really, really difficult to identify where the state is getting modified. We can’t search the document for any particular behavior because anyone could write their function to work in any way they like and potentially blow away all state for any part of the app. Yikes!

Make State Mutation Obvious

Simply handing around an object pointer is anything but obvious. We are all quite familiar with working with object literals, and most SPAs end up littered with them, so our state object will quickly become part of the crowd. Losing track of your state object is a dangerous game that will lead to some very difficult to identify bugs. Ideally your state should have a well-defined interface to interact with so it is easy to identify when state properties are added or existing properties are accessed and modified.

The post about creating a new generic type introduced structs as an object concept. Structs are a good step in the right direction. Each property has its own accessor and mutator, so we get a lot of what we need for free. Let’s update our state factory build method to return a struct and lock down our state interface.

The interface is a little more verbose than directly manipulating an object literal, but the clarity we get when interacting with the state outweighs the pain of an extra couple key strokes. We can now easily find every instance of state access by looking for get.keyName() and every instance of state modification by looking for set.keyName().

With an interface distinct to the struct, our state interactions become obvious. This kind of clarity makes programming much easier since our local data can be anything we like, while we keep our state data clean and obvious. There is still one issue: we still have a lot of access to do terrible things to the data stored in our state. The struct definition comes with a merge function so we could easily do the following:

Now all of the data for three keys has been blown away which could quickly send our app into a tailspin. There is no way to be certain our objects will continue to line up with the state our app expects in other places. This kind of uncertainty makes for a scary programming experience which could bring even the most senior programmer to tears.

Keep State Out of Meddling Hands

There is a saying that goes a little like this, “if one person gets a cold everyone gets a cold.” What this really means is, when everyone is working closely together, if someone introduces something bad into the system it becomes everyone’s problem. There is no place this becomes more obvious than when dealing with state.

When dealing with state access, handing off a pointer becomes a liability. What we really need to do is wrap up our struct with a state-specific interface. Structs as created in the related post, are a mutable data type. This means anyone can go in and change them any way they want. What we really want to do is temper that mutability and define how our program actually interacts with our struct.

The best way to do this is to wrap our struct up in a state object, and expose methods which tightly control how our data flow works. This new construct uses the clone function from JFP v2.4.0 to allow copied data out through a getter, but stops data propagation back up the object tree. This forces programmers working with our state object to explicitly set the new state once they have modified the object copy.

Our new state object also introduces the concept of an initial state. This allows us to bootstrap our application into a particular state by making use of the struct merge behavior. We will want our stateFactory to account for this and only allow state bootstrapping once, throwing an error if someone tries to fully re-bootstrap the app in the middle of the user’s experience. Let’s take a look at our final build behavior.

This final iteration gives us the remaining stability we need to really handle state well. Now when we access state, we get a copy of the data, which may be all we need. If we need to actually update the data, we have exposed methods to merge new data back into the state system. Finally, we can make use of the merge functionality in our structs, which allows us to bootstrap our state with little effort, which takes the user to exactly the place they wanted to be when we pick up from a non-trivial state. Let’s take a look at what our new interactions would look like.

By taking time to build out a stable state system, we shield our application from sudden, unexpected changes in our state. This means each of our modules can act on state data independently, and then feed the updates back into the system. Ideally, we would take care to merge state updates into our local data before we merge our changes back into the core state, but that’s another post for a different day.

As you build your client-side applications, don’t only think about state that your routing system provides, but also consider the true, robust state of your application. Create state data your application can access, make it obvious when that data changes and keep the original state data out of meddling hands. With just a little bit of preparation, your applications will be stable and fun to work on.

Similar posts in Coding, Data Structures, Javascript

Data, Types, Objects and Creating A New Generic Type

Javascript comes with a variety of types, and with ES 2015 and beyond, we are seeing new types emerge like sets and symbols. The primitive data types come with a standard set of comparison rules and default behaviors which make them especially nice to work with, however, complex data like arrays and objects are not so friendly. Objects have no means for comparison (unsurprising) and arrays have no simple, idiomatic way to distinguish them from objects. Even null, which is referred to as a primitive data type, can lie to us. See the example below.

It gets worse if we want to compare any of these. If we compare two distinct arrays which contain the same data in the same order, we will get false every single time. Comparisons only happen on pointer references in Javascript and it makes checking equality really tough. There really isn’t much you can do about all of this except monkey patching data prototypes, which I always discourage for a variety of reasons.

I know I write a lot of posts about functional programming and, typically, it is the way to enlightenment and a land of wonder and elegance. Even with functional programming paradigms, sometimes what you need is not a function, but a data type… An object. The data type Javascript doesn’t have, but is older than (programming) time itself is a struct.

Structs are complex data types of a generic sort, which are used to store data in a predictable way. C and C++ have them, Ruby has them and even Go has them. Javascript has object literals which are similar, however, they lack something that is built into other languages. Let’s look at structs in a couple different languages, Ruby and C.

As I looked at these rather convenient data structures I wonder why we can’t have nice things in Javascript. This is especially apparent when we look at things like someStruct.merge() in Ruby, since we typically need a third party function to accomplish this for objects or arrays. What if we just had a really cool new data type?

This kind of construction is where object oriented programming really shines. Instead of lusting after something other languages have, but Javascript is missing, let’s just create our own data type! First thing we are going to want is a foundation to put our data into. Let’s create a constructor that sets up our struct with the stuff we need.

This really isn’s anything more than an object literal wrapped in a class, but it sets us up for a lot more. The first big problem we encounter with this is, we have a data object backing our struct object, but the way we access our data is kind of strange; myStruct.dataStore[key]. Let’s do a little bit of data hiding and add accessors and mutators so we can define an interface for our struct. By creating an API with a standard naming convention, we make our struct stable and predictable.

If this step feels pretty abstract, that’s because it is. We have wandered into the world of metaprogramming and generic programming. We won’t go into detail on those topics because they are whole realms of computer science unto themselves. Instead, let’s discuss what we added and why.

AttachProperty adds a key initialized to null to our backing object literal, then it takes the pointer to the object literal and creates two object-bound methods: get.keyName and set.keyName. This gives us a clean, obvious API to interact with. Even better than that, we now know exactly which keys are supported by our struct and if someone tries to interact with a property which isn’t defined, they will get a useful error. This is a lot more stable than just allowing someone to come and modify the data directly. Let’s take a look at creating a point with our new struct.

Hey! Our struct is starting to come together. We can create a new structure on the fly, it sets up our object with the right keys, adds appropriate functions and behaves something akin to a data-oriented class. That’s pretty spiffy.

We could, theoretically, stop right here and be correct in saying we have defined our own data type, but there are so many things we are still missing. That setter behavior is fine for something small like a point. It’s actually pretty terse. However, what if we have a bunch of keys and we want to be able to modify them all at once? This is what merge is good for. Let’s define new syntax to handle batch mutation for properties.

MergeKey is little more than an alias for our mutator functions, but it gives us everything we need to keep merge nice and tidy. It also gives us a way to pluck values from an object at run-time and update just a single property in our struct. Merge on the other hand is built exclusively for power. We can hand in an object and merge will lift all of the properties and batch update our struct. This added syntax provides a short, powerful way to handle our struct data at initialization time, and on the fly for big updates.

Now that we’ve gotten this far, we have a fully functioning struct with a couple of conveniences. Rather than stepping through each new function we add, let’s just take a look at a final struct class. Our final struct will have comparison extension and type checking through duck-typing. This post, quite possibly, could be broken into a whole set of posts discussing each of the minute topics within our struct, but I think it might be better to just see the final product and visit some of the deeper ideas at another time.

Leveling Up With Reduce

It was pointed out to me the other day that I suffer from the curse of knowledge. What this basically means is, I know something so I don’t understand what it means to NOT know that thing. This can happen in any aspect of life and it’s common for people, especially software developers, to experience this. Many of us have been working with computers in some way or another for most or all of our lives. This means, when we talk to people who haven’t shared our experiences, we don’t understand their position, i.e. we talk and they don’t have any clue what we are saying.

Within various programming communities this can also happen when more experienced developers talk to developers who are still learning and growing. The experienced developer says something they think is imparting great wisdom and knowledge on the person they are talking with, meanwhile the inexperienced developer is baffled and lost.

Functional programming has become one of these dividing lines in the community. There are people who have dug in deep and have an understanding of the paradigm which they then have trouble conveying to people who haven’t had the same experiences. Ultimately the message falls on deaf ears.

One of the least understood, but, possibly, easiest to comprehend concepts is reduce. We perform reductions every day. We reduce lists of values to sums. We reduce records down to a single selected record based on user preferences or our need. Programming and reduction really go hand in hand.

To come to grips with the kinds of behavior we’re talking about, let’s have a look at some common patterns programmers use in their day to day development. The following block of code contains functions for taking the sum of an array, finding a maximum number and filtering an array of integers. If you have written loops, conditionals and functions before, these will probably be completely unsurprising.

These functions are written in an imperative style, and express every minute detail of the reduction process. We start with some sort of accumulator, whether it’s an array or a number, our variable is meant to capture outcome as we move through our iteration. We iterate over the array, performing some action at each step, then returning the result at the end.

These functions aren’t beautiful, but they are effective and predictable. For many readers, this pattern feels warm and cozy like a winter blanket. The problem we run into is, this methodology is really verbose and bloats the code. It also introduces a lot of noise. Do we really care about the inner workings of the iteration process or do we merely care about the output of our functions?

Let’s take a couple examples from our initial three functions, and rewrite them. It has been said that any recursive algorithm, may be rewritten as an iterative loop. I have no evidence to support the inverse, but I can say, with certainty, that we can rewrite all of these as recursive functions.

Just to catch everyone up, recursion is when a function calls itself internally to perform an iterative operation. We discussed recursion relatively recently in a different post. Essentially what we are going to do is put more focus on what happens in each step of the iteration, and make the iteration process less prominent in our code. Let’s take a look at a recursive strategy for sum and max behaviors.

An interesting point to note is, these functions are actually destructive in nature. We could have written them in a way that is not destructive, however it would have added complexity we don’t need to dig through at the moment. Instead, we can slice the array we are sending in to ensure the original array is not modified by the pop behavior.

Each of these recursive algorithms do something very similar. They highlight a single step in the process, allowing the programmer to focus on the immediate problem of reducing no more than two values at a time. This allows us to actually identify the real behavior we are interested in.

Recursion, of course, leaves us in a position where we have to identify a stopping condition, which was more obvious in the original, imperative, code. Nonetheless, if we choose to halt the process on the occurrence of an empty array, we can just replicate the behavior without needing to put too much extra thought in.

When we review these recursive functions, it becomes apparent the main difference is the accumulation versus comparison behavior. Without too much work, we can strip out this unique behavior and create a generic recursion function which accepts a behavior parameter as part of its argument list. Although this makes our recursion function fairly abstract, and possibly a little harder to read, it reduces the load when we start thinking about what we want to do. The recursion function can disappear as a referentially transparent black box function.

This level of abstraction allows the implementation details of our recursion to be safely separated from the details of our immediate functional need. Functions of this type, which take functions as arguments, are called higher-order functions. Higher order functions are commonly highly-abstract and can lead down a rabbit hole known as generic programming. Let’s not go there today, instead let’s cut to the chase and see our abstraction!

This generic recursion is actually the final step toward our goal, the reduce function. Technically, our generic recursor, given the way it behaves will perform a right-reduction, but that is more than we need to bite off at the moment. We could easily rename genericRecursor to rightReduce and we would truly have a reduction function. The problem we would encounter is, our function is backwards! If we really want to replicate the behavior from our original function we need to make a small modification. Let’s rewrite our genericRecursor as a first, and final hand-build reduce function.

The two key changes we made were renaming and changing from pop to shift. Shift is notoriously slower than pop, so this function is useful for illustration, but it lacks characteristics we would like to see in a production-ready reduce function. Instead, let’s jump from our hand-rolled reduce function to the Javascript native implementation.

Javascript’s native implementation really is a black box function if you are working only from the Javascript side. Implemented in C++, reduce works only on arrays, and has a couple of shortcomings we won’t address here. Nevertheless, the native reduce is key to leveling up your fluent Javascript skills, and is a valuable tool for reducing cognitive load and SLOC bloat. Let’s take a look at a couple of examples of using reduce.

If we return to our original filtering function, we can easily replicate the behavior using reduce. We will also introduce a mapping function. Reduce is so incredibly flexible we can actually accomplish many of the iterative tasks we do every day. The primary pitfall of using reduce for all of the iterative tasks is we will begin to introduce bloat again as we replicate more generic behavior. We won’t dig into the details today. Instead, let’s take a look at some of the power we get from reduce as a tool. It’s kind of the electric drill of the programming world: many uses, all of which save time and energy better spent elsewhere.

This discussion is merely the tip of the iceberg, but it exposes the kind of work which can be done with reduce and the energy we can save by using it more often. For as frequently as complex data types like arrays and objects appear in our code, it only makes sense to work smarter and faster. With the power that comes from first class functions and higher-order functions, we can accomplish large amounts of work with small, but highly declarative behaviors.

As you look at your code, try to spot places where behaviors are repeated and the real focus should be on the data you are working with. Perhaps reduce is an appropriate solution. You might even be able to use it in an interview. I leave you with FizzBuzz performed using reduce.

Comments Off on Leveling Up With Reduce

Callback Streams With Function Decoration

Regardless of whether you prefer raw callbacks or promises, there comes a time where asynchronous behavior pops up in your application. It’s an artifact of working on the web and working with Javascript. This means that, although a function was originally written to solve a particular problem, eventually that function may need to be extended. If we follow the open/closed principle, we should not modify the original function since it almost certainly still solves the original problem for which it was designed. What to do…

Function decoration through composition gives us a powerful way to enhance existing function behavior without modifying the original function. This provides guarantees that our program remains more stable for more use cases and only introduces changes in a surgical, requirements-driven way.

Let’s start off with a core request service. It’s important to note that this is written with certain assumptions being made, i.e. we have specific modules which are already defined and that we only care about the service because it is the foundation for our request construction. This service only does a single thing: it makes a get call to a predefined endpoint with a provided ID. It’s uninteresting, but it helps to illuminate how we are going to construct our function stack.

This is the last piece of object oriented code we are going to look at in this post. We are going to assume from here forward that this service has been instantiated with the correct dependencies injected. Now, let’s create a basic function that uses an instance of our service to make a request. This would look like the following.

So far I know nothing about what the callback does, but that’s okay. This is a simple wrapper to handle our request in some business-layer service. The view-model or controller code will be able to blindly call this service with a request matching the contract.

Technically we have already exposed everything that needs to be known about callback streams, but it’s a little early to end the post, since there isn’t much to be gained here, yet. If all we did was wrap up our request in another function, the goodness isn’t going to be readily obvious to someone who is coming fresh to this concept. Let’s take a look at what a callback stream looks like as an image before we start really digging in.

Callback Decoration Diagram

The important thing to take away from our diagram is no one layer needs to know anything more than what is passed from the layer above. It is unimportant to understand what the layer above does or why. It is, however, very important to know how to respond to the callback that is passed in. This is why contracts become so important in decoration. If, at any point, we break a contract, our stream will break and our application will fail. Fortunately, this adheres to the same requirements as calling any other function, so we are not introducing any greater rule strictness than we had before.

So, back to our business-layer abstraction. Suppose something changed at the data layer and a property name in the JSON that is returned was changed. Although we would like to hope this would never happen, we live in the real world and things are never perfect. Fortunately our abstraction layer allows us to handle this gracefully, rather than having our entire application break because of a database or service change.

Here’s a transformation function.

You’ve probably already noticed our transformation function isn’t tied in with our callback at all. That’s actually a good thing. This function is simple. but if there were complex logic, it would be important to isolate it, and unit test it appropriately. This function does exactly one thing and the declaration is clear. Since callback streams already introduce an abstraction layer, anything we can do at each layer to make the code clear and clean will make debugging easier.

Now, let’s take a look at an approach to handle transformation decoration. We will start off with a simple pattern and expand from there. If Josh Kerievsky taught us anything it’s that we should identify patterns as they appear in the code and refactor to them instead of doing extra, unnecessary work. Let’s write some code.

By making changes this way, we silently introduce changes to fix our application without having to go and touch every place where this call is made. All of a sudden data changes become a much smaller liability to mitigate. We have broken a hard dependency that would be scattered throughout our code by adding an abstraction between our view layer and our data access layer. This is one of the biggest wins the common n-tier architecture provides to us. Let’s take a look at what happens when we have a bunch of changes that happen over time.

The amount of cut and paste I had to do to create all those functions made me die a little inside. This is really smelly code. This is where we can start recognizing patterns and cut out a bunch of duplication. What we really care about is the set of data transformations that need to be managed in our call. The rest of this has become boilerplate. Unnecessary boilerplate in Javascript is bad. Don’t do it. Let’s make a change and fix all this. I like to do this one step at a time. Sometimes things appear as you refactor that might not have been immediately obvious.

That’s a lot better already. Now we don’t have to struggle with a bunch of function duplication and copy/paste hoopla. All we care about is the set of transformations we are going to use on the data. We can practically read off the transformation functions we are using in order. This is actually more descriptive of what we intended to begin with anyway!

Let’s actually do one more refactoring on our code. By eliminating one duplication problem, we introduced another, although less-painful, duplication.

Now we’re cooking with gas! Our getSomeData function can be extended with almost no effort whatsoever now. We can simply create a new transform and then decorate the callback as many times as we need to. This decoration process relies on our original idea: callback streams. Since each layer only cares about adhering to a single contract, and we can wrap the callback as many times as we want, multiple decorations, all behaving asynchronously, can be created as a stream of decorated callbacks without worrying about a failure somewhere in the middle of it all.

The more important item to note is, this could be a single step in a long line of behaviors within a stream. We are adhering to the callback contract in our getSomeData function, so we could, just as easily, use this as an intermediate step between the requesting function and the final request. We really only care about the behavior that happens at the edges of our function, so it really doesn’t matter where this code lives!

This discussion fits in the middle of a couple of different common issues. First, this kind of decoration and function streams behavior directly combats the “pyramids of doom” callback issue many people encounter. The other issue this deals with is exposed promise objects that worm their way through many modern Javascript projects which force us to tightly couple our data access layer to our view code. The abstractions are lost unless a new promise is created and resolved at every level throughout the stack. By thinking about the goal of your code, you take back the power of tiered applications and provide smart, well-isolated functionality which can be enhanced while keeping the rest of your codebase blissfully unaware of the ever-changing data that lives just beyond the edges of your application.

Comments Off on Callback Streams With Function Decoration

Extending Functions with Decoration through Composition

In object oriented design, the decorator pattern is commonly used to extend and enhance classes so they perform some new or more refined functionality. In functional programming it is possible to decorate functions as well. The decoration must follow a few rules, but the result is a very powerful technique to enhance functions statically and at run time.

At the core of functional decoration is function composition. Composition is a straightforward practice that relies on pure functions which take a predictable input and output. A trivial example is something like the following:

This is so foundational to what we know of math and computing there is actually a special notation in mathematics to compose functions this way. It’s common to see this kind of thing annotated as f ∘ g.

I was trying to think of a toy problem that could be used to demonstrate the kind of power we can get from function composition, but then it dawned on me. A great example of real powerful decoration through composition can be demonstrated through a very common problem in statistics: Computing x-bar for a set of data points.

We actually already have almost everything we need to do it. Let’s create a divide function to round out the basic functions we will need to compute x-bar.

That’s it. Let’s do some statistics.

The first thing we are going to need to compute x-bar is a simple mean. This is typically referred to as an average in daily life. We’re all pretty familiar with taking a basic average: take the sum of all values and divide by the number of values. Let’s build the simpleMean function.

SimpleMean is our first big example of function decoration through composition. As it turns out the line gets rather fuzzy when dealing with composition and decoration. The important distinction to make is when a function is decorated, the new function will honor the original function contract. What we can see here is sum takes a list and returns an integer. SimpleMean also takes a list and returns an integer. Most importantly, simpleMean wraps up the behavior of sum with another function to extend the behavior of sum without modifying or duplicating it.

Let’s dig a little deeper into our statistical function and create a new function that normalizes the values in our list using the simpleMean function and map. It’s really important to note that normalize values is a composite function but it is not a decorated function. Although we are using simpleMean to create our new function, the resulting contract does not adhere to the original simpleMean contract.

By creating this new function, we are now ready to start piecing together our final function which will provide x-bar. X-bar is the sum of the squares of normalized values. We have our normalized values, so let’s get to squaring and summing!

The power we are ultimately seeing here is something that comes out of strong composition and an adherence to the contracts of our functions to produce richer and richer behaviors with only one or two lines at a time. The important thing to note while looking at these functions is the extremely limited use of variables. By building functions through composing and decorating functions, state is eliminated and the spaces where bugs and incorrect logic is reduced to the smallest footprint.

As you work on new systems of behaviors, think about what your goal is and break it down into small, easy to understand steps which can be composed to create a powerful, easy-to-understand function which avoids state and provides fast, crisp, high-quality behavior.

Comments Off on Extending Functions with Decoration through Composition

Mainstay Monday: Hiatus

I have spent the last several months writing two blog posts per week. I have really enjoyed writing both the Mainstay Monday and the regular Wednesday posts, but time is a limited resource. I have started a new writing project which will consume a considerable amount of time, so I had to make a choice: I could either continue writing two blog posts a week, or I could pursue my new project.

By cutting back to a single post per week, I free up enough time to pursue my new project while maintaining my blog regularly with high-quality content. Though this means I will publish less frequently, it will serve not only me, but the Javascript community at large in the long-run.

There is no current estimate when Mainstay Monday will resume, but it is not gone forever. I am working on sorting out how I will present foundation material, and whether it makes sense to ever resume the two-post pace. Only time will tell, but in the meanwhile, we’ll meet on Wednesday, just like always!

Comments Off on Mainstay Monday: Hiatus
Similar posts in Mainstay Monday