Learning D3.js Mapping

January 10, 2018 ~ Oscar Villarreal ~ Leave a comment

A few years ago I was lucky enough to co-write my book Learning d3.js Mapping alongside one of my favorite colleagues Thomas Newton. It was an incredible experience filled with many hours of digging deep into a library, finding the best way to articulate a concept, trying to keep readers entertained while at the same time explaining d3.js as best as we could. At the end of it, we had a product that we’re very proud of and that has served thousands of d3.js enthusiasts around the world learn this beautiful library and build some stunning visualizations. I swore I would never do it again though, it was simply too much work and time that was very hard to for me to afford (since writing isn’t my profession and my normal life involves traveling 4 days a week). Yet here I am once again, this time collaborating with Lars Vespol and Thomas Newton for our second edition of Learning d3.js 4 Mapping, with the new d3.v4 version.

We couldn’t have found a better co-author than Lars. He brought in a new approach on d3 via Canvas which is seldom spoken of and the energy we were so desperately looking to obtain. These new topics give our readers the ability to improve the performance of their visualizations by leveraging hardware accelerated graphics via canvas and minimizing DOM impact.

The end result is a very complete learning progression of d3 that we all truly hope you’ll enjoy.

On a theory of consciousness

November 24, 2017 ~ Oscar Villarreal ~ Leave a comment

Found this old draft that I never posted and given my current readings on The Self Illusion it felt like a great time to actually post this and do some more digging on Panpsychism.

This is the second time this week that I’ve stumbled upon the word Panpsychism. If you don’t know the meaning of this word it is basically the idea that consciousness lies within everything. Not just humans, but plants, objects, atoms, quarks etc.
It all depends on information and on that information is organized. The more complicated an information structure (say for instance all the info in a human) the more conscious that thing ends up being.

This amongst many others is a question that Philosophy has been trying to nail down for a long long time. A good video on the subject is here

Out of this video what made me start thinking was the idea of shared consciousness. Are we, you the reader and I the blogger right now, sharing a little bit of the same consciousness just by reading this blog.

In my mind that makes absolute sense. Consciousness can recreate scenarios, surroundings, and context of a given action one has undertaken or will undertake. That context permeates the conscious ether, filling it with life and structure and shapes.
Because your consciousness is right now in the context of my blog, it only makes sense that we up to a point are sharing the same consciousness.

If consciousness is a fundamental law, permeation of it is up to discussion. In my view just like gravity and time are fundamental laws, that doesn’t mean they produce or act in exactly the same way. Time slows down at higher gravitational pull. Gravity is dependant on mass. So can consciousness be dependant on its creator, in this case all beings, whether innate or not. To this idea the concept of consciousness as a product of information structures fits quite well (all this described as a theory in video).

Will it ever be possible to have the subjective be described by the objective ?

The next Angular

January 29, 2015 ~ Oscar Villarreal ~ 1 Comment

I hear this question so much these days. What is the next Angular ?
Or what JS framework would you recommend we use?
Tho which I always answer, there is no silver bullet, there never has been.

Its all about the context in which the application is going to be developed. What good is it to build something in Angular with Node, or Ember with Rails or Backbone with Python, or what ever other stack if the people who are going to implement the code aren’t going to follow standards or simply, don’t know how to tie all the pieces together.

Some things permeate languages and libraries and that is what always should be sought after. A house is judged not by the hammer that built it, but by its architecture and livability.
Small methods, tested code, documentation, good use of memory etc. All these are examples of architecture and style not of tools.

The question we should be asking ourselves is how do we develop this application so we can pluck out pieces of it and put in newer/better pieces without having it fall apart.
It needs a solid base that is loosely coupled and well structured. Clear interfaces and code flow. Single priority methods, good analysis of cyclomatic complexity, multiple device testing automation etc.

In reality you can have your app be small pieces of a lot of things, so long as you can assert it does what its supposed to do and so long as they are all used for the right thing. Focus on having good people that know what they are doing instead of finding silver bullets. Focus on having good test coverage so that when you do pull the pieces out, you know exactly what is broken or what to fix.

In the world of corporate applications, tech is rarely the problem, normally its people and how they are managed or how competent they are at their job.

The beloved spinner

October 28, 2014November 6, 2014 ~ Oscar Villarreal ~ Leave a comment

So many times we confront this issue.

In the world of AJAX where requests take time, we’ve come to develop a handful of incredible solutions such as promises and deferred responses in order to continue to execute JavaScript with out blocking the thread. This is an incredible achievement for true async comes to existence

But what about the user? The user doesn’t give a damn about your miraculous promises. The user wants the page to do as he/she says.. NOW!

A few months ago a colleague of mine was chasing a pretty interesting bug. We were trying to figure out why some database items were getting duplicated or erased every time a user would submit a form. It made no sense for we had a bunch of form submissions that were pristine, yet some were simply corrupted.

We debugged through the AngularJS code and it was pretty straightforward. The form was getting sent to a service and that service, responded with a 200 if it was saved o,r with the corresponding error code if it was not. So what the hell was going on?

Double click was the cause of the problem. Our users were a bit older around the ages of 50-60 and they are used to double clicking everything. Hence when they submitted the form they actually submitted it twice and this cause all kinds of unhandled issues in the back end. Yes in theory there should have been a check to make sure that user has or has not submitted a form and then updated accordingly but, the UI could have been a little bit more understanding of the crowd and helped out more in handling the situation.

A simple swap from button to spinner disables the user from being able to submit twice and informs the user that the system has noted his/her request and is working on it.

Some times its trickier. For instance when we have multiple events that can take place on the same page. In this particular case, a global method that detects whether a spinner is present in the DOM or not and if so alerts the user to hold on does the trick quite effectively and elegantly.

An example of a good interaction is Ladda. Here, as the event happens, the button gets injected a spinner right there.

Another cool spinner solution for all device, resolutions and browser versions is spin.js. You should definitely give it a spin (get it?).

Some word of advice, always think about spinners early in the development cycle, and include them in every action that can potentially have latency. Your users will thank you, and you’ll avoid having to debug very weird issues.

Happy hacking!

Yeoman, Angular and D3 – A solid stack for web apps

November 16, 2013 ~ Oscar Villarreal ~ 7 Comments

Slides from my workshop given at JSChannel in Bangalore India

5 reasons to use AngularJS in the corporate app world

May 7, 2013 ~ Oscar Villarreal ~ 18 Comments

The current situation in the world of corporate apps can be described with one word, chaotic.

Between legacy code that has been around for N years jumping from developer to developer with no documentation. To over complicated interfaces that are over complicating the code. To a mixture of N technologies in one single app. Lets face it, these things were not built with maintainability and support in mind. The problem we face today is, how do we fix all these apps? Where do we start? How do we bring sanity into them? How do we bring them into state of the art tech stacks?

This post represents my opinion of why using AngularJS in corporate applications will not only bring sanity to your apps, but it will allow them to evolve into the future.

1) Start little by little

In an Enterprise you are probably starting with an ugly brownfield project, not a nice green field. Something that has been cooked and baked many times over. Making sense out of it is so hard that developers normally tend to just throw away and rewrite. This of course affects cost and time.

With AngularJS this is not the case. You can start by sprinkling it in some places of your app that you can slowly start to rewrite. Believe me I’ve done this before and it actually WORKS. Not only does it work but it allows you to put the seed of a soon to become amazing application. Given the fact that you can put your ng-app anywhere, you can say for instance – Hey, why don’t we start updating this app with the home screen. Then you get your back end guy to write the web services (yes I am a front end developer), you start writing your tests with Karma Test Runner (thank you Voijta Jina) and in no time you suddenly have a Controller a View and Model for the home section of your app, which kicks in when a the given home route is reached. All of this controlled by AngularJS and living nicely encapsulated in its own little world. My friend you have just done a very very good deed.

2) Fast developer adoption, at just a click away

Technology changes so fast that keeping up to speed has become quite a hard task. You either have the right ways of learning or you suffer buying all kinds of O’reilly books and trying to figure out how this thing works.

One of the cool things that have popped not so recently into the scene are Javascript MVC frameworks. These came to be because the community realized how much Javascript was becoming hell to maintain and make sense out of. Javascript MVC frameworks are a good thing and need to be learned and applied when necessary, such as in our case of web applications.

With AngularJS if you know absolutely nothing about JavaScript MVC frameworks you can basically just hop into the AngularJS site, take the tutorial (which is quite cool by the way) and in 10 lessons or so (About 15-30 mins) you will know exactly how to start breaking out and building up an application using a Model a View and a Controller (MVC) and the best part, having all of it be testable.

This allows for large teams to go online at any point in time, take the tutorial and, because they are learning from the same source, they can ask and feed from each other at their own pace. This is very important specially if you have a team that is working remotely from each other.

3) Productivity and Scalability.

Having the ability to scaffold a project quickly with all the goodies that you need is key to getting started and producing efficiently. I’ve been in projects that take about 1 week just for everyone to jump on and install everything that is needed for the given project. This slows down productivity and it is a clear sign that something is still missing from the recipe.

In order to fix this we can use Yeoman. Yeoman is a workflow comprised of a collection of tools that allow you to quickly start, run and build a project.

Yeoman glues pretty well with AngularJS (makes sense since they are both part of the same family). Yeoman will allow you (via generators) to create an AngularJS repo that includes all the latest Twitter Bootstrap stuff, angular goodies, Karma Test Runner etc. And glue it all together just by initializing it under a directory with any name (preferably something related to your app). It will use that name to namespace the application and with that, it will create a sample view tied to a sample controller, tied to a sample test (thank you Brian Ford).

This structure not only allows you to quickly assemble something up, but it lays the roots for something way bigger to evolve. On top of that, once you are ready to deliver, you just run a grunt command and your files get compressed and combined (use ngmin) and put into a separate distribution folder. How sweet is that!

Aside from everything because you are building a single page application, everything should be RESTFUL which means that this app can and will interact with ANY back end code as long as it gives the front end the JSON it needs. You are decoupling your code and logic making it easier to scale and evolve.

4) Tests tests tests

Situation: There are thousands maybe hundreds of thousands of lines of code in a project. You are dropped smack in the middle of it. There is no documentation and there is a new feature that needs to get implemented ASAP. You start fiddling around with the code, next thing you know something else where breaks and you have no clue why… now you have to spend hours maybe days navigating through spaghetti code to see how your change broke something else.. bad, very very bad.

One of the most important things for maintaining a large code base throughout time is the usage of tests. By using Karma Test Runner given to you by the AngularJS generators in Yeoman, you can have your unit tests running all the time in the background, that way if you change anything and hit save, your tests will automatically re-run and you will know if you are breaking something, where you are breaking it, and if the tests are built correctly maybe even why you are breaking it.

This is super useful because normally this is how developers test insert a new piece of code. They add in 10-15 lines, they save then whoops, something breaks. Then they comment out 8 lines of code or so. Save, rerun still breaks? No? Ok so the error must be in these 2-5 lines.

Way way simpler to do this than to have to navigate through N files of hidden dependencies and figuring out what the previous developer was thinking when they were building this out.

5) Good community support

So many times you find yourself stuck with a piece of code and have absolutely no idea why it is not working. The community has never failed me so far when it comes to helping me out with a problem. I ask a question in Stackoverflow or a Google group and within a few hours or maybe even minutes, I get the answer I was needing, sometimes even from the core AngularJS team its self! This not only helps you out but by solving your issue, others might solve their issues as well. This speeds up the learning curves and the detection of any glitches that the tool might have. It basically feeds on its self and continues to evolve and grow.

Conclusion: The usage of a very opinionated Javascript MVC framework such as AngularJS is an excellent way to start cleaning up legacy code for it allows you to start in small pieces and bring sanity piece by piece. If your tech team is not on the bleeding edge they can take the AngularJS tutorial online at their own pace and learn how to use it. Once they are done and ready to produce they can incorporate Yeoman and kick start this upgrade within a few minutes, using AngularJS, Twitter Bootstrap, Karma Test runner and many other goodies that Yeoman scaffolds out. As the developer starts building they can use Test Driven Development to create unit tests and scenario tests for everything they build out in AngularJS. This will guarantee that in the future if anyone new jumps on the project and needs to make modifications, they will be able to do so more easily and with more awareness of what their lines of code are affecting.

AngularJS is not the final bullet proof solution, but it is helping pave the way for cooler things that are coming in newer versions of Javascript such as object.Observe, better controlled scopes etc. These will in theory (and I do believe so my self) bring order in a more natural way into all applications using Javascript.

D3 Waterfall chart

May 3, 2013May 30, 2013 ~ Oscar Villarreal ~ 6 Comments

When it comes to waterfall charts and d3 you have very few options.
1) Your are on your own, which means you are free to draw it how ever you want (which is pretty much the coolness of d3)
2) You use this repo https://github.com/sattybhens/waterfall which will give you a horizontal chart. Has a mix of ruby and coffescript
3) You use this plugin that resembles to what most jQuery users or Highcharts users are used to which is a config object that generates a chart. https://github.com/climboid/d3Waterfall
Currently it only supports vertical waterfall charts. Feel free to use and contribute.

Cheers

Automated d3 map of any country

February 19, 2013May 30, 2013 ~ Oscar Villarreal ~ 6 Comments

So I previously wrote a post on how to create a map of anywhere of the world using d3. I’ve decided to take that a step further and created a python script that automates the entire thing.
Follow the instructions in this github repo and you should be good.
Note: You will need to install GDAL and have Python running.

Enjoy!

CSS Challenge

February 19, 2013February 19, 2013 ~ Oscar Villarreal ~ 3 Comments

A good friend of mine, Laurent Brucher was tasked to interview a candidate with regards to his front end skills. We talked for a while, shared ideas and in the end he came up with a clever challenge for the candidate to code out.

Inspired in his challenge I’ve crated this one below

Create a JS fiddle with your solution. Use only HTML and CSS
Whom ever uses less classes and less attributes will be declared the winner.

Create any map of the world in SVG

July 12, 2012November 10, 2012 ~ Oscar Villarreal ~ 34 Comments

There is nothing cooler than having a map fully powered by data. D3.js does a great job at this by accepting geoJSON data and converting that into SVG. The thing is how the hell do we acquire a map that is not the one of the USA (which is the most typical example). There are many tools out there that do svg maps and they are quite good at it, but this tutorial is going to focus on D3.js, .shp files and geoJSON.

Two of my favorite resources to obtain maps are Natural Earth Data and Diva. From what I’ve seen Natural Earth Data does a good job for country layouts (just the outer border) and Diva for details within a country (regions, lakes, rivers, train tracks etc). For this example we are going to use Natural Earth Data because it already provides us with a map of the world.

Step One

Go to Natural Earth Data
Select Downloads
We want a good detailed map so select from Large scale data, 1:10m the cultural option
Focus on Admin 0-Countries from the list and click Download Countries (this will give you a full map of the world with all of the countries in it)

Step Two

Install QGIS

If you are using a mac make sure to install all of the dependencies first ( GDAL Complete and GSL )

Step Three

Manipulate map with QGIS

Extract the zip file downloaded from Natural Earth Data. Open the .shp file with QGIS. You should see a map of the world like so

After opening on the left hand side make sure that you have enabled the layers panel, right click on the layer you are viewing and select Toggle Editing. You should see something like this:

If you are unable to see the layers panel bar you can show it by clicking on View > Panels >Layers. That should enable the panel.

Now with our map ready to be edited go to the move feature located in the tool bar, select it by clicking it and then point at the country you want to use. For this example we are going to focus on Madagascar

Click on Madagascar and move it away from the rest of the world. Say some where up North near Greenland or something. This is just to create enough space in order to remove the rest of the world’s vectors from our map.

Once moved you should end up with something like this:

Next we will remove the rest of the world by selecting the “Select Features by Rectangle” option . If you have a small screen like I do right now it might be hidden behind the “Identify Features” icon . Select the chevron and that will show you the “Select Features by Rectangle” option.

With the “Select Features by Rectangle” option selected we proceed to select the rest of the world by clicking, dragging and surrounding the world EXCLUDING MADAGASCAR.

Your world aside from Madagascar should have turned into some kind of highlighted color (varies every time). Click the Delete option and remove all of the vertices that are selected.

Use the “Pan Map” option to center Madagascar right in the middle. Next zoom in by scrolling up/down in order to see better our final Madagascar.shp file.

You should end up with something like so

This map is now free from all the crap that it doesn’t need. Still it might be a little bit to heavy if say for instance you want all of Australia. For scenarios where we are using to many vertices there is a tool called <a href=”http://mapshaper.com/test/MapShaper.swf”>Map Shaper</a> that allows you to upload a .shp file and minimize the number of vertices that it contains as well as export back into .shp format. This will come in handy so keep it in mind! =)

Next we proceed to save our .shp file into geoJSON format and we do so by right clicking on the Layers panel the only layer we have and choosing save as

This will pop up a window. Choose GeoJson from the Format drop down like so

And voila! we now have Madagascar in geoJSON format.

Next we will use the power of d3.js to convert our geoJSON data into an SVG map.

Step Four

If you don’t have d3.js head over to its site and download the minified or full versions. We are going to need also jquery in order to do some DOM manipulation; so add that into the equation as well.

Lately I’ve been doing a lot of coffee script and thanks to my good friends Alex and Thomas I am now an auto proclaimed coffee script evangelist, which is why our d3 code will be written in coffee script. Both Alex and Thomas have some really good posts on maps and d3. Make sure to check out this post from Thomas Positioning and Scaling maps in d3 and this one from Alex Interactive Maps with d3 and ember. You will find a lot of valuable cool stuff in them!

We will need a base index.html file to gather everything together so create one like so

<!doctype html>
<head>
	<meta charset="utf-8">
</head>
<body>
	<div id="vis"></div>
	<script src="jquery.js"></script>
	<script src="d3.js"></script>
	<script src="script.js"></script>
</body>
</html>

We now proceed to fill out script.js with our Coffee script code as follows:

xy = d3.geo.mercator().scale(8500).translate([0,-1200])
path = d3.geo.path().projection(xy)

vis = d3.select("#vis").append("svg:svg").attr("width", 960).attr("height", 600)

d3.json "madagascar.json", (json) ->
 vis.append("svg:g").attr("class", "tracts")
 .selectAll("path").data(json.features)
 .enter().append("svg:path").attr("d", path)
 .attr("fill-opacity", 0.5)
 .attr("fill", "#85C3C0")
 .attr("stroke", "#222")

or the javascript version if you prefer

var path, vis, xy;

xy = d3.geo.mercator().scale(8500).translate([0, -1200]);

path = d3.geo.path().projection(xy);

vis = d3.select("#vis").append("svg:svg").attr("width", 960).attr("height", 600);

d3.json("madagascar.json", function(json) {
  return vis.append("svg:g").attr("class", "tracts").selectAll("path").data(json.features).enter().append("svg:path").attr("d", path).attr("fill-opacity", 0.5).attr("fill", "#85C3C0").attr("stroke", "#222");
});

There are a few interesting parts in this code. Notice the projection (mercator) the scale(8500) and the translate([0,-1200]). To be 100% honest I’m not totally sure of what these values do exactly but I know that tweaking the scale and the translation accordingly will enlarge (scale) and center (translate) your svg object within its dimensions (960 width and 600 height). The projection mercator is in reference to the entire world, so Madagascar is located not so far from the center. You will most likely end up doing trial and error until you get the right scale of the country you want. I recommend starting with a small scale, say 200 and a translation of [0,0] so that you start from the top left corner and they progress your way.For more details on this please refer to d3.js

After compiling this coffee script code into JavaScript code you should end up with something like this

That’s all for this post and if you have any cool idea of how the projection, translation and scaling exactly works or if there is an algorithm to it, so that we can know what values to put in there instantly please drop a comment and let me know and I will gladly add it in your name!

Good luck with your hacks!

Some useful links

Big Thinking – strange maps