Create any map of the world in SVG

There is nothing cooler than having a map fully powered by data. D3.js does a great job at this by accepting geoJSON data and converting that into SVG. The thing is how the hell do we acquire a map that is not the one of the USA (which is the most typical example). There are many tools out there that do svg maps and they are quite good at it, but this tutorial is going to focus on D3.js, .shp files and geoJSON.

Two of my favorite resources to obtain maps are Natural Earth Data and Diva. From what I’ve seen Natural Earth Data does a good job for country layouts (just the outer border) and Diva for details within a country (regions, lakes, rivers, train tracks etc). For this example we are going to use Natural Earth Data because it already provides us with a map of the world.

Step One

  • Go to Natural Earth Data
  • Select Downloads
  • We want a good detailed map so select from Large scale data, 1:10m the cultural option
  • Focus on Admin 0-Countries from the list and click Download Countries (this will give you a full map of the world with all of the countries in it)

Step Two

Install QGIS

If you are using a mac make sure to install all of the dependencies first ( GDAL Complete and GSL )

Step Three

Manipulate map with QGIS

Extract the zip file downloaded from Natural Earth Data. Open the .shp file with QGIS. You should see a map of the world like so

After opening on the left hand side make sure that you have enabled the layers panel, right click on the layer you are viewing and select Toggle Editing. You should see something like this:

If you are unable to see the layers panel bar you can show it by clicking on View > Panels >Layers. That should enable the panel.

Now with our map ready to be edited go to the move feature  located in the tool bar, select it by clicking it and then point at the country you want to use. For this example we are going to focus on Madagascar

Click on Madagascar and move it away from the rest of the world. Say some where up North near Greenland or something. This is just to create enough space in order to remove the rest of the world’s vectors from our map.

Once moved you should end up with something like this:

Next we will remove the rest of the world by selecting the “Select Features by Rectangle” option . If you have a small screen like I do right now it might be hidden behind the “Identify Features” icon . Select the chevron and that will show you the “Select Features by Rectangle” option.

With the “Select Features by Rectangle” option selected we proceed to select the rest of the world by clicking, dragging and surrounding the world EXCLUDING MADAGASCAR.

Your world aside from Madagascar should have turned into some kind of highlighted color (varies every time). Click the Delete option  and remove all of the vertices that are selected.

Use the “Pan Map” option  to center Madagascar right in the middle. Next zoom in by scrolling up/down in order to see better our final Madagascar.shp file.

You should end up with something like so

This map is now free from all the crap that it doesn’t need. Still it might be a little bit to heavy if say for instance you want all of Australia. For scenarios where we are using to many vertices there is a tool called <a href=”http://mapshaper.com/test/MapShaper.swf”>Map Shaper</a> that allows you to upload a .shp file and minimize the number of vertices that it contains as well as export back into .shp format. This will come in handy so keep it in mind! =)

Next we proceed to save our .shp file into geoJSON format and we do so by right clicking on the Layers panel the only layer we have and choosing save as

This will pop up a window. Choose GeoJson from the Format drop down like so

And voila! we now have Madagascar in geoJSON format.

Next we will use the power of d3.js to convert our geoJSON data into an SVG map.

Step Four

If you don’t have d3.js head over to its site and download the minified or full versions. We are going to need also jquery in order to do some DOM manipulation; so add that into the equation as well.

Lately I’ve been doing a lot of coffee script and thanks to my good friends Alex and Thomas I am now an auto proclaimed coffee script evangelist, which is why our d3 code will be written in coffee script. Both Alex and Thomas have some really good posts on maps and d3. Make sure to check out this post from Thomas Positioning and Scaling maps in d3 and this one from Alex Interactive Maps with d3 and ember. You will find a lot of valuable cool stuff in them!

We will need a base index.html file to gather everything together so create one like so

<!doctype html>
<head>
	<meta charset="utf-8">
</head>
<body>
	<div id="vis"></div>
	<script src="jquery.js"></script>
	<script src="d3.js"></script>
	<script src="script.js"></script>
</body>
</html>

We now proceed to fill out script.js with our Coffee script code as follows:

xy = d3.geo.mercator().scale(8500).translate([0,-1200])
path = d3.geo.path().projection(xy)

vis = d3.select("#vis").append("svg:svg").attr("width", 960).attr("height", 600)

d3.json "madagascar.json", (json) ->
 vis.append("svg:g").attr("class", "tracts")
 .selectAll("path").data(json.features)
 .enter().append("svg:path").attr("d", path)
 .attr("fill-opacity", 0.5)
 .attr("fill", "#85C3C0")
 .attr("stroke", "#222")

or the javascript version if you prefer

var path, vis, xy;

xy = d3.geo.mercator().scale(8500).translate([0, -1200]);

path = d3.geo.path().projection(xy);

vis = d3.select("#vis").append("svg:svg").attr("width", 960).attr("height", 600);

d3.json("madagascar.json", function(json) {
  return vis.append("svg:g").attr("class", "tracts").selectAll("path").data(json.features).enter().append("svg:path").attr("d", path).attr("fill-opacity", 0.5).attr("fill", "#85C3C0").attr("stroke", "#222");
});

There are a few interesting parts in this code. Notice the projection (mercator) the scale(8500) and the translate([0,-1200]). To be 100% honest I’m not totally sure of what these values do exactly but I know that tweaking the scale and the translation accordingly will enlarge (scale) and center (translate) your svg object within its dimensions (960 width and 600 height). The projection mercator is in reference to the entire world, so Madagascar is located not so far from the center. You will most likely end up doing trial and error until you get the right scale of the country you want. I recommend starting with a small scale, say 200 and a translation of [0,0] so that you start from the top left corner and they progress your way.For more details on this please refer to d3.js

After compiling this coffee script code into JavaScript code you should end up with something like this

That’s all for this post and if you have any cool idea of how the projection, translation and scaling exactly works or if there is an algorithm to it, so that we can know what values to put in there instantly please drop a comment and let me know and I will gladly add it in your name!

Good luck with your hacks!

Some useful links

Big Thinking – strange maps

Forget console.log

I remember the day I learned that I could use console.log to test my app instead of having to alert everything. Man did I feel good and smart. Everything was very smooth, and fast. No longer did I have to close every single alert in order to get to the next one, nor did I had to worry about leaving some rouge alert saying You made it into the if!

But then again every time I tried to test in IE beauty became the beast. Instead of getting a good log in the IE console which… ooops doesnt exist, I got a nasty js error saing what the hell is console.log ?

So I got into the habit of cleaning up my console.logs after using them which really didn’t help me testing out something in staging or dev because when you are making heavy client side apps, everything is ajax based, and you always need to know if you are showing the exact data that you received or if your js code is doing something funky.

Solution simply use another function that checks if console.log exists and if so log it, else don’t


function cLog( text ){
if( (window['console'] !== undefined) ){
console.log( text );
}
}

Updated with a better solution

if(!this.console){
window.console={
log:function(){}
};
}

— Update as of sep 28 2014,

Using a grunt task to remove all your logs helps prevent deployment of logs, and avoid IE9 errors.

https://www.npmjs.org/package/grunt-remove-logging

Enjoy!

Good js maps

Recently in my latest project I was asked to build a world map where a user could select different countries for which we have certain data of and visualize that data. I tried every single mapping/charting tool I could find. This is a compendium that compares each one of the tools I went through and hopefully it will help you out in deciding which one to choose.

Google Geo Map

The first think I learned with Google Geo Map is that not because it has the label Google on it means that it is bound to be kick ass. I mean sure, a lot of their tools are kick ass and they do a great job with documentation and examples. But in this particular senario Google Geo Map just didn’t cut it. For instance if you have data loaded for Sout Africa and Lesotho as well, hovering over South Africa will dissapear Lesotho from the map…. yep bugs. Kosovo is no where to be found and well if you want the map to work in IE7 simply forget it. All of the tooltip labes that show uppon hovering will come out blurry and there is just no way to fix it. The work around is done by browser sniffing and implementing a flash solution for IE7. Now as crazy as this may sound given the fact that the map creation is 99% similar you can create either an SVG object or  FLASH object quite easily with the same code base… still this gets messy even if you try not to because the country codes for the flash map are different from those of the svg map. Hence you need to have two sets of json files….

Openlayers

Given the fact that I had to make the app work with IE7 and that Google was simply not going to provide me with an easy implementation that would suffice my needs, I turned around to openlayers.

Open layers is an amazing library that is extremely well documented. If you have the time to go through its entire documentation I strongly recommend you do and as that prepare your self in case you ever have to build a tool with a world map. Now at first glance I though I had found the tool I was so eagerly looking for but again I was deceived. Time was against me and I had basically two days to get this thing working and I was back to cero. Getting a simple map to work took for ever and finding the key points in the documentation that would allow me to fulfill the user experience that I wanted to create was turning out quite impossible.  A simple action like clicking on a country and getting its position or country code needed a combination of iso cods and geographies and on top of it there were some bugs that I just wasn’t able to pin point and fix like limiting the zoom level even though it clearly states in the documentation how to do it.

jVector map

In comes the hero.

Finally a map that is extremely easy to implement, has all the key functionalities that you would expect a world map to have (like clicking on a country or hovering over a country), highly customizable and friendly and works all the way down to IE7 like a charm. The only downside would be that it doesn’t have by default a lot of individual region maps. For instance if you want to show only the map of Africa, there is no individual SVG map for it. This is still a very particular use case because the map does have zoom in and zoom out functions in order to pan into the map and drag around to move from one part of the map to the other.

Hope this helps in deciding what tool to use!

vmware and localhost

Recently I’ve included into my suite of software VMWare Fusion. I was previously using the open source Virtual Box and a copy of Windows XP to emulate IE7 inside of my mac. I’m a web developer so IE counts wether I like it or not.

Even though I had these tools I realized to hitting into localhost was not as easy as just typing localhost:9000 or what ever port into my emulated IE. So I did a bit of research and found this post from Rowlandos Blog which I am simply going to paste here as a reference

Type ifconfig vmnet1 into a Terminal window. You should get a return like this:

vmnet1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> 
mtu 1500 inet 192.168.115.1 netmask 0xffffff00 broadcast 
192.168.115.255 ether 00:50:56:c0:00:01 

The “inet” number is your “secret” IP (in my case, 192.168.115.1).

Now that we have that number, we can edit the hosts file on the VM. We find it in: C:/WINDOWS/system32/drivers/etc. Just open the host file with Notepad, and add each virtual host (domain) on it’s own line at the end of the document, like so:

192.168.115.1 dev.triumphofgrace.com 

… and save. Afterwards refresh the VM’s DNS cache by typing ipconfig /flushdns in a command line window.

Boom, you can now point to your macs localhost from your emulator and test all that you need to test in Windows.

Best of luck!

Infinity in technology

Ray Kurzweil just faded out from my iPad as I was watching one of his lectures/speaches. He was talking as usual about how fast technology has evolved an will continue to evolve as time passes by. He also talked about the nature of technologies exponential growth and even game some insight into what the future might look like. Michio Kaku replaced Ray in my iPad. This time the subject of type I, type II, and type III civilizations came to be mentioned as he gave a speach about his new book titled Physics of the Future. What do both of these intelectual subjects have in common? And is there a conclusion we can draw from them?

If we take into consideration both characters ability to predict the future we must say that Ray Kurzweil has a high probability of being correct and even a bit conservative. If that is the case then within 25 years we will have the ability to blend with AI and blur the line completely into what it means to be human and real. In correlation with Michio, this will not necessary lead us into a type I civilization which is one that can control the planets energy (volcanos, rain, plants etc) to our and the planets benefit, but given the technology that AI will bring with its evolution, its hard not to think that it will certainly give us a big boost into a type I kind of civilization.

Now lets take a step back. Ray Kurzweil mentions a lot second life and how it has grown into encompassing many if not close to all activities that a regular/average human has. Increasing its graphics in order to make it seem more real and expanding the number of users it has, second life is a good example of how we humans can exist in a parallel reality with technology. The smarter the AI gets, the better the ability of the characters to interact with their enviorment and why not the cooler the posibilities of creating a non existing enviorment where for instance the laws of physics do not apply. What kind of civilization is this in Michio’s view? Is this a type I or II or III? Or rather is it a type -1, one that is manipulated by another kind of being in this case the human.

The characters of Second Life react to laws (coded) that can easily be meshed with the laws of physics and as it evolves and we get to understand more human behavior, themes such as feelings and attituedes and personality will also be able to be coded into the characters, giving them a sort of life/existance. But what good does that do to us?

Now here is were it gets a bit crazy. Imagine that you are a very advanced beeing and that the technology that you currently use allows you to have a super advanced second life character. One that wakes up every morning, has breakfast, goes to work, has relationships, suffers, enjoys, creates mathematical models to predict the future, seeks out side and inside of him self in order to understand etc etc. And what if this character could feed to you what the character learns and experiences. This character allows you to learn from what the character interacts with in his/hers every day life given that all that it computes is feed to you via an amazing device that is hooked up to your brain. An amazing device kind of  like emotive but reversed (which by the way can be acquired for 500 bucks… the developer version =).

Instead of putting info into the computer, you take out of it and use that info to feed your brain. Wouldn’t this allow you to experiment some on else’s life entirely? or to what ever extent the definition of being alive in that concept allows you to be a part of? Now take that same idea and multiply it times the n number of possible imaginative worlds that one can create in a virtual reality. Imagine that in each one of those worlds you have a character that personifies a kind of being that is different from the rest of your characters. This would allow you to fully understand concepts from a broad range of personalities and rationals in order to comprehend the absolute definition of a concept.

Lets step back a bit an analyze how the less human we become the more we know and the more we “do”. As humans have evolved, the use of their bodies has diminished and the use of their brain has increased. Now a days one can simply type a thought and feed it to the world. Something like this 20 years ago would have been completely impossible and or would have taken several years in order to  arrange and spread. We basically do more with less and that keeps getting faster as technology allows us to do more and more and more.

What if for instance we developed a device that allowed us to read a book within seconds. By understangind a brains synapsis and the process of laearning we can potentially speed it up as well and generate it with some kind of device that activates that same synapsis but faster and more efficiently. Yes I know sounds very much like the matrix but to be honest, it is not far from reality. If we are able to send messages to a computer (please look at the emotive link ) how far are we from putting information from the computer into the brain?

Now if this were the case and we were able to absorbe more information faster and not only absorbe that informatino but comprehend it (which is the key) then we can do more with it. But this is only correlated with our human selves. We can do more with that information in order to improve our human lives, our work condition, our daily habits, our knowledge on how the body works etc. And this in its self will lead to a faster evolution of technology and the way we live.

But what about our virtual life? our seond life? Well in order to be able to have a grasp on what we are experiencing we would have to let it follow a path of its own and with the spark of creativity (if we ever manage to code such an algorithm) we can leave that beeing to evolve as we continue to evolve our own current life. Hooking up to it every now and then in order to see how it is learning what it is learning and why it is learning it that way instead of another way.

By beeing able to tap into this secod life character and dump all of its information into our brain we would be actually living a second life. And if we are able to do this with multiple characters…. well you get the point.

This arises to bring the concept of the infinite you. The you that has evolved through out time and that has created multiple beings within their holographic universe that allows them to continue evolving at their own speed and that ultimately feed you with the knowledge that they acquire. Scientia potentia est  or knowledge is power, attributed to Sir Francis Bacon, one of the worlds most thoughtful philosophers and scientist of his time would have quite a precedent in the concept of infinite you’s due to technology.

How to start off the UI of your project

Back in the day one would have to create a series of fixed clases that would help you out through out the construction of your site. Classes that you now if chained with other clases would allow you to create very interesting and cool looking layouts. Today the answer is quite simple use html 5  boilerplate. The boiler plate comes packed with a bunch of useful css resets, browser sniffs for versions of IE, and if iE6 a built in chrome frame tag that allows the end user to install chrome frame into their browser if they desire to have a better experience.

The boilerplate is regularly maintained by Paul Irish ( a very bad ass developer ) and others, and it will definitely give you a fast start to build a web app with all of the standard folder structures you would expect a high quality web app to have.

One thing though, the boiler plate does not come with a CSS grid framework. For many developers that are becoming or have to become Front End develoepers for their projects this is quite an issue. Many developers rely on the use of some kind of css grid to give them a logical structure they can follow in order to build their websites. If you are using html5 boilerplate I recommend you get  super light grid framework such as 1140 grid or something a bit more complete like blueprint.

If you are like me and don’t like using grid frameworks because you like to build your own grid when you code then the boilerplate by it self will be very useful.

Keep in mind that there are many things included in the boilerplate and that you should with out a doubt go over all of the css code that is provided and make sure that everything that is in there is needed by you or your project.

Last but not least use CSS SPRITES!!! Its amazing that in todays world many people still seize to use css sprites. Ok maybe if you are thinking that your images need to scale in order to fit in multiple screen resolutions your not going to be using sprites for them. But normally there are tons of icons used in a web app and those icons by default should be implementing using sprites.

The technology for sprites is quite simple. Create one big image using Photoshop, or Fireworks or your image editor of preference. I use those two because I’ve been working with them for a while but if you want an open source solution the Gimp is definitely an awesome editor. In this big image you are going to stick in all of your little icons, preferably organized in a nice grid structure way. Make sure to leave some gutter between the icons just so that they are easier to identify and extrapolate in the future with css.

Once this image is done save it in the web format you prefer. Thinking of performance go for gif 256 or if you want better quality Photoshop has a built in “Save for Web and Devices” option that allows you to save as PNG and at the same time compresses quite well.

Ok so now you have a big image that has all of your little images inside of it. Now what?

Using css you create a parent class that will contain that image. If you want a nice name that makes good reference use the class

.sprite{background-image:url(/images/pathToYourSprite.gif)}

This will allow you to have all of the icons within one class. Next create the individual icon classes that you will be using. For instance if I were to have a light bulb icon in my sprite use the class

.lightbulbIcon{background-position:0 0; width:16px; height:16px}.

The background position marks the top-left starting point of the icon you want to pinpoint. Next the width and height are the width and height of the icon you are using.
In order to use include both classes within a html tag such as follows

<div class="sprite lightbulbIcon"></div>

This will render your lightbulb icon inside of the div tag. If you want more flexibility for icons in different tags include an inline-block attribute in your style like so

.sprite{background-image:url(/images/pathToYourSprite.gif);display:inline-block}

That will let you use anchor tags, spans etc.

Hope this helps!