When I meet with people and talk about our work, I get asked a lot what technology we use to create interactive and dynamic data visualizations. At Interactive Things, we have a set of preferred libraries, applications and services that we use regularly in our work. We will select the most fitting tool for the job depending on the requirements of the project. Sometimes a really simple tool is all you need to create something meaningful. On other occasions, a more multifaceted repertoire is needed. But how does one choose the right thing to use? An endless list of every tool available, does not answer that question and a recommendation from a friend is oftentimes more valuable.

That’s why we have put together a selection of tools that we use the most and that we enjoy working with. We called it selection.datavisualization.ch. It includes libraries for plotting data on maps, frameworks for creating charts, graphs and diagrams and tools to simplify the handling of data. Even if you’re not into programming, you’ll find applications that can be used without writing one single line of code. We will keep this list as a living repository and add / remove things as technology develops. We hope this will help you find the best tool for your next job.

Let me answer the most likely questions right away: No, not everything find its’ way into this list, so you might not find your personal favorite. Yes, some of the libraries mentioned are brand spanking new and may not be ready for production while others already have successors, but they still provide a good service.

We recently attended an interdisciplinary visualization workshop that was all about creating a dialogue between scientists, technologists and designers. It was interesting to discuss the different ways in which these groups think about visualization and how they use it for different purposes. Very bluntly put, each group lacks something another group knows and cares deeply about, be it an understanding of colour [we met in the UK] or an understanding of statistics.

We had a deep discussion on interdisciplinarity, whether one person should feel comfortable and knowledgable enough doing work in all fields, or whether it’s really about people and communication skills that are needed to successfully collaborate with specialists in their own fields. Regarding scientists and visualization in particular, one point that stuck out was how scientists had to embrace programming as a tool for their experiments, even though they are not computer scientists. Will they have to embrace visualization techniques in the same way?

Of course, scientists already use visualizations in their work. Even though they may not be aesthetically the most pleasing, they do serve their purpose – or do they? Computer scientist Miriah Meyer and designer Bang Wong are doing research to uncover the unused potential of visualizations for scientists that can help them in understanding and validating their scientific data and also support them in developing new hypotheses and insights through visual tools.

Miriah strongly believes in the power of multidisciplinary teams instead of knowing how to do everything yourself. But still, she thinks that a basic understanding of visualization is very important for every undergraduate science student and should be taught, so they know it exists, what potential it has and that there are actually people they can ask to help them with doing research using visualizations.

Pathline

Pathline is a tool that Miriah and Bang developed in close collaboration with geneticists applying a design process. They explored and designed before implementing anything and also incorporated user feedback into the tool. They didn’t use such a design process before and were very happy to discover, how it allowed them to move faster than if they had started the implementation part right away. What came out of this is a, in their own words, “visualization tool for comparative functional genomics that supports analysis of three types of biological data at once: functional data such as gene activity measurements; pathway data that presents a series of reactions within a cellular process; and phylogenetic data describing ancestral relationships between species.”

In addition to the above “curvemap”, the “linearized pathway” was developed. It has a very simple structure, always ranging from 0 to 1 from left to right. It can be enhanced by adding more detail to its structure.

This added detail may look more complex – but only if you don’t know what to look for. Our favourite [still trying to be British here] example for this is a topographic map: there’s an incredible amount of detail in there, but if you know that you’re looking for place A and B and the road between them, you’ll find them easily with a little bit of training, because you know how they are visually encoded. If you’re going by bike, you’ll probably also check the contour lines to choose a not too steep road.

Pathline also hides structures that may be very useful to biology students, but are well known by the end of their studies. At this point, you know how a certain pathway works and all you need is hints that help you see, which part of a pathway you’re looking at.

Case Studies

Two main advantages of the new tool were found. First, there was a massive gain in efficiency. The study of a heatmap took up to a half-hour before but can be done at a glance at the curvemap now. More importantly, though, the scientists made new discoveries of gene properties they didn’t know about before. What was hidden in the data before, is now very clear, even to an untrained eye.

What we see is that genes g5 and g6 are very similarly expressed across all species – except for species s7. Thanks to this tool, the scientists were able to see this anomaly and to conduct more research into why this is so.

Conclusion

We think Pathline is an excellent example of researchers pushing science forward through visualization. It’s striking how much better the visualizations are now compared to what they were before. From our experience, a collaboration between scientists and designers is very difficult, as design schools are usually not in the same places as more science oriented universities, so we need to find ways to collaborate more. But we hope that by covering a topic like this, readers are encouraged to go out and look for projects in an interdisciplinary scientific context.

Pathline is available as an open source Processing application from the Pathline website – hopefully soon on GitHub as well, so we can contribute back more easily if we see opportunities for improvement.

I just left the Stamen studio where I had a brief chat with founder and CEO Eric Rodenbeck. He enthusiastically showed me their newest work: maps.stamen.com. As part of the CityTracking project, the savvy folks at Stamen have put create three sets of map tiles to be used with OpenStreetMap data. These tiles are not your average map using custom colored vector paths. Instead Eric and his team tried to push the boundaries of what we consider feasible in mapping on the web.

Toner is a high-contrast black and white tile set that is optimized to focus on the actual shapes found in maps without being distracted by a plethora of colored lines and areas. Eric mentioned that working with simplified maps like these, allows the creator to focus on the actual information that is represented on multiple levels to find the right balance between no information and an overload thereof.

Terrain contains hill shading and natural vegetation colors as seen many great static maps from the past (like the ones in this excellent book; no affiliation in any way, I just love this book).

Watercolor is what got me really excited: The watercolor tiles were created by applying raster effect area washes and organic edges over a paper texture. What started as an exploration of how water color behaves eventually was automated and simulated by the awesome work of Zach Watson. He would actually describe the process rather as faking than simulating, but the outcome is what counts.

Along the demo of the maps, Stamen also provides simple instructions how to use them with you favorite mapping framework like ModestMaps, Leaflet or Google Maps. They are available under a Creative Commons Attribution (CC BY 3.0) license.

Yesterday Visualizing.org launched version 1.0 of their Visualizing Player, a media player designed specifically for embedding and displaying infographics and data visualizations.

Perhaps its best feature is that it’s nearly agnostic of the visualization format. Right now it supports HTML5, Java, Flash, PDF, Video, Image, and whole websites. Functionality of the player adapts to each format; interactive visualizations and videos are launched via a play button, high resolution images can be zoomed and panned to explore details.

It also features a full-screen mode which launches a seperate browser window and is especially suited for interactive visualizations. It allows the user to favorite (account required), share and embed, and download a visualization. As Visualizing.org is one of the few platforms that cares for proper licensing (all content is CC-licensed), you won’t have to worry about being allowed to embed and share visualizations with the Visualizing Player.

Here’s what an embedded interactive Flash visualization looks like:

And a static infographic (with zoom and pan functionality):

As with any 1.0 release, there are still a few quirks and loose ends. Especially when displaying website and HTML content, there sometimes seem to be scaling issues and too many scrollbars appear. Some visualizations work around this by launching directly in full screen mode (like Jan Willem Tulp’s Ghost Counties). Also, when panning an image, it inadvertently sticks to the cursor when the mouse is released outside of the Player and then re-enters. But I’m sure the people at Visualizing.org are working hard to release an even better version 1.1.

Small issues aside, we’re pretty excited and think the Visualizing Player is a terrific tool for people to embed and share visualizations. It offers great features for displaying properly licensed interactive and static content, and goes beyond embedding mere screenshots and links. So what do you think? Do you have any suggestions for improvements? Share them here in the comments or directly on Visualizing.org.

For the past year or so I have been dabbling with protovis. I don’t have a heavy CS background but protovis is supposedly easy to pick up for people like me, who are vaguely aware that computers can make calculations but who need to check the manual for the most mundane programming instructions.

What I found was, while it’s reasonably easy to modify the most basic examples to make stuff happen, it is much harder to understand or adapt the more complex ones, let alone to create a fairly complex visualization. So I started documenting my struggle with data, first for my own use, and eventually realized I could share what I learned.

—Jerome Cukier

Be aware that the content is laser focused on how to deal with data instead of shiny visualization goodness. That said, if you plan to create custom visualizations with Protovis, I highly recommend to have a look at Jerome’s notes.

One aspect that the release of the Google Books Ngram Viewer last week shows really well is the power of visualization: instead of offering a huge but abstract data set like back in 2006, Google created a simple visualization tool that shows the data and makes it easily queryable. It’s not as visually appealing as what people like Chris Harrison have done with similar data, but it doesn’t have to be! The purpose of this tool is to give first insights and spawn ideas, which can then lead to a deeper analysis.

What I find most exciting about this project, is that Google enables everyone (no programming skills necessary) to ask questions and dig into a century old corpus of accumulated wisdom in over 5 million books in 6 languages.

While playing with the Ngram Viewer and looking through other peoples’ queries (click on the charts to go to the source), I noticed that there are different kinds of questions people tend to ask, so I came up with this incomplete and unscientific categorization of what the Ngram Viewer is, that I’d like to put up for discussion.

It’s About Comparing Things

A very simple but powerful use case of the Ngram Viewer is to compare ideas, products, concepts, etc. over time. People like to think in comparisons like “good” and “bad”, so this is an ideal entry point for people who don’t quite know what to do with this tool. As a case in point, I wanted to look at how the pie chart stacks up against other visualization methods, and made a first observation: these charts are always opinionated, you can (have to) leave words out, forget them, or spell them differently than others.

Another comparison I wanted to make was about what the development of communication media looks like over the years. Here, I noticed a difficulty: The Ngram Viewer is case-sensitive, so be careful how you spell “Internet”, as there will be fewer results when written in lower-case.

It’s About Patterns

Many people discover interesting patterns, like the occurrence of year numbers. Seems logical, when you see it, but did you think of this before?

It’s About Correlations

If you suspect, that one thing could have an influence on another, just go to the website,try out some terms, and see, whether they occur in literature during the same time periods. This, of course, is not a definite answer, but it’s a good start to investigate.

It’s About Phrases

The term “n-gram” describes words (or characters) that occur in a specific sequence. The Google data is available for n-grams of up to 5 words, which means that it is possible to not only search for single words, but for phrases and sayings.

It’s About Language

Because the data repository goes back to the 17th century, this tool can give us an interesting insight into the development of languages, like in the visualization below, that shows how the medial s (ſ) was superseded by the “normal” s. When looking for insights using this tool, always be aware that words may have been written differently, centuries ago, so they may not show up, if you don’t know what to look for.

It’s About History

Books reflect the history of the world, so I queried the Ngram Viewer for “guerre” (which is French for “war”), a (sadly) omnipresent event of human history. I did the query in French, because a lot of historic wars happened there, and it shows indeed: the French Revolution in 1789–1799, the Napoleonic Wars (1792–1815), the Franco-Prussian War (1870), and then, of course, the two World Wars. If you do the same query in American English, you’ll also notice a strong bump in the 1970s, the Vietnam War, which didn’t have the same impact on France as it did on the USA.

I also made a query for “baïonette” (bayonet), a tool of war, and indeed, it correlates with the wars, and we also see, when it became available, and that it’s less used today (I guess that it still shows up because it’s written about in history books).

This shows another interesting use case for the Ngram Viewer: let a teacher ask her students “what do you see?” They’ll (hopefully) know about the two World Wars, but then they’ll have to go and do some research about what those earlier spikes might mean.

It’s About Society

A last example that I want to go into, is one, that isn’t possible with the current version of the Ngram Viewer: the comparison of societal change within different language areas. I supposed, that “racism” would have  had different impacts in different regions of the world, the USA specifically. And indeed, when we superimpose queries in American English, British English, German and French using Photoshop (be sure to adjust the percentage scales correctly), we can see the bump in the late Sixties in American, but not in British literature. Also interesting is the development in France, which is strangely linear, and different from all the others.

Conclusion

I hope you had as much fun and insights as I had while researching this article. I strongly believe, that by making a visual viewer available for this huge data set, Google did a lot of people a great service, who wouldn’t otherwise have a chance to dig into this data at all.

So, go try the tool yourself and post interesting queries in the comments or to the Ngrams Tumblelog. Also be sure to read Google’s introduction to the Ngram Viewer, which has some interesting background information. And don’t forget, that you can click the links at the bottom of the charts, which will take you to the sources in the huge repository of books, that Google has digitized.

Polymaps is a free, open-source JavaScript library for making dynamic, interactive maps. It is the result of a collaboration between Stamen Design and SimpleGeo. Founded by Erik Rodenbeck, Stamen is one of the most respected data visualization studios out there. SimpleGeo is a service that provides location infrastructure that makes it easy to add geo-aware features to applications.

Description

Polymaps provides speedy display of multi-zoom datasets over maps, and supports a variety of visual presentations for tiled vector data, in addition to the usual cartography from OpenStreetMap, CloudMade, Bing, and other providers of image-based web maps.

Because Polymaps can load data at a full range of scales, it’s ideal for showing information from country level on down to states, cities, neighborhoods, and individual streets. Because Polymaps uses SVG (Scalable Vector Graphics) to display information, you can use familiar, comfortable CSS rules to define the design of your data. And because Polymaps uses the well known spherical mercator tile format for its imagery and its data, publishing information is a snap.

Examples

The provided examples give a hint about the possibilities you have with Polymaps. Here are two elegant examples for maps based on custom tiles made with CloudMade:

The following example shows different ways to display data. The used tiles are served by another great piece of code by Stamen named TileStache:

This visualization shows the pavement quality of San Francisco streets, mashing up two datasets from DataSF. Streets with high pavement quality are shown in green, while those in the worst quality are red.

Flickr Shapetiles is a slippy map of all the shapefiles that were generated from geotagged Flickr photos.

You can start using Polymaps right away by downloading the source code from Polymaps.org or checkout the latest version from the public repository on GitHub.

Together with Kim Rees from Periscopic I have reviewed the social data visualization application Swivel. Swivel is a simple to use web application that lets you visualize public or private data sets and collaborate openly or in closed user groups. My review is the first post in a series of reviews that will be published over at Information Aesthetics. Following up are reviews of the online tool Many Eyes and the desktop application Tableau Public.

Read the full review on Infosthetics.com and let me know your feedback and thoughts over there in the comments.

Humble software development released a new visualization library called HumbleFinance. It is visually and functionally inspired by the Domestic Trends visualization of Google Finance. Unlike Google’s version, which uses Flash, it is written entirely in JavaScript and uses HTML5′s canvas tag, so no Plug-In is required to display the visualization.

Although the HumbleFinance graph is especially suited to display financial time series, any two two-dimensional sets of numerical data which share an axis can be displayed.

The graph is divided in three areas: On top a section of the data sets can be compared side by side. At the bottom an overview over one data set is displayed. It also supports full interactivity, i.e. hovering over a data point in the demo shows connected points and the associated data. You can also scrub through the timeline and zoom the section of the data.

On its trail to organize the world’s information, Google has just added a new experimental product to their Lab. The Public Data Explorer makes “large datasets easy to explore, visualize and communicate”. It is designed to help people comprehend data and statistics through rich visualizations.

The Explorer uses Google’s ability of syndicating public datasets previously introduced in their search results. The visualizations are built on top of their Visualization API and the animated charts are based on the Trendalyzer technology Google acquired from the Gapminder Foundation. It has previously been available in the Motion Chart in Google Spreadsheets.

Creating a visualization is straight forward and a matter of minutes. A user can chose the data and how to visually represent it. Additional data can then be applied to different attributes of the visualization. The created views can be shared via permalink or embedded in any webpage.

The data

Google selected the data sets by analyzing anonymous search logs to find patterns in the kinds of searches people are doing. This way they can cover a majority of the demand for statistical data with a limited amount of different data providers and data sets. Here’s a list of the currently available data sets:

The World Bank, the U.S. Bureau of Labor Statistics, the U.S. Census Bureau, the Organization for Economic Co-Operation and Development (OECD), the California Department of Education, Eurostat, the U.S. Center for Disease Control, and the U.S. Bureau of Economic Analysis.

Conclusion

In my opinion Google is in a strong position to evangelize the advantages of transparency in information distribution. Let’s hope this encourages more insitutions to open acces to their data sets. I am also happy to see a user friendly addition to the solutions already available for developers to access public data programmatically. It makes it really easy to create and share visualizations based on public data. As a note: If you’re looking to display your own data, I recommend having a look at the recently launched Tableau Public.

What do you think? Are the available data sources sufficient enough to be real value to the average user? If not, what would you like to see added to the list? Please share your thoughts in the comments below.