Python Modules are code someone else has written and distributed, in order to make life easier for the rest of us. You may be familiar with the standard modules that come with Python, like math or datetime, but there are numerous more resources out there for the GIS minded developers. I will be discussing some of the modules I find essential in my work apart from the famous ArcGISScripting module by ESRI: GDAL, numpy, NetworkX, xlrd and xlwt. Let’s dive in!

GDAL – Geospatial Data Abstraction Layer

It will come a time in every GIS Professional’s career when they will need to quickly access information from a random shapefile they have, but do not have access to any GIS software or geoprocessing functionality (think of a laptop on the road, a remote machine not running Windows, etc). GDAL comes to the rescue, providing us such functionalities.

GDAL is a translator library with Python bindings that allows access to raster data using a unified abstract layer. Bundled with it is OGR, which provides similar functionality for vector data. Download it here.

A quick example of using GDAL:

import gdal
from gdalconst import *
# Open the raster dataset
dataset = gdal.Open(filename, GA_ReadOnly)
# Print the projection of the data
print dataset.GetProjection()

Using OGR:

import ogr
# Get the driver
driver = ogr.GetDriverByName('ESRI Shapefile')
# Open a shapefile
dataset = driver.Open(shapefileName, 0)

numpy – Numerical Python

I cannot think of any GIS practitioner that did not have to manipulate raster data in a peculiar way, only finding that the software at hand doesn’t allow you to perform many customized functions. If one can interpret raster data (which GDAL above helps us with), then one can use them in Python as a matrix (algebraic matrix). numpy is the best Python package out there for this sort of situation.

numpy is a package that enables n-dimensional array manipulation in Python, as essential part of any scientific endeavor. It also provides linear algebra functionalities, Fourier transforms and random number generators. Get it here.

An example of the use of numpy:

from numpy import *
# Sample IO Table data
ioSample = [[1,2], [3,4]]
# Turn into a numpy array
ioMatrix = array(ioSample)
# Find the inverse of ioMatrix
ioMatrixInv = linalg.inv(ioMatrix)

NetworkX – Complex Networks Analysis

While most GIS software out there provides the ability to build networks, sometimes it is easier to build networks quickly and dirty, without having to involve complex GIS software. An analysis of participation by space in an experiment can easily be achieved using the simple, yet powerful NetworkX module.

NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. It is hosted by the Los Alamos National Laboratory, and sees active development (presumably sponsored somehow by Los Alamos). Download it here.

A quick sample is shown below:

import networkx as nx
# Create a graph
g = nx.Graph()
# Populate the graph
g.add_node(1)
g.add_node(2)
g.add_node(3)
# Create edges
g.add_edge(1,2)
g.add_edge(1,3)
# Print the neighbors of node 1 (returns 2)
print g.neighbors(1)

xlrd – Excel™ File Reader

All GIS practitioners have been sent “GIS data” in an Excel file, either a geocoding result or GPS waypoints, or anything similar. While ideally whoever sent the data would be educated on why it is a bad idea, most often we have to deal with the data without any additional help. xlrd comes into play, allowing you to read the said Excel formatted data into Python with little effort.

xlrd is a Python module that allows one to read Excel files without the need of Microsoft Excel or Windows. It provides access to XLS files for Microsoft Office 2003 or earlier. Download it here.

A quick example that will read an XLS file and print it to screen:

import xlrd
# Open the Excel file
book = xlrd.open_workbook("excelFile.xls")
# Read the first sheet in the Excel workbook
sheet = book.sheet_by_index(0)
# Read the first row from column A to E
rowValues = sheet.row_values(0, start_colx=0, end_colx=4)
# Print the row values
for value in rowValues:
    print value

xlwt – Excel™ file writer

Business requirements often want results in an Excel file, so some other person, in another department, can run some sort of analysis on the data. Building a distance matrix is fine, but Joe from accounting is using complicated Excel spreadsheets and does not want to bother with database connections or DBF files. This is a situation in which xlwt excels, writing data to an Excel spreadsheet without the need of Excel or manipulation by the mouse.

xlwt is a Python module that, similarly to xlrd mention above, allows for cross platform Excel file creation without the need of Microsoft Office. Download it here.

An example follows:

import xlwt
# Create a new workbook
book = xlwt.Workbook()
# Add a new sheet
sheet = book.add_sheet("My Sheet")
# Write the number 5 in the first row, first column
sheet.write(0, 0, 5)
# Save the file
book.save("myExcelFile.xls")

These are my essential Python modules for GIS development. Notice I left quite a few behind, especially those relating to web-based GIS developments. Those will appear in another post in the future. But please, do share your thoughts on what Python modules are essential for your GIS work.

Related posts:

1. Manipulating Excel files using Python part 1: Reading Excel Files

The discussion ranged from skills to succeed in an organization using GIS to support business decisions (City of Seattle), ESRI, the leading GIS software producers and freelance development using GIS technologies. There is significant overlap for the required and desirable skills, but also some slight differences. I will list the skills in no significant order and provide a brief explanation. If a skill is something I personally added, it will start with an asterisk.

GIS Skills

• 
• Spatial Data and Algorithms understanding: Understand the special case of spatial data, how they work and their internals. Also, be familiar with how certain operations are carried out and when they are applicable. Many operations will run in the software, but not necessarily produce valid results. (Contributed by reader Duane Marble)
• Data entry: Be able to enter data into a database successfully with minimal errors. This includes editing said data as needs arise.
• Data conversion: The ability to convert data from either older sources (digitization) or from multiple sources to either a common format or common schema. It is extremely useful to be able to work with data coming from GPS and performing data corrections as needed. (With contribution by Jimmy Xu)
• Data maintenance: Be able to maintain data, correctly archive and ensure quality control.
• *Metadata creation and editing: Maintain logs of data processing and relevant information to include in metadata and ensure accurate creation and maintenance of said metadata.
• GIS Analysis: Be able to perform GIS Analysis as it is often used to solve common problems. An ability to extend and alter the standard analysis to meet requirements is a plus. Remember, data analysis can be performed on vector or raster data, therefore some remote sensing skills are required. (With contribution by Jimmy Xu)
• GIS Workflow: Understand the workflow to perform some procedure and be able to follow it and enhance it as needed.
• Model Building: Be able to create models of processes to allow for a workflow to be built. Also, model building in the ArcGIS sense is very helpful in this regard.
• Cartography and Graphic Design: Familiarize yourself with cartographic principles and graphic design principles. Maps are used in a variety of ways and presented in a multitude of media. You need to be able to work with that. Think of color, symbology, fonts, etc. Bad cartographic design will often make your analysis hard to decipher and interpret. (With contribution by DavidM)

Programming Skills

• Basic understanding of programming: Be able to understand what programming is and what it can do to solve certain problems. Know the strengths and limitations of programming custom solutions to problems, as well as time requirements. (More about programming)
• Programming language: Familiarize yourself with a programming or scripting language, as it is often used to build workflows or custom solutions to problems. For scripting language, both ESRI and the open source community tend to gravitate toward Python. For programming languages, C++ will give you an opportunity to work in multiple environments, while C# and the .Net languages offer you good development tools and interaction with Windows based software. (More about programming languages)
• Object Oriented programming: Learn the concepts of object oriented programming and be able to apply them in conjunction with your programming language of choice. Most GIS development is leaning toward this paradigm, and you should too. (More about object oriented programming)
• Basic GIS architecture (desktop and web): Understand the architecture of GIS and the method of communication between the different parts of GIS. Be able to distinguish when one can introduce internet-based communication in the mix and how it would work. (More about GIS architecture)
• Web Services knowledge and experience: Web services are everywhere these days, and GIS is not escaping. Learn about them, how they work, and try to implement some of your own. HTML, CSS, JavaScript, XML and related AJAX technologies are a valuable tool. (More about web services) (With contribution by Andy Anderson)

Database Skills

• Able to understand data models and structure: When given a database, you should be able to explore the data models within it and understand the structure of the database. Often times, structure will be represented in diagrams (UML), discussed below. (More about data models and structure)
• Ability to design data models: Given specific requirements for data, you should be able to design data models to fit your data.
• Database Design tools knowledge: You should familiarize yourself with database design tools, like Microsoft Visio. Most design work for data models uses it. (Check out Visio here)
• Structured Query Language (SQL) knowledge: Almost all modern Database Management Systems (DBMS) understand SQL for data queries, inputs, deletions, etc. One should be familiar with SQL and be able to perform SELECT, INSERT, MODIFY and DELETE statements. JOINS, RELATES and further SQL knowledge is greatly valued and useful on the field. (Contributed by Andy Anderson)

Project Management and Design

• Ability to translate user needs to solutions: More often than not, you will be supporting some client (or boss) that is not familiar with the details of GIS. You need to be able to translate their needs into solutions that can work in your domain. If it is not possible, you also need to be able to say so and offer alternatives. This is similar to requirements analysis used in software development.
• Good communication skills: You need to be able to communicate effectively and with confidence with your team and clients. There is no substitution for this skill.
• Good writing skills: Communicating is not restricted to verbal communication. You need to be able to clearly communicate in writing not only for communicating with your clients, but also to be able to produce metadata and reports of your work.
• Project management skills: Often overlooked in the GIS world, formal data management training is desired and required to run successful projects on time and within budget.

Other Skills

• Ability to apply expertise in multiple domains: GIS skills, while important, are not useful if they can not be applied to different domains. Your knowledge of other domains (like biology, forestry, etc) will allow you to think of creative ways to apply your GIS skills in a multi-disciplinary functions, which is greatly needed. Think outside the box (Yawer S. Ansari commented to reiterate this)
• Portability of skills on multi-platforms and online/offline world: Your skills need to be applicable to different platforms. Not only should you learn how to do GIS Analysis, but you should be able with limited help to achieve similar results using other platforms (be it moving from an ESRI training to Idrisi, or from a Windows machine to a Linux machine, or a desktop application to a server based one). An ability to traverse between online and offline worlds is a valuable asset to have.
• Detail oriented: This needs no explanation, but in the GIS world, detail oriented can get you very far. The quality of your work will show (especially when you think of metadata or workflows).
• Customer Support skills:  In most cases, GIS is used as a support tool within large organizations. As such, GIS Analysts oftentimes need to interact with clients, either internal or external. Having good customer support skills ensures you establish strong relations and opportunities.
• Don’t be afraid to explore

Now that you’ve read all the skills mentioned at the panel discussion, please provide your own or your feedback below.

UPDATE: Fixed spelling. Thank you skobola for the corrections.
UPDATE: Incorporated user comments into the body of the text by Yawer S. Ansari, Duane Marble, Andy Anderson, DavidM and Jimmy Xu.

Related posts:

1. The Essential Python Modules for GIS

Google Earth Logo

Before we begin, let us make a distinction. Google Maps and Google Earth are not one of the same. Google Maps is a web-based application that allows people to view maps on a web page. There are three main features Google Maps offers:

1. Users can search by location (i.e. for restaurants within their current map view).
2. Users can ask for directions in a multimodal transportation system (i.e. driving, public transport, etc).
3. Users can add their own data to the map, but only if they can master the Google Maps API (not a hard task, but not something the average person will do).

Google Earth on the other hand is a desktop application that provides the first two functionalities (search and direction), as well as an easy way to add your own data, one point at a time, to your map. There are of course some more capabilities present, like measurements of paths and the ability to upload and share your data with other Google Earth users. But Google Earth is not only what most people user. There are 2 more editions.

1. Google Earth Pro
   Provides some data import functionality from existing GIS data sources (shapefiles, etc) and spreadsheets.
2. Google Earth Enterprise
   The infrastructure that power Google Earth and Google maps at your fingertips, to create your own versions of them. The main advantage is data management of terrabytes of data, with 2 dimensional web maps and 3 dimensional globes.

All of Google’s offerings in the arena of geographic data are impressive to say the least, and worthy of attention. But are they truly GIS?

If one wants to define GIS, one needs to run back to definitions used throughout the years by academics and software publishers alike. Leaving the issue of people running the system and organizational contexts that build the system in a certain way, GIS is defined by the following components:

• Data: The very basic component is of course measurements. Without them, a GIS cannot function. Undoubtedly, Google has made data available in an unprecedented way.
• Representation: Measurements need to be represented in a format that the computer understands in order to operate on. Google Maps/Earth do have this. The famous KML format is one example on how Google does it.
• Operations: This is where Google Maps/Earth get a bit left behind. Granted, there is the search functionality and route finding operations, but is this the only thing a GIS can do? Out of the box, most GIS offerings have operations for slope calculation, linear referencing, buffering, and many more. Google Maps/Earth lack this functionality at their current state.
• Transformations: In this case, we do not simply refer to transformations from one data format to another, but also transformations from simple measurements out in the world, to the results of an analytical process (a report if you may). The pathway from measurement to data to information is not clear, and unfortunately not present in many cases.

If one uses then one of the commonly accepted definitions of GIS, one cannot easily place Google Maps/Earth in this realm. This is not to say that they do not partially provide functionalities that a GIS would, but rather that they are limited and not the whole picture. A pollution dispersion problem would not be solved by Google Maps/Earth. A study of the distribution of populations among some characteristic cannot be rigorously performed (visually perhaps, but can anyone report error numbers visually?).

I therefore suggest that Google Maps/Earth, as well as offerings by other vendors, like Virtual Earth, Bing Maps, Yahoo! Maps and MapQuest should not be called GIS, but rather Mapping Information Systems, or even Networked Mapping Tools. Partially a GIS, but lacking the full functionality needed.

Related posts:

1. Google Spatial Data in-house – Goodbye to TeleAtlas?

Legibility can be defined as the ability a human reader to read something without effort. It can depend on many things. Often, the size of font chosen restricts legibility. For our purposes though, legibility is discussed in light of typeface choice.

Readability can be defined not on a letter by letter basis, but how he combination of letter are read within a larger body of text. In other words, readability is defined by the amount of effort one needs to make to read text, not single characters.

This may seem confusing at first, but looking an example of text below may help:

The above text is exactly the same in both cases, yet if one tries to read it, one finds some differences.The one on the left is a serif font (Times New Roman), while the one on the right is sans-serif (Helvetica). When reaching the end of each line, from my experience at least, it is easier to identify the correct next line in the text on the left. The one on the right creates some problems to read the line, even though letters are easier to understand. On the left we have what is called readable type, while on the right we have a legible type.

The use of lettering (and hence typography, legibility and readability) appears in many parts of a map: title, legend, labels and any other explanatory text (credits, projections, etc). Below are 4 examples from the major web mapping applications out there: Google Maps, MapQuest, Microsoft Live Search Maps and Yahoo! Maps. All four zoom in on a location on the University of Washington campus, showcasing labels for street features.

Which one is the best typographically is left to the reader. What is of note here though is that all labels, no matter what web mapping application you may use, come in only sans-serif fonts, encouraging legibility over readability. Labels not only need to provide a quick and easy way for us to identify specific features, but they also do not always follow traditional text in being horizontal. What is of note as well is the fact that the lettering is always pushed to fit within the width of the road features (not always successfully it seems). What most people note though is that labels of features tend to use non-serif fonts that allow readers to easily recognize names (as the purpose of labels is to quickly identify features by their name).

This can also be seen in the following two examples, even though they are much older and I focused on the title mostly. Here, the two maps are of Seattle, Washington, circa 1970 (USGS map) an 1917 (Automobile Blue Book, Volume 8 Section 2), both from the excellent Perry-Castañeda Library Map Collection at the University of Texas.

The above map is the USGS map for Seattle, Washington. The extracted version is of the southwestern most part of the map, to show the title and scale. Notice the map title is using a sans-serif font, as well as the scale and place names, like cities or airports. Serif fonts are used only in the names of islands and water features, like the Puget Sound or Fox Island. There is a mix of both types of typefaces that make distinctions that help the reader identify information.

The map above is the 1918 map of Seattle, WA cropped again to the southwestern most portion of the map to include the title and scale. Note that the title this time is serif along with other place names, while the street name and scale are using a sans-serif font. Again, a combination of the two that provides a visual hierarchy to the map reader that helps discern information faster and easier.

What sort of advice can I therefore give to a cartography student in regard to typography? Not much really. If we remember that cartography is an attempt to convey certain messages through visual hierarchies, then one needs to remember that sans-serif typefaces are more legible and therefore easier for spot reading, while serif typefaces are more readable and easier to read when one has long texts. As any cartographer knows though, a 72 points serif font on the map will be more legible than a 12 points sans-serif font, so when you choose your font, choose wisely: what typeface should you use, at which size and what weight, not independently from the rest of the lettering of your map. All lettering is related, and all together provide your visual hierarchy.

Online Resources

Cisalpin Typeface: Designed for Cartographic Usages
An often-used typeface in Cartography: Optima
Fedra Sans: A modern typeface with multiple variations
A Typophile Discussion Thread about Cartographic Typefaces

Related posts:

1. Cartographic Typefaces: Some examples

I am again following definitions of GIS that can be found everywhere. Basically, GIS is defined as a combination of three following main components: data, processes and output, all controlled by specific actors (or users if you may). The P*GIS thus far has been concentrated on the output, the production of visual displays by not a specific actor but rather the public, as well as the initial goal statement of what is to be produced. And this is why I argue that P*GIS has not yet reached a maturity level as described in the early days of P*GIS. But there is a lot of hope in the horizon.

The idea of data as a component of P*GIS has found a cozy home in what is termed Volunteer Geographic Information (VGI). The VGI idea comes within academia as a response to the widespread introduction of user-generated data to web-based mapping solutions, like Google Earth/Maps. The VGI literature discusses the collection of data by individuals, out of their own interest, to supplement data generated by other agents (either government or private agents). A good example is OpenStreetMap (website), a collaboration of individuals from across the globe to collect street network data to compete with commercial offerings. While this is not directed toward a specific sort of analysis, there are multiple uses for it, limited only to the imagination of the analyst.

The idea of processing as component of P*GIS has seen some work done by GIS services. There are numerous examples of either free, or commercial services offered. Geocoder.us (website) offers one basic GIS service: geocoding. Google, through its web APIs provides geocoding services as well,along with a route-finding algorithm. ESRI introduced the commercial ArcGIS Server, which provides the full functionality of their GIS software through a network, which is not restricted to the data on their servers like Google’s solution. These are just few of the examples of GIS processing functionality provided as a service to the public, either for free or a fee. But this is where P*GIS finds its other component.

Therefore, while there are different areas of research focusing on participation on processes, there is no unified P*GIS yet. And this is the opportunity. To combine (or mash) the capabilities built by these diverse actors into one, truly participatory, GIS.

Related posts:

1. Geoweb, web mapping and web GIS

In essense, the geoweb consists of locationally aware web technologies usually manifested on the world wide web. Web mapping then refers to those online applications that permit users to view or create maps on a web platform, usually with limited or no GIS analysis. Web GIS then refers to Geographic Information Systems that use web technologies as a method of communication between the elements of a GIS.

GIS has been defined numerous times by several scholars and organizations. ESRI, the world’s largest producer of GIS Software defined GIS through the lens of data, map view and model view (source). The United States Geological Survey (USGS) defines GIS as a system capable of capturing, storing, analyzing and displaying geographically referenced information (source). Nicholas Chrisman, a leading scholar in GIS, in his book Exploring Geographic Information Systems, defined GIS as “the organized activity by which people measure aspects of geographic phenomena and processes; represent these measurements, usually in the form of a computer database, to emphasize spatial themes, entities and relationships; operate upon these representations to produce more measurements and to discover new relationships by integrating disparate sources; and transform these representations to conform to other frameworks of entities and relationships. These activities reflect the larger context (institutions and cultures) in which these people carry out their work. In turn, the GIS may influence these structures.” (source) As can be seen by all the definitions, a Geographic Information System has multiple components, including human actors, that work together for a common purpose. These definitions then provide us a first step into clarifying what is the difference between a web GIS and web mapping.

If any of the above definitions are to be considered valid, then mapping is a single component in what is termed GIS, the visualization aspect. Therefore one can define web mapping applications as applications that enable the visualization of geographically referenced data through a web interface available online. While most web mapping applications today allow users to perform some spatial analysis (short path finding algorithms), this does not constitute a GIS application in my mind, as other, simple sorts of analysis are not permitted (imagine spatial queries, buffer analysis, etc).

As web mapping has been clarified, we can now explain what the geoweb is. First of all, the geoweb is considered as a collection of web applications and/or services that are geographically aware. What that means in other words is that applications that somehow are aware of geographic locations, either through geo-ip location or supplemental information as tags or EXIF data in photographs. The uses are multiple, like browsing Flickr photos by location information, load-balancing servers (redirect traffic to the nearest available server to answer requests, etc). The geoweb is therefore an amalgamation of location-aware services available to the public that provide location-based decisions to be made.

The last definition, web GIS, is all that remains. Using any definition of GIS provided above, one will notice there are multiple components. The interaction between the components is usually very direct in desktop GIS. Data, maps and analysis happen on the same computer, so communication is done internally. Enterprise GIS often allows the user to communicate with data remotely, and sometimes even analysis is done remotely. Web GIS enables the communication of all components to happen through the web, enabling diverse data, analysis algorithms, users and visualization techniques that may be hosted at any location on the web.

Of course, the above definitions are the way I understand the world of GIS and location-aware applications. If you have any suggestions or corrections to make on the above, feel free to leave me comments or email me directly.

Related posts:

1. Google Maps and GIS