ArcGIS Online Beta Screenshot

In order to make use of the new service, you either create a new account, or register your esri.com account with ArcGIS.com as well (which is what I did). You are presented with a few options when you log in, including a Gallery (viewing), Map (creation), Groups (collaboration) and My Content (management). All functionality seems to be working find from my end at least, and I encountered no errors through my limited testing of their website (way to go ESRI).

Gallery (viewing)

The Gallery allow

ArcGIS.com Gallery

s you to view maps, tools and applications provided by other users. There are some nice examples of maps out there, notably the Gulf Oil Spill Forecast map provided by ESRI and the Gulf of Mexico Oil Spill map provided by the user community (nice ones to compare and contrast). Some cool tools include the Decision Tree election application that lets people create priority maps by moving around sliders and the Recovery.gov mapping application. There is also a number of Mobile Apps available, which I have not had the pleasure of testing (if anyone wants to sponsor me for an iPhone or Windows Mobile phone.. ).

Map (creation)

Map Creation on ArcGIS.com Online

Creating your own maps is part of the offered functionality, in which you choose from a number of base maps (including Bing Maps basemaps, OpenStreetMap or the usual ESRI offerings), and then adding your own layers from online services (meaning you need someone to serve data for you). This opens up quite a few possibilities, and may (hopefully) mean that ESRI does plan on releasing a cheap (or better yet, free) way for people to serve their data (which is My Content). For now, you will have to trust ESRI in hosting your data if you wish, by visiting the “My Content” tab. Maps can be created fast and saved with permission options (including saving for personal use, share with group(s) or publicly available.

Groups (collaboration)

ArcGIS Online - My Groups

The My Groups section allows you to collaborate with other users. There are already 899 groups as of this post. Some interesting ones to note are the “ESRI Featured Content” which provides a lot of maps, applications and tools, the “National Maps for USA” by the aptly named  Federal_User_Community user, and many more. Unfortunately, the search method for groups implies you are already familiar with a group name or keywords from their description, if any. I would love to see a Groups browser, which allows you to browse groups on these different parameters: spatial extent, content theme, tools available or in development, applications available or in development, whether they seek help or simply offer services. This is perhaps the next move by ESRI, to create a community of developers rather than simply users of the technology.

My Content (data management)

ArcGIS.com My Content

My Content is where one can upload their own content, as well as manage content. Note this is an extension of simply data management, as tools, applications and maps can be managed through this page. The supported items are as follows:

1. Maps
   1. Web map
   2. ArcGIS Server Map, Image or Feature Service
   3. Map Package (mpk)
   4. Layer Package (lpk)
   5. ArcGIS Explorer Document (nmf) or Application Configuration (ncfg)
   6. ArcGIS Windows Mobile Package (wmpk)
   7. ArcMap Document (mxd and msd) – not recommended
   8. Layer File – not recommended
   9. ArcGIS Explorer Layet (nmc) – not recommended
   10. ArcGlobe Document (3dd) – not recommended
   11. ArcScene Document (sxd) – not recommended
   12. ArcReader Document (pmf) – not recommended
2. Web mapping applications
   1. ArcGIS API for Javascript, Flex or Silverlight
   2. Java Web Application Development Framework
   3. .Net Web Application Development Framework
3. Mobile applications
   1. ArcGIS Apple iPhone
   2. ArcGIS Windows Mobile
4. Tools
   1. ArcGIS Server – Geocoding, Network Analyst, Geoprocessing or Geodata Service

All in all, this looks tremendously useful and a great resource for developers and students alike. Jack Dangermond seems to be leading ESRI to the new world of GIS wonderfully, creating something closer to a web GIS than anyone else has thus far.

Related posts:

1. ArcGIS.com now online

Python Modules are code someone else has written and distributed, in order to make life easier for the rest of us. You may be familiar with the standard modules that come with Python, like math or datetime, but there are numerous more resources out there for the GIS minded developers. I will be discussing some of the modules I find essential in my work apart from the famous ArcGISScripting module by ESRI: GDAL, numpy, NetworkX, xlrd and xlwt. Let’s dive in!

GDAL – Geospatial Data Abstraction Layer

It will come a time in every GIS Professional’s career when they will need to quickly access information from a random shapefile they have, but do not have access to any GIS software or geoprocessing functionality (think of a laptop on the road, a remote machine not running Windows, etc). GDAL comes to the rescue, providing us such functionalities.

GDAL is a translator library with Python bindings that allows access to raster data using a unified abstract layer. Bundled with it is OGR, which provides similar functionality for vector data. Download it here.

A quick example of using GDAL:

import gdal
from gdalconst import *
# Open the raster dataset
dataset = gdal.Open(filename, GA_ReadOnly)
# Print the projection of the data
print dataset.GetProjection()

Using OGR:

import ogr
# Get the driver
driver = ogr.GetDriverByName('ESRI Shapefile')
# Open a shapefile
dataset = driver.Open(shapefileName, 0)

numpy – Numerical Python

I cannot think of any GIS practitioner that did not have to manipulate raster data in a peculiar way, only finding that the software at hand doesn’t allow you to perform many customized functions. If one can interpret raster data (which GDAL above helps us with), then one can use them in Python as a matrix (algebraic matrix). numpy is the best Python package out there for this sort of situation.

numpy is a package that enables n-dimensional array manipulation in Python, as essential part of any scientific endeavor. It also provides linear algebra functionalities, Fourier transforms and random number generators. Get it here.

An example of the use of numpy:

from numpy import *
# Sample IO Table data
ioSample = [[1,2], [3,4]]
# Turn into a numpy array
ioMatrix = array(ioSample)
# Find the inverse of ioMatrix
ioMatrixInv = linalg.inv(ioMatrix)

NetworkX – Complex Networks Analysis

While most GIS software out there provides the ability to build networks, sometimes it is easier to build networks quickly and dirty, without having to involve complex GIS software. An analysis of participation by space in an experiment can easily be achieved using the simple, yet powerful NetworkX module.

NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. It is hosted by the Los Alamos National Laboratory, and sees active development (presumably sponsored somehow by Los Alamos). Download it here.

A quick sample is shown below:

import networkx as nx
# Create a graph
g = nx.Graph()
# Populate the graph
g.add_node(1)
g.add_node(2)
g.add_node(3)
# Create edges
g.add_edge(1,2)
g.add_edge(1,3)
# Print the neighbors of node 1 (returns 2)
print g.neighbors(1)

xlrd – Excel™ File Reader

All GIS practitioners have been sent “GIS data” in an Excel file, either a geocoding result or GPS waypoints, or anything similar. While ideally whoever sent the data would be educated on why it is a bad idea, most often we have to deal with the data without any additional help. xlrd comes into play, allowing you to read the said Excel formatted data into Python with little effort.

xlrd is a Python module that allows one to read Excel files without the need of Microsoft Excel or Windows. It provides access to XLS files for Microsoft Office 2003 or earlier. Download it here.

A quick example that will read an XLS file and print it to screen:

import xlrd
# Open the Excel file
book = xlrd.open_workbook("excelFile.xls")
# Read the first sheet in the Excel workbook
sheet = book.sheet_by_index(0)
# Read the first row from column A to E
rowValues = sheet.row_values(0, start_colx=0, end_colx=4)
# Print the row values
for value in rowValues:
    print value

xlwt – Excel™ file writer

Business requirements often want results in an Excel file, so some other person, in another department, can run some sort of analysis on the data. Building a distance matrix is fine, but Joe from accounting is using complicated Excel spreadsheets and does not want to bother with database connections or DBF files. This is a situation in which xlwt excels, writing data to an Excel spreadsheet without the need of Excel or manipulation by the mouse.

xlwt is a Python module that, similarly to xlrd mention above, allows for cross platform Excel file creation without the need of Microsoft Office. Download it here.

An example follows:

import xlwt
# Create a new workbook
book = xlwt.Workbook()
# Add a new sheet
sheet = book.add_sheet("My Sheet")
# Write the number 5 in the first row, first column
sheet.write(0, 0, 5)
# Save the file
book.save("myExcelFile.xls")

These are my essential Python modules for GIS development. Notice I left quite a few behind, especially those relating to web-based GIS developments. Those will appear in another post in the future. But please, do share your thoughts on what Python modules are essential for your GIS work.

Related posts:

1. Manipulating Excel files using Python part 1: Reading Excel Files

The new ESRI Support website

The ESRI family of websites have been seeing a lot of new updates as of late. The screenshot on the right is of the support site that ESRI offers, which hadn’t changed in quite a while. Notice the changes, including the lack of links to the web help (which has proven so helpful in the past to show people help files), and no direct link to the forums prominent in the page (it is located on the left-hand side navigation bar). Instead, people are directed to the new ESRI Resources site, which appears to be the new organization of resources for ESRI products. It is nice to see ESRI trying to separate the official support requests from the support resources, even though I am not sure which design will be more useful to the majority of users.

The new ArcGIS Resource Center

The new resources page, for those that did not notice it, is no longer on the esri.com domain, but rather moves to the arcgis.com domain we discussed in the previous post. In the meantime, arcgis.com is still under construction, which the “Coming soon” phrase over an oblique image of the earth. If anyone has any more insights to this, please do share them on the comments section below.

Related posts:

1. ArcGIS.com now online

Executive Compensation Mashup by Dave Bouman, winner of the 2010 ESRI Mashup Challenge

ESRI announced the winners of their 2010 Mashup Challenge. 1st place goes to Dave Bouwman and his executive compensation mashup available here.

Second place winner was a social media mapper, that presents real time results from social media websites to a map (see it live here), which seems like a great time killer to me.

Congratulations to all the winners and applicants, and many thanks to ESRI for running this challenge and allowing people to be creative with their products.

Related posts:

1. ESRI Updating Website ahead of new release

ArcGIS.com Online Screenshot from 03/23/10

ArcGIS.com is now online, with a new look and design, and everyone’s favorite web phrase: “Coming Soon…“. A little bit of searching around the internet reveals some information from Vector1Media regarding a cloud-version of ArcGIS in collaboration with Amazon, providing on-the-go GIS functionality.

This is a dramatic change from the previous functionality of the ArcGIS.com domain, which redirected to the esri.com section about ArcGIS. The Internet Archive has a number of older snapshots available here.

Note that the site is currently only available to ESRI employees, that may actually have to request access too. Curiously, the http://maps.arcgis.com/ domain mentioned in this ESRI Blog entry redirects to the same page, even though the screenshots seem to differ. This of course will be different from ArcGIS Online, which is available now in Beta.

Related posts:

1. ESRI Updating Website ahead of new release

The ArcToolbox

The ArcToolbox of ArcGIS 9.3

The ArcToolbox is a collection of tools provided by ESRI that achieve a number of geoprocessing operations. They are logically organized in themes based on the functionality, and are available to use without any customization. These operations can be very powerful and achieve great results. As you can see from the image on the right, there is a number of tools available to you to use. Depending on the extensions you have installed, you will see some different results, but the Geocoding Tools we mentioned above are available with any kind of license. To use any of these geoprocessing tools, you simply double-click on its name, and a window will pop up asking you for the required inputs and outputs. Pay attention to all the options available to you, and make use of the help button to clarify what the operation does, and how it is achieved.

The Model Builder

The Model Builder Window

The Model Builder is a wonderful program written by ESRI that allows you to graphically define geoprocessing operations. Using a simple tool, a user can drag and drop operations and data on a canvas, and connect them with directional arrows (signifying direction of data). On the example on the right, we follow the Address Geocoding operation. There are three data inputs for the operation, including an address locator, address fields to be used, and finally the input table with the addresses. There is one more data element in the picture, the output feature, on the far right. Note that all data are represented by ovals. The operation, which is in the middle, receives data from all the arrows pointing to it, and returns data to the arrow pointing outwards. Notice that the operation is in a rectangular box rather than an oval, to distinguish it.

The Command Line

The Command Line Window

The command line window is one more way to achieve geoprocessing when one is using ArcGIS. The window simply allows one to type commands, an hit enter to execute them. Some users are more familiar with this interface, and like the speed of which you can type operations. Typing a few letters in the top box will give you a list of all commands starting with those letters. Pressing space will bring the tooltip shown in the top of the image on your right. As you can see, it tells you what the tool expects to run. In this case, it expects an input table (in_table), an address locator (address_locator), the address fields (in_address_fields) and where to save the output (output_feature). Optionally, you can set the out_relation_type (notice the curly braces this time).

Python Scripting

Python Scripting Geocoding

The perhaps most complex way to perform geocoding is done through Python scripting. The example on the right actually presents geoprocessing code. What happens when one uses Python Scripting to create geoprocessing operations is similar with the command line option talked about earlier, with some slight changes. Python scripting gives you the option to interact with the data deeper. What this means is that the user can now access an individual point by latitude and longitude. And since all vector spatial data are built from points, this means you have deep access to any spatial data using Python (to see more about Accessing Geometries using the Geoprocessor click here). Also, as Python is a general use scripting language, you can find reusable code by other users (in the form of modules) to achieve things like reading Excel files (click here for an overview).

And that is what geoprocessing is. Operations that work on spatial data. In the ArcGIS realm, that means either using ArcToolbox, the Model Builder, the Command Line interface or creating your own Python Scripts to achieve your desired results.

Related posts:

1. Understanding the Geoprocessor Programming Model part 1

The method I use to load the geoprocessor, while verbose, ensures that if anyone loads the geoprocessor before they use any of my module functions, any of their settings remain intact. This is achieved as follows:

1. Check if the arcgisscripting module was loaded.
   1. If it was not loaded, obviously the geoprocessor is not instantiated. Import, instantiate and return.
2. If the arcgisscripting module was loaded, check the variables the user created in the main program.
   1. If a variable is found to be of type ‘geoprocessor object‘, then return that object.
3. If no variable is found of type ‘geoprocessor object‘, create it and return it.

The code can be found below. Please review and use as you wish. If you have any comments or questions about it, please leave a comment.

"""
Loads the geoprocessor safely, ensuring that it is not loaded
twice (to avoid overwriting parameters).

Usage: load()

Returns: a geoprocessor object.

Note: Geoprocessor Version returned is 9.3

"""
import sys

def load(version=None):
    # Check if the arcgisscripting module was loaded
    if "arcgisscripting" not in sys.modules:
        # The arcgisscripting module was never loaded Try to load it.
        print "arcgisscripting not loaded"
        print "Trying to load arcgisscripting"
        try:
            import arcgisscripting
        except ImportError, errorDetail:
            sys.exit("Error loading arcgisscriping. Is ArcGIS installed?\n" + errorDetail)
        # Try to create a geoprocessor instantiation
        print "Trying to create the geoprocessor instance"
        try:
            gp = arcgisscripting.create(9.3)
        except:
            sys.exit("Error creating geoprocessor object. Exiting\n" + str(sys.exc_info()))
        # Return the geoprocessor instance
        print "Done!"
        return gp
    else:
        # arcgisscripting was loaded. Is the geoprocessor instantiated?
        print "arcgisscripting was already loaded. Searching for geoprocessor..."
        # Get the global variables
        globalVars = vars(sys.modules["__main__"])
        for item, value in globalVars.iteritems():
            # Loop through global variables, ignore internals
            if not item.startswith("__"):
                # Check the type of the variable
                if type(globalVars[item]).__name__ == 'geoprocessing object':
                    # If the variable is the geoprocessor, return it
                    print "Found geoprocessor instance as", item
                    return globalVars[item]
        # If we are here, the module is loaded but no gp instantiated
        # Try to create a geoprocessor instantiation
        try:
            print "arcgisscripting loaded but not instantiated. Instantiating now..."
            gp = sys.modules["arcgisscripting"].create(9.3)
        except:
            sys.exit("Error creating geoprocessor object. Exiting\n" + str(sys.exc_info()))
        # Return the geoprocessor instance
        return gp
 

Related posts:

1. Understanding the Geoprocessor Programming Model part 1

The image below shows how the labs features are presented:

Related posts:

1. Google Maps API now supports multiple languages

1. A free geoprocessor: While this is unlikely to happen in the foreseeable future, there are elements of the geoprocessor that can be made freely available, either as a separate Python module or through a license check for parts of the objects. Specifically, I am referring to some of the List* methods (ListRasters(), ListFeatureClasses(), etc.) that do not require deep access to data (this would include ProductInfo(), validators, and hopefully even the Describe object). Why would ESRI want to do this? To ensure that limited functionality is available to everyone independent of cost. Why restrict people from seeing the power of the geoprocessor? Let them join the fun, and perhaps become customers too.
2. Path traversal options: The List*() methods come in the list again, with a slight change. Since many users use directory structures to store data that are deep (subfolder within a subfolder within a folder), it would be nice if the geoprocessor could read through them all and return refernces to them based on the current folder. Before one jumps in to say that the Python os module can help, please remember that most users are not aware of os.walk, and trying to deal with ERI GRIDS (which are a folder themselves) presents a challenge even for experienced Python programmers. Why would ESRI want to do this? To make life easier for the majority of users out there with data in directory trees.
3. Expose numpy in the geoprocessor: The geoprocessor as is right now (9.3.1) uses the excellent numpy module to perform matrix algebra (think of raster manipulation). Yet, when one wishes to run numpy commands, one needs to manually read raster files with GDAL, import them as numpy arrays (default), perform operations, and translate back to raster. ESRI must have modules for dealing with this, and we want them. Why would ESRI want to do this? Right now, raster manipulation through Python is done outside the geoprocessor. Most people turn to open source tools to manipulate data, which leads to less and less users relying on ESRI for this. Why pay when free software will do it? The capability is there, and we need to access it too.
4. A faster starting geoprocessor: Right now, on my dual-core, 4GB RAM machine, creating the geoprocessing object takes a noticeably long time (in the orders of a few seconds). Why is that the case today? If the geoprocessor is written as C module, the bottleneck is somewhere else. Regardless, this needs to be address, especially for using an interactive shell or multiple parallel processes involving the geoprocessor. Why would ESRI want to do this? Simply because a faster startup time is time-saving and psychologically makes people feel better about the tools at hand.
5. Multi-core aware geoprocessor: Right now, ESRI operates on one core per command (on the desktop). Similarly, the geoprocessor is a one-core operation. Yet many operations can run in parallel, and indeed, many people on the ESRI forums try endlessly to achieve multi-core operations. This is not easy of course, and not many people know how to do it. Why would ESRI want to do this? Increase in speed for one, but most importantly, to introduce a way to achieve this for everyone that wants it. Right now, there are as many solutions to this as there are people needing it. One simple and good way to achieve this would allow for unification of effort, and some good documentation be written.
6. Exposure to the desktop application when run through it: If you remember the good old VBA days (which will not be included on any version after 10, beware), you could manipulate the application via scripting. You could add data to a map, reference the 3rd item on the table of contents, etc. We want a return to this regime, but this time with Python. Why would ESRI want to do this? Something will need to replace VBA, and what better language than the chosen one for geoprocessing scripting? Also, access to such functionality opens up development to a larger audience, which can help bolster the industry by itself.
7. Make every method return a Python list for multiple data: The cursor methods in ArcGIS right now return an enumeration object. While this is understandable (ESRI doesn’t want people manipulating the order of items or inserting items in the middle of the list), it is not to the best interest of the geoprocessing model to return both enumeration objects and Python lists. At the very least, the enumeration object should have more Pythonic methods, like append() to add an element in the end or pop() to remove the last item. Why would ESRI want to do this? Consistency. There was an attempt to become more Pythonic in 9.3, so why not keep up with the theme and be more consistent at the same time?

Do you have any suggestions to include in this list? Please leave a comment. Or if you think I am off my head and don’t know what I am talking about.

Related posts:

1. Understanding the Geoprocessor Programming Model part 1

The Describe Realm

As mentioned earlier, the geoprocessor gives us the ability to access or create data to perform various operations. These operation vary in type and content, but more often than not are performed on data we are not intimately familiar with. This can become a problem quite quickly, as we need to know the names of fields in the datasets, as well as other characteristics (the type of data we are dealing with, spatial references, geographical extent and other). The solution to this lack of knowledge comes through the describe object which provides users with methods to identify all unknown information beforehand. In the most basic form of usage, one uses the describe object on a dataset, accessing the bottom left box in the diagram, the Describe Object Properties. The Python code to achieve this follows:

import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

dataset = r"c:\data\testData.shp"
# The location of the data. This is usually
#   not hardcoded but discovered with List methods

dsc = gp.Describe(dataset)
# Create a describe object for the dataset

print dsc.DataType
# Print the data type of our dataset

print dsc.CatalogPath
# Print the path to the dataset

With the above simple example, we managed to read the Data Type and Catalog Path of a dataset. While this seems simple, it can become a very powerful tool, when on considers the possible return values (listed below).

As you can see, the return values as listed above correspond to different objects in our diagram (the boxes in the first image above). This means that it would be trivial for us to move between the first access we have into the Describe realm (through the Describe object) and move to any other box of our choice when we are not familiar with what possible data we have.

Retrieving Information about data: Feature Classes

FeatureClass Object Properties

The most commonly used types of data in GIS now seems to be feature classes, either standalone (in the form of shapefiles) or in a geodatabase. As such, we will use the example of a feature class to retrieve information about out data. This is of course not the only type of data one can use, so please refer to the image on the top for more information.

As you can see from the diagram on the left, feature classes have specific information associated with them, including what feature type they are of, their shape type, and the shape field name we used in part two of this series. We will dive right into it by trying to identify what shape type our feature class has, and what type of feature it is.

import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

dataset = r"c:\data\testData.shp"
# The location of the data. This is usually
#   not hardcoded but discovered with List methods

dsc = gp.Describe(dataset)
# Create a describe object for the dataset
# As we know the dataset is of type FeatureClass,
#   all the properties of the FeatureClass object
#   are available to us now.

print dsc.ShapeType
# Print what type of shape we have
#   (point, polygon, polyline, etc)

print dsc.FeatureType
# Print what type of features are stored
#   (simple, annotation, dimension, etc)

With the above code we can identify what type of shape we have and the type of feature stored. For possible values to these properties, please check this help file.

Table Properties

In the FeatureClass Properties object image above, on the bottom, you can see that there are also Table Properties and Dataset Properties listed. This means that when you have a feature class, you automatically gain access to those properties as well (as a feature class has them embedded). Therefore, we can use this to our advantage to further retrieve information about our feature class, like the name of all fields within the feature class. The example below shows you how to achieve this.

import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

data = r"c:\data\testData.shp"
# The location of the data. This is usually
#   not hardcoded but discovered with List methods

dsc = gp.Describe(data)
# Create a describe object for the data
# As we know the dataset is of type FeatureClass,
#   all the properties of the FeatureClass object
#   and inherited objects are available to us now.

dataFlds = dsc.Fields
# Retrieve a list of fields in the table of the
#   feature class as a Python list. Note the fields
#   are of Field Object type.

for fld in dataFields:
    print fld.Name

Dataset Properties

What he have done on the above example is run the ListFields() method mentioned in an older article (Using the Geoprocessor’s List Methods). Similarly, one can follow the Dataset Properties object (shown on your right) to access information like the extent of the data, or the spatial reference system. Don’t forget, both of those return an object, which has a corresponding box in the geoprocessor programming model diagram, so you need to treat them as such.

In this installment, I have shown you how one can use the Describe object the geoprocessor makes available to us to retrieve information about our data. While I did not provide complete examples of every single possible option available, hopefully by reading this post you are able to navigate your options better to achieve your goals.

As always, comments are encouraged and I am willing to answer any questions you may have.

Related posts:

1. Accessing Geometries using the Geoprocessor (Updated)