Python Modules are code someone else has written and distributed, in order to make life easier for the rest of us. You may be familiar with the standard modules that come with Python, like math or datetime, but there are numerous more resources out there for the GIS minded developers. I will be discussing some of the modules I find essential in my work apart from the famous ArcGISScripting module by ESRI: GDAL, numpy, NetworkX, xlrd and xlwt. Let’s dive in!

GDAL – Geospatial Data Abstraction Layer

It will come a time in every GIS Professional’s career when they will need to quickly access information from a random shapefile they have, but do not have access to any GIS software or geoprocessing functionality (think of a laptop on the road, a remote machine not running Windows, etc). GDAL comes to the rescue, providing us such functionalities.

GDAL is a translator library with Python bindings that allows access to raster data using a unified abstract layer. Bundled with it is OGR, which provides similar functionality for vector data. Download it here.

A quick example of using GDAL:

import gdal
from gdalconst import *
# Open the raster dataset
dataset = gdal.Open(filename, GA_ReadOnly)
# Print the projection of the data
print dataset.GetProjection()

Using OGR:

import ogr
# Get the driver
driver = ogr.GetDriverByName('ESRI Shapefile')
# Open a shapefile
dataset = driver.Open(shapefileName, 0)

numpy – Numerical Python

I cannot think of any GIS practitioner that did not have to manipulate raster data in a peculiar way, only finding that the software at hand doesn’t allow you to perform many customized functions. If one can interpret raster data (which GDAL above helps us with), then one can use them in Python as a matrix (algebraic matrix). numpy is the best Python package out there for this sort of situation.

numpy is a package that enables n-dimensional array manipulation in Python, as essential part of any scientific endeavor. It also provides linear algebra functionalities, Fourier transforms and random number generators. Get it here.

An example of the use of numpy:

from numpy import *
# Sample IO Table data
ioSample = [[1,2], [3,4]]
# Turn into a numpy array
ioMatrix = array(ioSample)
# Find the inverse of ioMatrix
ioMatrixInv = linalg.inv(ioMatrix)

NetworkX – Complex Networks Analysis

While most GIS software out there provides the ability to build networks, sometimes it is easier to build networks quickly and dirty, without having to involve complex GIS software. An analysis of participation by space in an experiment can easily be achieved using the simple, yet powerful NetworkX module.

NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. It is hosted by the Los Alamos National Laboratory, and sees active development (presumably sponsored somehow by Los Alamos). Download it here.

A quick sample is shown below:

import networkx as nx
# Create a graph
g = nx.Graph()
# Populate the graph
g.add_node(1)
g.add_node(2)
g.add_node(3)
# Create edges
g.add_edge(1,2)
g.add_edge(1,3)
# Print the neighbors of node 1 (returns 2)
print g.neighbors(1)

xlrd – Excel™ File Reader

All GIS practitioners have been sent “GIS data” in an Excel file, either a geocoding result or GPS waypoints, or anything similar. While ideally whoever sent the data would be educated on why it is a bad idea, most often we have to deal with the data without any additional help. xlrd comes into play, allowing you to read the said Excel formatted data into Python with little effort.

xlrd is a Python module that allows one to read Excel files without the need of Microsoft Excel or Windows. It provides access to XLS files for Microsoft Office 2003 or earlier. Download it here.

A quick example that will read an XLS file and print it to screen:

import xlrd
# Open the Excel file
book = xlrd.open_workbook("excelFile.xls")
# Read the first sheet in the Excel workbook
sheet = book.sheet_by_index(0)
# Read the first row from column A to E
rowValues = sheet.row_values(0, start_colx=0, end_colx=4)
# Print the row values
for value in rowValues:
    print value

xlwt – Excel™ file writer

Business requirements often want results in an Excel file, so some other person, in another department, can run some sort of analysis on the data. Building a distance matrix is fine, but Joe from accounting is using complicated Excel spreadsheets and does not want to bother with database connections or DBF files. This is a situation in which xlwt excels, writing data to an Excel spreadsheet without the need of Excel or manipulation by the mouse.

xlwt is a Python module that, similarly to xlrd mention above, allows for cross platform Excel file creation without the need of Microsoft Office. Download it here.

An example follows:

import xlwt
# Create a new workbook
book = xlwt.Workbook()
# Add a new sheet
sheet = book.add_sheet("My Sheet")
# Write the number 5 in the first row, first column
sheet.write(0, 0, 5)
# Save the file
book.save("myExcelFile.xls")

These are my essential Python modules for GIS development. Notice I left quite a few behind, especially those relating to web-based GIS developments. Those will appear in another post in the future. But please, do share your thoughts on what Python modules are essential for your GIS work.

Related posts:

1. Manipulating Excel files using Python part 1: Reading Excel Files

The method I use to load the geoprocessor, while verbose, ensures that if anyone loads the geoprocessor before they use any of my module functions, any of their settings remain intact. This is achieved as follows:

1. Check if the arcgisscripting module was loaded.
   1. If it was not loaded, obviously the geoprocessor is not instantiated. Import, instantiate and return.
2. If the arcgisscripting module was loaded, check the variables the user created in the main program.
   1. If a variable is found to be of type ‘geoprocessor object‘, then return that object.
3. If no variable is found of type ‘geoprocessor object‘, create it and return it.

The code can be found below. Please review and use as you wish. If you have any comments or questions about it, please leave a comment.

"""
Loads the geoprocessor safely, ensuring that it is not loaded
twice (to avoid overwriting parameters).

Usage: load()

Returns: a geoprocessor object.

Note: Geoprocessor Version returned is 9.3

"""
import sys

def load(version=None):
    # Check if the arcgisscripting module was loaded
    if "arcgisscripting" not in sys.modules:
        # The arcgisscripting module was never loaded Try to load it.
        print "arcgisscripting not loaded"
        print "Trying to load arcgisscripting"
        try:
            import arcgisscripting
        except ImportError, errorDetail:
            sys.exit("Error loading arcgisscriping. Is ArcGIS installed?\n" + errorDetail)
        # Try to create a geoprocessor instantiation
        print "Trying to create the geoprocessor instance"
        try:
            gp = arcgisscripting.create(9.3)
        except:
            sys.exit("Error creating geoprocessor object. Exiting\n" + str(sys.exc_info()))
        # Return the geoprocessor instance
        print "Done!"
        return gp
    else:
        # arcgisscripting was loaded. Is the geoprocessor instantiated?
        print "arcgisscripting was already loaded. Searching for geoprocessor..."
        # Get the global variables
        globalVars = vars(sys.modules["__main__"])
        for item, value in globalVars.iteritems():
            # Loop through global variables, ignore internals
            if not item.startswith("__"):
                # Check the type of the variable
                if type(globalVars[item]).__name__ == 'geoprocessing object':
                    # If the variable is the geoprocessor, return it
                    print "Found geoprocessor instance as", item
                    return globalVars[item]
        # If we are here, the module is loaded but no gp instantiated
        # Try to create a geoprocessor instantiation
        try:
            print "arcgisscripting loaded but not instantiated. Instantiating now..."
            gp = sys.modules["arcgisscripting"].create(9.3)
        except:
            sys.exit("Error creating geoprocessor object. Exiting\n" + str(sys.exc_info()))
        # Return the geoprocessor instance
        return gp
 

Related posts:

1. Understanding the Geoprocessor Programming Model part 1

The Describe Realm

As mentioned earlier, the geoprocessor gives us the ability to access or create data to perform various operations. These operation vary in type and content, but more often than not are performed on data we are not intimately familiar with. This can become a problem quite quickly, as we need to know the names of fields in the datasets, as well as other characteristics (the type of data we are dealing with, spatial references, geographical extent and other). The solution to this lack of knowledge comes through the describe object which provides users with methods to identify all unknown information beforehand. In the most basic form of usage, one uses the describe object on a dataset, accessing the bottom left box in the diagram, the Describe Object Properties. The Python code to achieve this follows:

import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

dataset = r"c:\data\testData.shp"
# The location of the data. This is usually
#   not hardcoded but discovered with List methods

dsc = gp.Describe(dataset)
# Create a describe object for the dataset

print dsc.DataType
# Print the data type of our dataset

print dsc.CatalogPath
# Print the path to the dataset

With the above simple example, we managed to read the Data Type and Catalog Path of a dataset. While this seems simple, it can become a very powerful tool, when on considers the possible return values (listed below).

As you can see, the return values as listed above correspond to different objects in our diagram (the boxes in the first image above). This means that it would be trivial for us to move between the first access we have into the Describe realm (through the Describe object) and move to any other box of our choice when we are not familiar with what possible data we have.

Retrieving Information about data: Feature Classes

FeatureClass Object Properties

The most commonly used types of data in GIS now seems to be feature classes, either standalone (in the form of shapefiles) or in a geodatabase. As such, we will use the example of a feature class to retrieve information about out data. This is of course not the only type of data one can use, so please refer to the image on the top for more information.

As you can see from the diagram on the left, feature classes have specific information associated with them, including what feature type they are of, their shape type, and the shape field name we used in part two of this series. We will dive right into it by trying to identify what shape type our feature class has, and what type of feature it is.

import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

dataset = r"c:\data\testData.shp"
# The location of the data. This is usually
#   not hardcoded but discovered with List methods

dsc = gp.Describe(dataset)
# Create a describe object for the dataset
# As we know the dataset is of type FeatureClass,
#   all the properties of the FeatureClass object
#   are available to us now.

print dsc.ShapeType
# Print what type of shape we have
#   (point, polygon, polyline, etc)

print dsc.FeatureType
# Print what type of features are stored
#   (simple, annotation, dimension, etc)

With the above code we can identify what type of shape we have and the type of feature stored. For possible values to these properties, please check this help file.

Table Properties

In the FeatureClass Properties object image above, on the bottom, you can see that there are also Table Properties and Dataset Properties listed. This means that when you have a feature class, you automatically gain access to those properties as well (as a feature class has them embedded). Therefore, we can use this to our advantage to further retrieve information about our feature class, like the name of all fields within the feature class. The example below shows you how to achieve this.

import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

data = r"c:\data\testData.shp"
# The location of the data. This is usually
#   not hardcoded but discovered with List methods

dsc = gp.Describe(data)
# Create a describe object for the data
# As we know the dataset is of type FeatureClass,
#   all the properties of the FeatureClass object
#   and inherited objects are available to us now.

dataFlds = dsc.Fields
# Retrieve a list of fields in the table of the
#   feature class as a Python list. Note the fields
#   are of Field Object type.

for fld in dataFields:
    print fld.Name

Dataset Properties

What he have done on the above example is run the ListFields() method mentioned in an older article (Using the Geoprocessor’s List Methods). Similarly, one can follow the Dataset Properties object (shown on your right) to access information like the extent of the data, or the spatial reference system. Don’t forget, both of those return an object, which has a corresponding box in the geoprocessor programming model diagram, so you need to treat them as such.

In this installment, I have shown you how one can use the Describe object the geoprocessor makes available to us to retrieve information about our data. While I did not provide complete examples of every single possible option available, hopefully by reading this post you are able to navigate your options better to achieve your goals.

As always, comments are encouraged and I am willing to answer any questions you may have.

Related posts:

1. Accessing Geometries using the Geoprocessor (Updated)

In this post I will finish covering the bottom left part of the model that deals with direct access to features and their geometry. In order to do that, we will discuss cursors, features and geometries.

Spatial Data Primer

Before we begin discussing deep access to data, we need to define what spatial data are. In essence, there are two types of spatial data, vector and raster data. An easy way to understand the difference is to imagine raster data as images, while vector data are defined by points, lines and polygons. What is special about spatial data is their dimension of space, or location. Raster data offer extensive coverage of an area, in which our spatial data offer a regular grid of cells that provide data. Space in this case is depended on the location of the cell in the raster file (and therefore implicit location). Vector files on the other hand contain explicit information about the location of things (providing coordinates). Data are then associated with the locations of features. We will be dealing with vector data mostly, so a further investigation is warranted.

In order to represent the world in spatial data, we choose a type of vector data we want to use (for example, choosing points to represent trees, lines to represent road networks, etc). Each of the features in the world get assigned two types of data: spatial characteristics (for a point, an x,y pairs, for a line, a series of points, etc.) and non-spatial characteristics (think of them as attributes). A file that contains all the trees in a neighborhood then includes a list of all the trees, with each tree having a location attribute, and a set of attributes that describe the tree (like type of tree, or age of tree, etc). Therefore, spatial data are defined as follows:

• A file dealing with a specific type of feature in the world.
  • The file has a list of features that all satisfy the requirement of belonging in that type of feature.
    • Each feature contains a spatial footprint for where it is located.
    • Along with location, each feature contains zero or more characteristics that are aspatial but refer to the feature itself.

Similarly, when one tries to access the geometry or the characteristics of a specific feature in our data, one has to first access the file we need, within the file, identify the right feature we need, and then eventually, within the feature find the geometry and/or the characteristics associated with it.

Accessing Data Using the Geoprocessor

From the Geoprocessor to Geometry

The image above shows the path one needs to travel within the Geoprocessor to access a Geometry. The cyan box on the left represents the commands available to the geoprocessor object, as discusses in the previous post. The commands that include the word Cursor have an arrow to their right, that show they can access rows, which is actually a representation of the spatial data file in Python. From there on, one can access a single row (the third box from the left), which can eventually lead to the Geometry object on the left. So in the geoprocessing world, in order to access a geometry (or spatial characterics of a feature), we need to do the following:

1. Create the geoprocessor object.
2. Call the appropriate cursor command to receive a rows object, which is a Python representation of the file we need to access.
3. Within the rows object, we call the Next() or NewRow() command to receive an individual feature, represented as a single row.
4. The row object allows us to directly access characteristics of the file, including the Geometry object. Please note that the characteristics are actually variables now, and not methods.

A question people often ask is how one knows which path to follow when they need to access a specific object (or box in the diagram). If you are not intimately familiar with the model, there is one method to follow: Identify the object you want, and then follow the arrows in reverse to find the methods that can get you there. Once you have those steps reverse them and you have a workflow. Remember, you will always need to end up in the cyan, geoprocessor object, as this is your only starting point.

From the Geoprocessor to Spatial File Access

There are three methods to access your spatial file, as shown in the diagram, explained below:

• InsertCursor: The insert cursor allows you to access a spatial file with the sole purpose of adding new features (or rows). Use this cursor to add more features to a file.
• SearchCursor: The search cursor allows you to access a spatial file with the sole purpose of reading features (rows). Use this cursor to read the characteristics of each feature.
• UpdateCursor: The update cursor allows you to access a spatial file in order to update values in each feature (row), or completely remove it. Use this cursor to update characteristics (including geometry) of a feature, or to remove the feature itself.

The required parameter for each of these cursor is the file you want to access, called InputValue in the diagram. Here is an example of how to use each cursor:

import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

fcWrite = gp.InsertCursor(r"c:\data\testWrite.shp")
# Create an insert cursor that points to the testWrite.shp file.
# The returned value fcWrite is actually a rows object.
# You are now able to add new features to the file.

fcRead = gp.SearchCursor(r"c:\data\testRead.shp")
# Create a search cursor that points to the testRead.shp file.
# The returned value fcRead is actually a rows object.
# You are now able to read the information about features from the file.

fcUpdate = gp.UpdateCursor(r"c:\data\testUpdate.shp")
# Create an update cursor that points to the testUpadate.shp file.
# The returned value fcUpdate is actually a rows object.
# You are now able to update values in features, of completely remove them from the file.

If you follow the above examples, you manage to escape from the geoprocessor object (the cyan box) and move to the rows object, the first yellow box on the left. Note that the rows object has different methods available depending on how you access it.

From the Rows Object to a Single Row

From the file to a single row

Now that you have access to the rows object, you can continue accessing the file by individual feature (or row). Depending on the Cursor method you used, you will have the following options available to you:

• SearchCursor Rows Object: This provides you with read-only access to the elements/features of your file. As such, there are two methods available: Next() and Reset(). As you may infer from their name, Next() will provide to you the next feature as a row object, while the Reset() method simply returns the pointed to the specific row in the beginning of the file.
• InsertCursor Rows Object: As the InsertCursor() method provides the ability to add new features in the file, the methods available are NewRow(), InsertRow() and Reset(). This creates some confusion, but it is easy to distinguish what happens when you understand the internals. NewRow() creates a new and empty row object in memory, which allows you to create your new feature. This row object remains in memory until one calls the InsertRow() function, which will save the row object to your file. The Reset() method functions as with the SearchCursor Rows object explained above.
• UpdateCursor Rows Object: As the UpdateCursor() provides you the most functionality, it comes with more methods: Next(), Reset(), UpdateRow() and DeleteRow(). The Next() and Reset() methods function the same way as with the SearchCursor Rows object. UpdateRow() is similar to the InsertRow() method of the InsertCursor Rows object. Given a row object, it will update the feature the cursor is set on. Similarly, the DeleteRow() method will delete the current row object the cursor points to.

Examples on how to use the commands follow:

Example 1: Writing a new feature

# Example 1: Create a new Row
import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

fcWrite = gp.InsertCursor(r"c:\data\testWrite.shp")
# Create an insert cursor that points to the testWrite.shp file.
# The returned value fcWrite is actually a rows object.

fcWrite.Reset()
# Reset the cursor to the rows object

fcRow = fcWrite.NewRow()
# Create a new row object in memory, saved in the fcRow variable

# Manipulate the row here.

fcWrite.InsertRow(fcRow)
# Save the row object back to our file

Example 2: Reading a feature

 # Example 2: Read the n feature of a file
 import arcgisscripting
 # Import the module provided by ESRI

 gp = arcgisscripting.create(9.3)
 # Create the geoprocessor object

 fcRead = gp.InsertCursor(r"c:\data\testRead.shp")
 # Create an insert cursor that points to the testRead.shp file.
 # The returned value fcRead is actually a rows object.

 fcRead.Reset()
 # Reset the cursor to the rows object

 fcRow = fcRead.Next()
 # Read the first row of the file and save it to the fcRow variable

 # Read the information for the feature here

# If you need to find a specific feature, you need to continue your search
# Use a while loop to achieve this
while fcRow:
    # Test whether this feature is the one you want
    # When you found the feature, you can use break to end the loop
    # If the feature is not the right one, keep trying
    fcRow = fcRead.Next()
 

Example 3: Updating and Deleting Rows

# Example 3: Updating and Deleting Rows
import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

fcUpdate = gp.UpdateCursor(r"c:\data\testUpdate.shp")
# Create an update cursor that points to the testUpadate.shp file.
# The returned value fcUpdate is actually a rows object.

fcUpdate.Reset()
# Reset the cursor to the rows object

fcRow = fcUpdate.Next()
# Receive the first row

# Loop through the rows until you find one to delete and one to update
while fcRow:
    # If this is the row that needs deleting, then
        fcUpdate.DeleteRow(fcRow)
        # Call the DeleteRow() method with the current row as a parameter
    # If this is the row that needs updating, then
        # Perform the update to the row
        fcUpdate.UpdateRow(fcRow)
        # Call the UpdateRow() method with the current row as a parameter

Manipulating Individual Features and Geometries

If you followed the above examples, then you are aware on how one can access an individual feature from a file. The question then remains on how one manipulates that row. Below are some explanations on how one can perform this.

Individual Features (row) and their Geometry

As with the Rows object, there are different versions of the Row object depending on who calls it. The two possibilities are:

• Row object derived from a SearchCursor: This object provides the GetValue() method that allows one to read values associated with a field. Along with the method, there is a collection of read-only variables that represent the fields/attributes the row represents.
• Row object derived from an InsertCursor or UpdateCursor: This object provides what the read-only version provides, with the variables representing fields/attributes being write-enabled as well. The SetValue() method is also provided to allow one to set new values to fields.

The Field Object

Before we continue on with the idea of geometries, we need to show how one can access the field names. Often we know the names of fields/attributes because we are familiar with the dataset or have access to the metadata. Another way to get familiar with the names of the fields though exists, and is through the ListFields() method available to the geoprocessor object. This command will return a Python list that includes all the attributes available to you, as Field objects. What this object provides to you is a set of variables that are read-only and make available to you information like the name, alias, domain, editable ability and more information about the field. Please have a look at the diagram on the right for a full listing of information included in a Field object.

# Example of finding out the fields available in a file
import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

fieldsList = gp.ListFields(r"c:\data\test.shp")
# Call the ListFields() method on the test.shp file
# The returned list is saved in fieldsList

for field in fieldsList:
# Loop through all the field
    print field.Name
# Print the name of the field

This is one way to also identify the field that includes the geometry. Unfortunately, depending on the type of file you have (or database for that matter), the name may change. There is a trick to find the actual name of the field that stores geometry, but it will be covered in another part. For now, here is how one can do it:

# Example of finding out the fields available in a file
import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

fcDsc = gp.Describe(r"c:\data\test.shp")
# Get the descriptor of the file
geometryField = fcDsc.ShapeFieldName
# Get the geometry(shape) field name from the description

As you now know what the name of the geometry field is, you can access is as mentioned before with the GetValue() method. Similarly, you can save a geometry using the SetValue() method. It is important to remember that geometries in the ESRI world are Geometry objects of multiple parts which are held in Array objects holding one or more Point objects. This means when you use the GetValue() method, you first receive a Geometry object. Within that object, you will call the GetPart() method to receive an Array object, which in turn holds a number of Point objects, which are called through the Next() method. An example follows to help you understand this.

# Example of accessing a Geometry of a feature / print geometry type
import arcgisscripting
# Import the module provided by ESRI

gp = arcgisscripting.create(9.3)
# Create the geoprocessor object

fcDsc = gp.Describe(r"c:\data\test.shp")
# Get the descriptor of the file
geometryField = fcDsc.ShapeFieldName
# Get the geometry(shape) field name from the description

srcCursor = gp.SearchCursor(r"c:\data\test.shp")
# Create a search cursor to the file

row = srcCursor.Next()
# Grab the first row

while row:
# Loop through the features/rows
rowGeom = row.Getvalue(geometryField)
# Grab the geometry of the row

print rowGeom.Type
# Print the type of the geometry

row = srcCursor.Next()
# Grab the next row

With this installment, you should have the knowledge to navigate the Geoprocessor Programming Model on the bottom, where accessor methods for files reside. Also, you should be able to access a Geometry object for every vector dataset you have available. In the next installment we will further navigate the programming model.

Note: Thanks to Eric for catching a Python error.

Related posts:

1. Accessing Geometries using the Geoprocessor (Updated)

A question often asked when people venture into the wonderful world of Python Geoprocessing with ESRI is how one can read the programming model they make available on their website. As it may not be easily interpreted when one begins programming, I will do my best to unpack it and explain how one can use it more effectively. All images presented here are extracts from the actual model presented by ESRI on their website.

Dual Usages for the Geoprocessor

The geoprocessor within Python can be used for two different purposes: to access tools provided by ESRI to manipulate data (the tools one sees in the ArcToolbox), and to programatically manipulate data using the underlying structures provided by ESRI. To further explain this process, a little understanding of how software works is needed. A software package usually involves multiple components. At the heart of software are basic operations and data models, the core of your software. This core deals with things like accessing and saving data, manipulating numbers and the like. This functionality is usually bundled together to create the tools the software produces. In the case of ArcMap, a tool like buffer could use components about reading and writing data, coordinate transformation routines, and probably some sort of functionality that allows manipulation of spatial data. To bring it back to the geoprocessor, and why it is so much more powerful than using the software by itself, what you get access to is not only the tools ESRI has created for you by compiling multiple elements, but you also get access to a lot of those elements yourself that allows you to write your own buffer tool for example.

Understanding what the Geoprocessor is

When one talks about the geoprocessor, one is referring to a Python object built by ESRI, that allows one to call many different functions to perform operations. As in most Python code, the definitions of the object reside in a module that is provided to you. As with any other module, in order to access it through Python you need to import it. This is done by simply typing import arcgisscripting in your Python code.

import arcgisscripting
# Imports the module provided by ESRI

In this case, the module provides a single command accessible to you called create. This exposed function allows one to instantiate the geoprocessor object. Simply calling the create() command is enough to return to you a geoprocessor object. You can also pass parameters to it, but the only one at the moment is the number 9.3, to create a geoprocessor object that returns values based on the 9.3 iteration of the programming model.

gp = arcgisscripting.create(9.3)
# Instantiates the geoprocessor and assigns the name gp to it

Beginning our Travel in the Geoprocessor Programming Model

Geoprocessor Object

The box on the right represents your geoprocessor. While it seems that these are the commands available to the arcgisscripting module, that is incorrect. Therefore, in the Geoprocessor Programming Model, one always enters through the cyan box, as seen on your right. There are few interesting things to note here:

1. There are a lot of names appearing on the list.
2. Each name has a symbol ahead of it.
3. Most name have a parenthesis to their right.
4. A few of those names also have more information on the very right.

Here is how one needs to interpret this part of the diagram. The symbols on the left define what the name represents.

Geoprocessor Variables

If there is a box connected to a dash, or two boxes, then the name represents a variable. What this means is that the name represents a value. There are a few options here:

1. Box on the left of the dash: This represents a built-in variable that is read-only. You as a user have no power to change these value, but can only read it. An example is the variable called MessageCount. The value represents how many messages the geoprocessor object has thus far.
2. Box on both sides of the dash: This represents a variable you can also write too. This is a value that the software can use, and you are allowed to overwrite it with your own value. An example is the OverwriteOutput variable, which allows you to tell the geoprocessor whether it has permission to overwrite existing files.

Example usages for geoprocessor variables is below (click to expand):

import arcgisscripting
# Import the module provided by ESRI
gp = arcgisscripting.create(9.3)
# Instantiate the geoprocessor object

gp.OverwriteOutput = True
# Set the OverwriteOutput variable to True

print gp.MessageCount
# Print the number of messages on the object

Geoprocessor methods

Boxes are not the only symbols that appear on the dashes. There is also an arrow, which represents a function or procedure you are allowed to call. What this means is that by using that name, one is telling the geoprocessor to perform an operation. Some operations also allow you to define variables to be used on the operations. For example, RemoveToolBox is an operation that will remove a Toolbox from the geoprocessor, as long as you can provide its name. There are two types of operations available to you, and to distinguish them one needs to see what appears to the right of the name. here are the possibilities:

1. Methods that do not return anything: A lot of functions do not return anything to the user, and you can see this by the lack of any more information to the right of the name of the function. They perform an operation the user asks for, but do not return something the user can work with, other than a message of their success or a preset value. ResetEnvironments() is an example of such a method.
2. Methods that return objects one can manipulate: Some of the functions in the list return a new object that the user can manipulate. These objects are often Python objects (like a Python List) that contain other ESRI objects, or sometimes an ESRI defined object directly that allows us to further manipulate what the software does. An example is the InsertCursor() method, that will return an object back called a rows object. The ListFields() method returns a Python List object populated with ESRI defined Field objects.

The methods that return an object or a collection of objects also have an arrow on their right linking them with the object they return. For example, all the cursor methods connect to the row object to allow the user to see what that object provides. Similarly, the list methods oftentimes have an arrow to connect them to other objects, like the field object. The special case is the method called CreateObject(), which allows you to create a large set of objects from scratch, like spatial reference, points, valute table, etc.

Example usages of methods follow:

 import arcgisscripting
 # Import the module provided by ESRI
 gp = arcgisscripting.create(9.3)
 # Instantiate the geoprocessor object

 gp.ResetEnvironments()
 # Reset all environments on the geoprocessor object

 tools = gp.ListTools()
 # Returns a Python list of all the tools available, save in the tools variable
for tool in tools:
print tool
# Loop through each tool and print it
 

In the net part of this series I will investigate the objects connected with cursors (meaning the rows object and all connected object, like the row object (singular) and geometry).

Geoprocessor Access to Tools

The geoprocessor object also allows users to call on tools that ESRI provides. Those tools are identical to the tools found in the ArcToolbox. A list of them, with a description, is available from ESRI here (note, it is a large PDF file). Note that the Python scripting syntax is not included in that file, but the help file contains a Python example for each tool. You will need to search for it yourself unfortunately, but the thing one needs to remember is that the tool name remains the same, and all parameters to be passed to it must be within parenthesis.

I hope you enjoyed this article and be sure to visit for the remaining parts.

Related posts:

1. Loading the Geoprocessor Safely

ListWorkspaces()

The purpose of ListWorkspaces() is to provide the user with a listing of the types of workspaces that ArcGIS can make use of. The command is simple and accepts two parameters, a wildcard, and the type of workspace one is using. Below is a listing of the parameters:

In order to execute the command, you also need to have your workspace defined. This may sounds a bit contradictory, as you are trying to find workspaces, but what this does is identify an initial location on the disk to start from.

Sample ListWorkspaces() Code

import arcgisscripting
gp = arcgisscripting.create(9.3)    # We are using the 9.3 version here
gp.workspace = "mydatafolder"   # The original location the search begins at
accessWorkspaces = gp.ListWorkspaces("*", "Access")   # Return all personal geodatabases type workspaces
coveraWorkspaces = gp.ListWorkspaces("*", "Coverage")  # Return all coverages type workspaces
fileGDBWorkspaces = gp.ListWorkspaces("*", "FileGDB")  # Return all file geodatabases type workspaces
folderWorkspaces = gp.ListWorkspaces("city*", "Folder")  # Return all shapefile folders starting with the word "city"
sdeWorkspaces = gp.ListWorkspaces("*", "SDE")    # Return all SDE type workspaces
allWorkspaces = gp.ListWorkspaces("*", "ALL")   # Return all workspaces of all types

Please note that if you are using an earlier version of the geoprocessor (versions 9.2 and below), there are some slight changes on how you access the result of these commands. In 9.3 and above, results are Python Lists, while in earlier versions the results are a geoprocessing enumeration.

ListTables()

The purpose of ListTabless() is to provide the user with a listing of tables ina workspace. The command is simple and accepts two parameters, a wildcard, and the type of table one is seeking. Below is a listing of the parameters:

In order to execute the command, you also need to have your workspace defined.

Sample ListTables() Code

 import arcgisscripting
 gp = arcgisscripting.create(9.3)    # We are using the 9.3 version here
 gp.workspace = "mydatafolder"   # The original location the search begins at
 dbfTables = gp.ListTables("*", "dBase")   # Return all dBase tables
infoTables = gp.ListTables("*", "INFO")  # Return all INFO tables
allTables = gp.ListTables("*", "ALL")   # Return all tables. This list is equal with the concatenation of the two above lists.
 

Please note that if you are using an earlier version of the geoprocessor (versions 9.2 and below), there are some slight changes on how you access the result of these commands. In 9.3 and above, results are Python Lists, while in earlier versions the results are a geoprocessing enumeration.

ListRasters()

The purpose of ListRasters() is to provide the user with a listing of raster type datasets in a workspace. The command is simple and accepts two parameters, a wildcard, and the type of raster file one is seeking. Below is a listing of the parameters:

In order to execute the command, you also need to have your workspace defined.

Sample ListRasters() Code

 import arcgisscripting
 gp = arcgisscripting.create(9.3)    # We are using the 9.3 version here
 gp.workspace = "mydatafolder"   # The original location the search begins at
bmpFiles = gp.ListRasters("*", "BMP") # Return only BMP files
gifFiles = gp.ListRasters("*", "GIF") # Return only GIF files
imgFiles = gp.ListRasters("*", "IMG") # Return only IMG files
jp2Files = gp.ListRasters("*", "JP2") # Return only JP2 files
jpgFiles = gp.ListRasters("*", "JPG") # Return only JPG files
pngFiles = gp.ListRasters("*", "PNG") # Return only PNG files
tiffFiles = gp.ListRasters("*", "TIFF") # Return only TIFF files
gridFiles = gp.ListRasters("*", "GRID") # Return only GRID files
allFiles = gp.ListRasters() # Returns all raster file type, since optional parameters not provided.

Please note that if you are using an earlier version of the geoprocessor (versions 9.2 and below), there are some slight changes on how you access the result of these commands. In 9.3 and above, results are Python Lists, while in earlier versions the results are a geoprocessing enumeration.

ListFeatureClasses()

The purpose of ListFeatureClasses() is to provide the user with a listing of feature classes in a workspace. The command is simple and accepts two parameters, a wildcard, and the type of feature class one is seeking. Below is a listing of the parameters:

In order to execute the command, you also need to have your workspace defined.

Sample ListFeatureClasses() Code

 import arcgisscripting
 gp = arcgisscripting.create(9.3)    # We are using the 9.3 version here
 gp.workspace = "mydatafolder"   # The original location the search begins at

pointFCs = gp.ListFeatureClasses("*", "POINT") # Return only Point feature classes
lineFCs = gp.ListFeatureClasses("*", "LINE") # Return only Line feature classes
polygonFCs = gp.ListFeatureClasses("*", "POLYGON") # Return only Polygon feature classes
# Similarly, change the type to one of the supported types listed above

Please note that if you are using an earlier version of the geoprocessor (versions 9.2 and below), there are some slight changes on how you access the result of these commands. In 9.3 and above, results are Python Lists, while in earlier versions the results are a geoprocessing enumeration.

ListDatasets()

The purpose of ListDatasets() is to provide the user with a listing of the datasets in a workspace that ArcGIS can make use of. The command is simple and accepts two parameters, a wildcard, and the type of dataset one is using. Below is a listing of the parameters:

In order to execute the command, you also need to have your workspace defined.

Sample ListDatasets() Code

import arcgisscripting
gp = arcgisscripting.create(9.3)    # We are using the 9.3 version here
gp.workspace = "myPersonalDB.mdb"   # The original location the search begins at - a geodatabse
pgdbDatasets = gp.ListDatasets("*", "Feature")   # Return all datasets of type Feature within a personal geodatabases
# Similarly, changing the workspace and type of dataset, one can retrieve a list of all datasets

Please note that if you are using an earlier version of the geoprocessor (versions 9.2 and below), there are some slight changes on how you access the result of these commands. In 9.3 and above, results are Python Lists, while in earlier versions the results are a geoprocessing enumeration.

ListFields()

The purpose of ListFields() is to provide the user with a listing of the fields within a table, shapefile or feature class that ArcGIS can make use of. The command is simple and accepts three parameters, an input table/shapefile/feature class, a wildcard, and the type of field one is trying to list. Below is a listing of the parameters:

In order to execute the command, you also need to have your workspace defined. This may sounds a bit contradictory, as you are trying to find workspaces, but what this does is identify an initial location on the disk to start from.

Sample ListFields() Code

import arcgisscripting
gp = arcgisscripting.create(9.3)    # We are using the 9.3 version here
gp.workspace = "mydatafolder"   # The original location the search begins at
fileToRead = "myShape.shp" # resides in the mydatafolder workspace
allFields = gp.ListFields(fileToRead, "*", "ALL") # List all fields in the fileToRead table
geoFields = gp.ListFields(fileToRead, "*", "Geometry") # List geometry fields in the fileToRead table
# Similarly, you can search and return any specific fields you need. This is often done when asked to add a field, to verify it does not exist.
fieldToAdd = "Name" # Give our field a name
# Check if the field exists
if len(gp.ListFields(fileToRead, fieldToAdd, "ALL)) == 0:
    # If the field does not exist, the length of the returned list is zero
    # Within this loop, we know the fields does not exist, so we need to add it
    gp.AddField(fileToRead, fieldToAdd, "TEXT", "#", "#", "256")
    # The above line adds a string field called "Name" with 256 characters maximum length

Please note that if you are using an earlier version of the geoprocessor (versions 9.2 and below), there are some slight changes on how you access the result of these commands. In 9.3 and above, results are Python Lists, while in earlier versions the results are a geoprocessing enumeration.

Related posts:

1. Geoprocessing Iteration with Python

1. The method of invoking the geoprocessor is very similar for both, with the exception of an optional parameters when creating the object.
2. Geoprocessing methods do not return Python built-in data types but rather ESRI iterable objects.
3. Methods that used to return True/False values now return the Python True or False object rather than a string or 0/1.

The following table illustrates the differences:

gp = arcgisscripting.create()

gp = arcgisscripting.create(9.3)

# Create a list of available feature classes
fcs = gp.ListFeatureClasses()
# Iterate through said feature classes
fcs.Reset()
fc = fcs.Next()
while fc:
    # Do work here
    fc = fcs.Next()

# Create a list of available feature classes
fcs = gp.ListFeatureClasses()
# Iterate through said feature classes
for fc in fcs:
    # Do work here

# Identify the fields of a feature class
dsc = gp.describe(fc)
# Get the fields from the description
flds = dsc.Fields
# Iterate through the fields
flds.Reset()
field = flds.Next()
while field:
    # Do work here
    field = flds.Next()

# Identify the files of a feature class
dsc = gp.describe(fc)
# Get the fields from the description
flds = dsc.Fields
# Iterate through the fields
for field in flds:
    # Do work here

# Tell the geoprocessor to overwrite output by default.
gp.OverWriteOutput = 1

# Tell the geoprocessor to overwrite output by default
gp.OverWriteOutput = True

There is not list of commands that need the changes to the best of my knowledge. But a quick look through the Geoprocessor Programming Model for ArcGIS 9.2 can show which methods return an enumeration unit. In brief, ListDatasets, ListFeatureClasses, ListRasters, ListTables, ListWorkspaces, ListEnvironments, ListToolboxes, ListTools and all other methods that return an Array Object. For a complete listing of the changes, please read the ESRI document for Geoprocessor changes in 9.3.

Related posts:

1. Geoprocessing Iteration with Python

import xlwt # Import the package
wbook = xlwt.Workbook() # Create a new workbook
sheet = wbook.add_sheet("Sample Sheet") # Create a sheet
data = "Sample data" # Something to write into the sheet
for rowx in range(5):
# Loop through the first five rows
for colx in range(5):
# Loop through the first five columns
# Write the data to rox, column
sheet.write(rowx, colx, data)
# Save our workbook on the harddrive
wbook.save("myFile.xls")

And that is all you need. You just created a brand new Excel file called myFile.xls within Python without the use of Excel itself. The file has the first 5 rows and columns filled out with the text “Sample data”. It is trivial then to change this so it can save any data you want with a little bit more of Python.

Related posts:

1. Manipulating Excel files using Python part 1: Reading Excel Files

Before continuing, let’s discuss the basic idea of Excel. This programs stores data in what is called a Workbook. Think of this as the file on your computer (the .xls). A Workbook can then have one or more Sheets, the little tabs on the bottom left corner usually. Within each Sheet, you can find an arrangement of Cells in a matrix form, which are referenced by Column Name and Row Number (so the top left cell is A1, the one directly to its right is A2, the one below is B2, etc. The image should help shed some light if you are not familiar with the concept of spreadsheets.

In order to read Excel sheets in Python without using any Microsoft Office code, we need to use the excellent xlrd packge by Lingfo. The basic workflow is as follows:

1. Open a workbook using book = xlrd.open_workbook(“filetoread.xls”)
2. Access a sheet in the workbook using sheet = book.sheet_by_index(0) to open the first sheet, or if you know the name, sheet = book.sheet_by_name(“Name of Sheet”)
3. Retrieve data using any of the multiple methods: cellValue =sheet.cell_value(rowx, colx), or to retrieve a whole column use colValues = sheet.col_values(colx, start_rowx=0, end_rowx=None) or the retrieve a whole row use rowValues = sheet.row_values(rowx, start_colx=0, end_colx=None)

The basic idea is very simple, and we can easily write a little Python script that will allow us to read Excel files without the need to use Excel itself. The following example will offer a preview of an Excel file’s first sheet using the first 5 rows and columns of a file.

import xlrd # Import the package
book = xlrd.open_workbook("sample.xls") # Open an .xls file
sheet = book.sheet_by_index(0) # Get the first sheet
for counter in range(5): # Loop for five times
# grab the current row
rowValues = sheet.row_values(counter,start_col=0, end_colx=4)
# Print the values of the row formatted to 10 characters wide
print "%-10s | %-10s | %-10s | %-10s | %-10s" % tuple(rowValues)
# Print row separator
print "-" *62

That is all there is to it. Quick, fast and no need for Excel whatsoever. Sample output is presented below:

Related posts:

1. Manipulating Excel files using Python part 2: Writing Excel Files

Geoprocessing in Python provides multiple methods that can help one iterate. Depending on the dataset types you have, you can access list of data easily and effortlessly. The following code will show you how this is done in version 9.3, although earlier versions are as easy to use as well.

In order to iterate a process, we need a list of files we will use, and then a series of commands to be run on those files. There are multiple ways to achieve this through the operating system (os and sys modules), but since we do have that pesky little problem of shapefiles having multiple files associated with them, or geodatabases being a single representation for the OS, we need a different solution. And here lies the power of the geoprocessor. It allows one to list data the way ESRI can recognize them: workspaces, (feature) datasets, feature classes, tables and rasters. So let’s begin.

Oftentimes, analysts store data in folders. If you are conducting a study in 10 different regions, it is often the case you save the data files for each region in a different folder. A folder structure like the following is common:

• \myStudy\
• \myStudy\region1\
• \myStudy\region2\
• \myStudy\region3\
• …

We therefore need a method that allows us to go into the master folder (\myStudy\ in this case) and read the files in each folder. To achive this, we will read the names of folders first. The command ListWorkspaces(widcard, type) can help us do that, by accepting a wildcard as a first argument (any combination of * can help narrow things down) and the type of workspace we want (from Access, Coverage, FileGDB Folder, SDE or ALL). An example follows:

import arcgisscripting # Import the geoprocessor capabilities
gp = arcgisscripting.create(9.3) # Create the geoprocessor object
masterFolder = r"path\to\folder" # Define the master folder
gp.workspace = masterFolder # Define the location to run the commands on
workspaces = gp.ListWorkspaces("*", "Folder") # List all FOLDERS in master workspace
for workspace in workspaces: # Iterate through the workspaces one at a time
print workspace # Print the workspace name
# Do you work here

At this point, you have a Python list object that holds the names of all the subfolders that you may want to use. The next step is to identify a list of all data within that folder. To achieve that, we use one of the following commands, depending on the type of data we have: ListDatasets(wildcard, type), ListFeatureClasses(wildcard, type), ListTables(wildcard, type) or ListRasters(wildcard, type). The wildcard is defined as above, but the type is different. Type actually refers to the type of listing. For datasets, the type is feature,TIN, raster, CAD or ALL. For feature classes the types are Annotation, arc, dimension, label, line, node, point, polygon, region, route or tic. Table types can be DBF, INFO or ALL. Similarly, raster types can be BMP, GIF, IMG, JP2, JPG, PNG, TIFF, GRID or ALL. We will contnue our example by searching for feature classes (shapefiles) in the folders we identified above.

import arcgisscripting # Import the geoprocessor capabilities
gp = arcgisscripting.create(9.3) # Create the geoprocessor object
masterFolder = r"path\to\folder" # Define the master folder
gp.workspace = masterFolder # Define the location to run the commands on
workspaces = gp.ListWorkspaces("*", "Folder") # List all FOLDERS in master workspace
for workspace in workspaces: # Iterate through the workspaces one at a time
gp.workspace = workspace # Change to the new workspace
fcs = gp.ListFeatureClasses() # No parameter defaults to all
for fc in fcs: # Iterate through each file in the folder now
print fc # print the feature class
# Do individual file work here

As you can see we managed to begin with a master folder and continue to analyze individual files within all subfolders of the initial master folder. This is all done in a few lines of code, minimize complexity and confussion for us. In less that 6 lines of Python code you can begin processing each file in your computer quickly and easily.

While listing of other types of files has not been shown, it is quite easy to do by using the appropriate command (as mentioned above). Dealing with databases is similar, as you can imagine and use the database as your master folder. Do not forget that “subfolders” in the database are called datasets, so you will need to use ListDatasets() instead of ListWorkspaces().

I hope this will be useful to some of you out there.

Related posts:

1. Using the geoprocessor’s List* Methods