Development guide

Authors:Michael JasonSmith;
Contact:Michael JasonSmith <>
Copyright:This document is licensed under a Creative Commons Attribution-Share Alike 4.0 International License by


In this guide I hope to introduce you to the tools used in the development process for GroupServer. I also hope to provide an overview of the system structure.

You can find out more about GroupServer development by reading the archive, and asking questions, in the GroupServer development group, joining the #gsdevel channel on, or both.

Development process

The development process for GroupServer is straight forward, but it may be different to what you are used to simply because of the size and structure of the project. Here we cover using a virtual environment, before introducing you to the version control system used for GroupServer.

Virtual environment

The development for GroupServer takes place within the Python virtual environment of a GroupServer installation (see GroupServer installation). When you start development you need to activate the virtual environment:

$ . ./bin/activate

The command-prompt may change to indicate that you are now in a virtual environment.

When you have done with development you can deactivate the environment:

$ deactivate

Version control system

Two version control systems are used to store the source-code for GroupServer:

GitHub hosts the canonical repository at <>
The Mercurial repository is hosted on a Redmine site that has a self-signed HTTPS certificate <>

You can use either system, but I recommend that people use GitHub because they are more likely to have an account there. However, you will almost certainly lack the permissions required to push any changes. To ensure your changes are not lost you should fork a repo and commit any changes you make there. Once you are done you can make a pull request.


I recommend that you use mr.developer (PyPI) to integrate your development with the buildout system that builds GroupServer. In this section we will cover its installation with buildout, its usage, and the development configuration file.


Enter your virtual environment using activate as above, then:

$ buildout -N -c develop.cfg

This builds your site (the same as a normal GroupServer installation) along with the mr.developer script develop. (Two scripts to help with internationalisation, i18ndude and tx are also installed; see the Translation guide for more information.)


First you will need code to work on. Use mr.developer to checkout the source-code from the repository:

$ develop checkout

This will checkout the product from its repository into the src/ directory within your GroupServer installation. (There is more on products below.)

Next you will have to rebuild GroupServer to update all the configuration to point to the new code:

$ buildout -N -c develop.cfg

Now any changes that you make to the code in src/ will change your version of GroupServer.

When you have finished making changes you want you should commit them, and push your changes up to a repository. To resume using the standard version of a product deactivate the source code version of the product and rebuild GroupServer:

$ develop deactivate
$ buildout -N -c develop.cfg

Development configuration file

The configuration for mr.developer is in the develop.cfg file, which is very similar to the configuration files that control the rest of the build.

The main configuration is in the [sources] section. This maps each product that makes up GroupServer to the repository location. Each line is of the form

name = kind url
The name of the product.
The kind of version control system to use. For GroupServer development this will be either hg or git. For your own development you may want to live dangerously and use fs for a product that lacks version control [1].
The location of the repository. This is specific to the version control system.

When you make your own changes you will need to change the URL to point to the repository provided by the version control system that you use. The default configuration that ships with GroupServer points to the canonical Git repositories for all products.

Default configuration

The default configuration for mr.developer is generated from the versions.cfg file using the following awk script. It specifies that git should be used with all the products.

  FS=" = "
  print "[buildout]"
  print "extensions = mr.developer"
  print "sources = sources"
  print "auto-checkout = "
  print "\n[sources]"
$1 ~ /^((gs)|(Products.G)|(Products.XWF)).*/ {
  printf "%s = %s %s%s\n", $1, vcs, dest, $1

To change Mr Developer to use Mercurial as the default version-control system, but use GitHub as the primary repository, carry out the following tasks.

  1. Install the Hg-Git plugin for Mercurial.

  2. Copy the above awk script to the file emit-devel.awk.

  3. Change the vcs variable to hg.

  4. Add git+ to the start of the value for the dest variable.

  5. Run the command:

    $ awk -f emit-devel.awk < versions.cfg > new-develop.cfg
  6. Check that the new configuration is to your liking and move the new configuration into place:

    $ mv new-develop.cfg develop.cfg

Adding a new product

GroupServer is split into multiple products, each controlling one aspect of the system. (There is more on products below.) To add your own functionality, or override existing functionality, you will need to add your own product.

To add your own new product to GroupServer carry out the following tasks.

  1. Create the product in the src directory. Normally I copy an existing product:

    • Rename the product, the directories in the product namespace, and the configuration in the
    • Delete the old code — keeping a blank at the top, and the files needed for the namespace.
    • Delete the contents of the <configure> element of the configure.zcml file, keeping the element itself.
  2. Add the name of your product to the custom-zope-eggs section of the custom.cfg file.

  3. Add the version-control information for the product to the development configuration file.

  4. Activate the product:

    $ develop activate
  5. Rebuild GroupServer:

    $ buildout -N -c develop.cfg

System structure

GroupServer belongs to a family of systems that share underlying technology:

The source-code for GroupServer is split into many products, with the documentation provided by reStructuredText. It is all coordinated by buildout.


Buildout assembles and deploys the different parts that make up GroupServer. The basic idea is that the high-level components are specified by buildout. For GroupServer all the components are Python products, which are installed as eggs. Each Python product specifies its dependencies in the However, buildout restricts the versions that are installed, ensuring that a known good set of products is installed.

Buildout is controlled by the *.cfg files at the top of your GroupServer installation directory. There are three main files, and seven supplementary files, all of which are in INI format.

Main configuration files

The three main files specify the ways the products can be deployed.


Controls the installation of all the components that make up GroupServer, up to the point that the unit tests can be run, but it stops short of creating the site:

$ buildout
$ testrunner -k --auto-color --udiff -m gs\\..*

Extends buildout.cfg by adding the recipes (buildout scripts) that actually initialise the database and create the GroupServer site (if necessary). This is the normal way to build GroupServer:

$ buildout -c site.cfg

Extends site.cfg by adding the components that allow for development: mr.developer, i18ndude, and transifex-client:

$ buildout -c develop.cfg

Supplementary configuration files

The seven supplementary configuration files are used by the three main files in a similar way to modules.

The main site configuration that is written during GroupServer installation. Used by site.cfg.
The products that customise the install go in this configuration file. Initially it is mostly blank. Used by buildout.cfg.
The parts that control the recipes that set up the base install. Used by buildout.cfg.
The parts that control the recipes that install the initial GroupServer site (the instance). Used by site.cfg.
The list of what specific versions should be installed. Used by buildout.cfg.
The configuration for the Zope2 web-application server that GroupServer is based on. Used by buildout.cfg.
The configuration for the Zope Toolkit. Used by buildout.cfg.


GroupServer is split into many (currently 146) products: small Python packages that deal with one aspect of the system. The general rule is that one product for each user interface (usually a form). While this may seem limiting, each product usually contains

  • The page templates that makes up the interface,
  • The JavaScript that is specific to the page,
  • The Python code that defines the behaviour of the interface,
  • The Python code that handles storing the data and retrieving the data (using SQLAlchemy),
  • The SQL code that defines any product-specific tables,
  • The user-help, and
  • The code documentation.

This tends to be more than enough for each product.

If more than one product relies on the same code then that code is normally refactored into a base product — which is normally given a name ending in .base, such as

Development is carried out on one, or a few, products at a time. If you are unsure what products provide aspect of GroupServer it would be best to ask in GroupServer development or in #gsdevel on IRC. However, there are some clues: normally name of the product will make up the part of the identifier or class-name of an element in the HTML source of a page. For example, the link to the ATOM feed of posts on the Group page has the identifier gs-group-messages-posts-link — which indicates that it is provided by the product.

<link id="gs-group-messages-posts-link" rel="alternate"
    title="Posts in GroupServer development"
    href="/s/search.atom?g=development&amp;t=0&amp;p=1" />

Each product makes use of namespaces, and ZCML. Each product usually contains some static resources, page templates, and some documentation


The products use namespace packages (PEP 420).

  • Each part of the namespace is separated by dots. For example, the code that produces for the plain-text version of an email message is

  • Each GroupServer product belongs beneath the gs namespace. Beneath that there are many others. The three large ones are for the three main agents in GroupServer:

    Groups, including mailing list functionality (


    People, as represented by their profiles.

    The code relating specifically to a site.

    The other major namespace is gs.content, which provides the client side code.

  • The root of each product contains the packaging information for that product.

    • The configuration for setuptools, particularly in the files, setup.cfg, and The first is the one with most of the information, in particular the dependencies for the package.
    • The README.rst, which appears on GitHub.
    • The COPYRIGHT.txt, and LICENSE.txt.
  • The documentation will be in the docs/ directory.

  • The Python code is within nested sub-directories beneath the product directory, such as gs/group/list/email/text. All but the last directory will have files that set the directory up as a namespace directory. The last directory (text in this example) will have an It is necessary to turn the directory into a Python module.

The Python code is made up of an that is often blank, with each class in its own file. (This is my habit, you do not have to follow it.) To determine the relationship between the files, and the rest of GroupServer, it is necessary to look at the ZCML file.


The Zope Configuration Markup Language (ZCML) defines the static resources, the page templates, the relationship that the Python files have to each other, and to other products. The configuration for each product is always called configure.zcml, and it is always in the same directory as all the Python files.

To begin with the three most important directives are as follows.

<browser:resource />:
A static resource.
<browser:page />:
A page on the web, pointing to a page template.
<browser:viewlet />:
Part of a page, which also points at a page template.

Static resources

Static resources are simply files with names, which are useful for JavaScript, CSS, and images. When requested Zope sends the static file to the browser.

The resource is defined by the <browser:resource/> directive in the ZCML.

  permission="zope2.Public" />
  • The name attribute is the of the resource. The URL is made up of / and the name. Normally the name of the product ( in this case) makes up part of the name to prevent namespace clashes, and so it is easier to work back from the filename to the product. The name should end with the date the resource was created so there are fewer caching issues when the resource is updated.
  • The file is the static file that is served. It is a path from the directory that holds the ZCML file. Resources are always within the browser sub-directory, within a javascript, images or css directory.
  • The permission is the permission on the resource. It is always zope2.Public. This will allow the resource to be cached.

In GroupServer the resources are always accessed from the root of the site, with ++resource++ added to the start of the name: <>

Page Templates

Pages themselves are defined by one of two directives in the ZCML: <browser:page/> and <browser:viewlet/>. The former links the Python code (class) with a template, giving it a name.

  permission="zope2.View" />

A viewlet is part of a page. It also links a class up with a name and template.

  title="Topic Summary Statistics" />

The pages are created using Zope Page Templates (ZPT), which is the same template system that Plone uses, and is very similar to Chameleon.

  • The page templates are always stored in a directory called browser/templates, within each product. Each has the extension .pt.

  • The template itself is XHTML 5: the XML form of HTML 5.

  • The dynamic parts of the template are defined by attributes, using the Template Attribute Language (TAL). This accesses attributes and methods of the Python code. In the following example the group-name is written into the <h1/> element by the tal:content attribute.

    <h1 id="gs-group-home-h" class="fn"
  • Within each attribute is one or more expressions that generates the text that is placed into the page. The Python code (the class in the ZCML above) is always referred to as view, and a / is used as an attribute separator (rather than . in Python code). In the above example the Python class ( is accessed to get the group-information attribute (groupInfo), and from that the group-name is retrieved.


Every package will always have documentation. The primary documentation is in the README.rst, and there will also be a Changelog in the docs/HISTORY.rst file. Often there will be full API documentation.

The development documentation for GroupServer is entirely in reStructuredText, with the autodoc plugin for Sphinx used to generate the source-code documentation where possible. The documentation is then usually pushed up to the the GroupServer project at Read The Docs; if it is then a link will be in the README.rst.

[1]I recommend that you use a local Mercurial repository on your local machine, rather than abandoning version control altogether.