Development guide¶
Authors: | Michael JasonSmith; |
---|---|
Contact: | Michael JasonSmith <mpj17@onlinegroups.net> |
Date: | 2016-01-26 |
Organization: | GroupServer.org |
Copyright: | This document is licensed under a Creative Commons Attribution-Share Alike 4.0 International License by OnlineGroups.net. |
Introduction¶
In this guide I hope to introduce you to the tools used in the development process for GroupServer. I also hope to provide an overview of the system structure.
You can find out more about GroupServer development by reading the archive, and asking questions, in the GroupServer development group, joining the #gsdevel channel on Freenode.net, or both.
Development process¶
The development process for GroupServer is straight forward, but it may be different to what you are used to simply because of the size and structure of the project. Here we cover using a virtual environment, before introducing you to the version control system used for GroupServer.
Virtual environment¶
The development for GroupServer takes place within the Python virtual environment of a GroupServer installation (see GroupServer installation). When you start development you need to activate the virtual environment:
$ . ./bin/activate
The command-prompt will change to indicate that you are now in a virtual environment.
When you have done with development you can deactivate the environment:
$ deactivate
Version control system¶
Two version control systems are used to store the source-code for GroupServer:
- Git:
- GitHub hosts the canonical repository at <https://github.com/groupserver>
- Mercurial:
- The Mercurial repository is hosted on a Redmine site that has a self-signed HTTPS certificate <https://source.iopen.net/groupserver>
You can use either system, but I recommend that people use GitHub because they are more likely to have an account there. However, you will almost certainly lack the permissions required to push any changes. To ensure your changes are not lost you should fork a repo and commit any changes you make there. Once you are done you can make a pull request.
Mr Developer¶
I recommend that you use mr.developer to integrate your
development with the Buildout system that builds
GroupServer. Its usage is simple (which is why we use
mr.developer
) and its behaviour is controlled by the
development configuration file.
Usage¶
Run the following To use mr.developer
to checkout the
code from the VCS, and active the source code so you can work on
it:
$ develop checkout gs.group.home
This will checkout the gs.group.home
product from its
repository into the gs.group.home
directory within the
src
directory of your GroupServer installation. (There is
more on products below.)
Next you will have to rebuild GroupServer to update all the configuration to point to the new code:
$ buildout -N
Now any changes that you make to the gs.group.home
product
will change your version of GroupServer.
When you have finished making changes you want you should commit them, and push your changes up to a repository.
To resume using the standard version of a product deactivate the source code version of the product and rebuild GroupServer:
$ develop deactivate gs.group.home
$ buildout -N
Development configuration file¶
The configuration for mr.developer
is in the develop.cfg
file, which is a configuration file that is very similar to the
configuration files that control the rest of the build.
The main configuration is in the [sources]
section. This maps
each product that makes up GroupServer to the repository
location. Each line is of the form
name = kind url
name
:- The name of the product.
kind
:- The kind of version control system to use. For GroupServer
development this will be either
hg
orgit
. For your own development you may want to live dangerously and usefs
for a product that lacks version control [1]. url
:- The location of the repository. This is specific to the version control system.
When you make your own changes you will need to change the URL to point to the repository provided by the version control system that you use. The default configuration that ships with GroupServer points to the canonical Git repositories for all products.
Default configuration¶
The default configuration for mr.developer
is generated from
the versions.cfg
file using the following awk
script. It specifies that git should be used with all
the products.
BEGIN {
FS=" = "
vcs="git"
dest="ssh://git@github.com:/groupserver/"
print "[buildout]"
print "extensions = mr.developer"
print "sources = sources"
print "auto-checkout = "
print "\n[sources]"
}
$1 ~ /^((gs)|(Products.G)|(Products.XWF)).*/ {
printf "%s = %s %s%s\n", $1, vcs, dest, $1
}
To change Mr Developer to use Mercurial as the default VCS, but use GitHub as the primary repository, carry out the following tasks.
Install the Hg-Git plugin for Mercurial.
Copy the above awk script to the file
emit-devel.awk
.Change the
vcs
variable tohg
.Add
git+
to the start of the value for thedest
variable.Run the command:
$ awk -f emit-devel.awk < versions.cfg > new-develop.cfg
Check that the new configuration is to your liking and move the new configuration into place:
$ mv new-develop.cfg develop.cfg
Adding a new product¶
GroupServer is split into multiple products, each controlling one aspect of the system. (There is more on products below.) To add your own functionality, or override existing functionality, you will need to add your own product.
To add your own new product to GroupServer carry out the following tasks.
Create the product in the
src
directory. Normally I copy an existing product:- Rename the product, the directories in the product
namespace, and the configuration in the
setup.py
. - Delete the old code — keeping a blank
__init__.py
at the top, and the__init__.py
files needed for the namespace. - Delete the contents of the
<configure>
element of theconfigure.zcml
file, keeping the element itself.
- Rename the product, the directories in the product
namespace, and the configuration in the
Add the name of your product to the
custom-zope-eggs
section of thecustom.cfg
file.Add the version-control information for the product to the development configuration file.
Activate the product:
$ develop activate your.product.name
Rebuild GroupServer:
$ buildout -N
System structure¶
GroupServer belongs to a family of systems that share underlying technology:
The source-code for GroupServer is split into many products, with the documentation provided by reStructuredText.
Products¶
GroupServer is split into many (currently 146) products: small Python packages that deal with one aspect of the system. The general rule is that one product for each user interface (usually a form). While this may seem limiting, each product usually contains
- The page templates that makes up the interface,
- The JavaScript that is specific to the page,
- The Python code that defines the behaviour of the interface,
- The Python code that handles storing the data and retrieving the data (using SQLAlchemy),
- The SQL code that defines any product-specific tables,
- The user-help, and
- The code documentation.
This tends to be more than enough for each product.
If more than one product relies on the same code then that code
is normally refactored into a base product — which is
normally given a name ending in .base
, such as
gs.group.list.base
.
Development is carried out on one, or a few, products at a
time. If you are unsure what products provide aspect of
GroupServer it would be best to ask in GroupServer
development or in #gsdevel
on IRC. However, there are some
clues: normally name of the product will make up the part of the
identifier or class-name of an element in the HTML source of a
page. For example, the link to the ATOM feed of posts on the
Group page has the identifier gs-group-messages-posts-link
— which indicates that it is provided by the
gs.group.messages.posts product.
<link id="gs-group-messages-posts-link" rel="alternate"
type="application/atom+xml"
title="Posts in GroupServer development"
href="/s/search.atom?g=development&t=0&p=1" />
Each product makes use of namespaces, and ZCML. Each product usually contains some static resources, page templates, and some documentation
Namespaces¶
The products use namespace packages (PEP 420).
Each part of the namespace is separated by dots. For example, the code that produces for the plain-text version of an email message is
gs.group.list.email.text
.Each GroupServer product belongs beneath the
gs
namespace. Beneath that there are many others. The three large ones are for the three main agents in GroupServer:gs.group
:Groups, including mailing list functionality (
gs.group.list
).gs.profile
:People, as represented by their profiles.
gs.site
:The code relating specifically to a site.
The other major namespace is
gs.content
, which provides the client side code.The root of each product contains the packaging information for that product.
- The configuration for setuptools, particularly in the files
setup.py
,setup.cfg
, andMANIFEST.in
. The first is the one with most of the information, in particular the dependencies for the package. - The
README.rst
, which appears on GitHub. - The
COPYRIGHT.txt
, andLICENSE.txt
.
- The configuration for setuptools, particularly in the files
The documentation will be in the
docs/
directory.The Python code is within nested sub-directories beneath the product directory, such as
gs/group/list/email/text
. All but the last directory will have__init__.py
files that set the directory up as a namespace directory. The last directory (text
in this example) will have an__init__.py
. It is necessary to turn the directory into a Python module.
The Python code is made up of an __init__.py
that is often
blank, with each class in its own file. (This is my habit, you do
not have to follow it.) To determine the relationship between the
files, and the rest of GroupServer, it is necessary to look at
the ZCML file.
ZCML¶
The Zope Configuration Markup Language (ZCML) defines the static
resources, the page templates, the relationship that the
Python files have to each other, and to other products. The
configuration for each product is always called
configure.zcml
, and it is always in the same directory as
all the Python files.
To begin with the three most important directives are as follows.
<browser:resource />
:- A static resource.
<browser:page />
:- A page on the web, pointing to a page template.
<browser:viewlet />
:- Part of a page, which also points at a page template.
Static resources¶
Static resources are simply files with names, which are useful for JavaScript, CSS, and images. When requested Zope sends the static file to the browser.
The resource is defined by the <browser:resource/>
directive
in the ZCML.
<browser:resource
name="gs-group-messages-topic-compose-20160127.js"
file="browser/javascript/compose.js"
permission="zope2.Public" />
- The
name
attribute is the of the resource. The URL is made up of/
and the name. Normally the name of the product (gs.group.messages.topic
in this case) makes up part of the name to prevent namespace clashes, and so it is easier to work back from the filename to the product. The name should end with the date the resource was created so there are fewer caching issues when the resource is updated. - The
file
is the static file that is served. It is a path from the directory that holds the ZCML file. Resources are always within thebrowser
sub-directory, within ajavascript
,images
orcss
directory. - The
permission
is the permission on the resource. It is alwayszope2.Public
. This will allow the resource to be cached.
In GroupServer the resources are always accessed from the root of
the site, with ++resource++
added to the start of the name:
<http://groupserver.org/++resource++gs-group-messages-topic-compose-20160127.js>
Page Templates¶
Pages themselves are defined by one of two directives in the
ZCML: <browser:page/>
and <browser:viewlet/>
. The former
links the Python code (class
) with a template
, giving it
a name
.
<browser:page
name="index.html"
for="gs.group.base.interfaces.IGSGroupMarker"
class="gs.group.base.page.GroupPage"
template="browser/templates/homepage.pt"
permission="zope2.View" />
A viewlet is part of a page. It also links a class
up
with a name
and template
.
<browser:viewlet
name="gs-group-message-topic-summary-stats"
manager=".interfaces.ITopicSummary"
template="browser/templates/summarystats.pt"
class=".summarystats.SummaryStats"
permission="zope2.View"
weight="0"
title="Topic Summary Statistics" />
The pages are created using Zope Page Templates (ZPT), which is the same template system that Plone uses, and is very similar to Chameleon.
The page templates are always stored in a directory called
browser/templates
, within each product. Each has the extension.pt
.The template itself is XHTML 5: the XML form of HTML 5.
The dynamic parts of the template are defined by attributes, using the Template Attribute Language (TAL). This accesses attributes and methods of the Python code. In the following example the group-name is written into the
<h1/>
element by thetal:content
attribute.<h1 id="gs-group-home-h" class="fn" tal:content="view/groupInfo/name">Group</h1>
Within each attribute is one or more expressions that generates the text that is placed into the page. The Python code (the
class
in the ZCML above) is always referred to asview
, and a/
is used as an attribute separator (rather than.
in Python code). In the above example the Python class (gs.group.base.page.GroupPage
) is accessed to get the group-information attribute (groupInfo
), and from that the group-name is retrieved.
Documentation¶
Every package will always have documentation. The primary
documentation is in the README.rst
, and there will also
be a Changelog in the docs/HISTORY.rst
file. Often there
will be full API documentation.
The development documentation for GroupServer is entirely in
reStructuredText, with the autodoc plugin for Sphinx used to
generate the source-code documentation where possible. The
documentation is then usually pushed up to the the GroupServer
project at Read The Docs; if it is then a link will be in the
README.rst
.
[1] | I recommend that you use a local Mercurial repository on your local machine, rather than abandoning version control altogether. |