Cows and Church.  Tingstade (northern Gotland).

Using the ArsDigita Community System

to build various kinds of Web sites, by Philip Greenspun for the Web Tools Review

Last Update: April 21, 2000 (to coincide with being able to run ACS with the Apache Web server)


A McDonald's in Stockholm Here's what's in this document:

Introduction

At MIT the easiest way to discourage students from taking a class is to hand them a long list of prerequisites. With that in mind, it pains me to tell you that this article won't make too much sense unless you've read Chapter 3 of Philip and Alex's Guide to Web Publishing where the architecture of the ArsDigita Community System is set forth.

If you're a lazy Wall Street analyst, the best way to think about ACS is "like Broadvision or Vignette but (a) much stronger on community, (b) free and open-source, (c) backed up by education worldwide, (d) a bit weaker on catalog shopping than Broadvision and on newspaper/magazine production workflow than Vignette."

If your laziness falls in the middle, here's a summary of the ArsDigita Community System:

The bottom line is that if you have money and your goals are reasonably similar to those that have been attacked by the toolkit, starting from our software will save you 6-9 months in time to market and several million dollars in cash. If you don't have money, using our software will enable you to do things that only the richest Web publishers are currently doing.

This document exists to give publishers some broad ideas of how the toolkit can be used and roughly how much work is required to adapt it.

example: online community

Botaniska Tradgarden.  Visby, Gotland. The software was originally created to support photo.net, a site with a large body of static, carefully authored content, comments on static pages, a lively set of discussion forums, classified advertising, and a recommendation service where people can talk about their experiences with buyers or sellers of photographic equipment. In order to better support photo.net, I recently added some modules for dishing out news (e.g., "Canon releases the EOS-3 body") and calendar items (e.g., "Ansel Adams show at National Gallery, March 5-June 9").

If your idea of an online community differs from photo.net in that you want tighter moderation (e.g., content doesn't go live until a moderator has approved it), you can achieve that simply by setting some switches in the ad.ini file. If you want banner ads, just reference the ad server that is included with the standard ACS distribution.

If your community is large and contains an overwhelming number of documents, it might be good to use the curriculum module to guide new users through a set of important tutorial documents. This software puts a little progress bar into every page that they fetch from the server until they "graduate" by reading each item in the publisher-established curriculum.

If your community contains several different classes of user, you might consider using the portals module to give each class an easy-to-update portal into your service (and back out to the wider Internet). You can optionally configure the portal software to let each user customize his or her own personal portal.

Configure the customer relationship management module so that you can target email to people depending on how much they've contributed, how recently, and whether they were a really active participant at some point in the past.

The modules that support an online community include the following:

example: offline community (intranet)

Sheep in the Wicklow Mountains south of Dublin, Ireland. What if you're running a site for a group of people who work together in one organization? You start with our intranet module, documented at http://www.photo.net/doc/intranet.html. This is one of the modules that we're most actively developing (since we use it to run arsdigita.com, a 51-person company with $10 million in sales).

A lot of the stuff that is useful for public online communities turns out to translate very nicely for intranets. Some modules that are more useful for intranets than public communities include the following:

Organizations that use the ACS for supporting collaboration among employees include the following: ArsDigita, Caltech, Hewlett-Packard, Oracle, Siemens.

example: ecommerce site

Picture stone.  Bunge Museum.  Northern Gotland.  Sweden The main support for ecommerce, naturally, is our /ecommerce module. It does pretty much everything that you'd expect from an average ecommerce site like amazon.com (see /doc/ecommerce.html).

You might not see it this way, but any interesting ecommerce site is actually an online community. Amazon.com, for example, derives most of its value from collecting reader comments on books and from using reader buying patterns and preferences to help other readers find relevant books.

The other weird thing about ecommerce is that there are a million trivial software packages to support online catalogs, shopping carts, and the rest of the stuff that most of us got bored with back in 1995. However, most products aren't priced or sold this way (the same price for everyone).

If you're a company like Cisco, for example, what you would do with the ACS is to set up an online community in which some of the registered users work for Cisco and registered users work at companies that have purchased Cisco routers. You use the ACS's user-groups module to define a user_group_type of "company". You use groups of type "company" to bundle users together according to where they work. This enables you to have private discussion forums for Cisco internal people. This bundling enables you to assign a standard discount to each company that is a customer. Then if joe@hp.com is logged in and wants to see prices, the price display code can check which user-group joe@hp.com belongs to and then grab the discount and adjust the prices accordingly.

We've rolled a nice ticket tracking system into the standard ACS distribution. This ties in again to the user-group tables. When joe@hp.com reports a problem with a Cisco router, the next available support person is notified. If the problem stays open longer than N minutes, a service coordinator for the HP account can be notified. If the problem stays open longer than that, the sales reps for HP can be notified via email. At any time, the system can display reports of customer problems and how they were resolved, summarizing by user-group (i.e., company). If the Cisco salesperson for HP is over on Page Mill Road trying to sell some more routers and the HP customer says "how good are you guys at resolving problems", the Cisco salesperson can show a report that the average response time was 6 minutes and the average time to resolution was 2 hours.

Configure the customer relationship management module so that you can market to people depending on how much they've bought, how recently, and whether they were a really great customer at some point in the past.

Examples of live sites using the ACS ecommerce module: http://www.eplay.com/ecommerce/; www.perfectionlearning.com.

Note 1: if your ecommerce vision is truly banal and extends only to hawking a list of products to customers with whom you're building no long-term relationship, then ArsDigita Shoppe might suit your needs better. It is also free and open-source.

Note 2: Cisco and HP aren't actually using the ACS this way; I just used them as a convenient example since these words are coming to you from an HP Unix server through a Cisco router.

Customer Success Story

Here's a talk abstract from Dave Weis (mailto:djweis@plconline.com):
	Perfection Learning Corporation has implemented an ecommerce
system that has processed hundreds of orders with real-time credit card
processing using entirely open-source software, plus Oracle. The
installation, integration, and customization took four weeks with a total
cost of $30,000. The price was evenly split between the web/database
server, the licensing and support for Oracle, and the developer time used.
	We were able to keep the costs low by using the ArsDigita
Community System (ACS) and the AOLserver web server, both available under
the GPL. Our web/database server is running RedHat Linux. 
	Installation of the ACS consisted of feeding the data model to
Oracle and putting the HTML and Tcl documents in the correct locations.
Adding our products to the catalog was done with a browser-driven import
routine. We have over 30,000 products in the database with no speed or
scalability problems so far. 
	The ACS ecommerce module was easy to integrate into our existing
site. By modifying our pages to include links for adding the items to the
shopping cart, we made all of our products orderable in a few days. 
	Site maintenance and modification is easy due to the fact that all
of the Tcl code and HTML is stored in flat files. Any errors can be fixed
with a normal text editor. The functionality and presentation of the
ecommerce section is separated by using templates. 
	By combining an open-source operating system such as Linux with
the ACS, AOLserver, and Oracle, you can deploy an ecommerce solution in a
fraction of the time it would take you to write your own or integrate
commercial software with your business.

example: participants in a class

Puffin.  Oregon Coast Aquarium.  Newport, Oregon If you have a bunch of people taking a high school or university course, the ACS can serve as the hub for distributing course materials, collecting questions, and fostering discussion. It has in fact been used this way at MIT, Harvard, and lots of schools that aren't in Cambridge!

Don't expect miracles, though. Internet and computers are great for connecting people who are separated in time and space. If you've been able to collect everyone in the same room at the same time with an expert, how can you expect plunking down 17" monitors and the limping products of the modern software industry to improve matters?

Anyway, the lessons from our experience are that the ACS works as well as any other collaboration software for supporting folks in a class. Just be sure that they have a reason for using a Web service in the first place.

Botaniska Tradgarden.  Visby, Gotland. Garden through window in Skansen in Stockholm Architecturally you could set up everyone at a university in one big ACS. Then use the user-groups module to define various group types. The most important structures in a modern university are committees. So you'd have those sorts of groups to accrete administrators. Then you'd probably want to think about a group of type "course". Students and teachers in a particular course would be lumped together in a user group and could have a private discussion group.

Another thing that would work is setting up a standalone ACS just for one class. It takes us at least half a day to set up an ACS (download .tar file, set up new AOLserver or Apache, create new Oracle user and tablespace, edit the parameters file to give the service a name, etc.). So it might be a little too much overhead for a class that is going to be over in 13 weeks anyway.

example: electronic medical record system

Ice cream sign, just off Fjallgatan on Sodermalm.  Stockholm, Sweden The users are doctors and other folks who work at the hospital. User groups are departments within a hospital, e.g., orthopedics or cardiology. You have to write a custom module to collect patient data. These are tied to users and user-groups. Note that patients do not become registered users of the ACS.

example: auction site

If you want to get rich like those Internet auction company founders, you will like the ACS. Out of the box, it comes with a classified ad facility. The classified ad facility includes the ability for ads to accept auction bids. Those bids are displayed underneath the ad copy. User A can click on User B's name and see that user's history with the community. It has been in use at photo.net since 1996 or so. Users who post ads or bids are all authenticated via the standard registration and login pages.

(If you're thinking that this is the complete set of features that eBay has, you're right. We built it before they did but we forgot to raise money and hype it; so they are multi-billionaires and we remain ... computer programmers.)

example: group of programmers

Software systems are typically developed by multiple programmers working from specs, bug reports, and feature requests. This is and always has been a collaborative process. We may be guaranteed that everyone involved in this collaboration has a computer system. Consequently it is only natural that we're building a special Software Development Manager extension to the ArsDigita Community System. This is especially well-suite to open-source software projects where contributors are spread out across the world.

FAQs

I want to exchange sensitive information with my customers. How secure are data in the ACS?
Langhammars. Faro. Northern Gotland.  Sweden Any AOLserver application can be requested as either HTTP or HTTPS. The .tcl and .adp pages don't know or care that the request has come in via SSL or not. So you can configure your ACS to only talk to the world via HTTPS and therefore nobody can sniff your data in transit.

Does that mean you're home free? No. Aside from any security holes that there might be in the ACS, keeping sensitive data on any Internet server presents big security problems. Someone could attack your Oracle database or your Unix server itself and essentially get all of your data. So you want to have really good sysadmins and security folks on staff. These people are expensive but without them there is no way that you can run any kind of Web service and guarantee confidentiality.

What do I need to run the ACS?
You need

If you want your service to be reliable, you probably need something like the ArsDigita Server Architecture.

Why is this software free when similar-sounding commercial products cost $100,000 and up?
Millesgarden. Stockholm, Sweden Some short answers first. Because most people can't accept the idea that others might be altruistic, we will note that there are some very big, very rich companies using our software (see http://arsdigita.com/projects.html for an up-to-date list). We are some of the highest paid programmers in the world, with 100% ownership of a ArsDigita.com, a 60-developer profitable company with $20 million in annual sales (February 2000; sales should double every six months). We do not need to spend every second of every day trying to make money.

Free software does not mean that programmers get paid less. In fact, programmers probably get paid more since users of free software don't have to feed a vendor's marketing, PR, advertising, HR, trade show, and management departments.

That said, we didn't go into this to make money. We wanted to solve some hard problems in computer-supported cooperative work. Commercial software is fine when what you're buying is a clone of something Xerox PARC did in the 1970s (e.g., Microsoft Word, Microsoft Windows). Free software is the only hope when you're attacking a brand new problem that nobody knows how to solve. To us, the dimensions of being a software development professional include

It simply isn't possible to achieve this in an 80-year human lifetime if one is obsessed with scamming users and Wall Street with a piece of packaged junkware.

Longer answer: towards the end of Chapter 2 of "the book".


Is training available?
Millesgarden. Stockholm, Sweden Yes! We love to train people in how to use our toolkit. We publish a free book (you can also buy a hardcopy in bookstores). We give one-day courses (see http://register.photo.net), arrange boot camps, and have exported our one-semester course from MIT to other universities (including Caltech).

Will you guys extend it for me?
Maybe. If your idea is really interesting or if we think lots of other people would want the same extension, then will push your pet project up our list (see http://www.photo.net/acsplans/ for a moderately current list; we update it before every user's group meeting). If you want us to commit to build and support a module for you by Date X, you have to become a customer of Arsdigita, the company.

Will you guys pay me to extend it for you?
Maybe. We offer a $10,000 prize for young people. The prize is open to anyone 18 or younger who has built an interesting Web service. If that service happens to be built on top of our toolkit and the author chooses to release the source code, so much the better!

Will you guys support the ACS and the infrastructure?
There are a bunch of things that you need to have a reliable Web service, which is why we wrote this document explaining the ArsDigita Server Architecture. All of the elements of this architecture are either free software (some of which we wrote), commercially available programs such as Oracle, or commercial co-location services. You can hire arsdigita.com to host your Web service. You can buy a support contract from arsdigita.com for AOLserver or the free software we wrote that uses it. But really the beauty of open-source software is that you don't need us to support the ACS. Find someone smarter and cheaper than us and pay him or her to do it!

What do independent reviewers say about ACS?
More FAQs are contained in /doc/arsdigita-faq.html.

ACS versus Commercial Packaged Closed-Source Systems

South Island, New Zealand In the summer of 1997 some guys at Hewlett-Packard were laughing at my plans to build a free open-source software toolkit for building all kinds of Web sites for collaboration, including business-to-consumer ecommerce sites. "You'll never make it, Greenspun," they said. "Everyone will be using iCat. That's what we're using and all of our customers are using it and it is the future. Get used to a closed-source commercial software world."

I didn't hear much more about iCat until I happened upon a cover story in the October 4, 1999 PC Week: "Risky Business, IT rolls dice when it picks tools from unproven firms" (by Jim Kerstetter and Matt Hicks). Here's how the story begins:

On May 13 [1999], Greg Streeter was jilted by his e-commerce software vendor.

"Dear Greg," the goodbye e-mail message said, "Intel is discontinuing the iCat Electronic Commerce Suite, effective immediately." The note went on to remind Streeter that his end-user licensing agreement, which bars copying the software, was still in effect, so there would be no tinkering with the defunct software.

"At least it was personal," sighed Streeter, IS manager at Herman Goelitz Inc., a candy maker based in Fairfield, Calif.

The e-mail was a painful reminder of a regrettable decision Streeter made two years earlier. Faced with an immediate need for e-commerce software that could support personalization and offer an attractive presentation, and unable to find such features in products from established vendors, he committed development of Goelitz's JellyBelly.com Web site to software from iCat Corp., then a small startup.

However, Seattle-based iCat fell prey to the myriad problems that face startups in any industry. It ran out of money, forcing iCat executives to shift strategy. Then, earlier this year, iCat was acquired by Intel Corp., a company that had no interest in iCat's customer base. In August, Intel, of Santa Clara, Calif., discontinued support of the iCat software product line. Customers, including Streeter, got burned.

Sheep Research Center.  Corvallis, Oregon I went to the iCat Web site and found that they had press releases proudly announcing the 6500 stores built with their software. They proudly announced their acquisition by Intel. They had no trouble calling themselves "the leading provider of e-commerce software". No mention of leaving behind some orphaned users when they shifted to a strategy of simply copying Yahoo! Store (developed in the mid-90s by some Harvard CS grad students, and authored in Common Lisp!).

Contrast Point 1: The source code and documentation for ACS have been scrutinized by thousands of programmers worldwide; the iCat source code has probably never gone beyond a handful of cubicles.

Contrast Point 2: iCat ran out of money and had to sell out to Intel to pay back their investors, then do whatever Intel told them to do. ArsDigita is a company with $15 million in annual revenue (as of November 1999), doubling every six months. Since we've funded our growth out of operating profit, we don't have any investors to pay back. It is tough for me to see how we could go bust, but if we did the thousands of publishers who are using our software could continue to operate and extend it in comfort. That's part of the beauty of open source. We have faith in ArsDigita but we aren't asking you to bet the future of your company on the future of our company.

For additional perspective on the state-of-the-art in the closed-source world, read http://www.peterme.com/bvsucks/.

Getting the Software

Millesgarden. Stockholm, Sweden The software is available in a TAR file from http://software.arsdigita.com/dist/. Pick the most recent version! We only keep the old ones around so that you can humiliate us with how stupid we used to be.

The software.arsdigita.com server is worth some discussion. Basically it exists so that you can browse around our source code without having to download and untar the software. The AOLserver behind it is configured to serve .tcl and .adp pages (normally dynamic content) as plain text. The server is also rooted one directory lower than normal so that you can see the parameters file and the private Tcl scripts that are loaded at server startup.

Check out a random page such as http://software.arsdigita.com/www/gc/index.tcl (source code for http://www.photo.net/gc/). Even more instructive is the /doc/sql directory, which contains the data model for the ACS.



Text and photos Copyright 1998-1999 Philip Greenspun.
philg@mit.edu

Reader's Comments

To do a really low cost collaborative web server, can the Solid database system be used instead of Oracle? Are drivers available for that software?

-- Peter Tower, February 9, 1999
You can use Solid and AOL distributes a driver for it. However, you'll have to go through our code and remove all the Oracle-specific stuff. It is probably cheaper and better to simply get an Oracle license.

-- Philip Greenspun, March 1, 1999
Also, Solid has changed its licensing fee structure and market focus; it's now $3,000 for a single-server license on any platform. Not such a bargain.

-- Nic Wolff, March 17, 1999
I'm curious about the design of you Software developement manager. Do you have any more detailed info on it ? Since it has its own page describing it this doesn't seem like the right place to go into more detail, but that page doesn't have a comment section ...

-- Lee Schumacher, April 6, 1999
Note that InterBase port of ACS is now available at http://www.lavsa.com/acs-interbase/. Although it has some limitations, I find it quite useful. Among other things, it runs InterBase Q&A discussion forum.

-- Sebastian Skracic, May 27, 1999
AOLServer is now open-source, I believe. I have not download it yet since its only beta version. Waiting for the regular version!

Sanjeev

-- Sanjeev Mohindra, July 30, 1999

When we are talking about lowering the cost why not work on open source databases like mysql ? Whatever one says oracle is not a very cheap database to buy and moreover the connectivity cost for hosting it is prohibitively high.

-- Richy Marx, January 13, 2000
Consider an alternate view to the ACS (ArsDigita Community System) assertion of insulation by using open source components. It is January 2000 and the required release of AOL server is the closed source version from October, 1998. Also the 'open source' license is strongly implied to be the GPL, but no copy appears on the referenced pages.

So, why trust your software to a system without a strong enough development community to get a full release out every year or so? The trade-offs would be to use the Apache webserver with its larger development and user community group, and to establish the ACS for a subset of Apache configurations.

It is wise to look past the open source label to see if it provides actual investment protection.

-- Charles Merriam, January 26, 2000

Although not specifically stated, the implication of "opensource" is that it also be popular. While I am very impressed with the technology and acomplishments with the ACS system, I fear students will be learning specfic technologies which are not in wide use. Rather, I suggest other more popular mainstream and rapidly advancing "TRUE" opensource technologies be considered. For example, the relatively new PHP scripting tool has recently topped the one million installion mark on worldwide Apache servers. Based on its relative youth, this milestone of achievement should be indication of its skyrocket growth in popularity. This tool interfaces with many database backends but the two most popular are MYSQL (for extreme performance)and PostgreSQL for transaction suppport. Visit http://www.php.net for more info.

-- Scott Perkins, January 26, 2000
Ummm. PHP is just a programming language, just like TCL (that ars-digita uses) is a programming language. You can't compare Ars Digita to PHP. You compare PHP to TCL, both of which are mega-popular.

-- Chris Bitmead, January 27, 2000
Actually there is much to be learned from using the ACS/AOlserver other than its solid proven technology. Vignette's StoryServer software is some of the hottest commercial software to be powering websites from dot.com's to Fortune 500 firms... and it uses... bingo... a TCL scripting engine. In fact, I just converted a commercial site from PHP to Storyserver and frankly it was much cleaner in TCL.

-- Jamie Ross, February 5, 2000
In response to Charles's comment above in which he refers to AOLserver and says, "why trust your software to a system without a strong enough development community to get a full release out every year". The only reason the last non-beta version of AOLserver was released (as of February 2000) in October 98 is because AOLserver is so stable and well designed it doesn't need to be released with bugfixes every two days like Apache. I'd be surprised if Apache could go for two weeks without a patch. The "must upgrade often" syndrome is the Microsoft way to hide bugs.

And, the ACS no longer "requires" the closed-source AS 2.3 version (you could get it running on AS3 in January too).

-- No One, February 9, 2000

Scott Perkins wrote "While I am very impressed with the technology and acomplishments with the ACS system, I fear students will be learning specfic technologies which are not in wide use". I think the value of something such as ACS goes far beyond the specific tools used. Anyone who learns to build web systems with this kind of an integrated approach is likely to implement effective solutions to difficult problems no matter whether they use ASP, Servlets, mod_perl or whatever.

On the other hand, a peek at the AOLserver source code will tell you that it is a superbly engineered piece of software with a very thoughtful implementation of the Tcl API. In combination with Oracle on Unix you have a very solid architecture.

-- V. Nair, February 16, 2000

Re:Databases - We're in the final stages of setting up our intranet using the ACS ported to Postgres. It's working great and we're really happy this far - a great system. The specs are (aolserver 3 beta6; postgreSQL 7 beta 3; ACS/pg 3.2.2). This circumvents the need to rely on either Oracle or Solid, and the online community support we've discoverd has been impressive.

-- Grant Schofield, May 3, 2000
Three years ago, Chris could compare PHP and TCL by saying that it acts of "an other" language...

It was already a fast comparison and now it's even more obvious: PHP is a programming language (with whole share) whereas TCL (and well of others) are only languages of script.

-- Fabrice PELLEAU, January 9, 2003

Add a comment

Related Links

Add a link