Multics > About
20 Sep 2020

Maintenance tasks

History | People | Library | Sites | About Search

Tom Van Vleck

This file describes maintenance tasks for multicians.org. General design principles are listed in Design Principles for the Multicians Website.

Roles

These are the current roles.

Hosting for multicians.org

Hosting for this site is provided by the hosting site admin, who pays the domain name registration fees and web hosting fees as a contribution to the online community. The web site has been hosted at Pair Networks Inc in Pittsburgh PA since 2001. Pair is referred to as the Information Service Provider (ISP).

We are limited in both storage space and bandwidth as a result of Pair's account parameters. The editor will take whatever steps are necessary to avoid exceeding the (fairly high) limits imposed by the ISP, since this could cost the editor a lot more money than he can spare.

The ISP has a fairly reasonable policy on "slashdotting" but more than one in a month could be a problem.

We are also bound by Pair's Acceptable Use Policies. So far this has not been a problem.

The advantages of hosting at Pair are

Pair has been in business since 1996 and says it does not plan to change management, merge, or be sold.

Pair supports

Site Maintenance tasks for Site Editor

Files on the site include graphics and PDFs, HTML provided by others, and HTML compiled from HTMX.

The open source expandfile tool translates HTMX to HTML. (Overview of HTMX. Details of HTMX Language.)

Translation is invoked using the Unix make command for any file whose source is newer than its object.

After translation, the Makefile invokes HTML Tidy to warn if there are errors. Cure these and re-install.

Managing Images

Images are stored in directory /mulimg in the object directory. (This directory is linked to from the source directory so that image macros can get the size of the image.) They can be .jpg, .gif, or .png files.

Maintaining Links in HTML files

Use the bitsaversmultics HTMX macro to generate HTML links to scanned Multics manuals at bitsavers.org,
e.g. %[*callv,bitsaversmultics,="AN61A_storageSysPLM_Sep78.pdf",="AN61, Storage System PLM"]%.

Use the mitsourcefile and mitsourcearc macros to generate links to Multics source at MIT.
e.g. %[*callv,mitsourcefile,="Multics/ldd/include/flagbox.incl.pl1",="flagbox"]%.

Generate other external links using the HTMX construct {!tag text!} which looks up "tag" in the external links table and inserts an HTML link with globe icon and TITLE tag.

Generate multicians links using {[tag text]} which looks up "tag" in the multicians table and inserts an HTML link with TITLE tag.

Generate glossary links using {{tag text}} which looks up "tag" in the glossary table and inserts an HTML link with TITLE tag.

Generate internal links using {@tag text@} which looks up "tag" in the pages table and inserts an HTML link with TITLE tag.

Use {:text:} to format items with the "cmd" class, {=text=} to format items with the "pathname" class, {+text+} to format items with the "code" class, all defined in mxstyle.css.

Updating the Main Page

Edit multics.htmx in the source directory.

Execute make and then check the local copy of the compiled files. Fix any errors or warnings. Then execute make install to publish the new story.

make will automatically update the main page if other pages change that would change totals in the main page. See Makefile.

(explain how to add a photo to the sliding gallery)

Adding or Updating an Offsite Link

Offsite links are specified in loadext.sql, loadbib.sql, or loadm.sql.

For an HTML link to a file not on multicians.org used in article text, edit loadext.sql to add or modify a row in the external links table. This row specifies an internal tag and the lin target URL. Then generate an HTML link to the external file, using the {!tag anchor text!} syntax. The globe icon, etc will be generated by expandfile.

Multicians home page URLs are specified in a row in the m table defined in loadm.sql.

External URLs of items in the bibliography are specified in a row in the biblio table defined in loadbib.sql.

After editing the SQL files, execute make install to recompile source files that reference them. In some cases it may be necessary to touch a source file to cause it to be re-expanded.

Check every few months for dead links. I use the Mac tool Integrity.

Adding or Updating a Multician

A site visitor may use the Multician add/update form at mform.html

If the visitor has JavaScript enabled, changing the person name in mform.htmx sends an XHR (Ajax) request to mxregister4.cgi at www.multicians.org to look for existing database entries in the m table and fill in the rest of the form. If multiple database entries match the partial person name, the form code presents a pick list.

When the visitor submits the completed form, this action invokes mxregister.cgi at www.multicians.org, which sends mail to the editor with SQL rows for database input. (It doesn't update the database.) The mail looks like this:

  loadm.sql
  ('fooch,melvint', 'Fooch, Melvin T', 'User (MIT): stuff', '', '', 'mail 2020-04-01 11:54', '', 'Y', '', null),

  hiddenmail.sql
  ('fooch,melvint', 'fooch@supertool.com', '', '2020-04-01 11:54'),

Alternatively, a Multician may send a mail message to the editor instead of using the form, or the editor may discover a Multician in some other publication, web page, or forum.

The editor verifies the form info, including whether the submitter is actually a Multics contributor. The editor may know the sender personally or know someone who does. They editor may send mail to the submitter and ask for details. The editor also has a standard form letter that says, "Sorry, liking Multics doesn't make you a Multician."

If a Multician's name or "did" contains apostrophes, they should be replaced by '. Loose ampersands should be replaced by &. Accented characters like the accented o in Corby's name must be replaced by character escapes such as ó.

The editor checks the URL if supplied, and may correspond with the submitter if it has a problem.

The editor adjusts the submitted "did" field to match terminology in other entries.. e.g. OU versus Oakland University, UC versus Calgary. Formatting tags such as <cite> and <em> in "did" fields canot be used: the user's browser will remove them when parsing the Ajax XML message. It is OK to use HTMX constructs such as {@page title@} to link to other site pages.

The editor checks the mail info add adds or updates the corresponding row in hiddenmail.sql in the source directory. Note that if a new multicians.org email address is added, the site editor must also log into the web account hosting control panel and add a mail forwarding recipe. See "mail reflector maintenance" below.

If a Multician has won an award or medal, the editor should add or update a row in the mxawards table in mxawards.sql in the source directory. For an award type not seen before, the editor must also add a row describing the award in the awards table.

When the editor executes make after updating loadm.sql or hiddenmail.sql, the Makefile will rsync updated copies of the m, mmail, and mgemail tables to the web hosting computer and then execute the remote command reloadm on the hosting computer via SSH. The reloadm command loads the SQL database on the hosting site with new copies of the m and mmail tables. Its output looks like this:

  total 892
  -rw-r--r--  1 thvv  users    7897 Apr 25 16:28 grpmail.sql
  -rw-r--r--  1 thvv  users   52710 Mar 31 19:44 hiddenmail.sql
  -rw-r--r--  1 thvv  users  258790 Apr 25 16:39 loadm.sql
  reloaded m on timayo
  reloaded mmail on timayo

The editor sends a welcome/thanks email to each new or updated Multician.

Adding or Updating Site Information

Verify the site info and add or update the site's row in loadsites.sql in the source directory.

Update the site history if there is one. Write a new site history page if you have enough information to make it interesting.

If new dates are added, fix site timeline datafile site-timeline.txt by hand.

Execute make and then check the local copy of the compiled files. Fix any errors or warnings. Then execute make install to publish the new story.

Adding or Updating the Bibliography

Contributors send documents to the editor, or send URLs to online documents, or the editor discovers documents on the Web or scans paper documents. If a document file is provided, find space for a copy, and ensure that the document is readable, searchable, and small enough to load quickly. (Providing copies on the site avoids issues of other sites taking the document down in the future.) Ensure that the rights of the document creator are respected, and that the content is accurate and informational.

Add or update a row in table biblio in loadbib.sql in the source directory. Give this row a document identifier like "Smith100" with an author name and unique sequence number.

For each author of the document, add a row to table "authors" giving the document identifier, the author name as it appears in the document, and the ordinal position of the author name in the citation.

For each author name, ensure that there is a row in table "aka" that links the document-style name with the canonical "name" field in loadm.sql.

Executing make will load the SQL tables and regenerate the bibliography, the MTB index, the MCB index, the MSPM TOC, the MSB index, the MCR index, the MAB index, and the publication counts in the Multicians list. The Makefile will execute the shell script checkbib.sh to check that there are no missing rows in the bibliography tables and that they match the Multicians table. Fix any problems reported. Then execute make install to publish the new files.

Adding or updating glossary entries

Edit or add paragraphs to g1.sql in the source directory. Use special tags in other files on the site to generate links to glossary entries and Multicians list entries, and to mark commands, pathnames, and code. One oddity: to insert a backslash in the output, you have to put EIGHT backslashes in the database file, because it gets expanded three times.

Execute make and then check the local copy of the bibliography. Fix any errors or warnings. Then execute make install to publish the new story.

Adding a new story

If necessary, propose changes to the submitter so that the context of the story is clear to a broad audience, and so that story fits into the rest of the site.

Contributed Text

If a Multician sends information in plain text, convert it to simple HTMX. (If the contributor provided plain text with empty lines as paragraph separators, convert those to </p><p>, and convert bulleted lists to <ul> <li> ... </li> </ul>.) You can use HTML Tidy on your converted file to have it generate standardized HTML.

If you're provided text in Microsoft Word, it is often best to just copy the text to the clipboard and paste it into a simple text file.

In either case, turn it into a HTMX file by and wrapping the contributor's text in a {:*block,&body,^END:} ... END construct, followed by an *include of the standard wrapper (look at an existing story for the pattern).

Contributed HTML

If a Multician provides a story in HTML, turn it into a HTMX file by removing the HEAD and BODY tags and wrapping the user's text in a {:*block,&body,^END:} ... END construct, followed by an *include of the standard wrapper (look at an existing story for the pattern).

CSS Definitions

Use the standard site CSS definitions; if the story uses specialized CSS constructs, create a {:*block,&extrastyle,^END:} (css) ... END block. Use <p> and </p> around paragraphs and similar simple markup ... HTML Tidy should find no errors or warnings in the converted file.

If the story is a long one, using H2 and H3 headings may make it easier to follow. References to other pages on the site, or to external documents, may help readers understand the relation of this page to other information on the Web.

If possible, include pictures. Getting useful pictures with sufficient pixels may require a mail discussion with the submitter., and the pictures will often need color and contrast adjustment to display well in HTML. Use macros to generate IMG tags for pictures, as described below.

Create new files storyname.html in the object directory and storyname.htmx in the source directory.

Do the other "new page" operations below. Adding the page to loadpages.sql with a nonblank "site" entry and a kind of "people" triggers its inclusion in the stories index. Set the nextstory field of an existing page to point to this one, and the nextstory field of the new entry to the old value, so "Next Story" link works. (touch the source of the existing page so its navigation links will be regenerated also.)

Spell check the local copy of the file and correct any errors. Execute make and then check the local copy of the compiled files. Fix any errors or warnings. Then execute make install to publish the new story.

Adding or updating other class 2 and class 3 pages

Edit or add filename.htmx in the source directory. Don't edit the HTML object files directly.

Search the rest of the site to see if links to the new page should be added.

Use the wrapper macros: set title, description, keywords, and headingdate, then define a block called body that contains the HTML of the content. At the bottom, include pagewrapper.htmi to cause the content to be formatted with standard headings and footers. If necessary, use optional blocks extrastyle for extra lines in the HEAD section, and extratail to add content to the footer.

Use the getimgtag, getimgdiv, getfancybox_li, or getimgpopdiv macros for references to images. For new images, make a 150x150 thumbnail and a -2x version of the file if possible, and update multhumbs.sql. If the images have picture of Multicians, add the appropriate 'gallery' field to the multhumbs row.

When a new HTML page is added,

Search the rest of the site to see if links to the new page should be added, and if so, add links referencing the pages table entry.

Spell check the local copy of the new file and correct any errors. Execute make. Fix any errors or warnings. If you made major updates, use W3 validator to check the new local copy of the file. Then execute make install to publish the new or updated story.

Adding or updating class 4 pages (formatted by others)

Either convert the file to a class 3 page, or

Place the HTML file directly in the object directory. Edit it to insert navigation to the rest of the site at the bottom, so that if a visitor finds the file from a search engine, they can find the rest of the site.

Add a row for the file to loadpages.sql and an entry in makemxtarextrafiles in the source directory.

Optionally edit the HTML to add mxstyle.css for uniform heading styles if this doesn't mess up the appearance of the document; discuss this with the file author.

Change mail addresses in the HTML to HREF links via the Multicians database, so that the submitter's mail address cannot be scraped by spammers.

Verify that the local copy of the file looks OK. Execute make install to publish the page.

Adding or updating a PDF

Ensure that the PDF has OCR text so that indexers can see its content.

Place the file directly in the object directory.

Add its name to loadpages.sql. For the "source" use something like "../multics/foo.pdf" -- that is, the file is both source and object. This entry ensures that it is in sitemap.html, and that the date modified displayed is that of the PDF file.

Add its name to makemxtarextrafiles in the source directory. This ensures that the PDF file will be included in the TAR of the whole site.

Execute make install to publish the page.

For All Changes

Run W3 Validator on each new class 1-3 HTML file. It checks for more errors than HTML Tidy. (Both Tidy and W3 Validator fuss about some things that we include anyway: for example, the Facebook link code and the Google search code use non-standard HTML, and we include a few deprecated CSS constructs necessary for correct display in old browsers like MSIE.)

Update loadc.sql with a one-line description of each change, and execute make to generate changes.html describing what's new. Changing this file will regenerate the HTML site map, RSS feed, and Google sitemap.xml.gz files. (Some minor edits, typo corrections, etc. do not need a line in 'changes.' For intermal machinery changes, enter a line in loadc.sql indicating that we did make a change but that changes.html need not be updated.)

The htmx, sql and other source files are stored in a local git repository (graphics, generated files, and binary files are not included). The make install command will do a git status to remind you to do a git commit -a with a summary of changes. Currently this repository is local to the site editor's machine.

To make a tarfile of the whole site, first touch makemxtarextrafiles and then do a make install. This sends the shell script make_mx_tar to the ISP. Run that script from the ISP command line.

To manage the Google Custom Search box, log into https://cse.google.com/cse/all. Add and delete sites that are searched.

Monitoring Implementation Features

Editors and admins must keep aware of and respond to the following issues:

Smoke Test by Site Editor

Before updating the hosting site, check the appearance of the local copy of the site. Check major updates in Firefox/Mac, Safari/Mac, Chrome/Mac. (I probably ought to check Chrome and Edge on Windows also, but I do not have a Windows 10 machine.) Also check when new browser versions of Chrome, Firefox, and Edge are released. Run the W3C HTML5 Validator on a sample of pages, in case the validator changes its criteria or the HTML language changes (which happens sometimes).

Check for:

If all looks OK, execute make install.

Occasionally check that the actual online site loads correctly and looks right, to make sure that nothing is down or misconfigured at the ISP. Especially check this after making any kind of change to the web server configuration or file access controls.

Details to Check on Hosting Site When Deploying

Hosting site admins should ensure the following.

Hosting site admins can execute shell script make_mx_tar to generate a tarfile for transmission to mirror sites when needed.

Site Maintenance by Hosting Site Admin

The main hosting site is currently at the ISP pair.net. The account has a control panel that manages incoming email, domain names, billing, and other details. The domain name multicians.org is registered at pairdomains.com and this site has its own password and control panel.

multicians.org mail reflector maintenance

The procmail script multicians.procmail in the home directory of the account hosting multicians.org redirects selected mail messages.

See multicians-org.html for details.

We are trying to phase this facility out since the ISP may not support it forever, and because less expensive plans at the ISP do not support procmail.

The redirector script is automatically generated by expandfile with procmail.tpt from the multicians table by the Makefile when loadm.sql is updated.

When a new multicians.org address is added, it must also be added to the list of valid entries in the ISP's mail recipe control panel.

Mail Forwarding Repairs

Very occasionally, Pair screws up the mail forwarding. This happened in 2004, 2007, 2014, and 2019 (often when I was traveling). Usually this happens when some other problem, such as disk errors or a spam attack, causes web server load to spike. Inexperienced sysadmins then panic, notice multiple procmail processes running (checking incoming mail for spam), decide that procmail is the problem, and either break .procmailrc or set access on /usr/local/bin/procmail to null. If they choose the first option, then all incoming mail including spam gets dumped into a single file with no filtering. If they choose the second option, then all incoming mail is bounced with a permanent fatal error from MAILER-DAEMON. The site editor must then call Pair support on the phone and teach them their jobs, which takes about an hour on hold. The right solution is usually to fix any I/O problems, and then wait 10 or 15 minutes for procmail to clean out the mail queue. Recovery from such a mess depends on how they panicked: in the first case, this means taking the huge mailbox and piping it through procmail; in the second case, the mails sent are lost forever, but Pair support can grep the qmail log to find out the senders of messages that were discarded.

alt.os.multics FAQ Maintenance

Currently the site admin causes an FAQ posting to alt.os.multics on the first of every month. At some point we will stop doing this, when USENET is no longer interesting.

In order to post to USENET you need an account at some USENET service. Currently we have a free account at etermal-september.org.

To post the short FAQ, run make auto2 on the first of each month. This operation invokes a program that updates the alt.os.multics FAQ with totals from the website.

Groups.io Multicians group administration

If applicants meet the group criteria (Multics contributor, including user) accept them. (Otherwise send a form letter.) Use the 'admin' control panel of groups.io to approve new applications. Add the user's subscription email to grpmail.sql (nothing uses this table yet).

LinkedIn Multicians group administration

If applicants meet the group criteria (Multics contributor, including user) accept them.

Domain name management

The domain name multicians.org is registered at pairdomains.com, which has a somewhat cryptic website. Some domain name operations need to be done partly with the domain registrar site and partly with the Pair control panel.

Pair website management supports https access and automatic renewal, for free, in the Pair control panel. So far, it has worked with no problems. It is valuable to enable this feature, since Google search rankings penalize non-https sites.

Search Engine Optimization

Most search engine optimization activity is not relevant to multicians.org. Keyword stuffing and fake links to our site from link farms are a bad idea. Make sure that META DESCRIPTION tags are accurate and incorporate the words that visitors might use in a query.

Google Webmaster Tools

We have a Google Webmaster Tools account. They sometimes send warnings that our pages have problems they detected when indexing our site. Some of these are issues that should be fixed.. others are not useful.

We don't use Google Analytics, because it would inject JavaScript into our pages, and because users may not wish to have Google tracking their web browsing. For the same reason, we removed the code that creates a Facebook "like".

Details

Rules for maintenance and content of multicians.org are in Web Site Design.
Details about the site build process and tools for multicians.org are listed in Build process for the Multicians Website.