Multics > About
02 Feb 2024

Web Site Build Process

Tom Van Vleck

These are my notes on the site building machinery, including usage, internal details, and possible future improvements.

Rules for maintenance and content of multicians.org are in Web Site Design.
Detailed instructions for maintenance tasks for multicians.org are listed in Maintenance tasks for the Multicians Website.

Summary

The multicians.org website is built on the site editor's computer from HTMX source using Unix tools and the open source program expandfile, and served by one or more web host computers. Expandfile is available from github.

Web Pages

The DNS A records for https://multicians.org specify that web requests are sent to a Virtual Private Server (VPS) at the Information Services Provider (ISP) pair.com. The VPS serves pages using a web server. Hosting at Pair costs me money every month. I think it is worth it: they have good uptime, multiple backbone connections, good bandwidth, and keep Ubuntu Linux updated with security fixes.

  1. The Multicians web site is served as static HTML pages. (i.e. no database accesses to display a page.)
  2. Other online static objects (PDFs, graphics, etc.) are not built from HTMX. (the biggest part of storage)
  3. The complete multicians.org site occupies about 5GB.

The HTML pages are created at compile time from HTMX source and configuration files. Local SQL databases are used to build some HTMX source into HTML at compile time offline.

expandfile is also used to build some HTMX files in from the local SQL database, e.g. the bibliography. I do not leave data in SQL overnight. (Old prejudice from the 80s.) The tables are small, reloading from text is quick, and schema changes are easier. There are about 25 tables, stored in text files, which the Makefile loads into MySQL if they have changed. About 71 SQL queries are used to build the site, mostly in the bibliography files. (The text files define column names that are used in SQL queries in the HTMX source; these must match.)

Mail Handling

The mail host is a web server specified in the MX record for multicians.org. It could be a different server from the one that serves multicians.org.

Site visitors may invoke a FORM to send mail to a registered Multician. The form's ACTION specifies a CGI that runs on the mail host. Sending mail using the form requires the sender to answer a "spam excluder" challenge. the CGI looks up the Multician's name to obtain a hidden external mail address, and sends the message. The mail host checks that every message being sent is not spam, and enforces rate limits.

When I set up Multicians mail forwarding in the 90s, I promised Multicians I would not let their addresses be visible to spammers scraping web pages. Multicians' actual mail addresses are stored in a hidden SQL database on the mail host.

Site visitors may invoke a FORM to add or update their Multicians registration, including an optional mail address. The form can look up a Multician's name in the SQL database and enter new data or corrections. The form does not display the hidden mail address. Submitting this form sends mail to the website maintainer, who verifies that the request is valid and updates the SQL database.

66 Multicians have requested forwarding mail addresses on multicians.org. Enabling a user's multicians.org address requires altering the configuration of the mail host to accept the address. If someone with a multicians.org address changes their forwarding address, the mail server need not be updated, just the SQL that generates the procmail instructions. Incoming mail for Multicians is checked for spam using by the mail host's procmail setup.

Unix tools flow

1. Build Process

1.1 General Principles

The build process attempts to

1.2 ISP Host Copy and Editor's copy

The domain multicians.org maps to a virtual web host operated by pair.com that serves the Multicians website. Web hosts run a web server process (Apache) that responds to visitor HTML requests with the content of the requested site resources.

The Multicians site has been mirrored at additional other ISP hosts in the past, but as of 2020 there are no mirrors. (The mirroring at stratus.com was done using an FTP server, unable to support server-side execution, and this constrained the site implementation. This turned out to be a good limitation.)

The site editor's computer has a complete copy of the multicians.org website content. To make a change, the editor prepares updated or added content on the local computer, checks it locally, and then publishes modified pages to the multicians account on the ISP using rsync. When the site was mirrored, updates were made available to the mirrors every few months.

1.3 Source and Object

Rather than editing HTML language files, the site editor works on local source files that are translated to HTML. Source files are written in the "HTMX" (HTML eXtended) language. .htmx files are translated to .html files by a program called expandfile. Expandfile has optional features designed for multicians.org: see Expandfile Multics Arguments.

Source and object are kept in separate directories on the editor's computer. (A copy of the source directory is also stored at the ISP, for backup, not available on the Web.)

A few HTML files, supplied by others, do not have HTMX source. PDF and graphic files do not have a corresponding source. Copies of these files are kept on the site editor's local computer and sync'd to the web host computer if they change.

1.4 Steps in updating an HTML file

The basic steps in generating an update to multicians.org are:

  1. Edit one or more local source files, and possibly add graphics or other files to the object directories.
  2. Execute the Unix make command to rebuild local object files by expanding SQL and HTMX files to HTML with expandfile.
  3. View and check the local site, including any object files generated by make and links to and from them.
  4. Iterate 1-3 until satisfied.
  5. Execute the make install command to push the local object image to the web host directory.

This process depends on software and configuration files installed on a site editor's local computer.

1.5 Use of Make Utility

The website's files on the web host computer(s) are kept up to date by executing the make command in the source directory on the editor's computer. (The make command is available on Mac, Linux, Unix, and Windows systems. The original make command was created for Unix by Multician Stuart Feldman.)

The make command is used for two major tasks:

Why do it this way? This setup

Editors don't need to know how the machinery works to use it for routine changes.

There are many features of make that I haven't learned yet. The Makefile could be better. Suggestions are welcome.

1.6 Use of Rsync utility

Why rsync?

2. Installing and Configuring the Build Machinery

A number of files have be be in the right places on the editor's computer for the build process to work.

2.1 Object files

2.1.1 HTML Object files

On the editor's computer, there is a complete local image of all files needed to display the web site, such that the web hosting computer can be updated using the Unix rsync command. These files are kept in the object directory. These files include .html files, .pdf files, JavaScript files, CSS files, and graphics.

After making changes and recompiling the site, the site editor uses a local web server to check the appearnce of the local copy of the website with a few web browsers, before updating it to the web host.

A "smoke test" listing a few breakable features is defined in Maintenance tasks for the Multicians Website. New pages should be checked with https://validator.w3.org/. The build script runs HTML Tidy on each file in silent mode, and will complain about mechanical errors.

A few files are excluded from the rsync command that updates the ISP object directory from the local copy: for instance, configuration files that define local directory structure, and artifacts of the Mac file system.

2.1.2 Subdirectories

The object directory has subdirectories for groups of files and graphics, such as the /mtbs directory and /reunion-04. These subdirectories contain their own index.html. Subdirectories symlink to CSS and graphic files and the /js directory in the parent directory.

2.1.3 Graphics

Graphics files are stored in the subdirectory /mulimg, with thumbnails in /thumbnails150. Many graphics have a standard version and a -2x version for High DPI displays. Graphics files are listed in multhumbs.sql so that the graphic indexes multics-images.html and people-pictures.html can be generated.

2.1.4 PDF Files

PDF files are stored in the object hierarchy but do not have a corresponding source. If PDFs are generated from other software, the site editor keeps the source on the local computer and does the generation manually.

PDF files scanned from original documents should have OCR text so that web indexers can find them. I have used Adobe Acrobat Pro, which works OK on many documents. Old documents with colored backgrounds, poorly printed text, strange fonts, multi-column documents, and figures with text labels often lead to nonsense text. Retyping a whole document and re-drawing the figures may be preferable.

2.2 Source files

On the editor's computer, there is a source directory containing source files that are used to generate the object files. For most html files in the object directory, there is one .htmx file in the source directory in the object directory. (Sometimes it's more complicated. See below.)

2.2.1 Subdirectories

Wherever there is a subdirectory in the object hierarchy, there is a corresponding subdirectory in the source tree. These subdirectories contain their own index.htmx and have sub-Makefiles pointed to by the main Makefile. In such subdirectories, there are symlinks to .htmi files and graphic and thumbnail directories in the main directory.

There is a symlink to menubar.htmi in the main source directory. This file, generated by gen-menubar.htmt, prefixes each filename with %[hrprefix]% which is set by configx.htmi to be ../. Every invocation of expandfile in the Makefile expands configx.htmi before expanding the source file. This ensures that all links in the menus point correctly to files in the parent directory.

2.3 Expandfile

The Makefile uses the expandfile utility program to create HTML object files from HTMX source. expandfile and the software it depends on must be installed on the editor's computer. This program is written in Perl and is open source provided by the editor. See expandfile Installation Instructions.

2.3.1 Expandfile wrappers

Each HTMX file for a web page defines its content inside several BLOCKs. One is named body and lays out the page content. The HTMX file ends with an *include of a wrapper file that defines the template look and feel of multicians.org, and expands the body block. There are several different wrappers, depending on the type of page. Most wrappers produce a header, footer, and drop-down and alternate menus. Most page wrappers contain HTML that adapts the page to mobile devices by setting a "viewport" and providing alternate CSS rules by testing screen size.

Wrapper fileNotes
pagewrapper.htmiStandard page wrapper
.. class2head.htmiLays out the DIV for the headbar including date and title, inserts menubar.htmi
headpagewrapper.htmiSpecial wrapper for index.html
codewrapper.htmiWrapper for a program listing, no viewport set
glospagewrapper.htmiWrapper for glossary pages
mspmwrapper.htmiWrapper for an MSPM page
mtbwrapper.htmiWrapper for an MTB, no viewport set
noncenteredpagewrapper.htmiWrapper for pages with long lines
paperwrapper.htmiWrapper for conference papers, no viewport set
widepagewrapper.htmiWrapper for site-timeline.htmx, cannot be reflowed, no viewport set

2.3.1.1 Auxiliary include files

Page wrappers and source files include additional files. Here are the most popular.

FileNotes
class2head.htmiLays out the DIV for the headbar including date and title, inserts menubar.htmi and altmenubar
menubar.htmiGenerated by gen-menubar.htmt from menubar.sql. Defines DIVs redmenubar and altmenubar.
mxstdfmt.htmiincluded by pagewrapper.htmi. Standard page layout, CSS that hides/shows Drop-down and and mobile menus, loads jquery,includes ddm.htmi
ddm.htmiJavaScript function for dropdown menus (uses jQuery).

2.3.2 Expandfile Macros

Expandfile provides useful HTML-generating macros. The Multics source file mxlib.htmi contains Multics-specific macros such as bibliolink and includes the library htmxlib.htmi, which provides macros for generating image references, such as getimgdiv. Generating multicians.org uses macro calls to generate IMG tags for photos. HTMX source files begin by including mxlib.htmi.

2.3.3 Tools invoked by Expandfile Scripts

Some expandfile templates invoke the *shell builtin function to execute helper tools that return character strings to be used in the expansion. These tools are either standard Unix utilities such as date, or tools provided by the site editor (many written in Perl). Tools specific to the website are stored in the subdirectory /tools of the source directory. The most important tool in this directory is expandfile and its components. (The /tools directory also contains a few other files for completeness, such as copies of .js files used in the object directory.)

2.4 SQL database

Loops Over Data Tables

About 45 expandfile template files use the *sqlloop builtin function one or more times to access my local MySQL database. The database tables are loaded from text .sql files which DROP, CREATE, and reload the tables each time the .sql file changes.

The Expandfile *sqlloop builtin expands templates for each selected row returned by an SQL query. Some MySQL tables are used to generate several HTMX templates: for example sites.html and site-timeline.html are created from loadsites.sql.

Formatting Lookup Macros

Special syntax in the source is expanded by expandfile into data retrieved from SQL queries. For example {{term anchor text}} expands to a hyperlink to the glossary entry for "term," with a TITLE attribute that previews the link target. Parameter values set in config2.htmi control which SQL tables are accessed for these definitions.

Database Connection Parameters

The *sqlloop builtin function and the special syntax references need the variables _xf_hostname, _xf_username, _xf_password, and _xf_database to be set to appropriate values for the local SQL database. Assignments to these values are kept in the local configuration file config2.htmi and most file expansions in the Makefile are performed by executing the little shell script expandfile_config, which invokes expandfile config2.htmi (rest of args)

The Makefile and shell scripts that execute mysql commands require that the file $HOME/.my.cnf is also set up with values for host, user, password, and database.

2.4.1 SQL tables

Text fileTablesUsageNotesRowsRef
g1.sqlglossaryOne row per paragraph in the Glossaryprintgloss3.pl writes 27 HTMX files1189{{ }}
home-slider.sqlhomesliderOne row per photo on multics.htmlPage build generates CSS sprites.15
loadbib.sqlbibtypes, biblio, authors, akaOne row per document, plus author infoexpand 7 templates to produce 7 HTMX files5058
loadc.sqlchangesOne row per changeCreates topchanges.htmi included by multics.htmx, changes.htmx, changes-old.htmx2231
loadext.sqlextrefOne row per external pageUsed in many pages.818{! !}
loadm.sqlmList of Multiciansproduces multicians.htmx2055{[ ]}
loadpages.sqlpagesPages of the websitesitemap.htmx, Google sitemap455{@ @}
loadsites.sqlsList of sitesextra rows for sites with complex history; used by sites.htmx and site-timeline.htmx102
chrono.sqlchdate, chevent, chflagChronological list of events.306
chrono2.sqlch2year, ch2thumbPictures in thumbnails150/ to decorate chrono.html.34
menubar.sqlmenubarLayout of jQuery dropdown menus and altmenubartwo levels80
msource.sqlmsourceOne row per source filesource-index.htmx5877
multhumbs.sqlmulthumbsOne row per thumbnailproduces multics-images.htmx663
mxawards.sqlmxawardsAwards earned by Multiciansused with loadm.sql253
publist.sqlpublistDocuments produced by Multiciansproduced by addwp.htmt, used with loadm.sql724
hiddenmail.sqlmmailEmail forwarding addresses (see 3.7).752

2.4.2 CSV files

Text fileUsageRows
grpsio.csvMembers of groups.io/multicians173

This list is created manually for now, by a group admin going to the "Admin" section of the mailing list page and clicking "Members" and "download". The file downloads as multicians\@groups.io.csv and is manually renamed to grpsio.csv. Each row of the file has an email address, user number, user display name, username, and then a list of flags.

2.5 Executable files

The editor's computer has a standard OS installation with programming tools and a command shell and standard Unix tools installed. The Unix or Mac operating system can be used. Tools such as Perl and MySQL are installed by a package manager in /usr/local/bin (I use Homebrew on the Mac). My tool expandfile comes with some helper executables written in Perl and is installed in $HOME/bin. Additional tools are in the /tools subdirectory of the source directory.

Windows users can use a Unix-like environment such as Cygwin. For Windows 10, there is "Windows Subsystem for Linux". I have never used it; supposedly it provides a Unix kernel and Bash-like shell, and supports Perl and MySQL.

The editor's account needs a bin directory below the home directory, and a search path set up to find commands in these directories.

expandfile Install instructions describes the setup of Perl, CPAN, MySQL and expandfile.

2.6 Environment

The editor's computer has the environment variable PERL5LIB set to point to the correct version of Perl, and has a PATH that includes a /bin directory with site building tools. .ssh keys are set up so that files can be sent from the editor's computer to the web host: they are automatically loaded using ssh-add when the editor logs in.

2.7 Symbolic links from source directory to object directory

The source directory needs a few links to files in the object directory. In particular, in order for the image reference macros such as getimgdiv to work correctly, they need to access the pixel dimensions of target images (by invoking the helper program (:gifsize2:) with the *shell builtin). Symbolic links to the object subdirectories /mulimg and /thumbnails150 are placed in the source directory.

3. How the Build Process Works

This section describes the inner workings of the build machinery.

3.1 Make

The Unix make utility generates the object directory from the source directory. It is driven by a file in the source directory called Makefile.

The Multics website's Makefile has recipes that expand any .htmx file that is older than the corresponding .html. In addition, each SQL table is loaded from a .sql text file, and the Makefile ensures that if the .sql file changes, it is loaded into SQL and then relevant .htmx files are generated. make takes care of ordering the generation steps.

3.1.1 Makefile Configuration

The Makefile refers to some external-world facts by symbolic configuration names. The actual values of these symbols is set in the file configmake.mk which is included by the Makefile. (This makes it possible to test new versions of the site or Makefile, and may someday support multiple editors.)

# Configuration constants for Makefile for Multics website
# place this in the multicians source directory
# this is where site-dependent values like path names are kept
PRIMARYWEBACCT = account@server
PRIMARYWEBPATH = directory path on server
PRIMARYWEBPATHSRC = directory path on server

3.1.2 Makefile Rule for Expanding Files

If you want to know how the build works, examine the Makefile. Makefiles define values for some variables, and then provide a set of rules. The Makefile for multicians.org rarely changes. Here is an important rule that invokes expandfile:

# Rule that makes the HTML files from HTMX files and includes. $(OBJ_DIR)/%.html: %.htmx $(CONFIG) $(INCLUDES) @chmod +w $@ $(EXPAND) $< > $@ @checknonempty $@ @-/usr/local/bin/tidy --markup no --quiet yes --show-warnings no $@ @chmod 444 $@

This rule means "Use expandfile to build the object file if the corresponding source, or any of the include files, or the config file, have changed."

Makefile rules have three parts: the "target" file(s) being made, a colon, a list of the "dependencies" -- files that the target depends on, and then an indented list of commands to execute to create the target from the dependencies.

The rule above says "do the commands if ANY of the dependency files is newer than the target."

(Note that if there is an .htmx file that does not have a corresponding .html file, make will do nothing. So when adding a new file to the site, the first step is to create an empty .html file that will be older than the new .htmx.)

make examines all the rules in Makefile and executes the rules in the "right" order, so that dependencies are built before the files that depend on them.

The percent sign in the target is a wildcard that matches any name. (Used in a dependency, on the other side of the colon, it has the same value.) This example will cause fred.html to be remade if fred.htmx is newer.

In the Makefile rule for making HTML files:

3.1.3 Makefile Variables

VariableContentsMeaning
CONFIGconfig2.htmiConstants for multicians.org used by expandfile, including the MySQL database connection parameters.
OBJ_DIR../multicsrelative pathname to the object dir from the source dir. config2.htmi has a var named obj_dir that must match this value.
INCLUDES menubar.touch class2head.htmi linktags.htmi textnav.htmi addrtag.htmi leftpanelstyle.htmi mspmstyle.htmi mxstdfmt.htmi pagewrapper.htmi headpagewrapper.htmi glospagewrapper.htmi noncenteredpagewrapper.htmi paperwrapper.htmi codewrapper.htmi htmxlib.htmi mxlib.htmia list of files that all expansions depend on
DATABASEthvv_userlistlocal MySQL database on the editor's computer
HOMEREL../prefix used by subdirectories to find the main directory
BINDIR$(HOME)/binlibrary directory on the editor's computer
EXPAND$(BINDIR)/expandfile_configShell script that invokes expandfile with first argument config2.htmi
SUBTGTSr2012d r2004d rvand mtbsd mabsd debugd blwd pgdlist of sub-make-targets that should be built as part of 'all'
LOGACTION$(BINDIR)/wtlog.shcommand executed to log updates

Some variables must be different on different build environments. These variables are set in the included configuration file configmake.mk. For example, OBJ_DIR might be set to ../multics_object, and EXPAND is set to $HOME/bin/expandfile.

Shell VariableMeaning
PRIMARYWEBACCTusername@DNS name of ISP web hosting computer
PRIMARYWEBPATHpathname of web directory on web host
PRIMARYWEBPATHSRCpathname of source directory on web host

(It may occur to you that the Makefile rule rebuilds some files that do not need rebuilding. For example, if any wrapper include file in the INCLUDES list changes, every file will be re-expanded, not just the ones that include the changed file. We could fix this, at the cost of increased complexity in the Makefile. Since it takes less than 2 minutes to re-expand all files, the savings are not worth the development and future maintenance effort.)

Besides rules for creating HTML files, there are also rules that specify additional dependencies for some files, in order to cause some files to be built after other files.

3.1.4 Other Unix Tools invoked by Makefile

As mentioned above, when .html files are generated, the Makefile executes the Unix HTML Tidy command to check if the files have HTML syntax errors.

The "install" recipe in the Makefile uses the rsync command to copy modified files from the object directory to the web host computer(s). The recipe causes a "make all" before executing, so to make a one-line change, the editor just edits the file and types "make install". (Check the version of rsync on the editor's computer. MacOS shipped an old version, 2.6.9. I needed to install version 3.2.7 from Homebrew, Pair has 3.2.5.)

3.2 Generation of HTMX from SQL

Certain HTMX files are generated by make from the SQL database if their .sql text file source changes. (These HTMX files are marked read-only so that an editor doesn't forget and edit them directly.) The Makefile notices that the .sql file has changed by noticing that the date modified for the .sql file is newer than the corresponding zero-length .touch file, loads the .sql file into the database, expands a template file for the modified table which generates one or more .htmx files, and executes touch filename.touch so the reload will be done only once. Updating the date modified on filename.touch also causes make rules that depend on this file to be executed.

Additional dependency rules for compiling .htmx files into .html are also present in the Makefile. The .touch file is listed as a dependency in a Makefile rule to cause HTML files to be rebuilt if they may depend on the table contents. There are about 10 such rules in Makefile. For example, the Makefile rule

  # .. rebuild about-multics for count of pages, graphics, or changes
  $(OBJ_DIR)/about-multics.html:  changes.touch pages.touch multhumbs.touch

declares that about-multics.htmx should be re-expanded into about-multics.html if any of the "changes", "pages", or "multhumbs" database tables change, in order to ensure that the counts of changes, pages, or images are re-computed. This rule adds reasons to regenerate its target to those declared in the standard rule for .html files, and ensures that about-multics.html is regenerated after its prerequisites are generated. (If we discover that we have to make twice to cause an output file to be correct, this is a sign that a dependency rule is missing.)

3.2.1 HTMX files generated from SQL

FilenameTableSQL fileTemplate
sites.htmxsitesloadsites.sqlglostpt.htmt
site-timeline.htmxsitesloadsites.sqlglostpt.htmt
mgloss.htmxmglossg1.sqlglostpt.htmt
mga.htmxmglossg1.sqlglostpt.htmt
mgb.htmxmglossg1.sqlglostpt.htmt
mgc.htmxmglossg1.sqlglostpt.htmt
mgd.htmxmglossg1.sqlglostpt.htmt
mge.htmxmglossg1.sqlglostpt.htmt
mgf.htmxmglossg1.sqlglostpt.htmt
mgg.htmxmglossg1.sqlglostpt.htmt
mgh.htmxmglossg1.sqlglostpt.htmt
mgi.htmxmglossg1.sqlglostpt.htmt
mgj.htmxmglossg1.sqlglostpt.htmt
mgk.htmxmglossg1.sqlglostpt.htmt
mgl.htmxmglossg1.sqlglostpt.htmt
mgm.htmxmglossg1.sqlglostpt.htmt
mgn.htmxmglossg1.sqlglostpt.htmt
mgo.htmxmglossg1.sqlglostpt.htmt
mgp.htmxmglossg1.sqlglostpt.htmt
mgq.htmxmglossg1.sqlglostpt.htmt
mgr.htmxmglossg1.sqlglostpt.htmt
mgs.htmxmglossg1.sqlglostpt.htmt
mgt.htmxmglossg1.sqlglostpt.htmt
mgu.htmxmglossg1.sqlglostpt.htmt
mgv.htmxmglossg1.sqlglostpt.htmt
mgw.htmxmglossg1.sqlglostpt.htmt
mgx.htmxmglossg1.sqlglostpt.htmt
mgy.htmxmglossg1.sqlglostpt.htmt
mgz.htmxmglossg1.sqlglostpt.htmt
biblio.htmxbiblio,bibtypes,authors,akaloadbib.sqlglostpt.htmt
mspmtoc.htmxbiblioloadbib.sqlmspmtoc.htmt
multicians.htmxm,mxawards,awards,publistloadm.sql,mxawards.sql,publist.sqlmulticians.htmt
changes.htmxchangesloadc.sqlchanges.htmt
sitemap.htmxpagesloadpages.sqlsitemap.htmt

3.2.2 HTMX files that fetch values from SQL

FilenameTableSQL fileValues
multics.htmxhomesliderhome-slider.sqlsliding picture file
about-multics.htmlchangesloadc.sqlcount of changes
articles.htmlpagesloadpages.sqlcount of articles
changes-old.htmlchangesloadc.sqlcounts of changes by year
mac50-videos.htmlmulthumbsmulthumbs.sqllist of videos
multics-images.htmlmulthumbsmulthumbs.sqlthumbnail image paths
multics-stories.htmlpagesloadpages.sqltables of stories
people-pictures.htmlmulthumbsmulthumbs.sqlimage paths
site-timeline.htmlsloadsites.sqlsite names and dates, data for pie and bar charts
sites.htmlsloadsites.sqlsite information
source-index.htmlmsourcemsource.sqlsource file names and info
chrono.htmxchronochrono.sql, chrono2.sqldates and events, thumbnails

(mult-links.html also includes a count of mailing list members in grpsio.csv described in section 2.4.2, obtained by using the *shell builtin in mult-links.htmx to count the lines in the CSV file.)

The following Multics specific expressions appear in many HTMX files and are replaced by the results of an SQL query.

ExpressionTableSQL file
{! ... !}extrefloadext.sql
{@ ... @}pagesloadpages.sql
{[ ... }}multiciansloadm.sql
{{ ... }}mglossg1.sql
bibliolink macro callsbiblioloadbib.sql
mtblink macro callsbiblioloadbib.sql

3.3 Makefile Rule for Sending files to the Web Host Computer(s)

The Makefile invokes the rsync command to publish files from the personal computer to the web host computer(s). rsync doesn't copy files that it doesn't need to. Files are sent securely so they can't be intercepted or tampered with, and the file transmission is compressed so it will go quickly.

The rsync command is issued by make instead of typed directly. The Makefile gets the rsync local and remote path names from Makefile variables. rsync uses SSH keys in $HOME/.ssh to be able to send files to the web host computer(s).

Specific Makefile rules can be executed by specifying their name as an argument to make. The command make install invokes rsync:

# make install pushes modified files to the primary website and also backs up the source install: all incrementversion vnfile expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv $(LOGACTION) updated multicians rsync -avzu --blocking-io -e ssh --exclude .DS_Store $(OBJ_DIR)/ $(WEBACCT):$(WEBPATH) rsync -avzu --blocking-io -e ssh --exclude .DS_Store --exclude $(CONFIG) . $(WEBACCT):$(WEBPATHSRC) date

This rule means "Use rsync to send any modified object files to the web host computer, and also update a copy of the source files on the web host." It also updates a file dashboard.csv which visitors can look at to see counts of files on the site and the time of last update.

See Using Unix tools with expandfile for an explanation of rsync args.

3.4 Other Makefile Rules

The Makefile has other interesting rules.

3.6.1 Version Number File

The rule that pushes files to the ISP calls the shell script incrementversion to update vnfile. The contents of the file vnfile look like this:

6416 2024-01-31 11:40

The sequential number is incremented every time a make install is done, so this is the number of installs done since some time in 2006, when I first set up this file. It is followed by the date and time of the push. The shell script incrementversion updates these values. vnfile is included into an HTML comment in multics.html when that file is built. (The sequential number is not the same as the number of file changes, since a single install may install many files, or I may do repeated installs to get a single change just right.)

3.4.2 Dashboard File

The rule that pushes files to the ISP executes expandfile dashboard.csvt > $(OBJ_DIR)/dashboard.csv to update dashboard.csv. The file dashboard.csv is created by expanding the template dashboard.csvt. It contains two lines, a list of column headers and a list of corresponding column values:

  changect,asof,multicians,multiciansmail,multiciansurl,multiciansdec,glossent,npages,\
   npdf,alldoc,alldoc-online,images,mspm,mspm-online,nsites
  5388,"2020-07-02 13:35",2039,749,169,90,839,466,601,4994,2669,639,838,54,84

This file repeats the version number, and then has counts of the number of various items on multicians.org, such as number of multicians. It is intended to assist in the construction of other sites that refer to our site's contents. A file with this format can be read using the Expandfile builtin *bindcsv.

3.4.3 Subdirectories

Each subdirectory has a sub-Makefile which is invoked via a "subdirectory target" in the Makefile. The $SUBTGTS variable names these targets, and this variable is listed as a dependency for the "all" target. The result of this setup is that just typing make will also invoke the sub-Makefiles and make whatever is out of date. Most subdirectories have symbolic links to standard .htmi files in the parent directory, as well as the image and thumbnail directories. Subdirectories have a configx.htmi file that may override some path definitions in config2.htmi so that subdirectory files refer correctly to other files on the site. If you are setting up a subdirectory, it is best to clone the Makefile from one that already works, and make the same symbolic links.

3.4.4 Special build instructions

Several rules in the Makefile create special files which are published to the host ISP, using Unix utility commands in addition to expandfile. For example, make_mx_tar is a shell script that crates a gzipped tarfile listing all files that should be shipped to mirror sites. Generating it automatically saves me from having to remember to edit another file whenever I add a page to the site.

FileFunction
sitemap.xml.gzGzipped XML sitemap used by Google crawler
multicians.procmailMulticians email forwarding file (not visible on web)
make_mx_tarshell script that creates a tarfile of the site, for mirrors (not visible on web)

3.4.5 Error checking performed by the build process

File building rules check for errors and alert the editor of any problems whenever make rebuilds a file. For ordinary HTML file creation, the auxiliary program checknonempty is invoked to check for the case where a target file is nonexistent or empty after expandfile is invoked. (This can happen when expandfile aborts due to a missing include file.) If this happens, make exits with an error. After you fix the original error, make sure that the HTML file was not deleted.

Newly created HTML files are also checked by the HTML Tidy utility, which prints a line if there are any warnings or errors. When this happens, the editor should investigate the failing file and correct the problem.

For a few cases where SQL files are reloaded, the make recipe runs custom shell scripts to check that foreign key relationships are satisfied. In particular, the script checkbib.sh ensures that the tables loaded by loadbib.sql have authors entries for each document, that aka rows are present for each author, and that m rows for each author exist. If any errors occur, the editor should investigate loadbib.sql and correct the problem.

The rule that pushes files to the ISP executes check_missing_extref.sh, which checks output files for the sequence ###, which is produced when a {@path anchor@} sequence is executed and does not find path. This warns that we have generated a dead intra-website link. (Some of these are OK, and represent unfinished improvements.)

The rule that pushes files to the ISP also executes check_quotes.sh, which checks HTMX files for characters such as curly quotes that display poorly in browsers. (This can happen if text is copied and pasted from mail messages or Word documents.) These characters should be replaced by HTML entities.

3.5 Source file suffixes

These suffixes are my convention for indicating the use of the content in its file name. expandfile doesn't interpret files differently depending on their names. Macros defined in mxlib.htmi have some suffix dependencies. The Makefile generates .html from .htmx files.

SuffixMeaning
htmxHTML eXtended, generates HTML
htmtHTMX Template, generates HTMX
htmiHTMX Include file
htmlHTML
tpttemplate, generates non-HTML files such as site map
cssstyle sheet
csvComma Separated Values
csvttemplate that generates CSV file
etext file (emacs)
faqUsenet source, posted to alt.os.multics
gifbitmap graphic
htmqtest htmx file
jpgJPEG graphic
jsJavaScript
mkMakefile include
plPerl language
pngportable network graphic
procmailmail host path for procmail
pyPython
shshell script for /bin/sh
sqlMySQL source
txttext file

3.6 Standard Template Expansion with Wrappers

Most pages' HTMX files conform to a standard pattern. Each file contains

Expansion of normal HTMX files proceeds as follows. expandfile

  1. Reads config2.htmi to set configuration constants, MySQL configuration, etc.
  2. Ignores comments in the input HTMX file.
  3. Sets variables and blocks. (Does not expand variable references when blocks are defined.)
  4. Optionally executes *sqlloop builtins to execute SQL queries that create variables containing the expansion of an iterator template block for each query row.
  5. Includes and expands the wrapper, which outputs standard HTML structure and expands variable references, including %[body]%. If an expanded variable like body contains variable references, these are expanded also. Page wrappers begin by querying the pages table:
    1. Executes SELECT wdate FROM pages WHERE sourcefile='%[mainfilename]%'
    2. Executes SELECT crumbs FROM pages WHERE sourcefile='%[mainfilename]%'
    3. Executes SELECT l2navfile FROM pages WHERE sourcefile='%[mainfilename]%'
    4. Outputs the HEAD section
      1. Expands title, description, and keywords obtained from the *set statements, and minwidth, maxwidth, and contentwidth obtained from config2.htmi
      2. Executes *include,=mxstdfmt.htmi, which includes standard CSS boilerplate for the site
      3. Expands extrastyle if it was defined in the source file. This block may include JavaScript files and define STYLE settings local to the file.
    5. Outputs the BODY section:
      1. Includes class2head.htmi, which defines the page's standard top bar (which will use the wdate and crumbs from above) and drop-down menus.
      2. Expands the body block, set by the HTMX file, and variable references contained in it. This block may include result variables set by *sqlloop.
      3. Expands the extratail if the file supplied it.
      4. Formats and outputs tail navigation, including l2navfile if it was obtained from pages.
    6. Closes the BODY and HTML.

Alternate wrappers support variations for special page formats, such as MSPM sections or Glossary pages.

3.6.1 Why do this?

Expanding items that expand other items may seem indirect and convoluted. However, this scheme spares the writer of simple pages from the details of the site implementation, reduces the chances of a simple change breaking a page, and enables global changes when site design or HTML standards change. In practice, a one-line change can be made and published in about a minute; adding a new page to the site, cross-linked to other pages, visible to Google, and findable by standard navigation, can be done in only a few more minutes.

3.7 Additional Functions of Website Host

HTML pages, whether hosted on the primary site or mirror sites, invoke CGI functions on the primary host site.

The primary host provides additional functions for the support of multicians.org:

Protected mail addresses are in a file called hiddenmail.sql which is not published on the web. This SQL file is loaded into a MySQL database on the primary host site. The database is not accessible from the Web; CGIs on the primary host site access it and return values to web pages, after checking that the request is valid. Mail forwarding addresses are not visible to web crawlers (unless the ISP account gets hacked).

3.8 Tools

Tools needed to maintain Multics site provided by others

These are installed in /usr/bin by the OS distribution or /usr/local/bin by package managers.

3.8.1 Tools needed to maintain Multics site provided by editor

See expandfile web pages. These are mostly installed in the /tools subdirectory of the source directory.

3.8.2 The /mxs/tools directory

Tools are distributed in a subdirectory of the source directory, but they need to be linked to from your $HOME/bin to run correctly, and $HOME/bin should be on your shell PATH. These tools include:

----- wtlog.sh, calls $HOME/wtx/wtlog (Perl)
----- expandfile_config (script, calls expandfile with config file)
----- check_missing_extref.sh
----- printsites2.pl
----- gen_site_timeline.pl
----- auto1a.sh, calls post.pl

3.8.3 Web Host Programs and data files in cgi-bin directory provided by editor

Some of these tools invoke the subroutine version of expandfile and link to expandfile.pm in $HOME/bin

3.8.4 Web Host Programs and data provided by web host and site admin

3.8.5 Web Host Runtime Functions Provided by Editor

These functions are installed in the js subdirectory of the object directory.

3.9 Performance

Expandfile is written in an interpretive language, so it is not blazingly fast, but it is fast enough. Re-expanding all 467 pages on the site takes about 1 minute. Pages that have a lot of *shell calls are slower: for example, a page like people-pictures.htmx that has to involve a lot of *shell calls to Perl helper program gifsize2 has to launch a shell and a Perl interpreter for each of over 100 pictures. If I built gifsize2 into expandfile this could be a lot faster. Not worth the effort at this time.

3.10 Git

The maintainer's Multics source directory contains a local Git repository that contains all the source .htmx and {.sql:} files. Doing a make install does a git status. After making and installing a set of changes, the editor should do a git commit -a with a brief message describing the changes.

The Makefile rule that pushes files to the ISP ends with a git status which reminds the editor to add or commit changes to the git repository.

--- the SQL source for mail addresses hiddenmail.sql is not included (see 3.7).
--- various temporary files, scratch files, obsolete files are named in .gitignore and don't go into the repo.
--- future: upstream repo on github, and a way for others to request changes and build their own copy.

3.11 Files and their purpose

(by THVV unless indicated)

3.11.1 Tools invoked by Makefile

auto1a.sh (shell script to post FAQ to USENET)
checknonempty (shell script halt make with error if a file is zero length)
config2.htmi (configuration constants for every page in the website)
expandfile (see Expandfile)
gen_site_timeline.pl (generate the site-timeline chart from sites.sql)
incrementversion (shell script to add one to the version number in vnfile)
nextpostingexpires (Perl program to generate expire date for USENET summary)
post.pl (post monthly update to USENET)
printgloss4.pl (generate 27 glossary htmx files from g1.sql)
printsites2.pl (generate sites.htmx from sites.sql)

3.11.2 Tools invoked by *shell in HTMX files

dbcounts (Perl: queries SQL databases for counts of various kinds of document)
get-facebook.sh (Shell script calls get-facebook-fan-count.pl)
filemodshort (Perl: comes with expandfile)
filemodyear (Perl: comes with expandfile)
filesizek (Perl: comes with expandfile)
firstletter (Perl: comes with expandfile)
gifsize2 (Perl: comes with expandfile)
nargs (Perl: comes with expandfile)

3.11.3 Files used to support ISP mail and signup on multicians.org host /cgi directory

errtpt.txt (template for error message if update fails)
mform-thanks.txt (template for successful multicians update)
mformtpt.txt (template for multicians update form)
mxaddrtag.htmi (editor email boilerplate for all HTML files)
mxclass2head.htmi (copy of class2 head for HTML pages)
mxh1style.htmi (copy of class2 head for HTML pages)
mxmailerr.txt (template for error message if mail fails)
mxolwrapper.htmi (wrapper file for all pages created by CGIs)
mxregerr.txt (template for error message if registration fails)
mxregister.cgi (Perl: sends update mail when mform.html for is submitted)
mxregister4.cgi (Perl: Ajax form checker called from mform.html)
mxstdfmt.htmi (copy of standard format for HTML pages)
mxtextnav.htmi (copy of tail text navigation for HTML pages)
reloadm (shell script invoked to update host database)
special.pmrc (Special procmail filter, for checking user outbound mail)
thvvutil.pm (utility functions in Perl)

3.12 Displaying graphics files on the site

There are five main methods the site uses to display graphics on a web page, depending on the desired appearance and visitor experience. These methods are supported by Expandfile macros. The macros generate HTML IMG tags to display the graphics: they make it easy for page authors by looking up the the image size at compilation time, and using additional versions of graphics to support High DPI displays. Using the macros makes page creation easier and generates sharp looking graphics.

Graphic images in jpg, png, and gif format are stored in the directory /mulimg, and are used at compile time and at page display time. Every thumbnail is stored in /thumbnails150 as a 150x150 thumbnail to be displayed in a 75x75 space.

There are five methods used to display graphics on a page, supported by different macros. Examine the source of existing pages to see how these macros are used. (For example, picnics.htmx uses both the second and third methods to display pictures.)

  1. Display a graphic in a DIV, possibly floated to the left or right. Use the getimgdiv macro to do this. Given a graphic name such as xxx.jpg, this macro looks for both xxx.jpg and xxx-2x.jpg. If both are found, an IMG tag that uses the SRCSET attribute is generated.

  2. Display a graphic in a DIV, possibly floated to the left or right, and if the graphic is clicked, display a larger version of the graphic in a popup with a Multics banner and a "close" control. In the extrastyle block of a multicians.org source page, include the file use-dhtml-lightbox.htmi. At the top of the body block of a multicians.org source page, include the file dhtml-lightbox.htmi. Use the getimgpopdiv macro to insert a graphic in a DIV block, with an optional CSS tag. Use the getimgpopdiv75 macro if the base graphic is a 150x150 thumbnail displayed in a 75x75 space. These macros look for the given file name and a -2x version of the file, and generate the appropriate IMG tags and popups.

  3. Some pages display galleries of thumbnails. (cisl-wake.html, cno-pictures.html, lcs25.html, lcs35.html, people-pictures.html, picnics.html, etc.) Clicking on any thumbnail opens a popup, with arrow keys that allow the visitor to go to the next item in the gallery. In the extrastyle block of a multicians.org source page, include the file usefancybox.htmi to include the JQuery JavaScript file fancybox.js. Create a DIV container for the gallery with a unique ID tag; inside the DIV, place a UL container. Then insert a call to the getfancybox_li macro for each thumbnail, specifying the thumbnail/image name and the caption, to generate an LI element for the thumbnail. This method and the "popdiv" method can be used on the same page. The thumbnails are displayed using CSS sprites, to speed up page loading. lightbox_li is used with an older image popup display used in several pages that display image galleries. Other pages use use-dhtml-lightbox.htmi and dhtml-lightbox.htmi and generate IMG tags with the macro getimgpopdiv. (For example, Picnics uses both.)

  4. On the home page multics.html, a 300x244 window with a sequence of sliding images is displayed by easySlider.js. The pictures and their attributes are specified in the homeslider table, queried when the home page is compiled. The query generates a packed graphic image with CSS sprite definitions for each individual picture. Each picture is 600x488, so that they will look sharp on a High DPI display. References to the pictures are in individual LI items in a UL element: EasySlider handles the formatting and picture changing. When the home page is compiled, a caption specified by the homeslider table is overlaid on each graphic: this caption's definition may invoke command line programs with *shell to get counts from the database, to produce legends like "84 Multics Stories."

  5. The Image Gallery page multics-images.html displays all thumbnails on the site, grouped by general type. It uses CSS sprites, so that instead of loading 462 small graphics, it loads five large ones, which is much faster. Each thumbnail links to a page that contains the graphic. All thumbnail links are generated at compile time by five SQL queries of the multhumbs table.

See How I use Affinity Photo for a cookbook on processing graphics, sharpening, etc.

3.12.1 Generating 2x Graphics from Adobe Illustrator files Using Affinity Designer and Affinity Photo

Here is my recipe for generating .png graphics from .ai files. (So I don't forget.) The goal is to take a diagram in Illustrator and make 2x and 1x versions of the graphic as PNGs.

  1. Read the xxx.ai file into Affinity Designer
  2. In the dialog box, accept the font substitution, set bold, set the DPI to 144, leave colorspace at 'estimate'
  3. Select all, export PNG, set Area to "selection with background", leave size alone
  4. Save with name to mulimg/xxx-2x.png
  5. Read in xxx-2x.png into Affinity Photo
  6. Resize to half size, sharpen, and save as mulimg/xxx.png (remember to make it PNG, not JPG)
  7. Edit the source file to invoke a macro such as getimgdiv referencing mulimg/xxx.png
  8. Recompile. On a machine with a HiDPI display, browsers will load xxx-2x.png and display it in 1/2 the CSS dimensions, because the SRCSET selects the 2x file; the browser knows the screen resolution and uses all the pixels to make a sharp image. On a machine without a HiDPI display, browsers will load xxx.png and display it, because the SRCSET selects the 1x.
  9. Generate a thumbnail for the image using gth2x xxx-2x.png and move /mulimg/thumb2-xxx-2x.png to /thumbnails150/xxx.png.
  10. Update multhumbs.sql to use the new thumbnail.
  11. Verify that everything displays correctly.
  12. If there are /mulimg/xxx.gif or /thumbnails150/xxx.gif files, remove them.

4. Expandfile Documentation