%[** Expandfile **]% %[*include,=htmxlib.htmi]% %[** **]% %[*set,&title,="expandfile"]% %[*set,&headingtitle,="expandfile"]% %[*set,&description,="Utility program that expands a text template."]% %[*set,&xxversion,="6"]% %[*set,&contentwidth,="800"]% %[*set,&maxwidth,="1000"]% %[*set,&minwidth,="200"]% %[** **]% %[** Copyright (c) 2002-2022, Tom Van Vleck **]% %[** THVV 01/06/20 add *bindcsv **]% %[** THVV 01/12/20 reformat and macroize **]% %[** THVV 09/01/20 changes for expandfile3 **]% %[** THVV 04/13/21 changes to *bindcsv; allow *shell, *fwrite, *fappend, *htmlescape to concatenate multiple args; remove comma and vbar syntax **]% %[** ================================================================ **]% %[*block,&extrastyle,^END]% %[*include,=leftpanelstyle.htmi]% END %[** ================================================================ **]% %[*block,&bif,^END]% %[** Function to doc each builtin **]% %[** *callv,bif,prototype,result **]% %[** **]% %[** .. where prototype is a comma separated list of args followed by (in|out|inout) **]% %[** **]% %[** .. writes HTML to stdout and also concats to biflist **]% %[** .. increments bifcount **]% %[** **]% %[*set,&cmdname,param1]% %[*subst,&cmdname,=",.*$",=""]% %[*format,&r1,="
$3.
\\nOutputs $2.
\\n",param1,param2,param3,cmdname]% %[**Builtin | Function | Output |
---|---|---|
** | Starts a comment | nothing |
{:file_name:} | name of the file |
{:file_type:} | 'f' for file, 'd' for directory, 'l' for link |
{:file_dev:} | device number |
{:file_ino:} | inode number |
{:file_mode:} | mode in Unix character format, e.g. "{:rwx--x---:}" |
{:file_nlink:} | number of hardlinks to the file |
{:file_uid:} | numeric file owner userid |
{:file_gid:} | numeric file owner groupid |
{:file_rdev:} | rdev (for special files) |
{:file_size:} | size in bytes |
{:file_atime:} | last access time |
{:file_mtime:} | last mod time |
{:file_ctime:} | inode change time |
{:file_blksize:} | preferred block size |
{:file_blocks:} | allocated blocks |
{:file_sec:} | mtime seconds (2 digits) |
{:file_min:} | mtime minutes (2 digits) |
{:file_hour:} | mtime hour (2 digits) |
{:file_mday:} | mtime day of month (2 digits) |
{:file_mon:} | mtime month (2 digits) |
{:file_year:} | mtime year (2 digits) |
{:file_wday:} | mtime day of week (0-6, Sunday is 0)) |
{:file_yday:} | mtime day of year |
{:file_isdst:} | 1 if mtime is DST |
{:file_datemod:} | mtime in the format "mm/dd/yy hh:mm" |
{:file_modshort:} | mtime in the format "mm/dd/yy" |
{:file_sizek:} | file size in K |
{:file_age:} | age in days |
The variable {:_xf_nrows:} is set to the count of files found.
Example:
{:\%[*dirloop,&content,format_one_file,=".",="."]%:}
Tutorial example: {:*dirloop:}.
%[** -------------------------------- **]% %[*callv,bif,="dump",="one line per defined variable",="Display the contents of all variables in the symbol table"]%
Example:
{:\%[*dump]%:}
Example:
{:\%[*exit]%:}
{:var:} may contain variable names and builtin invocations in {:\%[... ]%:}. Nested expansions are possible if the variable invokes {:\%[*expand,...]%:} or other builtins.
Outputs the value of expanding {:var:}. If {:var:} contains {:\%[ ... something ...]%:} constructs, they will be expanded.
Example: output the contents of a variable
{:\%[*expand,footnote_separator]%:}.
Example: output the contents of a variable whose name depends on another variable
{:\%[*expand,footnote_\%[ftn]%]%:}.
{:var:} may contain variable names and builtin invocations in {:\%[... ]%:}. Nested expansions are possible if the variable invokes {:\%[*expand,...]%:} or other builtins.
Sets its first argument to the expansion of {:var:}. Outputs nothing. If {:var:} contains {:\%[ ... something ...]%:} constructs, they will be expanded.
Example:
{:\%[*expandv,&formatted_name,convert_cityname]%:}
Example:
{:\%[*fappend,=tracefile.txt,timestamp,=" ",traceoutput]%:}
Example: combine cityname and statename:
{:\%[*format,&line,="$1, $2",cityname,statename]%:}
Example: Generate an HTML IMG tag:
{:\%[*format,&imgtag,="<img src=\\"$1\\" width=\\"$2\\" height=\\"$3\\">",filename,height,width]%:}
Example: Generate GraphViz (dot) input:
{:\%[*if,ne,t.st,="invis",*format,&x,="$1 -> $2 [xlabel=\\"$3\\",style=\\"$4\\",color=\\"$5\\"];\\n",y.n1,y.n2,y.b,y.e,t.o]%:}
The second example statement was used in an iterator block in an application that produced a block diagram from an SQL file.
%[** -------------------------------- **]% %[*callv,bif,="fread,&result(out),=filename(in)",="nothing",="Read the contents of {:filename:} into {:result:}"]%If the input file is not found, set content to the empty string. Does not expand blocks, builtins, or variables.
Sets its first argument. Outputs nothing.
Example:
{:\%[*fread,&pienumber,=pienumberfile]%:}
Replaces any previous contents of {:filename:}.
Example:
{:\%[*fwrite,=pienumberfile,counter%:}
For instance, html-escaping "{:<fred>:}" yields "{:<fred>:}".
Example:
{:\%[*htmlescape,filename]%:}
Perform the comparison v1 relop v2 and if it is TRUE, expand {:statement:}. {:statement:} can be any set of HTMX evaluations, including more "if" builtins. relop is the name of a comparison operator. The supported operators are:
Outputs whatever {:statement:} outputs.
Example:
{:\%[*if,eq,city,="Chicago",*set,&airport,="ORD"]%:}.
Nested {:*if:} statements:
{:\%[*if,ne,gnm,param1,*if,ne,tnms,="",*callv,rec_grpname2grpid,param1]%:}
Another nested {:*if:} statement, setting up a format string for the {:*format:} builtin:
{:\%[*if,gt,x3,param3,*if,gt,x1,=0,*set,&fmt,=" <span class=\"inred\">($1$2%)</a>"]%:}
Expands blocks, builtins, and variables.
If the file is not found, this is a fatal error and {:expandfile:} exits.
Example:
{:\%[*include,=page-wrapper.htmi]%:}.
Tutorial example: {:*include:}.
%[** -------------------------------- **]% %[*callv,bif,="includeraw,=filename(in)",="{:filename:} content, unexpanded",="Output the content of {:filename:}, unexpanded"]%Does not expand blocks, builtins, or variables.
If the file is not found, this is a fatal error and {:expandfile:} exits.
Example:
{:\%[*includeraw,=data_table.txt]%:}.
Add the value in {:value:} to the value in {:result:} and store the result into {:result:}. Uses Perl semantics.
Example:
{:\%[*increment,&pageno,="1"]%:}
Concatenate {:n:} copies of the value in {:value:} and store the result into {:result:}.
Example:
{:\%[*ncopies,&amt,="*",width]%:}
If the value of {:var:} has changed, execute the {:statement:}. This builtin is useful in iterators invoked by {:*sqlloop:}, {:*csvloop:}, {:*xmlloop:}, {:*dirloop:}, and {:*csvloop:}.
Example:
{:\%[*onchange,x,*callv,wrap,x,=" <dt>",="</dt>\\\\n"]%:}
outputs a line that surrounds the value of {:x:} with <dt> and </dt> if {:x:} is nonempty and changed.
(See the definition of the {:wrap:} macro in Macros in expandfile).
If the value of {:var:} has NOT changed, execute the {:statement:}. This builtin is useful in iterators invoked by {:*sqlloop:}, {:*csvloop:}, {:*xmlloop:}, {:*dirloop:}, and {:*csvloop:}.
If an iterator block uses both {:*onchange:} and {:*onnochange:}, put the {:onnochange:} call first.
Example:
{:\%[*onnochange,x,*increment,&titles,=1]%:}
Variable {:ssv:} should contain a list of strings separated by a separator character whose value is in {:_xf_ssvsep:}. (By default the character is a space -- SSV means "space separated values.") Remove the leftmost element of {:ssv:}, store it in {:result:}, and rewrite {:ssv:} with the rest. If {:ssv:} contains only one element, store it in {:result:} and leave {:ssv:} empty.
See also the {:*ssvloop:} builtin.
Example:
{:\%[*popssv,&next_player,&jersey_numbers]%]%:}
Multiply the value in {:result:} by the value in {:value:} and store the result in {:result:}.
Example:
{:\%[*product,&mins,hours,=60]%:}
Divide the value in {:dividend:} by the value in {:divisor:} and store the result in {:result:}. Discard the fractional part. If divide by zero is attempted, the result is 0.
Example:
{:\%[*quotient,&cows,hooves,=4]%:}
Divide the value in {:dividend:} by the value in {:divisor:} and store the result in {:result:}, rounded to the nearest integer. That is, compute {:int((dividend / divisor) + 0.5):}; If divide by zero is attempted, the result is 0.
Example:
{:\%[*quotientrounded,&cows,hooves,=4]%:}
Compute {:int(((n*base)/range)+0.5):} and store the result in {:result:}. {:range:} is the maximum value for the variable {:n:} and {:base:} is the maximum scaled value.
For example, if you are drawing an HTML horizontal bar graph that will be 500 pixels wide (the base), of variables that run from 0 to 1000 units, then each graph pixel will represent 2 units.
Example: Computing the width of a bar in the graph:
{:\%[*scale,&barwidth_pixels,wtdayhist.dhits,.maxdhits,=500]%:}
{:<img src="redpixel.gif" height="10" width="\%[barwidth_pixels]%">:}
This uses up to 500 pixels to display a graph bar whose longest bar represents {:.maxdhits:}.
If a {:value:} begins with an {:=:} then it is a literal value. Otherwise it is a variable name whose value is used.
Example: set a variable to a constant string.
{:\%[*set,&title,="Expandfile Usage"]%:}.
Example: set a variable to the value of another variable.
{:\%[*set,&title,datafield]%:}.
Example: set a variable to the concatenated values of several arguments. ({:quote:} is a builtin value.)
{:\%[*set,&htt,="https://multicians.org/thvv/htmx/expandfile.html"]%:}
{:\%[*set,&anchorstring,="Expandfile"]%:}
{:\%[*set,&test6,="<a href=",quote,htt,quote,=">",anchorstring,="</a>"]%:}.
{:quote:} is a builtin variable whose value is a double quote character. I could have written {:="\\"":} instead.
%[** -------------------------------- **]% %[*callv,bif,="shell,&result(out),command(in)...",="nothing",="Execute shell {:command:} and set {:result:} to its output"]%The string sent to the command processor is the concatenation of the values of {:command:} arguments.
If multiple lines are returned, newline characters are replaced by the contents of {:_xf_ssvsep:}. The command output is stored in {:result:}.
Be careful about security if you execute shell commands based on user input.
You can write your own functions, or use Unix programs such as {:grep:}, {:sed:}, {:awk:}, or {:sh:}.
Several useful utility functions are supplied with {:expandfile:}.
Example: get the date modified of a file ({:*shell:} is called with one argument, which contains an expansion of the variable {:parm1:})
{:\%[*shell,&filed,=filemodshort \%[param1]%.htmx]%:}
Example: get the age in days of the index file in a directory (note the concatenation of constants and variable values)
{:\%[*shell,&xage,="filedaysold ",obj_dir,="/",param1,="/index.html"]%:}
Example: use {:perl:} to modify a disk file to remove the string "NAMESP:" everywhere (note that double-quote is preceded by \\ inside a string)
{:\%[*shell,&xout,="perl -pi -e \\"s/NAMESP://g\\" ",filenamevar]%:}
Example: invoke the {:mysql:} command to load a disk file containing MySQL statements (note that we don't quote the argument)
{:\%[*shell,&xout,=mysql --execute \\"source \%[sqlloadfile]%\\"]%:}
Example: invoke the {:curl:} command to download output from a web service API
{:\%[*shell,&xout,=curl \%[endpoint]%/\%[cmd]% -X POST -d \%[JSON_rq]% -H \\"Content-type: application/json\\" > xx.json]%:}
Performance was not a consideration in the design of the {:*shell:} builtin. Executing command lines this way is done by launching a fairly large shell process, and then launching additional processes for each command in the arguments to the shell. (I expand one template, {:sitemap.htmx:}, which invokes {:filemodshort:} and {:filesizek:} for each of over 1000 pages. There is a pause of 10 seconds or so when the template is expanded.)
%[** -------------------------------- **]% %[*callv,bif,="sqlloop,&result(out),iter_block(in),query(in)",="nothing",="Set {:result:} to concatenated expansions of {:iter_block:} for each row returned by MySQL {:query:}"]%Execute the MySQL query {:query:}, which returns a number of rows. Each row returns a set of variables. Bind the variables in the symbol table using names like {:table.varname1:}, and then expand {:iter_block:}, which will refer to these variables. Append the result of the expansion to {:result:}. (Because {:_xf_colnames:} is set before {:iter_block:} is expanded, the iterator need not know the complete schema of the database.)
The variable {:_xf_nrows:} is set to the count of rows found by the query.
The variable {:_xf_colnames:} is set to a space separated list of column names bound by the query.
Computed values such as COUNT that have no tablename are bound to names like {:.count:}.
The variables {:_xf_hostname:}, {:_xf_database:}, {:_xf_username:}, and {:_xf_password:} must be set up to point to the MySQL database server before {:sqlloop:} is invoked, or {:expandfile:} will exit with an error message.
Some database errors are retryable, for instance if the server goes down. {:sqlloop:} will retry these errors 10 times before exiting with an error.
If there is a fatal database or query error, {:expandfile:} exits with an error message.
If the query is empty, a warning will be printed and _xf_nrows will be set to 0. It is not an error for a query to return 0 rows, but a warning will be printed.
Example: generate a bar chart summarizing population by country name from table "s":
{:\%[*sqlloop,&chartout,chart_iter,="SELECT country, COUNT(*) AS v FROM s WHERE fake=0 GROUP BY country"]%:}
To execute a query just to set a total,
specify a variable for the loop output that you never use, and a null iterator, and refer to the query result, e.g.
{:\%[*sqlloop,&junk,="",="SELECT COUNT(*) AS minors FROM tbl WHERE age < 21"]%:}
{:\%[*set,&minorcount,\%[.minors]%]%:}
Tutorial example: {:*sqlloop:}.
%[** -------------------------------- **]% %[*callv,bif,="ssvloop,&result(out),iter_block(in),ssv(in)",="nothing",="Set {:result:} to concatenated expansions of {:iter_block:} for each element of {:ssv:}"]%An SSV (space separated values) list is a variable value composed of tokens separated by the value in {:_xf_ssvsep:} (usually space).
Operate on each token in the SSV {:varname:}. For each token, bind {:_xf_ssvitem:} to the value (null tokens are skipped), and then expand {:iter_block:}, which can refer to {:_xf_ssvitem:}. Append the result of the expansion to {:result:}. This loop works on a copy of {:varname:}, so the input SSV is not changed.
The variable {:_xf_nssv:} is set to the count of tokens found in the SSV.
See also the {:*popssv:} builtin.
Example:
{:\%[*ssvloop,&nextstorybody,nextstory-iter,filenamesbydate]%:}
Tutorial example: {:*ssvloop:}.
%[** -------------------------------- **]% %[*callv,bif,="subst,&var(inout),leftside(in),rightside(in)",="nothing, rewrites first argument",="Substitute {:right:} for {:left:} in {:var:}"]%Replace the value in {:result:} by the result of the Perl substitution {:s/left/right/ig:}. Slashes in {:left:} and {:right:} must be escaped -- use {:\\\\:}.
{:left:} can contain parenthesized patterns, as in Perl substitutions; these patterns can be referenced in {:right:} as \\\\1, \\\\2, etc.
Example: truncate a name to its first 40 characters
{:\%[*subst,&name,="^(........................................).*$",="\\\\1"]%:}
Example: trim off the directory portions of a pathname
{:\%[*subst,&pname,="^.*\\\\/",=""]%:}
Be careful about security if you read external values from the Internet and use them in arguments to {:*subst:}. Watch out for backticks.
%[** -------------------------------- **]% %[*callv,bif,="urlfetch,&result(out),url(in)",="nothing",="Set {:result:} to contents of {:url:}"]%Fetch the contents of the Internet URL {:url:} into {:result:}. If the target is not found, set content to the empty string. Does not expand variables or blocks.
Be careful about security if you read external values from the Internet and expand them. Sanitize your values before expanding. Watch out for {:\%[]%:} and backticks.
Example:
{:\%[*urlfetch,&contents,=\%[_xf_ssvitem]%]%:}
{:filename:} is an ASCII text file in XML format. It may be gzipped.
Operate on each item the file as found by {:Xpath:}: if this argument is missing, the default is "{:/*/*:}". For each item found by the Xpath, the loop binds the values of sub-items "{:./*:}" and binds the values of attributes "{:./@*:}", and then expands {:iter_block:}, which will refer to these variables. Append the result of the expansion to {:result:}.
If no {:Xpath:} is provided, the outermost structure in the file should be something like <list> ... </list> and it should contain repeated items <item> ... </item> which in turn contain multiple fields like <person> ... </person> and <address> ... </address>. For each item, bind the values of fields in the symbol table using names like {:person:}.
The variable {:_xf_nxml:} is set to the count of items found by the query.
The variable {:_xf_xmlfields:} is set to a space separated list of variable names bound by the query.
If the XML file is empty or malformed, {:_xf_nxml:} is set to 0 and nothing is done. If the XML file is missing, {:expandfile:} exits with an error message. This is ugly: I should think of a better solution.
Example:
{:\%[*xmlloop,&junk,iter_gacc,=gacc.xml,="*/computers/computer"]%:}
I use a simple Perl program called {:json2xml:} to translate JSON data (fetched from a web API) into XML data, which I then process with {:*xmlloop:}.
Tutorial example: {:*xmlloop:}.
%[** -------------------------------- **]% %[*callv,bif,="warn,args...(in)",="nothing",="Write {:args:} to STDERR"]%Write a warning message {:args...:} on STDERR.
Example:
{:\%[*warn,No results for \%[query]%]%:}
Some builtin functions write output to the output file. Others produce no output but instead change an argument variable. A few functions conditionally expand the rest of their arguments, so these produce output wherever their arguments direct.
Builtin functions that modify an argument use {:&:} before the variable name to remind you that the value will be modified. If you omit the {:&:}, {:expandfile:} prints a warning. Some builtins with {:&:} have in/out args, others are out.
%[biflist]%(If I had it to do over, I might make all functions take an output argument, and have none of them write to the output.)
%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%These values are set up by {:expandfile:} when it is invoked. Probably a bad idea to store into any of these. (Example value in parentheses.)
{:%\[year]%:} | year (%[year]%) |
{:%\[prevyear]%:} | previous year (%[prevyear]%) |
{:%\[day]%:} | day (%[day]%) |
{:%\[month]%:} | month (%[month]%) |
{:%\[prevmonth]%:} | previous month (%[prevmonth]%) |
{:%\[monthx]%:} | numeric month (%[monthx]%) |
{:%\[hour]%:} | hour (%[hour]%) |
{:%\[min]%:} | min (%[min]%) |
{:%\[date]%:} | date in text format (%[date]%) |
{:%\[timestamp]%:} | timestamp (%[timestamp]%) |
{:%\[pct]%:} | percent (%[pct]%) |
{:%\[lbkt]%:} | left bracket (%[lbkt]%) |
{:%\[rbkt]%:} | right bracket (%[rbkt]%) |
{:%\[lbrace]%:} | left brace (%[lbrace]%) |
{:%\[rbrace]%:} | right brace (%[rbrace]%) |
{:%\[quote]%:} | double quote (%[quote]%) |
{:%\[_xf_currentfilename]%:} | current file name being expanded (%[_xf_currentfilename]%) |
{:%\[_xf_version]%:} | version of {:expandfile:} (%[_xf_version]%) |
{:%\[_xf_expand_multics]%:} | user config | If nonblank, enable Multics expansions. |
{:%\[_xf_debug]%:} | user config | If nonblank, enable warnings about unset variables. |
{:%\[_xf_tracebind]%:} | user config | If set to a nonblank value, causes {:*set:}, {:*block:}, {:*sqlloop:}, {:*csvloop:}, {:*xmlloop:}, {:*dirloop:}, and {:*csvloop:}
to print a message on STDERR when they bind a value to a variable. |
{:%\[_xf_ssvsep]%:} | user config | Separator between elements of a Space Separated Values list. Default is " ". |
{:%\[_xf_nssv]%:} | Set by {:*ssvloop:} | Number of elements processed. |
{:%\[_xf_ssvitem]%:} | Set by {:*ssvloop:} | Current item inside an iterator block being expanded. |
{:%\[_xf_nrows]%:} | Set by {:*sqlloop:}, {:*csvloop:}, {:*dirloop:} | Rows read |
{:%\[_xf_colnames]%:} | Set by {:*sqlloop:}, {:*csvloop:}, and {:bindcsv:} | SSV of column names bound. |
{:%\[_xf_xmlfields]%:} | Set by {:*xmlloop:} | SSV of XML item names bound. |
{:%\[_xf_nxml]%:} | Set by {:*xmlloop:} | Number of XML items read. |
{:%\[_xf_n_callv_args]%:} | Set by {:*callv:} | Number of 'paramN' args to macro. |
{:%\[_xf_currentfilename]%:} | Set by {:expandfile:} | Name of file currently being read. |
One way to extend the function of {:expandfile:} is to write macros. As mentioned above, you can define these macros using {:*block:}, and expand them using {:*callv:}. If you decide to use the macros for more than one file, it is natural to move the macros to a separate file (I use the file suffix {:.htmi:} meaning "HTMX Include") and incorporate them into the files that invoke them with a command like {:\%[*include,=macro_lib.htmi]%:}.
A library of macros called {:htmxlib.htmi:} is provided with {:expandfile:}. It contains macros you can use or adapt to your own needs.
For example, one macro is {:getimgdiv:}, which can generate HTML IMG tags with the WIDTH and HEIGHT of a graphic. The IMG tag will use SRCSET when a graphic image is represented in multiple sizes, to show the optimum image for the viewer's screen resolution. (see High DPI Pictures.) Call it by
\%[*callv,getimgdiv,path,target,alttag,titletag,class,caption]%
See "Macros in Expandfile" for more information.
%[** ---------------------------------------------------------------- **]%{:expandfile:} also expands nine special text formatting macros developed to simplify the source of the multicians.org website, its first application. See "expandfile Features for Multicians.org" for more information. These macros are applied to all text, whether inside \%\[ ... ]% or not.
To prevent these macros from being expanded and interpret them as literal text, precede them by a backslash (\\). For example, \\\{\: ... :}, \\\{\\\= ... =}, \\\{\\\+ ... +}, \\\{\\\- ... -}, \\\{\\\* ... *}, \\\{\\\[ ... ]}, \\\{\\\{ ... }}, \\\{\\\! ... !}, \\\{\\\@ ... @} .
%[** ---------------------------------------------------------------- **]%Basically, you set up some OS tools and prerequisite software, and then download {:expandfile:} and its Perl module files and some auxiliary files from {:github.com/thvv/expandfile:}, and install them on your search path.
Create a directory called {:/bin:} in your home directory, and then add it to your PATH environment variable, by issuing the following commands in a Terminal or shell window (assuming your shell is {:bash:}):
cd $HOME mkdir bin echo "export PATH=$HOME/bin:$PATH" >> .bash_profile . .bash_profile%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%
Arrange to set configuration values for Perl in your shell environment. Linux and Windows will be similar. For example, if you are using Perl 5.26 on a 64-bit Mac using Homebrew, you would set up
export VERSIONER_PERL_PREFER_32_BIT="no" export PERL5LIB=/Users/thvv/bin:/usr/local/lib/perl5/5.34.0:/usr/local/Cellar/perl/5.34.0/lib/perl5/site_perl/5.34.0 export PERL_LOCAL_LIB_ROOT="/usr/local/lib/perl5/5.34" export PERL_MB_OPT="--install_base \"/usr/local/lib/perl5/5.34\"" export PERL_MM_OPT="INSTALL_BASE=/usr/local/lib/perl5/5.34"
in your {:.bash_profile:}.
%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%{:expandfile:} is written in the Perl language. Your computer must have Perl installed, and your Perl library has to have a few modules installed. Macs and most Linux systems come with Perl, with the library set up. To check if Perl is installed and what version it is, type {:perl -v:}; my computer says
This is perl 5, version 34, subversion 0 (v5.34.0) built for darwin-thread-multi-2level
For Macintosh on of macOS Big Sur, I install Perl with Homebrew. Once Homebrew is installed, you can add packages such as ImageMagick. See {:https://formyfriendswithmacs.com/homebrew.html:}.
{:expandfile:} and its helper programs in Perl have a "shebang" line of {:#!/usr/local/bin/perl:},
which selects which version of Perl will be executed when the shell executes a command.
On my Mac, Homebrew sets up {:/usr/local/bin/perl:}.
On Linux, I link {:/usr/local/bin/perl:} to {:/usr/bin/perl:}.
Ensure that your shell environment variable {:PERL5LIB:} points to your $HOME/bin and to libraries for the same version of Perl.
On my Mac using Big Sur, my PERL5LIB is {:/Users/thvv/bin:/usr/local/lib/perl5/5.34.0:/usr/local/Cellar/perl/5.34.0/lib/perl5/site_perl/5.34.0:}.
On Linux, my PERL5LIB is {:$HOME/bin:/usr/local/lib/perl5:}.
Set your shell environment variables VERSIONER_PERL_PREFER_32_BIT, PERL_LOCAL_LIB_ROOT, PERL_MB_OP, and PERL_MM_OPT for your local environment.
%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%{:expandfile:} needs MySQL even if you never invoke {:*sqlloop:}, because it loads the CPAN module {:DBD::mysql:}, which won't install unless MySQL is available. (I am looking into a way to remove this dependency.) There is no standard Perl way to only load a module if it is needed at runtime. Download and install MySQL.
To configure MySQL, define a "root" user and password, and create a database. (Installing {:mysql:} may generate a temporary root password you have to change. This seems to be different for each MySQL release.) Then set up the file {:.my.cnf:} in your home directory so that you can access the database without giving your password every time. You'll want to set up a configuration file like {:config.htmi:} that sets values for {:expandfile:}, so that your programs don't have to have the credentials baked in.
"On Unix, MySQL programs treat the host name localhost specially, in a way that is likely different from what you expect compared to other network-based programs. For connections to localhost, MySQL programs attempt to connect to the local server by using a Unix socket file. This occurs even if a --port or -P option is given to specify a port number. To ensure that the client makes a TCP/IP connection to the local server, use --host or -h to specify a host name value of 127.0.0.1, or the IP address or name of the local server. You can also specify the connection protocol explicitly, even for localhost, by using the --protocol=TCP option."
%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%Expandfile depends on having several Perl library modules installed. If these are not installed already, install them using CPAN:
Module | Description |
---|---|
{:LWP::Simple:} | Support for {:*urlfetch:} and {:*bindcsv:} |
{:Term::ANSIColor:} | Support for error messages |
{:DBI:} and {:DBD::mysql:} | Support for {:*sqlloop:} |
{:XML::LibXML:} | Support for {:*xmlloop:} |
{:XML::Simple:} | If you use {:json2xml:} |
{:JSON:} | If you use {:json2xml:} |
Install and configure {:cpan:} if it is not installed, and then install these modules using the {:cpan:} command. e.g.
sudo -H cpan install XML::LibXML
On a Mac, see {:https://formyfriendswithmacs.com/cpan-sl.html:}.
On Fedora Linux, I found that {:XML::Simple::get:} was failing on {:https:} URLs. A little test got the message {:501 Protocol scheme 'https' is not supported (LWP::Protocol::https not installed) at t.pl line 8:}. Trying to install {:LWP::Protocol::https:} failed with CPAN errors. I had to manually install {:Net::SSLeay:} to get it to work.
%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%
Visit {!expandfile-gh {:https://github.com/thvv/expandfile:}!} in your browser.
Click the green "Code" button.
Choose "Clone" or "Download ZIP."
Move the downloaded files into your {:bin:} directory.
This will give you the following files in your {:/bin:} directory:
Filename | Description |
---|---|
{:expandfile:} | Command line program |
{:expandfile.pm:} | Internals of expandfile |
{:readbindsql.pm:} | Internals of expandfile for {:*sqlloop:} |
{:readbindxml.pm:} | Internals of expandfile for {:*xmlloop:} |
You also get some useful helper programs in Perl, for invocation by the {:*shell:} builtin.
Filename | Description | Example |
---|---|---|
{:checknonempty:} | exit with error if arg is missing or empty; used in Makefiles | |
{:csv2sql.htmt:} | HTMX macro to convert a CSV file to a SQL table declaration | -- |
{:filedaysold:} | return file age | {:\%[*shell,&x,=filedaysold example-page.htmx]%:} {:%[*shell,&x,=filedaysold example-page.htmx]%%[x]%:} |
{:filemodiso:} | return file mod date as {:yyyy-mm-dd:}/td> | {:\%[*shell,&x,=filemodiso example-page.htmx]%:} {:%[*shell,&x,=filemodiso example-page.htmx]%%[x]%:} |
{:filemodshort:} | return file mod date as {:mm/dd/yy:} | {:\%[*shell,&x,=filemodshort example-page.htmx]%:} {:%[*shell,&x,=filemodshort example-page.htmx]%%[x]%:} |
{:filemodyear:} | return year of file modification | {:\%[*shell,&x,=filemodyear example-page.htmx]%:} {:%[*shell,&x,=filemodyear example-page.htmx]%%[x]%:} |
{:filesizek:} | return file size | {:\%[*shell,&x,=filesizek example-page.htmx]%:} {:%[*shell,&x,=filesizek example-page.htmx]%%[x]%:} |
{:firstletter:} | return first letter of a string | {:\%[*shell,&x,=firstletter abcdef]%:} {:%[*shell,&x,=firstletter abcdef]%%[x]%:} |
{:firstofnextmonth:} | return date of the first of next month | {:\%[*shell,&x,=firstofnextmonth]%:} {:%[*shell,&x,=firstofnextmonth]%%[x]%:} |
{:fmtnum:} | return a number formatted with commas, as a file size, or as a date | {:\%[*shell,&x,=fmtnum 1234567 num]%:} {:%[*shell,&x,=fmtnum 1234567 num]%%[x]%:} |
{:fmtsql:} | return a string formatted so that it can be input safely to mysql | {:\%[*shell,&x,=fmtsql "isn\'t"]%:} {:%[*shell,&x,=fmtsql "isn\\'t"]%%[x]%:} |
{:gifsize2:} | return image size: used by {:getimgdiv:} macro | {:\%[*shell,&x,=gifsize2 tinymultics.gif]%:} {:%[*shell,&x,=gifsize2 tinymultics.gif]%%[x]%:} |
{:gifsize:} | display length of a graphic | {:\%[*shell,&x,=gifsize tinymultics.gif]%:} {:%[*shell,&x,=gifsize tinymultics.gif]%%[x]%:} |
{:gth2x:} | shell script to generate 150x150 thumbnails (uses ImageMagick {:convert:}) | -- |
{:gthumb:} | shell script to generate thumbnails (uses ImageMagick {:convert:}) | -- |
{:htmxlib.htmi:} | macro library | -- |
{:lowercase:} | return lowercase version of a string | {:\%[*shell,&x,=lowercase Boston]%:} {:%[*shell,&x,=lowercase Boston]%%[x]%:} |
{:nargs:} | return the number of args | {:\%[*shell,&x,=nargs 1 2 3]%:} {:%[*shell,&x,=nargs 1 2 3]%%[x]%:} |
{:padstring:} | pad a text field with blanks to a given width | {:\%[*shell,&x,=padstring 6 "X"]%:} \"{:X :}\" |
{:padleft:} | pad a text field with blanks on the left to a given width | {:\%[*shell,&x,=padleft 6 "Q"]%:} \"{: Q:}\" |
{:uppercase:} | return uppercase version of a string | {:\%[*shell,&x,=uppercase hello]%:} {:%[*shell,&x,=uppercase hello]%%[x]%:} |
{:xml2sql:} | translate a simple XML file into a SQL table declaration | -- |
You can write other little commands to invoke with {:\%\[*shell\]\%:} to extend HTMX, as shell scripts or programs in C, Perl, Python, and so on, and place them in your {:/bin:} directory. You can also invoke Unix commands with {:\%\[*shell\]\%:}, such as {:date:}, {:grep:}, {:sed:}, {:awk:}, {:cut:}, and {:sort:}.
%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%You should be able to type the command
expandfile
and get a USAGE message, and no errors. This will show that {:expandfile:} and its needed Perl modules are correctly installed.
Type the command
expandfile foo
where {:foo:} is a plain text file without {:\%[...]\%:} or {:\\:} and you should get the same output as the contents of {:foo:}.
Then try some of the examples in the expandfile Tutorial.
%[** - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - **]%The HTMX source that generates this page is here.
A little explanation: