mg_passes(1)

NAME
mg_passes - builds mg databases

SYNOPSIS
mg_passes [ -h ] [ -G ] [ -S ] [ -D ] [ -W ]
[ -1 | -2 | -3 ] [ -C compstatpoint ]
[ -n tracename ] [ -b bufsize ] [ -m memlimit ]
[ -c numchunks ] [ -a stemmer ] [ -s stemmethod ] [ -t tracepos ]
[ -T1 ] [ -T2 ] [ -I1 ] [ -I2 ]
[ -d directory ] -f name [ filename(s) ]

DESCRIPTION
mg_passes is the program that does most of the work when building mg(1) database systems. The input documents can come from either stdin or from a list of files on the command line. Individual documents must be separated with control-B characters. In general, mg_passes must be run twice to build a database, first with the -T1 and -I1 options, and second with the -T2 and -I2 options. Several other programs must be run in order to get an mg(1) database that is ready for the mgquery(1) program. The EXAMPLE section below gives an example of how to build a complete mg(1) database.

OPTIONS
Options may appear in any order, but the filename(s), if specified, must be last.

-h  This displays a usage line on stderr.

-G  Treat SGML tags as non-words when building the inverted file. An SGML tag is anything between angle brackets, i.e., `<' and `>'.

-S  This option causes a special pass to be executed. It is up to a user to modify mg.special.c in the source code to do something with the documents it is given.

-D  If mg_passes fails, then print the document that caused the failure to the trace file if tracing is active, or to stderr if it is not.

-W  This option enables the generation of the weights file when -I2 is specified. It causes -I2 to use a little more memory and CPU.

-1  Produce a level-1 inverted file. This option is only useful when specified with -I1 . A level-1 inverted file makes it possible for mgquery(1) to do Boolean queries. Ranked queries can still be done, although the quality of the ranking is abysmal.

-2  Produce a level-2 inverted file. This option is only useful when specified with -I1 . This is the default when neither -1, -2 or -3 is specified. A level-2 inverted file makes it possible for mgquery(1) to do Boolean queries and cosine-ranked queries.

-3  Produce a level-3 inverted file. This option is only useful when specified with -I1 . This has been implemented to enable paragraph-level inversion. Paragraphs are delimited by control-C characters in the source text.

-C compstatpoint
    This option causes statistics on the compression performance to be output to a file called *.compression.stats. compstatpoint specifies the interval between outputting each line of statistics. The units of compstatpoint are kilobytes of source text. E.g., if compstatpoint is 10, then a line is output to the file every 10 KB of input source. Each line of the file consists of 4 numbers The first number is the amount of input text, in bytes, processed so far. The second number is the amount of input text, in bytes, processed since the last line was output to the file. The third number is the number of output bytes generated since the last line was output to the file, and the fourth number is the compression achieved since the last line was output, i.e., the third number divided by the second number.

-n tracename
    This specifies the filename to use for the trace log, if tracing is enabled using the -t option. If -n tracename is not given and tracing is enabled, a default trace filename will be used.

-s stemmethod
    This specifies the method to use to “stem” the words in the inverted file dictionary. This is a bit mask specifying the operations to do on words as they are parsed out of the text, where bit number 0 is the low-order (rightmost) bit. Bit 0 does case folding, and bit 1 does simple stemming, so the value 3 for stemmethod does both case folding and stemming.

-a stemmer
    This specifies the stemmer to use when stemming words. This is a description of the language the stemmer is intended for or a description of the stemmer. Valid options include: english, lovin, french, and simplefrench.

-b bufsize
    Specify the size of the document buffer in kilobytes. If any document is larger than bufsize, the program will abort with an error message. This should probably be replaced with some system which automatically increases the buffer size as required. The default size is 3072 KB (3 MB).

-m memlimit
    Maximum amount of memory to use for the pass-2 file inversion in megabytes. This option is only useful when used in conjunction with the option -I1. The larger this value, the faster the pass-2 inversion will proceed. The default value is 5 MB.

-c numchunks
    The maximum number of inversion chunks to write to disk. Each chunk will be approximately as large as memlimit. This option is only useful when used in conjunction with the option -I2. The larger this value, the faster the pass-2 inversion will proceed. The default value is 5 MB.

-t tracepos
    This option activates tracing. A line will be generated in the trace file for every tracepos input bytes processed. The default name for the trace file can be overridden using the -n tracename option.

-T1 Generate the *.text.stats file.

-T2 Generate the *.text, *.text.idx, and possibly the *.text.dict.aux files. Using this option requires that the *.text.dict file be present.

-I1 Generate the *.invf.dict, *.invf.chunk, and *.invf.chunk.trans files.

-I2 Generate the *.invf and *.invf.idx files. Using this option requires that the *.invf.dict.hash, *.invf.chunk, and *.invf.chunk.trans files be present. The *.invf.dict.hash file is generated by mg_perf_hash_build(1) from the *.invf.dict.build file. If the -W option is specified, the *.weight file will also be generated.

-d directory
    This specifies the directory where the document collection is to be written.

-f name
    This specifies the base name of the document collection that will be created.

filename(s)
    This specifies the source text. If this is not specified, then the program expects the source text from stdin.

EXAMPLE
What follows is a UNIX csh(1) script as an example of how to build an mg(1) document collection.

#! /bin/csh
# The first argument on the command line specifies the
#   source of the text
set source = ($1)

# The second argument is the name of the collection
set text = ($2)

# Create *.text.stats,  *.invf.dict.build,
#   *.invf.chunk and *.invf.chunks.trans
${source} | mg_passes -T1 -I1 -m 1 -t 1 -f ${text}

# Create *.text.dict
mg_compression_dict -f ${text}

# Create *.invf.dict.hash
mg_perf_hash_build -f ${text}

# Create *.text,  *.text.idx,
#   *.invf and *.invf.idx
${source} | mg_passes -T2 -I2 -c 2 -t 1 -f ${text}

# Create *.text.idx.wgt and *.weight.approx
mg_weights_build -f ${text} -b 8

# Create *.invf.dict
mg_invf_dict -f ${text} -b 4096

# Create *.text.dict
mg_fast_comp_dict -f ${text}

ENVIRONMENT

MGDATA
    If this environment variable exists, then its value is used as the default directory where the mg(1) collection files are. If this variable does not exist, then the directory “.” is used by default. The command line option -d directory overrides the directory in MGDATA.

FILES

*.invf
    Inverted file.

*.invf.chunk
    Inverted file chunk descriptor file. When the inverted file is created it is created in chunks that use no more than a set amount of memory. This file describes those chunks.

*.invf.chunk.trans
    Word-occurrence-order to lexical-order translation file. The *.invf.chunk file is written in word-occurrence order but is required by -I2 to be in lexical order.

*.invf.dict.build
    Compressed stemmed dictionary.

*.invf.dict.hash
    Data for an order-preserving perfect hash function.

*.invf.idx
    The index into the inverted file.

*.weight
    The exact weights file.

*.text
    Compressed documents.

*.text.stats
    Statistics about the text.

*.text.dict
    Compressed compression dictionary.

*.text.idx
    Index into the compressed documents.

*.trace
    The default trace file.

*.compression.stats
    Statistics about the compression of the text.

SEE ALSO
mg(1), mg_compression_dict(1), mg_fast_comp_dict(1), mg_get(1), mg_invf_dict(1), mg_invf_dump(1), mg_invf_rebuild(1), mg_perf_hash_build(1), mg_text_estimate(1), mg_weights_build(1), mgbilevel(1), mgbuild(1), mgdictlist(1), mgfelics(1), mgquery(1), mgstat(1), mgtic(1), mgticbuild(1), mgticdump(1), mgticprune(1), mgticstat(1).