mg_compression_dict(1)

NAME
mg_compression_dict - build a compression dictionary.

SYNOPSIS
mg_compression_dict [ -h ] [ -C | -P | -S ]
[ -0 | -1 | -2 | -3 ] [ -H | -B | -D | -Y | -M ]
[ -l lookback ] [ -k mem ] [ -d directory ] -f name

DESCRIPTION
mg_compression_dict builds a compression dictionary based on the statistics gathered during the first pass over the text. The options to the program are mainly concerned with limiting the amount of memory the dictionary will use and with how the text compressor will cope with any novel words found during the compression phase.

OPTIONS
Options may appear in any order.

-h  This displays a usage line on stderr.

-C  Build a complete dictionary from the statistics file. If during the text compression phase a novel word is found, then the compressor will produce an error message and stop.

-P  Build a partial dictionary from the statistics file. This dictionary assumes that the statistics file are based on the entire text. The statistics of words not includes in the dictionary are used to calculate the escape probability. If novel words are being coded character by character, then there may not be a Huffman code for every possible character. This means that the compressor may fail if a novel word contains a novel character.

-S  Build a seed dictionary from the statistics file. This dictionary assumes that the statistics file is based on only a portion of the text to be compressed. The probability of a novel word is based on the number of words that have only occurred once. If novel words are being coded character by character, then the Huffman codes for characters are based on the frequency of characters in the dictionary.

-0  All words from the statistics file are included in the built dictionary.

-1  Words are included in the dictionary until the dictionary reaches the desired size. Words are selected for the dictionary based on the order they occurred in the source text.

-2  Words are included in the dictionary until the dictionary reaches the desired size. The most frequent words are included in the dictionary first; where there is a tie for frequency, the shortest word is included first.

-3  Words are included in the dictionary until the dictionary reaches the desired size. The most frequent words are included in the dictionary first; where there is a tie for frequency, the shortest word is included first. Words are the shuffled back and forth between the `keep' and `discard' lists to find the `optimal' set of words that should be in the dictionary.

-H  This specifies that novel words will be coded character by character using Huffman codes.

-B  This specifies that an auxiliary dictionary will be built by the compressor. Each novel word found will be placed at the end of the auxiliary dictionary. Novel words will be coded in the compressed text using binary codes. The binary code represents their occurrence position in the auxiliary dictionary.

-D  This specifies that an auxiliary dictionary will be built by the compressor. Each novel word found will be placed at the end of the auxiliary dictionary. Novel words will be coded in the compressed text using delta codes. The delta code represents their occurrence position in the auxiliary dictionary.

-Y  This specifies that an auxiliary dictionary will be built by the compressor. Each novel word found will be placed at the end of the auxiliary dictionary. Novel words will be coded in the compressed text using a combination of gamma and binary codes. The code represents their occurrence position in the auxiliary dictionary. This generally produces better compression than -B or -D.

-M  This specifies that an auxiliary dictionary will be built by the compressor. Each novel word found will be placed at the end of the auxiliary dictionary. Novel words will be coded in the compressed text using a combination of gamma and binary codes. The code represents their occurrence position in the auxiliary dictionary. This method is adaptive within documents, and generally produces better compression than -B, -D or -Y.

-l lookback
    The generated dictionary is designed to be front coded when it is loaded into memory. Under normal circumstances, a front-coded dictionary would require scanning from the beginning in order to find any particular word. However, every lookback words in the dictionary, the whole word is stored and a pointer to that word maintained. E.g., if lookback is 4, then every fourth word is stored in its entirety.

-k mem
    This limits the amount of memory to use for the generated dictionary. Words are selected for the dictionary based of the text statistics, and whether -0, -1, -2 or -3 is specified. The memory is calculated assuming a lookback of 0, irrespective of what actual lookback is specified. This means that if a non-zero lookback is given, the dictionary will actually occupy less space than specified by -k.

-d directory
    This specifies the directory where the document collection can be found.

-f name
    This specifies the base name of the document collection.

ENVIRONMENT

MGDATA
    If this environment variable exists, then its value is used as the default directory where the mg(1) collection files are. If this variable does not exist, then the directory “.” is used by default. The command line option -d directory overrides the directory in MGDATA.

FILES

*.text.stats
    Statistics about the source text.

*.text.dict
    Compression dictionary for the source text.

SEE ALSO
mg(1), mg_fast_comp_dict(1), mg_get(1), mg_invf_dict(1), mg_invf_dump(1), mg_invf_rebuild(1), mg_passes(1), mg_perf_hash_build(1), mg_text_estimate(1), mg_weights_build(1), mgbilevel(1), mgbuild(1), mgdictlist(1), mgfelics(1), mgquery(1), mgstat(1), mgtic(1), mgticbuild(1), mgticdump(1), mgticprune(1), mgticstat(1).