TextTools

The handful of Unix text idioms whose Python equivalent is non-obvious: awk-style field extraction, multi-key sort, and uniq.

TextTools is intentionally small. It wraps only the operations where native Python is fiddly to hand-roll. For the trivial cases, reach for the language directly: filter lines with re.search in a comprehension, slice with lines[:n] / lines[-n:], and substitute with re.sub.

Usage

from pymdm import TextTools

tools = TextTools()

# awk -F: '{print $1}' /etc/passwd
usernames = tools.awk(passwd_text, field=1, delimiter=":")

# the classic `sort | uniq -c` count-occurrences pipeline
sorted_lines = tools.sort(["http", "ssh", "http", "ftp", "ssh", "http"])
counts = tools.uniq(sorted_lines, count=True)
# -> ["1 ftp", "3 http", "2 ssh"]

Each method accepts input as either a multi-line str or a list[str]. Pass an optional MdmLogger to the constructor for debug-level call tracing.

class TextTools(logger: MdmLogger | None = None)

Bases: object

Text utilities for the Unix idioms Python doesn’t make obvious.

Parameters:

logger (MdmLogger | None)

awk(text: str | list[str], *, field: int | None = None, fields: list[int] | None = None, delimiter: str | None = None, filter: str | None = None, output_separator: str = ' ') → list[str]

Extract one or more 1-indexed fields per line (mirrors a simplified awk -F<delim> '/filter/ {print $field}').

This is a deliberately small subset of awk: field extraction with an optional row filter. For more complex transformations, use a list comprehension.

Parameters:

• text (str | list[str]) – Input text as a multi-line string or a list of lines

• field (int | None, optional) – A single 1-indexed field to extract. Mutually exclusive with fields.

• fields (list[int] | None, optional) – A list of 1-indexed fields to extract, joined by output_separator. Mutually exclusive with field.

• delimiter (str | None, optional) – Field delimiter. None (default) splits on any whitespace (matches default awk behavior).

• filter (str | None, optional) – Optional regex; only rows matching filter contribute to the output (like awk '/pattern/ {...}').

• output_separator (str, optional) – Separator used when joining multiple fields per row, defaults to a single space

Returns:

List of extracted strings, one per matched row. Rows with out-of-range field indices contribute empty strings for those fields (matches awk behavior).

Return type:

list[str]

Raises:

• ValueError – When neither field nor fields is given, or when both are given

• re.error – When filter is not a valid Python regex

sort(text: str | list[str], *, reverse: bool = False, numeric: bool = False, unique: bool = False, field: int | None = None, delimiter: str | None = None) → list[str]

Sort lines (mirrors sort).

Parameters:

• text (str | list[str]) – Input text as a multi-line string or a list of lines

• reverse (bool, optional) – Sort in descending order (like sort -r), defaults to False

• numeric (bool, optional) – Sort numerically rather than lexicographically (like sort -n). Non-numeric keys sort after numeric keys. Defaults to False

• unique (bool, optional) – Remove duplicate lines after sorting (like sort -u), defaults to False

• field (int | None, optional) – 1-indexed sort key field. When given, sort by that field instead of the whole line. Defaults to None.

• delimiter (str | None, optional) – Field delimiter when field is given. None splits on any whitespace.

Returns:

Sorted list of lines

Return type:

list[str]

uniq(text: str | list[str], *, count: bool = False, duplicates_only: bool = False, unique_only: bool = False) → list[str]

Collapse consecutive duplicate lines (mirrors uniq).

Like the bash uniq, this assumes input is already sorted — it only collapses consecutive duplicates. Pair with sort() for the common sort | uniq idiom.

Parameters:

• text (str | list[str]) – Input text as a multi-line string or a list of lines

• count (bool, optional) – Prefix each line with its occurrence count (like uniq -c), defaults to False

• duplicates_only (bool, optional) – Emit only lines that appeared more than once (like uniq -d). Mutually exclusive with unique_only.

• unique_only (bool, optional) – Emit only lines that appeared exactly once (like uniq -u). Mutually exclusive with duplicates_only.

Returns:

Deduplicated list of lines

Return type:

list[str]

Raises:

ValueError – When both duplicates_only and unique_only are True