API Documentation

Log Quantity Abstract Interfaces

class logpyle.LogQuantity(name: str, unit: str | None = None, description: str | None = None)

A source of a loggable scalar that is gathered at the start of each time step.

Quantity values are gathered in LogManager.tick_before().

__init__(name: str, unit: str | None = None, description: str | None = None) → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

tick() → None

Perform updates required at every LogManager tick.

property default_aggregator: None

Default rank aggregation function.

__call__() → Any

Return the current value of the diagnostic represented by this LogQuantity or None if no value is available.

This is only called if the invocation interval calls for it.

class logpyle.PostLogQuantity(name: str, unit: str | None = None, description: str | None = None)

A source of a loggable scalar that is gathered after each time step.

Quantity values are gathered in LogManager.tick_after().

__init__(name: str, unit: str | None = None, description: str | None = None) → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

tick() → None

Perform updates required at every LogManager tick.

property default_aggregator: None

Default rank aggregation function.

__call__() → Any

Return the current value of the diagnostic represented by this LogQuantity or None if no value is available.

This is only called if the invocation interval calls for it.

prepare_for_tick() → None

Perform (optional) update at LogManager.tick_before().

class logpyle.MultiLogQuantity(names: List[str], units: Sequence[str | None] | None = None, descriptions: Sequence[str | None] | None = None)

A source of a list of loggable scalars gathered at the start of each time step.

Quantity values are gathered in LogManager.tick_before().

__init__(names: List[str], units: Sequence[str | None] | None = None, descriptions: Sequence[str | None] | None = None) → None

Create a new quantity.

Parameters:

• names – List of quantity names.

• units – List of quantity units.

• descriptions – List of quantity descriptions.

tick() → None

Perform updates required at every LogManager tick.

property default_aggregators: List[None]

List of default aggregators.

__call__() → Iterable[float | None]

Return an iterable of the current values of the diagnostic represented by this MultiLogQuantity.

This is only called if the invocation interval calls for it.

class logpyle.MultiPostLogQuantity(names: List[str], units: Sequence[str | None] | None = None, descriptions: Sequence[str | None] | None = None)

A source of a list of loggable scalars gathered after each time step.

Quantity values are gathered in LogManager.tick_after().

__init__(names: List[str], units: Sequence[str | None] | None = None, descriptions: Sequence[str | None] | None = None) → None

Create a new quantity.

Parameters:

• names – List of quantity names.

• units – List of quantity units.

• descriptions – List of quantity descriptions.

tick() → None

Perform updates required at every LogManager tick.

property default_aggregators: List[None]

List of default aggregators.

__call__() → Iterable[float | None]

Return an iterable of the current values of the diagnostic represented by this MultiLogQuantity.

This is only called if the invocation interval calls for it.

prepare_for_tick() → None

Perform (optional) update at LogManager.tick_before().

Log Manager

class logpyle.LogManager(filename: str | None = None, mode: str = 'r', mpi_comm: mpi4py.MPI.Comm | None = None, capture_warnings: bool = True, watch_interval: float = 1.0, capture_logging: bool = True)

A distributed-memory-capable diagnostic time-series logging facility. It is meant to log data from a computation, with certain log quantities available before a cycle, and certain other ones afterwards. A timeline of invocations looks as follows:

tick_before()
compute...
tick_after()

tick_before()
compute...
tick_after()

...

In a time-dependent simulation, each group of tick_before() tick_after() calls captures data for a single time state, namely that in which the data may have been before the “compute” step. However, some data (such as the length of the timestep taken in a time-adaptive method) may only be available after the completion of the “compute…” stage, which is why tick_after() exists.

A LogManager logs any number of named time series of floats to a file. Non-time-series data, in the form of constants, is also supported and saved.

If MPI parallelism is used, the “head rank” below always refers to rank 0.

Command line tools called runalyzer are available for looking at the data in a saved log.

__init__(filename: str | None = None, mode: str = 'r', mpi_comm: mpi4py.MPI.Comm | None = None, capture_warnings: bool = True, watch_interval: float = 1.0, capture_logging: bool = True) → None

Initialize this log manager instance.

Parameters:

• filename – If given, the filename to which this log is bound. If this database exists, the current state is loaded from it.

• mode – One of “w”, “r” for write, read. “w” assumes that the database is initially empty. May also be “wu” to indicate that a unique filename should be chosen automatically. May also be “wo” to indicate that the file should be overwritten.

• mpi_comm – An optional mpi4py.MPI.Comm object. If given, logs are periodically synchronized to the head node, which then writes them out to disk.

• capture_warnings – Tap the Python warnings facility and save warnings to the log file. Note that when multiple LogManager instances have warnings capture enabled, the warnings will be saved to all instances.

• watch_interval – print watches every N seconds.

• capture_logging – Tap the Python logging facility and save logging messages to the log file. Note that when multiple LogManager instances have logging capture enabled, the logging messages will be saved to all instances.

save() → None

Commit the current state of the log.

close() → None

Close this LogManager instance.

Data retrieval

get_table(q_name: str) → DataTable

Return a DataTable of the data logged for the quantity q_name.

get_warnings() → DataTable

Return a DataTable of warnings logged by this LogManager instance.

get_logging() → DataTable

Return a DataTable of logging messages logged by this LogManager instance.

get_expr_dataset(expression: Expression, description: str | None = None, unit: str | None = None) → Tuple[str | Any, str | Any | None, List[Tuple[int, Any]]]

Prepare a time-series dataset for a given expression.

Parameters:

expression – A pymbolic expression that may involve the time-series variables and the constants in this LogManager. If there is data from multiple ranks for a quantity occurring in this expression, an aggregator may have to be specified.

Returns:

(description, unit, table), where table is a list of tuples (tick_nbr, value).

Aggregators are specified as follows:

• qty.min, qty.max, qty.avg, qty.sum, qty.norm2, qty.median

• qty[rank_nbr]

• qty.loc

get_joint_dataset(expressions: Sequence[Expression]) → List[Any]

Return a joint data set for a list of expressions.

Parameters:

expressions – a list of either strings representing expressions directly, or triples (descr, unit, expr). In the former case, the description and the unit are found automatically, if possible. In the latter case, they are used as specified.

Returns:

A triple (descriptions, units, table), where table is a a list of [(tstep, (val_expr1, val_expr2,...)...].

Configuration

capture_warnings(enable: bool = True) → None

Enable or disable warnings capture.

capture_logging(enable: bool = True) → None

Enable or disable logging capture.

add_watches(watches: List[str | Tuple[str, str]]) → None

Add quantities that are printed after every time step.

Parameters:

watches – List of expressions to watch. Each element can either be a string of the expression to watch, or a tuple of the expression and a format string. In the format string, you can use the custom fields {display}, {value}, and {unit} to indicate where the watch expression, value, and unit should be printed. The default format string for each watch is {display}={value:g}{unit}.

set_watch_interval(interval: float) → None

Set the interval (in seconds) between the time watches are printed.

Parameters:

interval – watch printing interval in seconds.

set_constant(name: str, value: Any) → None

Make a named, constant value available in the log.

Parameters:

• name – the name of the constant.

• value – the value of the constant.

add_quantity(quantity: LogQuantity, interval: int = 1) → None

Add a LogQuantity to this manager.

Parameters:

• quantity – add the specified LogQuantity.

• interval – interval (in time steps) when to gather this quantity.

enable_save_on_sigterm() → Callable[[...], Any] | int | None

Enable saving the log on SIGTERM.

Returns:

The previous SIGTERM handler.

Time Loop

tick_before() → None

Record data points from each added LogQuantity that is not an instance of PostLogQuantity. Also, invoke PostLogQuantity.prepare_for_tick() on PostLogQuantity instances.

tick_after() → None

Record data points from each added LogQuantity that is an instance of PostLogQuantity.

May also checkpoint data to disk.

logpyle.add_run_info(mgr: LogManager) → None

Add generic run metadata, such as command line, host, and time.

Built-in Log General-Purpose Quantities

class logpyle.IntervalTimer(name: str, description: str | None = None)

Records elapsed times supplied by the user either through sub-timers, or by explicitly calling add_time().

__init__(name: str, description: str | None = None) → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

get_sub_timer() → _SubTimer

start_sub_timer() → _SubTimer

add_time(t: float) → None

class logpyle.LogUpdateDuration(mgr: LogManager, name: str = 't_log')

Records how long the last log update in LogManager took.

__init__(mgr: LogManager, name: str = 't_log') → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

class logpyle.EventCounter(name: str = 'interval', description: str | None = None)

Counts events signaled by add().

__init__(name: str = 'interval', description: str | None = None) → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

add(n: int = 1) → None

transfer(counter: EventCounter) → None

pop() → int

class logpyle.TimestepCounter(name: str = 'step')

Counts the number of times LogManager ticks.

class logpyle.StepToStepDuration(name: str = 't_2step')

Records the wall time between the starts of consecutive time steps, i.e., the wall time between LogManager.tick_before() of step x and LogManager.tick_before() of step x+1. The value stored is the value for step x+1.

Note

In most cases, this quantity should approximately match t_step + t_log. If it does not, it might indicate that the application performs operations outside LogManager.tick_before() and LogManager.tick_after(), or that some other time is not being accounted for.

__init__(name: str = 't_2step') → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

class logpyle.TimestepDuration(name: str = 't_step')

Records the wall time between invocations of LogManager.tick_before() and LogManager.tick_after(), i.e., the duration of the time step.

__init__(name: str = 't_step') → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

class logpyle.InitTime(name: str = 't_init')

Stores the time it took for the application to initialize.

Measures the time from process start to the start of the first time step.

__init__(name: str = 't_init') → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

class logpyle.WallTime(name: str = 't_wall')

Records (monotonically increasing) wall time since the quantity was initialized.

__init__(name: str = 't_wall') → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

class logpyle.ETA(total_steps: int, name: str = 't_eta')

Records an estimate of how long the computation will still take.

__init__(total_steps: int, name: str = 't_eta') → None

Create a new quantity.

Parameters:

• name – Quantity name.

• unit – Quantity unit.

• description – Quantity description.

class logpyle.MemoryHwm(name: str = 'memory_usage_hwm')

Record (monotonically increasing) memory high water mark (HWM) in MBytes.

class logpyle.GCStats

Record Garbage Collection statistics.

Information regarding the meaning of these values can be found at:

• https://docs.python.org/3/library/gc.html

• https://alex.dzyoba.com/blog/arc-vs-gc

• https://stackoverflow.com/questions/64561488/pythons-gc-get-objects-from-get-count

• https://github.com/python/cpython/blob/main/Modules/gcmodule.c

logpyle.add_general_quantities(mgr: LogManager) → None

Add generally applicable LogQuantity objects to mgr.

Built-in Log Simulation-Related Quantities

class logpyle.SimulationTime(name: str = 't_sim', start: float = 0)

Record (monotonically increasing) simulation time.

class logpyle.Timestep(name: str = 'dt', unit: str = 's')

Record the magnitude of the simulated time step.

logpyle.set_dt(mgr: LogManager, dt: float) → None

Set the simulation timestep on LogManager mgr to dt.

Parameters:

• mgr – the LogManager instance.

• dt – the simulation timestep.

logpyle.add_simulation_quantities(mgr: LogManager) → None

Add LogQuantity objects relating to simulation time.

Parameters:

mgr – the LogManager instance.

Internal stuff that is only here because the documentation tool wants it

class logpyle._SubTimer(itimer: IntervalTimer)

Runalyzer Functions

logpyle.runalyzer.is_gathered(conn: Connection) → bool

Returns whether a connection to an existing database has been gathered.

Parameters:

conn – SQLite3 connection object

Database Upgrade Functions

logpyle.upgrade_db.upgrade_db(dbfile: str, suffix: str, overwrite: bool) → None

Upgrade a database file to the most recent format. If the overwrite parameter is True, it simply modifies the existing database and uses the same file name for the upgraded database. Otherwise, a new database is created with a separate filename by appending the given suffix to the original file’s base name using filename + suffix + “.” + file_ext.

Parameters:

• dbfile – A database file path

• suffix – a suffix to be appended to the filename for the upgraded database

• overwrite – a boolean value indicating whether to overwrite the original database or not

Note

Currently, upgrades all schema versions to version 3. Upgrading from version <=1 is untested.

Overview of known changes between schema versions

Schema version	Logpyle version	Changes
0	pre v1 (pytools.log)	Initial version, no schema_version yet
1	v1 – v9 (pytools.log)	Added warnings table
2	v10 – 2023.1	Added warnings.rank column
3	2023.2 –	Added warnings.unixtime column and logging table