pixie16.read.list_mode package

pixie16.read.list_mode.list_mode_data_reader module

Read list mode data

The file provides a python based list mode data parser.

The input can either come from a data stream or from one or several binary files (in case a large binary data set is split across multiple files).

We keep all the data in numpy arrays, whic are either memmaped or pre-allocated, to make data access fast.

exception pixie16.read.list_mode.list_mode_data_reader.EmptyError

Bases: Exception

We reached the end of the last file.

class pixie16.read.list_mode.list_mode_data_reader.Event(channel: int = -1, crate: int = -1, slot: int = -1, timestamp: float = -1.0, CFD_fraction: int = -1, energy: int = -1, trace: int = -1, CFD_error: int = -1, pileup: int = -1, trace_flag: int = -1, Esum_trailing: int = -1, Esum_leading: int = -1, Esum_gap: int = -1, baseline: float = -1.0, QDCSum0: int = -1, QDCSum1: int = -1, QDCSum2: int = -1, QDCSum3: int = -1, QDCSum4: int = -1, QDCSum5: int = -1, QDCSum6: int = -1, QDCSum7: int = -1, ext_timestamp: float = -1.0, chunk_timestamp: float = -1)

Bases: Struct

Data storage for a single event that has attribute access.

This is similar to a namedtuple, but the msgspec implemenation is faster.

CFD_error: int
CFD_fraction: int
Esum_gap: int
Esum_leading: int
Esum_trailing: int
QDCSum0: int
QDCSum1: int
QDCSum2: int
QDCSum3: int
QDCSum4: int
QDCSum5: int
QDCSum6: int
QDCSum7: int
baseline: float
channel: int
chunk_timestamp: float
crate: int
energy: int
ext_timestamp: float
pileup: int
slot: int
timestamp: float
to_list()
trace: int
trace_flag: int
class pixie16.read.list_mode.list_mode_data_reader.FileReader(files: Iterable[Path])

Bases: object

Read binary data from multiple files.

Use mmap for fast data access to binary data on disc. The data can be distributed across several files.

This class should be used as a context manager: >>> files = (“file1.bin”, “file2.bin”) >>> with FileReader(files) as r: >>> r.read(100)

or one needs to call self.open_files() and self.close_files() manually.

advance(size: int)

Not needed, since all the data is in the files and we can just advance the file position during read()

close_files()
open_files()
read(size: int, offset: int = 0) → bytes

Retrieve the next size bytes from the files and advance the file position.

Parameters:

  • size – How many bytes should be returned.

  • offset – ignored here (used in Streamreader)

Raises:

EmptyError – When data of the requested size is not available in the data stream.

exception pixie16.read.list_mode.list_mode_data_reader.LeftoverBytesError

Bases: Exception

To be raised if a partial event is left in the byte stream/file.

class pixie16.read.list_mode.list_mode_data_reader.ListModeDataReader(reader: FileReader | StreamReader)

Bases: object

iterevents()

Will raise an exception to exit loop.

pop() → Event

Pop the top event from queue

pop_all()
class pixie16.read.list_mode.list_mode_data_reader.StreamReader(initial_buffer_size=25000000)

Bases: object

Handle streamed binary data.

Provides the same interface as FileReader and can be used as a direct replacement.

New data can be added using the put method.

Internally we store the data in a np.array of np.uint32.

We make the array large enough to hold the data and have the buffer grow and shrink as needed.

This is similar to a ring buffer, but we just allocate twice the buffer size memory and then keep track of the left and right position in the buffer and shift the data to the far left every now and then.

adjust_and_shift_buffer() → None

Create a new buffer and copy the old buffer to beginning.

This also takes care of changing the size of the buffer

advance(size: int) → None

Advance self.left once a full event is read (possible after multiple read statement)

put(input: numpy.ndarray) → None

Add data to the buffer.

Also increase/decrease the buffer size as needed.

read(size: int, offset: int = 0) → numpy.ndarray

Return size 32-bit words from the buffer as a numpy array.

In the streamreader, there might be an incomplete event at the end of the buffer. However, we have several read statements to read a single event. We cannot advance self.left directly, unless we read a whole event (up to 3 read()). offset is used to read partial data and advance() is used to move self.left once a full event is read.

pixie16.read.list_mode.list_mode_data_reader.data_rate_from_file(files)

Calculate the data rate in MB/s from a list of binary files. Returns a dictionary with the data rate for each second in the data set.

pixie16.read.list_mode.list_mode_data_reader.data_rate_per_channel_from_file(files)

Calculate the data rate in MB/s per channel from a list of binary files. Returns a nested dictionary {channel: {second: MB/s}}.

pixie16.read.list_mode.list_mode_data_reader.events_from_files_generator(files)

Yields events from a list of binary files

pixie16.read.list_mode.list_mode_data_reader.read_list_mode_data(files, buffer_size=1000000000)

Loads a list of binary files into a pandas DataFrame.

Note: the binary files need to be ordered correctly, in case an event is distributed across two files.

pixie16.read.list_mode.list_mode_data_reader.read_list_mode_data_as_events(files, buffer_size=1000000000, max_size=None)

Returns a list of at most max_size events from files.

Note: the binary files need to be ordered correctly, in case an event is distributed across two files.

pixie16.read.list_mode.list_mode_data_reader.sort_events_by_channel(events, channel_list=None)

Sort events by channel.

Takes the output of read_list_mode_data_as_events and sorts them into a dictionary that has the channel number as key and as value a list of events (still including the channel number).

Parameters:

  • events – List of events as returned by read_list_mode_data_as_events

  • channel_list – Optional list of channels to include. This can help to reduce the amount of data.

pixie16.read.list_mode.list_mode_data_reader.timestamps_and_sizes_from_files_generator(files)

Yields (timestamp, event_length_words) for each event in binary files.

pixie16.read.list_mode.list_mode_data_reader.timestamps_sizes_and_channel_from_files_generator(files)

Yields (channel, timestamp, event_length_words) for each event in binary files.

pixie16.read.list_mode.list_mode_data_reader.to_tuple(obj)
pixie16.read.list_mode.list_mode_data_reader.unpack_energy(data)

Extract energy sum fields from 16 bytes (4 32-bit words).

Format: u32 u32 u32 f32 Fields: Esum_trailing, Esum_leading, Esum_gap, baseline

pixie16.read.list_mode.list_mode_data_reader.unpack_ext_time(data)

Extract extended timestamp from 8 bytes (2 32-bit words).

Format: u32 p16 u16 (p16 means 16 padding bits) Fields: Ext_TS_Lo, Ext_TS_Hi

pixie16.read.list_mode.list_mode_data_reader.unpack_extra_header(data, extra_header_length)

Unpack additional header data based on length.

Parameters:

  • data – Byte data (from byteswapped numpy array)

  • extra_header_length – Number of 32-bit words in extra header

Raises:

KeyError – If extra_header_length is not a valid value (indicates corrupt data)

pixie16.read.list_mode.list_mode_data_reader.unpack_header(data)

Extract header fields from 16 bytes using bitwise operations.

Format: b1u14u5u4u4u4u32u3u13u16b1u15u16 Fields: pileup, Event Length, Header Length, crate, slot, channel,

EVTTIME_LO, CFD trigger source bits, CFD Fractional Time, EVTTIME_HI, trace_flag, Trace Length, energy

Note: Works on little-endian data bytes directly from mmap.

pixie16.read.list_mode.list_mode_data_reader.unpack_qsums(data)

Extract QDC sum fields from 32 bytes (8 32-bit words).

Format: u32 * 8 Fields: QDCSum0-7