Simulation Architecture

The biologger-sim architecture is designed to support both high-throughput batch processing (Lab Mode) and real-time visualization (Simulation Mode).

Dual-Mode System

Feature	Lab Mode (Batch)	Simulation Mode (Real-Time)
Primary Goal	R-Compatible Analysis & CSV Generation	Real-Time Visualization & Streaming
Loop Architecture	Linear Loop (Single Entity)	C-Merged Time Loop (Multi-Entity)
Networking	Optional (for debugging)	Mandatory (Serialize + ZMQ Push)
Performance	~94,000 rec/sec	~33,000+ rec/sec (Scalable)

Event Loop: The “Scream” Architecture

To support scalable multi-animal simulation in pure Python, we utilize a Time-Sorted Merge architecture rather than a manual priority queue.

Prior Approach (Slow)

The naive implementation used a Python while loop with heapq.heappush and heapq.heappop for every single record to maintain time synchronization between entities.

• Overhead: O(log K) per record in Python interpreter steps.

• Result: 3x performance penalty for adding a second animal.

Current Approach (Fast)

We now use heapq.merge, which is a C-optimized generator.

1. Generators: Each SimulationEntity yields a lazy stream of (timestamp, index, record) tuples.

2. C-Merge: heapq.merge(*streams) consumes these streams and performs the K-way merge sort entirely in C space.

3. Python Loop: The main loop simply iterates over this pre-sorted stream, dispatching records to processors.

This results in sublinear scaling overhead for additional animals.