Platform Architecture & Tech Stack
Provider adapters collect prices, asset data, news, filings, short interest, fundamentals, alternative data, and general economic data into shared stores. Airflow schedules ingestion and maintenance; Kafka delivers news revisions for processing. A Java query service exports database results as Parquet, and a Python SDK returns them as Polars DataFrames.
System map
Select a path to highlight its components, or select a component for its responsibilities.
PostgreSQL
Holds assets, canonical events, source associations, share/float observations, earnings, scanner snapshots, processed news/model inputs, inference results, and durable operational queues.
Price ingestion and adjustment maintenance
The price path uses staged batch loads. Alpaca supplies raw minute bars and daily price series; task manifests retain ingestion lineage rather than repeating it on every row. Airflow invokes the existing Python jobs.
- DownloadFetch completed historical dates into task-specific temporary Parquet partitions.
- ValidateCheck finite positive price fields, OHLC ordering, activity fields, and daily exchange-session dates.
- Load & verifyInsert accepted partitions into ClickHouse and verify their database contents.
- Reconcile factorsDerive stored daily price/volume adjustment factors from accepted raw and adjusted daily observations.
- Certify intervalRun warehouse validation before advancing the incremental completion cursor.
- Maintain adjustmentsAudit for corporate-action changes; refresh and verify affected daily histories and factors.
- Clean stagingRemove temporary partitions only after the relevant load and maintenance work succeeds.
Airflow task dependency
advance_prices
↓
maintain_adjustments
↓
cleanup_stagingThe incremental job retries its uncommitted date interval after failure. An exclusive staging lock prevents concurrent executable price pipelines from interfering with each other.
Stored series and derived views
Raw M1, raw D1, split-adjusted D1, all-adjusted D1, and daily factors are stored. Other intraday bars aggregate raw minutes at exchange-session boundaries; adjustments are applied after aggregation. Weekly, monthly, and yearly views aggregate the selected daily series.
Corporate-action maintenance preserves raw M1 and targets daily-history corrections and factor rebuilding. Derived VWAP is labeled as estimated rather than canonical vendor VWAP.
Extending the streaming layer to live prices
Architecture extension · not currently a live-price deployment. Kafka is not specific to news. The publisher/topic/consumer boundary can be reused for live market data, while keeping the existing historical price-ingestion and certification path.
- Market-data publisherA live feed adapter validates and timestamps incoming price events.
- Price topicsPartition by a stable instrument key to keep each instrument’s events together under a fixed partitioning scheme.
- Validation consumersA consumer group checks schemas, duplicates, event times, sequence gaps, and corrections.
- Storage consumersBatch accepted bars into ClickHouse with retry-safe writes and explicit handling of bar finality.
- Independent groupsFeature builders or alerting applications can consume the stream separately without changing the feed publisher.
What is reusable
Versioned event contracts, keyed publication, acknowledged checkpoints, durable consumer offsets, and idempotent processing already exist in the news path. Separate price topics and groups would let price processing scale independently of article preprocessing.
What live prices still require
A price-event schema and feed publisher; event-time, sequence, late-arrival, correction, and duplicate handling; and integration with the existing canonical bar and adjustment contracts. Topic retention must match replay needs—news compaction settings should not be copied blindly to a price-history stream.
The checked-in Kafka foundation currently starts with one broker, one news partition, and replication factor one. More parallel workers require a deliberate partition plan; resilient multi-broker operation requires replication and capacity planning. Repartitioning also needs an ordering/migration strategy. Load tests, lag monitoring, retention sizing, and recovery verification are needed before claiming live-price throughput or availability.
Kafka: publishers, topics, and consumers
Kafka separates data arrival from the work performed on it. Publishers append keyed records to topics; consumers read those records independently. In Squeeze, this currently connects live news ingestion to preprocessing. The same event-streaming pattern is extensible to live prices.
Publishers produce events
A publisher serializes a versioned event and sends it to a topic. Squeeze’s news publisher reads canonical database revisions, keys each record by article event ID, and advances its checkpoint after broker acknowledgement. Ingestion can keep collecting while downstream processing runs independently.
Consumers process at their own pace
Workers subscribe as a consumer group, read batches, and record their position using offsets. Squeeze’s preprocessing consumer validates revisions, cleans text, resolves targets, and persists accepted outputs before committing offsets. After a restart it resumes from the committed position.
- PublishersIngestion adapters or revision publishers append versioned, keyed events.
- Topic partitionsThe broker retains ordered records within each partition, separating arrival from processing.
- Consumer groupWorkers share partitions to distribute preprocessing work; each partition has one assigned consumer in the group.
- Preprocessed dataConsumers validate, transform, and durably store outputs before committing progress.
Why this supports efficient processing
Batching amortizes delivery and database-write overhead. A retained log absorbs bursts while consumers work through a backlog; consumer lag shows how far processing is behind ingestion. This keeps slower transformations off the ingestion path. Sustained throughput still depends on broker, worker, network, and database capacity.
Partitions provide parallelism; groups provide independent applications
Partitioning assigns work across workers while preserving record order within each partition. Separate groups can independently consume retained events for preprocessing, analytics, or alerting. Adding workers to one group increases useful parallelism only up to its partition count; adding groups creates independent consumption, not shared work. See Apache Kafka’s event-streaming overview and consumer and batching design.
Current live-news implementation
Provider polling and continuous collection first create canonical events in PostgreSQL. A separate continuous publisher reads changed news records through the query service and SDK. Kafka carries the revision stream; preprocessing materializes rebuildable state back into PostgreSQL.
- Canonical eventUpsert provider events and asset associations; retain database revision timestamps.
- Keyset pageRead changes ordered by update time and event ID through a fixed scan boundary.
- Publish & acknowledgePublish records keyed by event ID; save the page cursor only after Kafka confirms delivery.
- Consume revisionDiscard already-accepted or older revisions; normalize the required changed events.
- MaterializePersist cleaned articles, story membership, model inputs, and rejection reasons in PostgreSQL.
- Commit offsetsStore Kafka offsets only after the accepted processing state is durable.
Deterministic preprocessing
HTML, Unicode, whitespace, boilerplate, and supported wrapper URLs are normalized. Conservative duplicate merging retains source-event provenance. Company names and explicit ticker mentions support target confirmation; input rows preserve text-quality flags and hashes.
Slow URL resolution uses a durable queue outside the Kafka offset transaction. A delayed network request therefore does not become part of the consumer’s offset commit.
Inference and research are separate consumers
Target-specific local-LLM inference uses a PostgreSQL queue with leases, bounded retries, and concurrent claims. Canonical result storage and queue completion share a transaction; model/prompt/input revisions remain traceable.
CatalystBERT-v1 instead consumes article-target examples and pre-event market data to construct return-labeled training datasets. It is a separate model workflow, not the Kafka consumer or the local-LLM scoring service.
Publisher restarts can repeat a page. Revision checks make repeats safe to process; this is an at-least-once delivery design, not a claim of exactly-once execution across every component.
SEC document processing
- Poll metadataCollect canonical filing identities, accession information, and associated assets.
- Download documentsThe SEC Airflow workflow runs document acquisition after successful metadata polling.
- Extract sectionsPrioritize 8-K/6-K exhibits and selected 10-K/10-Q sections; preserve document names and hashes.
- Build target inputsMaterialize filing-target rows with preprocessing version and deterministic input hash.
- Serve through SDKExpose processed filing inputs through the same query-service and Python-client path.
SEC documents follow their own extraction path. Their availability in the platform does not establish SEC-specific results for the news-based CatalystBERT-v1 experiment.
Storage responsibilities
| Component | Owns | Reason for the boundary |
|---|---|---|
| ClickHouse | Canonical price bars and adjustment factors. | Column-oriented analytical access to price histories; derived timeframes use the canonical stored basis. |
| PostgreSQL | Assets, events, asset associations, fundamentals, scanner snapshots, processed inputs, inference results, and queues. | Relational identity, joins, revision state, and transactional workflow updates. |
| Kafka | Event-stream delivery and replay; currently canonical live-news revisions. | Separates publishers from consumer groups; extensible to live-price topics and workers. |
| Parquet | Temporary ingestion partitions and generated analytical result files. | Columnar files separate provider downloads from loads and query execution from Python analysis. |
PostgreSQL is the production preprocessing materialization store. SQLite is retained in tests and older audit tooling; it is not the production news state described here.
Query service and Python SDK
Research callers request a named dataset with typed parameters. SQL stays in a service-owned catalog; clients do not need database credentials or internal asset IDs. The query service reads the relevant database and streams the result into a Snappy-compressed Parquet file.
from sq_db_client import SqDbClient
client = SqDbClient()
bars = client.price_bars(
["AAPL", "MSFT"],
timeframe="M60",
adjustment="all",
session="RTH",
start_at="2025-01-01",
end_at="2026-01-01",
)
# bars is a polars.DataFrameRequest and file lifecycle
List parameters use a JSON POST body. Requests for the same query ID are serialized. A unique temporary export becomes a completed file only after successful generation, with atomic publication where the filesystem supports it. The SDK downloads the exact filename returned by that request.
This is synchronous export followed by file download. An asynchronous query-job API is documented as future work; the diagram does not assume one.
Session-aware price queries
The price interface supports 11 timeframes, raw/split/all adjustments, bounded date windows, and exchange-session selection. Derived intraday intervals are anchored to each session boundary; longer intervals are limited to completed periods.
Short-interest ratios use share/float observations dated no later than the settlement date and expose snapshot age. This as-of denominator join does not by itself establish when a FINRA report was publicly tradable.
Recovery and operational boundaries
Ingestion checkpoint
The price cursor advances after certification. A failed interval remains eligible for retry.
Publisher checkpoint
The news cursor advances after acknowledged page publication. A crash can repeat work before that point.
Consumer checkpoint
Processed revisions become durable before Kafka offsets commit.
Inference transaction
Result persistence and queue completion commit together, with input/model/prompt provenance.
Airflow DAGs invoke existing jobs and define dependencies, timeouts, and concurrency limits. Replacement DAGs start paused until a successful manual verification; scheduler definitions alone do not confirm current operational deployment.
Price staging locks, database revisions, partition verification, and file-publication boundaries provide different forms of recovery. None is a universal exactly-once or performance guarantee.
Engineering
Storage design, data provenance, and recovery from interrupted processing.
Checkpointing and recovery
Price manifests track partitions and validation. News publishers checkpoint after delivery acknowledgements; consumers commit offsets after durable processing. Retries can repeat work safely.
Checkpoints · replay · idempotencyPrice storage and timeframe reconstruction
Keep raw minute bars, daily observations, and adjustment factors. Reconstruct other timeframes at query time, respecting exchange sessions and early closes rather than duplicating every view.
Canonical bars · on-demand reconstructionData provenance and historical alignment
Keep source revisions and model-input provenance. Use historical share snapshots for short-interest ratios, retain missing values, and mask ambiguous reaction windows when constructing model targets.
As-of joins · target masks · provenance