DCAS/FCAS - An HFT Analysis Framework

# DCAS: First Introduction

This document introduces a pair of Jupyter notebooks that together form a full stack for high‑frequency market data capture, storage, analysis, and visualization on top of Apache Iceberg and Polars.

The two main notebooks are:

- DCAS.ipynb – a robust high‑frequency trading data pipeline that turns raw market feeds into an organized, queryable data lake.

- Metadata_based_Analysis.ipynb – a generic, reusable factor‑analysis and visualization framework built on the DCAS data lake.

***

## What the DCAS notebook implements

DCAS.ipynb focuses on building a production‑oriented, high‑frequency data capture and enrichment pipeline that can run continuously while feeding an Iceberg‑backed data lake.

Key capabilities:

- Multi‑source data ingestion:

FileWatcher classes tail live CSV/NDJSON files (trades, quotes, bursts, metrics) in chunks, with pause/resume and restart‑from‑checkpoint behavior so capture can run continuously without data loss.

- API‑aware enrichment and buffering:

EnricherPool and BufferManager coordinate concurrent calls to ThetaData‑style APIs with per‑API semaphores, symbol‑tiering, and hysteresis‑based “mode” logic (historical, near‑real‑time, real‑time) so you stay within rate limits while keeping hot symbols in memory.

- Exchange‑aware trading calendar:

TradingCalendar wraps pandas‑market‑calendars to give trading‑day checks, previous/next trading day, holiday lists, and accurate market close times (including early closes) in Eastern Time, which is crucial for EOD logic and scheduling.

- Iceberg‑integrated archiving:

Archiver buffers enriched Polars DataFrames into time windows and flushes them to Iceberg tables with event‑time‑aligned segment metadata, automatic schema evolution, and table routing based on filenames (trading, metrics, bursts, bulk snapshots).

- Metadata catalog for segments and tables:

A Catalog component tracks offsets (last processed row per source/date), segments (file path, time window, row ranges), and Iceberg tables/snapshots with atomic write patterns and diagnostics for data lake health.

- System‑wide telemetry and control:

MQTTController publishes ingestion progress and listens for control commands (pause, resume, shutdown), enabling external monitoring dashboards and coordinated operations, similar to large vendor platforms.

- Backfill and EOD enrichment:

Utilities perform backfill operations (for example, enriching historical windows with previous‑close data), delete and rewrite date ranges in Iceberg tables, and maintain closing‑price caches on disk for fast restarts.

Compared to many off‑the‑shelf data systems that are either “batch only” or “API only,” this design lets you run continuous tick ingestion, ad‑hoc backfills, and daily snapshot jobs against the same Iceberg‑backed catalog.

***

## What the metadata‑based analysis notebook adds

Metadata_based_Analysis.ipynb builds a generic factor‑analysis and visualization framework on top of the DCAS data lake.

Core components:

- FactorContext for data and metadata:

FactorContext manages base sources (raw LazyFrames from Iceberg), node outputs, and column‑level metadata (name, role, category, colors, units). It provides methods like get_input, get_output, and store_output so nodes can be wired together in a clean, testable way.

- BaseFactorNode with rich output descriptions:

Each factor node declares its output columns and associated ColumnMetadata (semantic role such as PRICE, BREADTH, OSCILLATOR, PNL; category; default color; line style), which later drive automatic visualization and reporting.

- BaseFactorDAG and ModularDAGExecutionEngine:

BaseFactorDAG defines a graph of nodes, computes a topological execution order, and provides an execute(context) method that runs nodes, stores outputs plus metadata, and keeps a DAG‑level visualization configuration (which roles or columns to show).

ModularDAGExecutionEngine orchestrates the full pipeline: retrieve data from Iceberg, prepare and align clocks, execute one or more DAGs, flatten outputs into a wide DataFrame, normalize values to 0–1 for plotting, compute statistics, and package everything for visualization and reporting.

- DAGMetadataAdapter and trace discovery:

DAGMetadataAdapter scans the metadata registry and output registry to discover which columns should be plotted as traces (for example, benchmark price, AD breadth, McClellan oscillator, smoothed derivatives, regimes, P\&L) and which columns form overlays (signals, regimes, arrow annotations).

This discovery is role‑aware: if you tag a column as ColumnRole.PRICE or ColumnRole.PNL, it will be treated differently than ColumnRole.METADATA without extra plotting code.

- Visualization‑ready DataFrames:

build_visualization_dataframe joins all node outputs on EventDateTime and produces a single DataFrame with consistent time alignment across breadth, momentum, regimes, signals, and P\&L, ready for BokehVisualizationEngine.

- Market breadth and SMA examples:

The notebook includes concrete DAGs such as MarketBreadthDAG / MarketBreadthFactorGraph (advance/decline counts, McClellan oscillator and summation, derivatives, trend classification, composite regimes, arrow‑based trade signals, and BreadthPnLNode) and a SimpleMovingAverageDAG to show how multiple DAGs can reuse the same prepared data.

To a user of other systems, this looks like a generalized factor graph engine over the DCAS data lake, with metadata‑driven visualization built in rather than bolted on.

***

## How the two notebooks work together

Together, DCAS.ipynb and Metadata_based_Analysis.ipynb form a cohesive platform.

- DCAS.ipynb handles capture, enrichment, buffering, and storage of multi‑asset high‑frequency data in Iceberg with strong operational guarantees.

- Metadata_based_Analysis.ipynb retrieves those time series, runs modular DAGs of factors and trading logic, and automatically turns outputs into rich, interactive visualizations.

A typical workflow:

1. DCAS CaptureEngine and BulkDataCollector ingest trades, quotes, bursts, and snapshots, archiving them via Archiver into tables like tradingdata, metricsdata, burstevents, and bulkohlcsnapshot in an Iceberg namespace.

2. A researcher instantiates ModularDAGExecutionEngine with an IcebergBridge and a Factor DAG (for example, MarketBreadthDAG), configures parameters (EMA alphas, KAMA window, regime thresholds), and calls load_initial_data or load_incremental_data for a chosen symbol universe and time window.

3. The engine prepares ETF and constituent series, executes the DAG, flattens and normalizes outputs, and hands a metadata‑rich visualization package to BokehVisualizationEngine for browser‑ready charts with regime backgrounds and annotated trades.

This separation of capture versus analysis means you can evolve your trading logic quickly while relying on a stable, production‑grade data backbone.

***

## Power and flexibility compared to other systems

For someone familiar with vendor terminals, basic HDF5/Parquet lakes, or retail backtesting platforms, several aspects stand out.

1. Storage‑agnostic, Iceberg‑first design

IcebergBridge and Archiver are written so the same code handles local directories, NAS, and S3‑compatible object storage; partitioning by EventDateTime enables high‑performance time‑range queries regardless of backend.

Segment metadata (event‑time start/end, partition hour, row counts) provides observability and precise filtering that many ad‑hoc Parquet setups lack.

2. DAG‑driven analytics instead of monolithic scripts

Factor computations are decomposed into reusable nodes with explicit inputs and outputs, making it easy to add or swap components (for example, plug a volatility node into an existing breadth DAG) without touching the rest of the pipeline.

Because execution order is computed from dependencies, you can extend the graph without worrying about manual ordering, which reduces maintenance cost as the library of factors grows.

3. Metadata‑driven visualization and reporting

ColumnMetadata and ColumnRole encode how each output should be interpreted; visualization layers discover traces and overlays automatically, so new factors immediately become visible with consistent styling.

Summary reports can be generated by role, category, or node (for example, “show me all oscillator‑type outputs across the DAG”), helping users understand complex factor sets more quickly than in systems where outputs are just columns in a table.

4. Real‑time‑aware enrichment and buffering

BufferManager’s hysteresis logic and per‑API semaphores are tuned for real production API constraints, not just offline backtesting. It adjusts fetching strategy as you move from historical replay to near‑real‑time monitoring without changing calling code.

5. End‑to‑end observability

MQTT telemetry, Iceberg diagnostics, segment catalogs, and latency percentiles provide a level of transparency that many in‑house scripts or GUI‑only analytics tools lack, making the system more supportable by operations and SRE teams.

***

## Summary

These notebooks implement a flexible, production‑oriented platform that unifies data engineering and factor analytics: DCAS handles the hard plumbing of high‑frequency capture and storage, while the metadata‑driven DAG engine lets quants express complex strategies and visual narratives on top of that data with relatively little boilerplate.