CamouflageCheck

Reproduce the numbers

Scripts, session-level data and statistical outputs are public at a fixed commit. Raw packet captures, trained detector files and credentials are not.

Get the code and data

The archived state is commit 639e9b4 of zilantix/praxis.

git clone https://github.com/zilantix/praxis.git
cd praxis
git checkout 639e9b4
pip install -r requirements.txt

Rerun the confirmation statistics

The analysis scripts were written for the lab machine and read inputs from /opt/praxis. Point the Phase 17 script at the public session file, then run it. The output matches the archived Phase 17 statistics.

# in artifacts/scripts/phase17_statistical_validation.py, set the input path to:
#   artifacts/data_csv/phase17_confirm_session_scores.csv
python artifacts/scripts/phase17_statistical_validation.py

Main data files

All in artifacts/data_csv/. One row is one session unless noted.

FileContents
phase17_confirm_session_scores.csv150 confirmation sessions: features, detector probability, benign distance, overhead, score
phase17_confirm_config_scores.csvPhase 17 means per configuration
phase20_holdout_*_session_scores.csvHoldout detector probabilities for the same sessions
phase20b_threshold_validation.csvCalibrated thresholds and detection rates (one row per threshold and configuration)
phase21_rq3_curl_catalog.csvResolution-path sessions
phase29_feature_delta_effect_sizes.csvFeature changes between configurations
phase29b_artifact_control_scores.csvArtifact-control detectors by feature set
phase30_multidetector_session_scores.csvFive detector families
phase31_mawi_cicids_rescored_sessions.csvSessions rescored against five benign references
mawi_public_flows_20240302_1m.csvMAWI day-2 benign features (one row per flow)

Public and restricted

Public

  • Analysis scripts
  • Session-level CSVs for each phase
  • Statistics tables and figures
  • MAWI day-2 benign features

Restricted

  • Raw packet captures, because of size
  • Trained detector files
  • Lab logs, credentials and the archive bucket
  • Full benign datasets, under their license terms

Benign references come from their original sources: MAWI Working Group traffic archive and CIC-IDS 2017.

How the workbench matches the study

The workbench reimplements the study’s feature extraction and score in JavaScript. Its benign reference is the median and interquartile range of the public MAWI day-2 file, embedded in the page. With the study’s 150 confirmation sessions it gives these configuration scores:

ConfigurationWorkbenchPhase 31, MAWI day 2Phase 17, MAWI day 1
D, default profile, multiplexing c20.43310.43310.4329
C, camouflaged profile, multiplexing c20.46280.46280.4626
B, default profile, multiplexing off0.65360.65360.6541

Phase 17 originally used MAWI day 1, which is not in the public repository, so the workbench uses day 2. The two references differ by less than 0.001 in every score.

The session CSV the workbench downloads uses the study’s column names (packet_count through len_20, pcap_size_bytes, public_robust_log_distance), so it can feed the repository’s scripts.