# DEA Denmark
This repo replicates the exhibits in the paper "Disaggregated Economic Accounts" in the Quarterly Journal of Economics. Visualizations of the data are at https://www.disaggregatedaccounts.com/denmark .

## Software Requirements

### Operating System
- **macOS**: Tested on macOS (Intel and Apple Silicon)
- **Windows**: Compatible via Git Bash, WSL, or native PowerShell (see setup notes)
- **Linux**: Fully supported

### Python
- **Python 3.11 or higher** (code was last run with Python 3.11.14)
- The `setup.py` file enforces this minimum version requirement

### Python Dependencies
All Python package dependencies are listed in `requirements.txt` with specific version constraints:

**Installation:** Run `pip install -r requirements.txt` or use the provided `setup.sh` script which handles both dependency installation and directory setup.

### Git LFS (Large File Support)
- **Required for data files**: The replication package uses Git LFS to store the large compressed data file (`flows.csv.gz`, ~308 MB)
- **Installation**:
  - macOS: `brew install git-lfs`
  - Windows: `choco install git-lfs` or download from https://git-lfs.github.com/
  - Linux: `sudo apt-get install git-lfs` (Debian/Ubuntu) or check your distribution's package manager
- **Setup**: After installing, run `git lfs install` and `git lfs pull --include="data/replication/flows.csv.gz"`

### LaTeX (Optional, for standalone PDFs)
- **LaTeX distribution**: Required only if you want to compile standalone table PDFs
- **Installation**: https://tug.org
- **DVI-PNG converter** (optional): https://www.ctan.org/pkg/dvipng

### Setup Script
The replication package contains `setup.sh` which:
- Installs all Python dependencies via `pip install -e .`
- Creates required output directory structure
- This script should be run once before the first execution

## Memory, Runtime, and Storage Requirements

### Summary

**Approximate time needed to reproduce the analyses on a standard 2026 desktop machine:** 
- Full run (generates multipliers on first run): 2-8 hours (~6.7 hours).
- Subsequent runs (reuse cached multipliers): 10-60 minutes (~25 minutes) 

**Approximate storage space needed:** 25 GB - 250 GB (approximately 31 GB peak usage)

### Details

The code was last run on a **M2 MacBook Pro with macOS 14.x** (2024), **16 GB RAM**, and **50 GB of free storage space**.

**Storage breakdown:**
- Initial repository clone: ~19 GB (includes Git LFS pointers and existing data)
- Decompressed data file (`flows.csv`): 3.4 GB (from 294 MB compressed `flows.csv.gz`)
- Intermediate files generated during execution: ~12 GB (`data/intermediate/`)
- Network files for Gephi: ~48 MB (`data/gephi/`)
- Final output files: ~33 MB (`python25/`)
- **Peak storage usage: ~31 GB**

**Runtime scenarios:**

The replication package supports two execution modes based on whether cached multipliers are already present:

**1. Full run with multipliers** (first run when caches are missing):
- Step 0 (Clean): < 1 second
- Step 1 (Circular Flow generation): 15m 20s
- Step 2 (Facts generation): 381m 0s (6.35 hours)
- Step 3 (Output verification): < 1 second
- Step 4 (Copy outputs): < 1 second
- Step 5 (Tableau format): 4m 4s
- **Total runtime: 400m 24s (6.67 hours)**
- Generates all core tables and figures and generates the multipliers

**2. Short run** (reuses cached multipliers from `data/intermediate/`):
- Step 0 (Clean): < 1 second
- Step 1 (Circular Flow generation): 15m 29s
- Step 2 (Facts generation): 5m 42s
- Step 3 (Output verification): < 1 second
- Step 4 (Copy outputs): < 1 second
- Step 5 (Tableau format): 4m 3s
- **Total runtime: 25m 15s**
- Generates all core tables and figures, skipping computationally intensive multiplier calculations

The `run_all.py` script generates a `run_all_timing.json` file with detailed timing for each step.

## Installation Instructions

1. Clone the repository
    - `git clone https://github.com/kilianhuber/dea-denmark.git`

2. Create a virtual environment (recommended)
    - `python3.11 -m venv dea_venv`
    - Activate: `source dea_venv/bin/activate` (macOS/Linux) or `dea_venv\Scripts\activate` (Windows)

3. Install dependencies and set up folders
    - `cd dea-denmark`
    - `bash setup.sh`

4. Install Git LFS and fetch data files
    - `git lfs install`
    - `git lfs pull --include="data/replication/flows.csv.gz"`

5. Run the full pipeline
    - `python run_all.py`
    - The first run will generate multiplier caches. Subsequent runs automatically reuse them and take much less time and resources.

When you're done and you wish to leave the environment, run `deactivate`.

**Note on In-Person Spending**: We do not replicate offline/in-person spending analysis due to privacy restrictions on the underlying transaction-level data.

## Dataset List

All data files are located in `data/replication/` and are provided in CSV or JSON formats for compatibility with open-source software.

| Data file | Source | Notes | Provided |
|-----------|--------|-------|----------|
| `flows.csv.gz` | Processed aggregate transaction flows | 308 MB compressed, 3.4 GB uncompressed. Contains all monetary flows between economic sectors. Git LFS required. | Yes |
| `flows.csv` |  | Decompressed version of flows.csv.gz, automatically created by replication scripts | Generated |
| `sector_Consumers.csv` |  | Consumer sector attributes including demographics, income, wealth, geographic characteristics (98 variables) | Yes |
| `sector_Producers.csv` |  | Producer sector attributes including industry classifications, regional characteristics | Yes |
| `sector_Government.csv` |  | Government sector attributes | Yes |
| `sector_Finance.csv` |  | Finance/capital accumulation sector attributes | Yes |
| `sector_ROW.csv` |  | Rest of World sector attributes for all imports | Yes |
| `sector_ROW_All_imports.csv` |  | Detailed import flow data | Yes |
| `sector_ROW_Imports_from_households.csv` |  | Consumer import data | Yes |
| `cf_config.json` |  | Circular flow configuration: sector definitions, flow structure | Yes |
| `demand_supply_config.json` |  | Demand system and supply function specifications | Yes |
| `df_mpcs.csv` |  | Marginal propensities to consume by consumer cell | Yes |
| `exports_imports_by_country.csv` |  | Trade flows disaggregated by country | Yes |
| `policy_shocks.csv` |  | Policy experiment shock definitions | Yes |
| `sector_names.csv` |  | Sector name mappings | Yes |
| `region_names.csv` |  | Geographic region name mappings | Yes |
| `flow_names.csv` |  | Flow type name mappings | Yes |
| `reference_flows_table.csv` |  | Reference table for validating aggregate flows | Yes |
| `reference_row_account.json` |  | Reference data for rest of world account validation | Yes |
| `validation_checksums.json` |  | Checksums and metadata for data validation | Yes |

**Data Format:** All CSV files use standard comma-separated format readable by pandas, R, and other open-source tools. JSON files follow standard JSON specification.

**Codebook:** Variable names in sector CSV files are descriptive (e.g., `cust_age`, `cust_laborinc`, `cust_educ_college`). Full variable documentation is embedded in column names where possible.

**Confidential Data:** The replication package does not include raw transaction-level microdata from Danske Bank or other confidential sources. All provided data files contain aggregated, processed data suitable for public distribution.

## Description of Programs/Code

### Overview

The replication package is organized into several directories, each serving a specific purpose in the data processing and analysis pipeline. The main entry point is `run_all.py`, which orchestrates the entire workflow.

### Main Script

- **`run_all.py`**: Master script that executes the complete replication pipeline in the correct order. It performs cleaning, runs all circular flow and facts generation scripts, verifies outputs, and copies final results to `python25/`. The script generates `run_all_timing.json` with detailed timing information for each step.

### Code Organization

**`flows/`**: Circular flow construction scripts that process raw data and build the economic network structure.
- `get_circular_flow_replication.py`: Main circular flow construction using replication data (decompresses and loads `flows.csv.gz`)
- `get_circular_flow_no_finance.py`: Circular flow excluding financial sector
- `get_circular_flow_industry_regional.py`: Industry-regional level circular flow
- `get_circular_flow_industry_regional_full.py`: Full industry-regional circular flow with additional detail
- `get_circular_flow_local.py`: Local (municipal) level circular flow
- `get_circular_flow_gravity.py`: Circular flow with gravity model structure
- `get_circular_flow_nofacts.py`: Circular flow without empirical facts overlays
- `get_circular_flow_openness.py`: Circular flow with trade openness measures

**`facts/`**: Analysis scripts that generate all tables and figures for the paper.
- Each script produces specific outputs (tables, figures, or both) referenced in the manuscript
- Scripts use the circular flow objects created by the `flows/` scripts
- Examples include production facts, gravity relationships, domestic spending patterns, assortativity measures, multiplier calculations, and dynamic responses

**`denmarkmacro/`**: Core module containing helper functions and utilities used throughout the analysis.
- Data processing functions (circular flow construction, demand systems)
- Plotting utilities (network plots, maps, binscatter)
- Statistical tools (gravity models, centrality measures, regression tables)
- LaTeX table formatting

**`gephi/`**: Scripts to export network data in formats compatible with Gephi visualization software.
  - `gephi/save_network_gexf.py` produces a `.gexf` network file for Gephi.
  - Gephi generates the layout positions file (e.g., `data/gephi/positions oct.graphml`).
  - `facts/main_plot.py` reads the positions file to render Figure II (`output/figure/main_plot_white.png`).

**Supporting files:**
- `setup.py`: Package installation configuration with Python version requirements
- `requirements.txt`: All Python dependencies with version constraints
- `setup.sh`: Automated setup script for dependencies and directory structure
- `test_outputs.py`: Output verification script that checks for required files
- `format_for_tableau.py`: Converts circular flow data to Tableau-compatible CSV format

### Execution Flow

1. **Setup** (Step 0): Cleans output directories and creates required folder structure
2. **Circular Flow Generation** (Step 1): Runs all scripts in `flows/` to construct economic network data structures
3. **Facts Generation** (Step 2): Runs all scripts in `facts/` to produce tables and figures
4. **Verification** (Step 3): Checks that all expected output files were created
5. **Output Copying** (Step 4): Copies final outputs to `python25/` directory
6. **Tableau Export** (Step 5): Formats data for Tableau visualization

All output files (tables, figures, intermediate data) are generated in the `output/` directory and its subdirectories, then copied to `python25/` for final delivery.

## List of Tables and Figures

### Main Tables

| Table | Description | Python Source | Output File(s) |
|-------|-------------|---------------|----------------|
| Table I | All flows in the Danish disaggregated economic accounts | flows/get_circular_flow_replication.py (lines 1529-1531) | flows in DK.tex |
| Table II | Consumer spending gravity regressions | facts/new_facts_gravity.py | stata_gravity_interact.tex |
| Table III | Multipliers for fiscal transfer programs | facts/app_static_multipliers_policies.py (line 149) | policy_experiments.tex |
| Table IV | Multipliers after U.S. tariff shock: Role of spending patterns | facts/app_us_exportshock.py (lines 243-262) | multiplier_us_regression_specs.tex |

### Appendix Tables

| Table | Description | Python Source | Output File(s) |
|-------|-------------|---------------|----------------|
| Table A.III | Determinants of domestic spending shares | facts/new_facts_dom_spending.py (line 113) | stata_domestic.tex |
| Table A.IV | Consumer spending gravity regression for inflows into urban regions | facts/new_facts_gravity.py (lines 197, 213) | gravity_density_onlydestination.tex |
| Table A.V | External trade surplus across regions by region size | facts/new_facts_production.py (lines 156-203) | absolute_external_surplus.tex |
| Table A.VI | Internal trade surplus across regions by region size | facts/new_facts_production.py (lines 156-203) | absolute_internal_surplus.tex |
| Table A.VIII | Multipliers after U.S. tariff shock: Association with cell characteristics | facts/app_us_exportshock.py | multiplier_us_regression_determinants.tex |
| Table A.IX | Aggregate consumer account | flows/get_circular_flow_replication.py (line 1327: CF.show_accounts()) | Account - Consumers.txt |
| Table A.X | Aggregate producer account | flows/get_circular_flow_replication.py (line 1327: CF.show_accounts()) | Account - Producers.txt |
| Table A.XI | Aggregate government account | flows/get_circular_flow_replication.py (line 1327: CF.show_accounts()) | Account - Government.txt |
| Table A.XII | Aggregate rest of the world account | flows/get_circular_flow_replication.py (line 1327: CF.show_accounts()) | Account - ROW: All imports.txt<br>Account - ROW: Imports from households.txt |
| Table A.XIII | Aggregate capital accumulation account | flows/get_circular_flow_replication.py (line 1327: CF.show_accounts()) | Account - Finance.txt |

### Main Figures

| Figure | Description | Python Source | Output File(s) |
|--------|-------------|---------------|----------------|
| Figure II | The disaggregated circular flow of money | facts/main_plot.py (uses `data/gephi/positions oct.graphml`) | main_plot_white.png |
| Figure III | Domestic spending share and consumer cell characteristics | facts/new_facts_dom_spending.py (lines 20-38) | direct_domestic_spending_logpop.pdf<br>direct_domestic_spending_age.pdf<br>direct_domestic_spending_college.pdf<br>for_spending_industry_shares.pdf |
| Figure IV | Examples of spending shares across regions | facts/new_facts_dom_spending.py (lines 33-45) | map domestic spending share *.pdf (multiple regions: Billund, Copenhagen) |
| Figure V | Gravity equations | (a) facts/new_facts_gravity.py (lines 32-41, 73-94) | (a) combined_gravity_no_norm.pdf<br>(b) gravity_by_industry Domestic spending.pdf |
| Figure VI | Assortative consumer spending | facts/new_facts_assortativity.py (lines 26-35) | assortativity_income.pdf<br>assortativity_age.pdf<br>assortativity_education.pdf |
| Figure VII | Population of home region and region receiving consumer spending | facts/new_facts_dom_spending.py (lines 183-203) | average_denstity_spending.pdf<br>average_density_spending_many_categories.pdf |
| Figure VIII | Regional balance of payments | facts/new_facts_production.py (lines 79-114) | BOP_*.pdf (2 files) |
| Figure X | Domestic spending intensity and consumer cell characteristics | facts/new_facts_kilian_centrality.py (lines 21-38) | dom_spending_intensity_*.pdf (3 files) |
| Figure XI | Targeted transfer multipliers | (a) facts/app_static_multipliers.py (lines 48-55)<br>(b) facts/app_static_multipliers.py (lines 61-65) | (a) distribution transfer multipliers.pdf<br>(b) multipliers vs centrality.pdf |
| Figure XII | Direct exposure, slackness, and multipliers for the U.S. tariff shock | facts/app_us_exportshock.py (lines 85-91, 124-129, 149-152) | (a) direct_exposure_us_exportshock.pdf<br>(b) slack_us_exportshock.pdf<br>(c) multiplier_us_exportshock.pdf |
| Figure XIII | Dynamic multipliers | facts/dynamics_individual_scatter_plots.py (lines 105-124) | mpc_dk dynamic_cum_mult vs Static multiplier.pdf<br>mpc_dk dynamic_cum_mult vs Domestic spending intensity.pdf<br>mpc_dk dynamic_cum_mult vs MPC.pdf |

### Appendix Figures

| Figure | Description | Python Source | Output File(s) |
|--------|-------------|---------------|----------------|
| Figure A.III | Labor income shares and consumer spending shares across regions | facts/new_facts_dom_spending.py (Lines 518-606) | labor_vs_spending_shares \*.pdf (\* is the industry name) |
| Figure A.VI | Domestic spending share by log density | facts/new_facts_dom_spending.py (lines 42-44) | direct_domestic_spending_density.pdf |
| Figure A.VII | Population of home and receiving region: all spending | facts/new_facts_dom_spending.py (lines 206-224) | pop_slopes_by_industry.pdf |
| Figure A.IX | Internal trade surpluses and population size by industry | (a) facts/new_facts_dom_spending.py (lines 267-299)<br>(b) facts/new_facts_production.py (lines 119-153) | (a) good_TB_by_industry.pdf<br>(b) good_TB_by_industry_IO.pdf |
| Figure A.X | Distribution of domestic spending intensity | facts/new_facts_kilian_centrality.py (line 17) | distribution domestic spending intensity.pdf |
| Figure A.XI | Match between model economy and disaggregated accounts | facts/app_steadystate_comparison.py (entire file) | scatter ss output Comparison only CFnf.pdf<br>scatter ss spending Comparison only CFnf.pdf |
| Figure A.XII | Multipliers and consumer characteristics | facts/app_static_multipliers.py (lines 90-111) | multipliers vs Log population.pdf<br>multipliers vs Age.pdf<br>multipliers vs College share.pdf<br>multipliers vs Log density.pdf |
| Figure A.XIV | Distribution of multipliers after U.S. tariff shock | facts/app_us_exportshock.py (lines 176-185) | distribution us export shock mult slack.pdf<br>distribution us export shock mult not slack.pdf |
| Figure A.XV | Dynamic multipliers by consumer cell characteristics | facts/dynamics_individual_scatter_plots.py (lines 105-124) | mpc_* dynamic_cum_mult vs +.pdf<br>(* = dk, france, usa)<br>(+ = Log population, Age, Share college educated, Static multiplier, Domestic spending intensity, MPC) |
| Figure A.XVI | Dynamic multipliers using MPCs from Lewis et al. (2025) | facts/dynamics_individual_scatter_plots.py (lines 105-124) | mpc_* dynamic_cum_mult vs +.pdf |
| Figure A.XVII | Dynamic multipliers using MPCs from Boehm et al. (2025) | facts/dynamics_individual_scatter_plots.py (lines 105-124) | mpc_* dynamic_cum_mult vs +.pdf |
| Figure A.XVIII | Multipliers under counterfactual accounts | facts/facts_dom_spending_CF_nofacts.py (lines 60-71) | distribution Transfer multipliers nofacts.pdf<br>nofacts_multipliers_Domestic spending intensity.pdf |
| Figure A.XIX | Multipliers under counterfactual accounts and consumer characteristics | facts/facts_dom_spending_CF_nofacts.py (lines 60-65) | nofacts_multipliers_*.pdf<br>(* = Log population, Age, College share) |
| Figure A.XX | Multipliers under simplified accounts | facts/app_static_multipliers.py (lines 123-151) | multipliers vs local consumption DEA multiplier.pdf<br>multipliers vs local consumption Domestic spending intensity.pdf |
| Figure A.XXI | Distribution of multipliers under simplified accounts | facts/app_static_multipliers.py (lines 153-157) | distribution Transfer multipliers local consumption.pdf |
| Figure A.XXII | Multipliers in less open economies | facts/app_multiplier_by_openness.py (lines 79-116) | multiplier openness centrality static.pdf<br>multiplier openness centrality dynamic.pdf |
| Figure A.XXIII | Multipliers using alternative elasticities | facts/app_static_mult_elast.py (lines 81-120) | multiplier elasticities Cobb-Douglas multipliers.pdf<br>multiplier elasticities Domestic spending intensity.pdf |

Note: Any figure / table from the paper not produced by the replication package is not listed here. 

**Run All Consolidation**
- **Overview:** The `run_all.py` script prepares, runs, verifies, and exports outputs for the paper.

- **Cleaning:** `run_all.py` now runs a non-interactive cleaning step at startup that removes files under `dea-denmark/output/` so each run starts from a clean slate.

- **Execution:** The script then runs the circular-flow and facts generation scripts. We do not replicate the in person spending due to privacy restrictions.

- **Verification:** `run_all.py` calls the output checker. If required outputs are missing the run stops and no copy occurs.

- **Copying:** If verification passes, `run_all.py` copies generated `.pdf`/`.tex` outputs into `paper_output/Tables and Figures/python25/`. A `lyx_copy_report.json` is produced. A lyx copy of the paper placed in the same directory in the `paper_output/` folder will pick up the outputs produced and render the paper.

When you're done and you wish to leave the environment, run `deactivate`.

## Verification and maintenance scripts

After running `python run_all.py`, use the following scripts to verify artifacts, keep the workspace clean, and prepare outputs for paper compilation.