# Data dictionary

Two kinds of data: **real** (NYC TLC yellow-taxi trips, January 2022) and **generated**
(orders, windows, demands, service times, vans; seeded). Generated data never touches the
trip files; real data never contains any order information.

## 1. Raw source files (real, `data/raw/`, gitignored)

| File | Source | Size |
|---|---|---|
| `yellow_tripdata_2022-01.parquet` | https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-01.parquet | 38.1 MB, 2,463,931 rows |
| `taxi_zone_lookup.csv` | https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv | 265 zones |
| `taxi_zones.zip` | https://d37ci6vzurychx.cloudfront.net/misc/taxi_zones.zip | shapefile, EPSG:2263 (feet) |

Columns read from the parquet (the script fails loudly if any is missing; all five exist in 2022-01):
`tpep_pickup_datetime`, `tpep_dropoff_datetime`, `PULocationID`, `DOLocationID`, `trip_distance`.

## 2. Filtering (`make_data.py` -> `delivery/data.py::preprocess`)

| Step | Rows left |
|---|---|
| raw | 2,463,931 |
| pickup inside January 2022 (drops mis-dated records) | 2,463,879 |
| weekdays only (Mon-Fri) | 1,754,221 |
| duration 1-120 min | 1,731,617 |
| distance > 0 and speed <= 60 mph (outliers) | 1,722,324 |
| both ends in the 20 busiest zones (pickups + drop-offs; unknown zones 264/265 excluded) | 670,489 |
| seeded uniform sample (seed 42) | **300,000** |

Split by day: train Jan 3-19 (173,350), validation Jan 20-21 (33,058, early stopping only),
test Jan 24-31 (93,592). Jan 1-2, 8-9, ... are weekends and are removed. Jan 17 (MLK Day) is kept.

The 20 zones (index 0-19): Clinton East, East Chelsea, Gramercy, JFK Airport, Lenox Hill East,
Lenox Hill West, Lincoln Square East, Midtown Center, Midtown East, Midtown North, Murray Hill,
Penn Station/Madison Sq West, Sutton Place/Turtle Bay North, Times Sq/Theatre District, Union Sq,
Upper East Side North, Upper East Side South, Upper West Side North, Upper West Side South,
Yorkville West. 19 are in Manhattan; JFK Airport is the only far zone.

## 3. Processed trips (real, `data/processed/trips.parquet`; sample in `tests/fixtures/sample_trips.csv`)

| Column | Type | Meaning |
|---|---|---|
| `pickup` | datetime | pickup timestamp (not in the fixture) |
| `date` | str | pickup date `YYYY-MM-DD` (used for the day split) |
| `pu_id`, `do_id` | int | original TLC LocationID of pickup / drop-off |
| `pu`, `do` | int 0-19 | zone index (sorted LocationID order) |
| `hour` | int 0-23 | pickup hour |
| `weekday` | int 0-4 | Mon=0 ... Fri=4 |
| `distance_mi` | float | reported trip distance (miles); used ONLY to build the pair mean distance |
| `duration_min` | float | **target**: drop-off minus pickup, minutes, in [1, 120] |
| `split` | str | `train` / `val` / `test` |

## 4. Zones and pair distance (real-derived)

`zones.csv`: `idx`, `LocationID`, `borough`, `zone`, `service_zone`, `cx_km`, `cy_km`
(area-weighted polygon centroid from the shapefile, km in NY State Plane), `n_pickups`,
`n_dropoffs` (in the 300k sample).

`pair_distance.npy` (20x20, miles): mean `distance_mi` per (pu, do) over the **train split only**.
All 400 pairs were observed in train; the fallback (linear fit on centroid distance:
`-0.176 + 1.428 x centroid miles`) exists for other months but was not needed.
Sparsity: 8,406 of 9,600 (pu, do, hour) cells appear in train, 2,026 of them with < 5 trips
(mostly night hours).

## 5. NN inputs (derived)

pickup zone one-hot (20), drop-off zone one-hot (20), hour one-hot (24) + sin/cos(hour),
weekday one-hot (5), pair mean distance standardized + log1p standardized. 73 features.
The actual trip distance is never an input (it is unknown when planning).

## 6. Predicted time matrices (`models/time_matrices.npz`)

Keys `nn`, `hist`, `dist`, each shaped (5 weekdays, 24 hours, 20, 20): predicted **taxi**
minutes from zone i to zone j departing in that hour. CSV of the NN matrix for Tuesday:
`outputs/time_matrix_nn_tuesday.csv` (`hour, pu_idx, do_idx, pu_zone, do_zone, minutes`).
Van time = taxi time x **1.2** (documented assumption `VAN_FACTOR`; vans are slower, park, etc.).

## 7. Generated orders and vans (`delivery/instances.py`, `instances/*.json`)

Generator `generate_instance(n_orders, seed, ...)` with `numpy.random.default_rng(seed)`:

| Field | How it is generated |
|---|---|
| `depot` | zone index; random unless given (demo uses 7 = Midtown Center) |
| `weekday` | uniform 0-4 (selects which weekday's matrix is used) |
| order `id` | 0..n-1 |
| order `zone` | drawn from the 20 zones with probability proportional to real drop-off counts |
| `earliest` | uniform in [09:00, 17:00 - width], rounded to 15 min (minutes after midnight) |
| `latest` | `earliest + width`, width uniform from {30, 45, 60, 90, 120} (standard) or a narrower set (tight/rush scenarios: {15, 30} or {15, 30, 45}; demo {15, 30, 45, 60}) |
| `demand` | 1, 2 or 3 units with probability 0.5 / 0.3 / 0.2 |
| `service` | 5, 7 or 10 minutes (service-time variation extension) |
| van `capacity` | `ceil(total demand x cap_slack / 3)`, `cap_slack` 1.25 by default, 1.0 in the CSP experiment |
| van `shift_start`/`shift_end` | 08:00-17:00, 09:00-18:00, 10:00-19:00 (van 0, 1, 2) |
| van `max_stops` | 10 (spec: up to 10 stops per van); experiment 2 uses min(10, ceil(n/3)+1) |
| `return_to_depot` | true: routes end back at the depot (return-trip extension) |

`generate_route_instance(n_stops, seed, tm)` (experiment 1, one van): builds a hidden random tour
with the time model and puts each window (60/90/120 min) around the hidden arrival time, so the
instance is feasible by construction; ids are shuffled afterwards.

The instance set: `instances/n{15,20,30}_s{0..9}.json` (+ `manifest.json`).