Two kinds of data: real (NYC TLC yellow-taxi trips, January 2022) and generated (orders, windows, demands, service times, vans; seeded). Generated data never touches the trip files; real data never contains any order information.
1. Raw source files (real, data/raw/, gitignored)
| File | Source | Size |
|---|---|---|
yellow_tripdata_2022-01.parquet |
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-01.parquet | 38.1 MB, 2,463,931 rows |
taxi_zone_lookup.csv |
https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv | 265 zones |
taxi_zones.zip |
https://d37ci6vzurychx.cloudfront.net/misc/taxi_zones.zip | shapefile, EPSG:2263 (feet) |
Columns read from the parquet (the script fails loudly if any is missing; all five exist in 2022-01): tpep_pickup_datetime, tpep_dropoff_datetime, PULocationID, DOLocationID, trip_distance.
2. Filtering (make_data.py -> delivery/data.py::preprocess)
| Step | Rows left |
|---|---|
| raw | 2,463,931 |
| pickup inside January 2022 (drops mis-dated records) | 2,463,879 |
| weekdays only (Mon-Fri) | 1,754,221 |
| duration 1-120 min | 1,731,617 |
| distance > 0 and speed <= 60 mph (outliers) | 1,722,324 |
| both ends in the 20 busiest zones (pickups + drop-offs; unknown zones 264/265 excluded) | 670,489 |
| seeded uniform sample (seed 42) | 300,000 |
Split by day: train Jan 3-19 (173,350), validation Jan 20-21 (33,058, early stopping only), test Jan 24-31 (93,592). Jan 1-2, 8-9, ... are weekends and are removed. Jan 17 (MLK Day) is kept.
The 20 zones (index 0-19): Clinton East, East Chelsea, Gramercy, JFK Airport, Lenox Hill East, Lenox Hill West, Lincoln Square East, Midtown Center, Midtown East, Midtown North, Murray Hill, Penn Station/Madison Sq West, Sutton Place/Turtle Bay North, Times Sq/Theatre District, Union Sq, Upper East Side North, Upper East Side South, Upper West Side North, Upper West Side South, Yorkville West. 19 are in Manhattan; JFK Airport is the only far zone.
3. Processed trips (real, data/processed/trips.parquet; sample in tests/fixtures/sample_trips.csv)
| Column | Type | Meaning |
|---|---|---|
pickup |
datetime | pickup timestamp (not in the fixture) |
date |
str | pickup date YYYY-MM-DD (used for the day split) |
pu_id, do_id |
int | original TLC LocationID of pickup / drop-off |
pu, do |
int 0-19 | zone index (sorted LocationID order) |
hour |
int 0-23 | pickup hour |
weekday |
int 0-4 | Mon=0 ... Fri=4 |
distance_mi |
float | reported trip distance (miles); used ONLY to build the pair mean distance |
duration_min |
float | target: drop-off minus pickup, minutes, in [1, 120] |
split |
str | train / val / test |
4. Zones and pair distance (real-derived)
zones.csv: idx, LocationID, borough, zone, service_zone, cx_km, cy_km (area-weighted polygon centroid from the shapefile, km in NY State Plane), n_pickups, n_dropoffs (in the 300k sample).
pair_distance.npy (20x20, miles): mean distance_mi per (pu, do) over the train split only. All 400 pairs were observed in train; the fallback (linear fit on centroid distance: -0.176 + 1.428 x centroid miles) exists for other months but was not needed. Sparsity: 8,406 of 9,600 (pu, do, hour) cells appear in train, 2,026 of them with < 5 trips (mostly night hours).
5. NN inputs (derived)
pickup zone one-hot (20), drop-off zone one-hot (20), hour one-hot (24) + sin/cos(hour), weekday one-hot (5), pair mean distance standardized + log1p standardized. 73 features. The actual trip distance is never an input (it is unknown when planning).
6. Predicted time matrices (models/time_matrices.npz)
Keys nn, hist, dist, each shaped (5 weekdays, 24 hours, 20, 20): predicted taxi minutes from zone i to zone j departing in that hour. CSV of the NN matrix for Tuesday: outputs/time_matrix_nn_tuesday.csv (hour, pu_idx, do_idx, pu_zone, do_zone, minutes). Van time = taxi time x 1.2 (documented assumption VAN_FACTOR; vans are slower, park, etc.).
7. Generated orders and vans (delivery/instances.py, instances/*.json)
Generator generate_instance(n_orders, seed, ...) with numpy.random.default_rng(seed):
| Field | How it is generated |
|---|---|
depot |
zone index; random unless given (demo uses 7 = Midtown Center) |
weekday |
uniform 0-4 (selects which weekday's matrix is used) |
order id |
0..n-1 |
order zone |
drawn from the 20 zones with probability proportional to real drop-off counts |
earliest |
uniform in [09:00, 17:00 - width], rounded to 15 min (minutes after midnight) |
latest |
earliest + width, width uniform from {30, 45, 60, 90, 120} (standard) or a narrower set (tight/rush scenarios: {15, 30} or {15, 30, 45}; demo {15, 30, 45, 60}) |
demand |
1, 2 or 3 units with probability 0.5 / 0.3 / 0.2 |
service |
5, 7 or 10 minutes (service-time variation extension) |
van capacity |
ceil(total demand x cap_slack / 3), cap_slack 1.25 by default, 1.0 in the CSP experiment |
van shift_start/shift_end |
08:00-17:00, 09:00-18:00, 10:00-19:00 (van 0, 1, 2) |
van max_stops |
10 (spec: up to 10 stops per van); experiment 2 uses min(10, ceil(n/3)+1) |
return_to_depot |
true: routes end back at the depot (return-trip extension) |
generate_route_instance(n_stops, seed, tm) (experiment 1, one van): builds a hidden random tour with the time model and puts each window (60/90/120 min) around the hidden arrival time, so the instance is feasible by construction; ids are shuffled afterwards.
The instance set: instances/n{15,20,30}_s{0..9}.json (+ manifest.json).