Oak Park Crime Data Docs

Data quality

What is actually wrong with this dataset, rather than which checks were written. Generated from the same run that built the docs.

Incidents
22,803
one row per incident in fct_incidents
Tests passing
52
definitions sit on every column in the docs
Tests warning
2
deliberate flags, not regressions
Rows flagged
11%
2,406 distinct rows carry a defect, almost all the report-date artifact

The report date is not the occurrence date

Share of incidents whose PDF report date lands a day later than the dashboard's exact timestamp, by the hour the incident actually happened. The later it gets, the more often the report rolls into the next day.

evening and overnight0%20%40%60% 00:00 · 39.4% offset · 106 of 26901:00 · 15.1% offset · 19 of 12602:00 · 9.1% offset · 10 of 11003:00 · 11.8% offset · 9 of 7604:00 · 14.4% offset · 13 of 9005:00 · 13.5% offset · 12 of 8906:00 · 21.4% offset · 22 of 10307:00 · 16.0% offset · 30 of 18708:00 · 20.7% offset · 56 of 27009:00 · 19.8% offset · 48 of 24310:00 · 18.1% offset · 52 of 28711:00 · 16.8% offset · 48 of 28612:00 · 32.5% offset · 141 of 43413:00 · 18.5% offset · 65 of 35214:00 · 25.2% offset · 91 of 36115:00 · 32.5% offset · 134 of 41216:00 · 35.4% offset · 140 of 39517:00 · 45.8% offset · 183 of 40018:00 · 48.8% offset · 206 of 42219:00 · 53.9% offset · 192 of 35620:00 · 56.9% offset · 193 of 33921:00 · 63.2% offset · 187 of 29622:00 · 64.0% offset · 183 of 28623:00 · 64.4% offset · 123 of 19164% at 23:0012% at 03:000003060912151821 hour the incident actually occurred

Known defect classes

Actual defects only, measured on every run. Two are deliberate flags rather than regressions. Correct-but-surprising characteristics are kept out of this table on purpose; see dataset characteristics and geocoding below.

pdf_dashboard_date_mismatchPDF and dashboard disagree on the incident date by a day or more · 2,301 rows (10.09%)2,301complaint_collisionsOne complaint number on materially different incidents · 69 rows (0.3%)69republished_pdf_rowsSame incident ingested twice under a revised PDF filename · 68 rows (0.3%)68ungeocodedNo coordinates resolved · 19 rows (0.08%)19date_sentinelDate failed to parse and fell back to 1900-01-01 · 5 rows (0.02%)5date_futureIncident date is in the future · 2 rows (0.01%)2
IssueWhat it meansRowsShareOwner
pdf_dashboard_date_mismatchPDF and dashboard disagree on the incident date by a day or more2,30110.09%flagged in merge_notes, not repaired
complaint_collisionsOne complaint number on materially different incidents690.3%source data, flag only
republished_pdf_rowsSame incident ingested twice under a revised PDF filename680.3%ingest bug, fix in drop_exact_source_duplicates
ungeocodedNo coordinates resolved190.08%run a Python cache-fill
date_sentinelDate failed to parse and fell back to 1900-01-0150.02%source data, flag only
date_futureIncident date is in the future20.01%source data, poisons naive freshness checks

Dataset characteristics

Correct but worth knowing. None of these is a defect, which is why they are not in the table above.

MeasureWhat it isRowsShare
geocodedIncidents with coordinates resolved22,78499.92%
in_villageGeocoded inside one of the 8 policing zone polygons20,04487.9%
pdf_onlyPresent in the PDF activity reports only, so no NIBRS coding or exact timestamp8,99439.44%
dashboard_onlyPresent in the Power BI export only, so no narrative text7,42932.58%
both_sourcesMatched across both ingest streams and merged6,38027.98%
zone_snappedAttached to a zone by the 55m nearest-zone fallback rather than strict containment1,6307.15%
outside_villageGeocoded outside the village. Correct: OPPD reports incidents in neighbouring towns1,1104.87%

Geocoding

99.92% of incidents have coordinates (22,784 of 22,803). Only 19 do not. The pipeline is cache-first, so a run resolves almost everything without calling the geocoding API.

The chart below is not a cache-miss rate. dbt can only join the cache on lower(trim(location)), while the pipeline keys on normalize_location(), which also strips punctuation and standardises street suffixes. So this measures the dbt-side key, not the pipeline: 15,649 of the unmatched rows are geocoded perfectly well. It is here to quantify what publishing a real location_key upstream would buy, nothing more.
unmatched · 15,665 (68.7%)69%matched · 7,135 (31.3%)31%cached_unresolvable · 3 (0.0%)unmatched · 15,665matched · 7,135cached_unresolvable · 3

Zone assignment

Snapped rows fell within 55m of a polygon and were attached by the nearest-zone fallback. Outside-village rows are correct, not errors.

in_village · 20,044 (87.9%)88%snapped · 1,630 (7.1%)outside_village · 1,110 (4.9%)ungeocoded · 19 (0.1%)in_village · 20,044snapped · 1,630outside_village · 1,110ungeocoded · 19

Test outcomes

The docs site renders test definitions but not results. These are the results.

CheckApplies toStatusFailures
singularassert_no_complaint_collisionswarn34
singularassert_no_republished_pdf_duplicateswarn68
singularassert_zone_assignment_paritypass
not_nullcrime_details.record_idpass
relationshipscrime_details.record_idpass
uniquecrime_details.record_idpass
accepted_valuescrimes.primary_crime_againstpass
accepted_valuescrimes.sourcepass
not_nullcrimes.datepass
not_nullcrimes.record_idpass
unique_combination_of_columnscrimespass
not_nulldata_profile_summary.measurepass
uniquedata_profile_summary.measurepass
not_nulldata_quality_summary.issuepass
not_nulldata_quality_summary.row_countpass
uniquedata_quality_summary.issuepass
not_nulldim_date.date_daypass
uniquedim_date.date_daypass
not_nulldim_offense.offense_textpass
uniquedim_offense.offense_textpass
accepted_valuesdim_zone.zone_idpass
not_nulldim_zone.zone_idpass
uniquedim_zone.zone_idpass
accepted_valuesfct_incidents.primary_crime_againstpass
accepted_valuesfct_incidents.sourcepass
accepted_valuesfct_incidents.zone_statuspass
not_nullfct_incidents.record_idpass
not_nullfct_incidents.zone_statuspass
uniquefct_incidents.record_idpass
unique_combination_of_columnsfct_incidentspass
accepted_valuesgeocode_coverage.naive_key_statuspass
not_nullgeocode_coverage.record_idpass
relationshipsgeocode_coverage.record_idpass
uniquegeocode_coverage.record_idpass
accepted_valuesint_complaint_quality.quality_classpass
not_nullint_complaint_quality.complaint_numpass
not_nullint_complaint_quality.row_countpass
uniqueint_complaint_quality.complaint_numpass
not_nullint_crimes_deduplicated.record_idpass
uniqueint_crimes_deduplicated.record_idpass
accepted_valuesint_crimes_zoned.zone_idpass
accepted_valuesint_crimes_zoned.zone_statuspass
not_nullint_crimes_zoned.record_idpass
not_nullint_crimes_zoned.zone_statuspass
uniqueint_crimes_zoned.record_idpass
not_nullstg_crime_details.record_idpass
uniquestg_crime_details.record_idpass
not_nullstg_crimes.record_idpass
uniquestg_crimes.record_idpass
not_nullstg_geocode_cache.location_keypass
uniquestg_geocode_cache.location_keypass
not_nullstg_policing_zones.zone_idpass
not_nullstg_policing_zones.zone_namepass
uniquestg_policing_zones.zone_idpass