arboretum_data

Data Cleanup Suggestions

Notes on data quality issues found while building this repo, kept here so the same checks can be reused as a template when publishing other datasets. Findings below are worked around client-side in the webmap (webmap/js/config.js); the source files in data/ are left untouched. Fixing them upstream, at the point of collection/export, would remove the need for that workaround entirely.

How these were found

Quick value-frequency checks against the raw GeoJSON, e.g.:

python3 -c "
import json, collections
d = json.load(open('data/geojson/TreeShrubInventory.geojson'))
c = collections.Counter(f['properties'].get('FIELD_NAME') for f in d['features'])
print(c.most_common(30))
"

Run this for every categorical/coded field before publishing a new dataset — it quickly surfaces mixed codes/words, inconsistent casing, and stray nulls.

Issues found in this dataset

1. Tree_Type mixes coded and spelled-out values

Meaning	Raw values	Count
Broadleaf	Angiosperm	2,943
Broadleaf	A	688
Conifer	Gymnosperm	653
Conifer	G	81
(blank)	null	42

Same category is represented two different ways depending on when/how the record was entered. Likely cause: a domain/coded-value list was added or changed partway through data collection without back-filling older records.

Recommendation: standardize to one representation (e.g. always the full word) at the source, or add a coded-value domain in the collection app so this can’t drift again.

2. CONDITION mixes coded and spelled-out values, plus two unexplained codes

Meaning	Raw values	Count
Excellent	Excellent / E	28 / 109
Very Good	Very Good / VG	473 / 673
Good	Good / G	737 / 1,638
Fair	Fair / F	224 / 308
Poor	Poor / P	28 / 77
Dead	Dead / D	10 / 63
Unknown	C	5
Unknown	R	1
(blank)	null	33

Same root cause as #1. Additionally, C and R don’t map to any obvious rating — possibly “Critical”/”Compromised” and “Removed”/”Residual”? These need a definitive answer from whoever maintains the source system before they can be reclassified.

Recommendation: same fix as #1, and confirm the meaning of C/R with AU Geospatial staff; document the full code list in docs/DATA_DICTIONARY.md once known.

General checklist for future datasets

When onboarding a new dataset into this kind of repo, check for:

• Mixed codes/words for the same category (run the frequency-count snippet above on every text field with a small number of distinct values).
• Inconsistent casing or whitespace ("Good" vs "good " vs "GOOD").
• Undocumented codes — every abbreviation should have a definition in the data dictionary; flag any that don’t.
• Null vs. empty string vs. placeholder text (e.g. "N/A", "None", "-") used interchangeably for missing data.
• Duplicate records (same location/ID appearing more than once).
• Coordinate sanity — points falling outside the expected campus/study area bounding box.
• Field type consistency — numeric fields stored as strings, or vice versa.
• Date format consistency across fields that record dates/timestamps.

Log anything found in a file like this one before publishing, and note in docs/METHODOLOGY.md whether the fix was applied upstream or worked around downstream (and where).

This site is open source. Improve this page.