[EPA_USAirPollutantEmissionTrends_NationalAndState] Updated config.json for latest updated data

[Krishnam24maheshwari]

Summary

Fixes the EPA_USAirPollutantEmissionTrends_NationalAndState import (b/574320356) to support the U.S. EPA's September 2026 (1970–2025 Air Pollutant Emissions Trends Data) release (national_tier1_caps_04sep2026.xlsx and state_tier1_1990_2025_04sep2026_ktons.xlsx), hardens the download and processing scripts against upstream URL/layout changes, and adds import-specific validation rules.

Root Cause

1. Download Failure (HTTP 404 Not Found):
   • After September 1, 2026, the U.S. EPA published its new 1970–2025 Air Pollutant Emissions Trends Data workbooks (04sep2026) and retired the previous February 2025 URLs (2025-04/national_tier1_caps_21feb2025.xlsx and 2025-04/state_tier1_21feb2005_ktons.xlsx), causing download.py to fail with HTTP Error 404: Not Found.
2. State Workbook Header Row Shift (skiprows=0 vs. skiprows=1):
   • In the new state_tier1_1990_2025_04sep2026_ktons.xlsx workbook, EPA removed the title row above the header row on the 'State Trends' worksheet, shifting the 'State FIPS' header row from row 1 (skiprows=1) to row 0 (skiprows=0).

Changes Made

• scripts/us_epa/air_emissions_inventory/national_state/download.py:
  • Added requests.Session() with exponential backoff retry (Retry(total=5, backoff_factor=2, status_forcelist=[429, 500, 502, 503, 504])).
  • Added automatic fallback discovery (scrape_epa_trends_urls / resolve_download_urls) that scrapes https://www.epa.gov/air-emissions-inventories/air-pollutant-emissions-trends-data for the latest national_tier1_caps*.xlsx and state_tier1*ktons.xlsx URLs whenever the GCS config URLs (gs://unresolved_mcf/us_epa/config.json) return non-200.
  • Added HEAD-first URL probing (with GET fallback on HTTP 403/405), query-string stripping on .xlsx filenames, atomic downloads via .tmp + os.replace, non-empty file size verification, and cleanup of stale .xlsx/.xls/.tmp files prior to download.
• scripts/us_epa/air_emissions_inventory/national_state/process.py:
  • Updated _state_emissions() to dynamically detect whether 'State FIPS' is on row 0 (skiprows=0) or row 1 (skiprows=1) and support both 'State Trends' and 'State_Trends' sheet names.
  • Hardened year-column parsing in _state_emissions() to handle both 2-digit (emissions99, emissions01) and 4-digit (emissions1990, emissions2025) year headers while explicitly rejecting unparseable column headers (years.isna()).
  • Replaced deprecated Pandas chained inplace=True calls with explicit assignments (pd.to_numeric(..., errors='coerce')) for Pandas 2.x/3.x compatibility, and removed unreachable raise statements after absl.logging.fatal().
• scripts/us_epa/air_emissions_inventory/national_state/validation_config.json & manifest.json:
  • Added validation_config.json and registered validation_config_file and source_files in manifest.json:
    • check_deleted_records_percent (DELETED_RECORDS_PERCENT, threshold: 0.1): Enforces a 0.1% deletion threshold across refreshes. (Note: For the initial transition from the Feb 2025 release to the Sept 2026 2023-NEI restatement, 909 [0.38%] state cells in 2022–2024 changed from 0.00 placeholder values to blank/NaN upstream as documented in the workbook's README and DevelopmentOfData sheets; latest_version.txt was updated to the verified Sept 2026 baseline).
    • check_max_date_consistent (MAX_DATE_CONSISTENT): Verifies MaxDate (2025) is uniform across all 169 StatVars.
    • check_max_date_freshness (SQL_VALIDATOR): Verifies MaxDate >= '2025' across all StatVars.
• scripts/us_epa/air_emissions_inventory/national_state/process_test.py:
  • Fixed relative import paths and added unit tests covering skiprows=0 state worksheet parsing, 2/4-digit year parsing, HTML URL scraping/fallback resolution (scrape_epa_trends_urls, resolve_download_urls), and stale file cleanup (_clean_download_directory).
• scripts/us_epa/air_emissions_inventory/national_state/README.md:
  • Documented the GCS config (gs://unresolved_mcf/us_epa/config.json) and automatic fallback URL discovery mechanism.

Verification & Cloud Batch Test Run

• Unit Tests: ./run_tests.sh -p scripts/us_epa/air_emissions_inventory/national_state (Ran 6 tests — OK)
• GCS Bucket: https://console.cloud.google.com/storage/browser/datcom-import-test/scripts/us_epa/air_emissions_inventory/national_state/EPA_USAirPollutantEmissionTrends_NationalAndState/2026_10_16T04_55_51_589052_07_00?project=datcom-infosys-dev
• Validation Output (input0/validation/validation_output.csv — all 6 rules PASSED): https://console.cloud.google.com/storage/browser/_details/datcom-import-test/scripts/us_epa/air_emissions_inventory/national_state/EPA_USAirPollutantEmissionTrends_NationalAndState/2026_10_16T04_55_51_589052_07_00/input0/validation/validation_output.csv?project=datcom-infosys-dev

[gemini-code-assist]

Code Review

This pull request enhances the US EPA Air Emissions Inventory pipeline by introducing an automatic fallback mechanism to scrape live EPA trends page URLs when GCS configuration URLs are unreachable, improving Excel sheet and column parsing, and adding a validation configuration. However, the review feedback highlights critical issues regarding exception handling and error propagation in both download.py and process.py. Specifically, several exception handlers swallow fatal errors instead of raising them, which would cause the pipeline to exit with a success status code (0) despite failing. Additionally, the review points out a potential casting bug when parsing the 'year' column and suggests more robust checks for empty input directories.