Repository navigation
[BUG] Global commodities column name mapping conflict #795
Description
Describe the bug
Older global_commodities input files will have headings Commodity, CommodityName etc.
The mapping from these old columns names to the new names is:
Commodity -> description
CommodityName - > commodity
There is a problem here: how does muse know whether a column called "commodity" is in the old parlance or the new parlance?
At present the mapping Commodity -> description does not seem to be listed in csv.py
If we added it, it would incorrectly map a new style column name, because the mapping is done after reducing to camel case.
One tempting fix would be to do the mapping before the case change. However, this would put us back into a situation of being case-sensitive, which we were trying to get away from...
Hopefully there is a better solution? (Or I have misunderstood the cause of the problem..?)
To Reproduce
Run the function standardize_dataframe() from csv.py on the input file global_commodities.csv
Expected behavior
`ValueError: Duplicate columns in Index(['commodity', 'commodity_type', 'commodity', 'emmission_factor'], dtype='object')'
Context
Please, complete the following to better understand the system you are using to run MUSE.
- Operating system (eg. Windows 10): Mac OSX
- MUSE version (eg. 1.0.1): b779f23
- Installation method (eg. pipx, pip, development mode): development mode
- Python version (you can get this running
python --version): 3.9.18
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
tsmbland commentedon Oct 3, 2025
So the old style format is:
Commodity: description of the commodity, just for user referenceCommodityName: name used for the commodity within MUSE and the other input filesIn this case, we only actually need the
CommodityNamefield within MUSE.This is a bit of a hack, but I've dealt with this in MUSE by dropping the
Commodity(/commodity) column in the case that there's also aCommodityName(/commodity_name) column present (i.e. where it's following the old format). I do this withinread_global_commodities_csvbefore passing tostandardize_dataframe.For your purposes, I think the possible solutions are:
a) Manually rename the columns before passing to
standardize_dataframeb) Use
read_global_commodities_csvto read the file, instead of reading with pandas and passing tostandardize_dataframe. In this case theCommoditycolumn will be dropped to avoid the clash, but I guess in doing so you'll lose the description info which may/may not be a problem. I could always modifyread_global_commodities_csvso it renames "commodity" -> "description" rather than dropping it. Would that be useful?martinstringer commentedon Oct 3, 2025
If we get modify read_global_commodities_csv to rename "commodity" -> "description" how will that script know that the input file was in the old format ? ("commodity" is a correct column name in the new format so we wouldn't want it to be changed).
Basically the problem I'm alluding to is that commodity (case insensitive) is a valid column name in both the new and old formats.
tsmbland commentedon Oct 3, 2025
martinstringer commentedon Oct 6, 2025
So do you think this issue will never arise as a problem when running muse in full?
tsmbland commentedon Oct 6, 2025
If someone's following either the old convention or the new convention then everything should be fine