Repository navigation
State of ulmo #161
Description
This is a very valuable resource, but it seems to be abandoned.
Builds are currently failing, the code doesn't seem to be getting updated, and there are some deprecation warnings coming in (e.g. #156 )
@kkremitzki wants to package it for Ubuntu (#158 ), which would be great, but there should be some assurance the package is being maintained.
Maybe @solomon-negusse or @dharhas could comment?
Activity
evanleeturner commentedon Oct 3, 2018
I'm also curious to see what plans @dharhas has for this package in the long term. I sat through his most recent presentation at Scipy and I think he is working toward more unified data standards. I use this package as well to access water data - but i'm not interested in inheriting it!!
In a larger conversation, it would be nice for the world if data warehouses agree on standards of communication - so ulmo would be unnecessary!
kkremitzki commentedon Oct 3, 2018
I've actually already packaged it for Debian, and so now it should be available in Ubuntu 18.10 and beyond: https://tracker.debian.org/pkg/python-ulmo
sdc50 commentedon Oct 3, 2018
Our current development efforts have been on a new package called Quest, which uses Ulmo as a dependency. We are planning to refactor Ulmo soon to incorporate some of the changes that we've included in Quest. That said, we are still planning to get the builds working and get some patches in before starting the refactor.
(@dharhas please correct me as necessary)
dharhas commentedon Oct 3, 2018
@sdc50 is correct. We have been working on another package called Quest that we were recently able to open source. We hope to use some of the lessons learned in Quest to refactor and cleanup the ulmo codebase. Ulmo still builds correctly locally but the CI is broken because something changed on the CI side, I think it is the url we retrieve miniconda from, but I haven't been able to check. There is a pull request that fixes #156 but I'm not keen on merging that until we get the CI stuff working again.
Unfortunately, I haven't had the cycles recently to work on any of this, and probably won't in the near future. It is on @sdc50 roadmap, but we don't have a concrete timeline for it.
dharhas commentedon Oct 11, 2018
So I fixed the CI system and merged it in. I'm dropping testing for Python 2 and adding 3.7.
Unfortunately, there are currently many test failures due to changes in various upstream websites. I'm not sure what our strategy will be going forward. If there is an easy fix we can probably fix it. If it is a complete rewrite of a module and it isn't one we use we may have to drop support unless someone steps up with a pull request.
@sdc50 @solomon-negusse any thoughts?
ColinTalbert commentedon Nov 24, 2018
I just happened to have found a bug caused by a change in the service for NASA ORNL's Daymet. It's an easy fix. If Ulmo will be continuing I can submit a pr.
If I should be moving to quest, I can do that as well.
I think this is a useful package but I don't get a chance to use it frequently.
dharhas commentedon Nov 27, 2018
@talbertc-usgs please submit a PR for ulmo. Quest and Ulmo have a lot of overlap but Quest is not yet a replacement for ulmo. Currently my plan when I have some cycles to spare is to do one more release of ulmo with a bunch on the recent fixes and then think about the long term.
solomon-negusse commentedon Nov 28, 2018
@dharhas i'm sorry for the long delay - i'm just seeing your message here now. dropping test support for 2.7 will be fine with me. we'll make sure that tests pass on py3 when we push any updates. in terms of maintenance, i'm on the same boat... we have only had enough bandwidth to update the services we use here when the need arises and we'll continue to do that.
emiliom commentedon Jan 30, 2019
@dharhas I know "cycles to spare" are in short supply. Just wondering what your current thinking is regarding merging PR's and possibly issuing a new release. Selfishly, I'm most interested in the PR I submitted a couple of weeks ago (#165), plus #157. I may be able to help, if you'd like, after mid February. Thanks!
dharhas commentedon Jan 30, 2019
@emiliom I have a work related deadline Feb 1st. Once I'm past that, I should be able to devote some time to merging some PR's and making a new release. My current thinking is to disable any services that no longer work and make a new release. I'll try and think through the best way to do that next week.
emiliom commentedon Feb 2, 2019
Regarding #156, I don't see an open PR that fixes it. You might be thinking of #157?
#156 involves https://github.com/ulmo-dev/ulmo/blob/master/ulmo/util/misc.py#L52. The fix is simple, changing that line to
return pandas.Timestamp(datetime).to_pydatetime(). I'll be testing this over the next couple of days and would be happy to submit a PR afterwards.FYI, I've created a branch on my fork which merges that fix, together with PRs #157, #164 and #165. That's what I (and someone I'm working with) will be testing over the next week. We'll be using ulmo heavily at a webinar on 2/7 next week, and wanted to be able to use the latest Pandas as well as have some of these fixes in place.
emiliom commentedon Feb 2, 2019
@talbertc-usgs can you go ahead and submit a PR for the NASA ORNL's Daymet fix you mentioned above at #161 (comment)? That way we can fold it into a next release fairly soon. Thanks!
dharhas commentedon Feb 4, 2019
I'm working on this today (02/04/2019). I should be able to get a release out before @emiliom webinar.
emiliom commentedon Feb 4, 2019
Thanks @dharhas! But no pressure. I'm good for my webinar with the temporary fork I've already created. I've got a conda environment file that installs that version of ulmo directly from my fork via pip install, and that's meeting our needs this week.
dharhas commentedon Feb 4, 2019
Ok I just pushed a bunch of fixes to bring things up to date and compatible with modern pandas and updates to various source web services. Tests now pass on my local machine. CI is running now, hopefully my fixes are cross platform.
I attempted to fix all the services that I could but wasn't able to get all of them.
I'm thinking of removing the lcra service completely unless someone wants to rewrite it. I'll leave the hdf5 and pytables in with a warning that they don't work on recent pandas versions
3 remaining items
evanleeturner commentedon Feb 5, 2019
@solomon-negusse Our LCRA script was using ulmo and that broke at some point in late 2018. They had made some sort of change because their data is no longer live reporting, so that may have altered their internal API. The website system they have is still functional: https://waterquality.lcra.org/parameterbay.aspx?qrySite=MB02 I haven't been able to get in touch with their people but I agree that I could do a fix for their data here.
emiliom commentedon Feb 5, 2019
@dharhas thanks for all that work! That's terrific.
I don't have anything else to suggest, beyond my prodding @talbertc-usgs to see if he can submit soonish the ORNL Daymet PR he offered. While I don't use Daymet, I think it'd be good to wait to see if he has time to submit that bug fix so it can be folded into this release (assuming that's not too disruptive to you).
If you'd like a second pair of eyes and tests on your recent fixes before issuing the release, I can offer to help; but it'd have to be next week.
dharhas commentedon Feb 5, 2019
@evanleeturner @solomon-negusse if you plan to get the LCRA fix in, I'll wait a few days before making a new release. Otherwise, I'll plan to remove those services.
evanleeturner commentedon Feb 15, 2019
Just do whatever you feel like. I don't have time to look at this until maybe late March at the earliest when we get our vacant positions filled.
emiliom commentedon Mar 6, 2019
@dharhas, I'm assuming you lost your momentum for issuing a new ulmo release? It looked like you were practically ready. Let me know if I can help pushing it out the door soon -- say, by helping draft release notes.
Looking longer term, I've been thinking of trying to persuade CUAHSI (technical staff in the Boston office) to play a role in helping to maintain ulmo ... Thoughts?
dharhas commentedon Mar 7, 2019
@emiliom I got busy. Drafting release notes would help. Long term, I'm happy to hand over the reigns to someone more invested in ulmo going forward. I don't use ulmo as part of my day job any longer so its difficult to find time to push it forward.
emiliom commentedon Mar 8, 2019
Alright, I'll try to start on Monday.
Yeah, that's why I've been thinking about sustainability. I'll reach out to CUAHSI in the next couple of weeks (I'm on its informatics committee).
emiliom commentedon Mar 20, 2019
@dharhas I looked at the commits listing since the last release, and came up with the release notes below. Edit as you see fit. I assume the release version tag will be v0.8.5? If you could issue the release by Friday, that'd be really helpful. Once the release is ready, I can follow up on the conda-forge feedstock to get the new conda package built.
FYI, I wrote to the CUAHSI Informatics committee to propose having CUAHSI adopt ulmo. We'll probably discuss this at our next meeting, next week, and hopefully make a recommendation to CUAHSI right after it. I'll report back.
Thanks!
This release includes commits from 8 contributors. It includes many bug fixes and changes to plugins, as well as to the CI, test, and package build infrastructures.
Bug Fixes and Changes to Plugins
Other Bug Fixes
emiliom commentedon Mar 25, 2019
A new ulmo release (0.8.5) is now out, and a available via pypi and conda-forge. Thanks @dharhas for your help getting set up!
Also, on Wednesday I'll be making a pitch to CUAHSI to adopt ulmo
ChristinaB commentedon Dec 4, 2019
@emiliom How did that pitch go? What does 'adopt' mean?
emiliom commentedon Dec 5, 2019
@ChristinaB thanks for asking. I made the pitch back then, and the reception was positive but non-committal. What "adoption" would entail was TBD, but it probably would focus around ulmo readers for CUAHSI data.
I was asked to send them a justification with background on ulmo. I started putting that written pitch together, including statistics about usage and contributions, but I never completed it. I still would like to do it. Also, my momentum was deflated by not getting any responses to my request (April) for testimonials from users, over on the ulmo Google groups.
I'll try to get back to this next week, once and for all. There's also a PR waiting for my approval and merge.
mroberge commentedon Dec 5, 2019
@emiliom Hey I've been lurking for a while and I just wanted to check in to say thank you to everyone for their work on this project. It is not easy to maintain open source software, but it can make a big difference.
I have my own OSS project (hydrofunctions) that I created to make it easier for my students to work with USGS stream gauge data. I've long been considering Ulmo to provide access to more data sources.
emiliom commentedon Dec 7, 2019
Thanks @mroberge! And thanks also for mentioning hydrofunctions. I didn't know about it, and it looks like a really nice tool for NWIS access. I'll try to check it out in more detail later on.
dharhas commentedon Jan 3, 2020
I think this issue can be closed now.
In summary, @emiliom has taken over active maintenance of ulmo and has also proposed to the CUAHSI Informatics group that they take the project under their umbrella (that has not happened yet, but is still a possibility). I still lurk and am available to answer historical questions about the codebase but am no longer actively involved in the project.