With the latest changes to the DepMap portal, to add a captcha layer, it is now no-longer possible to programmatically download files. The bulk download option mentioned in the captcha also redirects to a captcha protected location.
In the end, this is okay, but we have been using the download page as the source for our version information on the depmap project. This is now not possible, and is also impacting open projects like bioversions bioversions/src/bioversions/sources/depmap.py at main · biopragmatics/bioversions · GitHub which used the same approach.
Would it be possible for the depmap team to provide a simple text file accessible without the captcha layer which simply states all, or the latest release which is available?
If I added a new endpoint at the address https://depmap.org/portal/api/no-captcha/download/files which returned identical content as https://depmap.org/portal/api/download/files except with the following two changes:
- There would be no captcha protection so it could be programmatically fetched.
- There would be no URL column
Would that address your needs?
(An alternative I can think of is a model where users can register and we can then issue API keys which can bypass the captcha by providing it as a bearer token in the authorization header field of the HTTP request. That’s appealing, but will take more effort to set up, so perhaps that will be a long term solution some day)
The new url I describe above should be easy to turn on, so while not particularly elegant, it would be easy to do.
All of this hassle is because in the last few months we’ve started experiencing a flood of requests which have been overwhelming the portal. And for our hosted downloads, we see the same files being downloaded over and over resulting in many many TBs of downloads – which was in turn creating substantial costs in egress fees.
Sorry about the inconvenience of the captchas, and I would like to find a solution which minimizes the inconvenience.
Thanks,
Phil
Hi Phil,
Thanks for getting back to us on this. Option 1 is incredibly valuable - we would use this immediately.
The bearer token auth would be the ultimate goal for us. We want to pull in new releases when (and only when) they are available - automated one time in our pipelines. Happy to help if ever needed with that
You also have some data available on Amazon Open Data - is there any potential to revive that? Similarly, we stumble across figshare depositions for some of the older resources , is this also completely out of the question now?
Thanks,
Matt
Hello,
Option 1 is implemented and live at https://depmap.org/portal/api/no-captcha/download/files
As I mentioned, the bearer token is our goal but it will likely be after the 26Q3 data release.
Yes, older data has been released on those platforms. However, for newly generated data, the DepMap data terms and conditions have changed. It continues to be freely available to anyone for research use, but it now requires separate licensing for commercial uses such as selling AI models trained on DepMap data.
From the current terms and conditions:
“The data made available on this website were generated for research purposes and are not intended for clinical or commercial uses, including direct sale, incorporation into a product, or the use of the data to train, develop, or enhance machine learning or AI models other than for internal research use (each a “Commercial Use”). For clarity, machine learning and AI models are permitted to be utilized with or on the Data for your own internal use, or shared for non-profit research purposes, including for process optimization and analysis. Commercial Use of the Data is not permitted under these terms and may require a separate license agreement from Broad or its contributors.”
Thanks,
Phil
