Data Mobility Exhibition Archive
Data Mobility Exhibition
The Globus endpoints and test data sets listed below are available for use by participants in the Data Mobility Exhibition. Thank you to the host institutions that that have generously made these available to us!
Usage Guidelines and Ground Rules
- Data sets are stored either in the endpoint root or /datasets directory. You have read-only access to this directory so you can initiate outbound transfers to your own destination endpoint.
- If you wish to use one of these the endpoints as the destination, please write files to a sub-directory within the root or /perftest directory; use a descriptive name for your subdirectory so that it's easily identifiable by all, e.g. esnet-team.
- Please be careful to not overwrite or otherwise tamper with another team's subdirectory.
- All participants in the exhibition are sharing the same storage allocation, so please be sure to delete all transferred files as soon as your transfer completes.
- While most endpoints have very large/open quotas, some may be limited as shown. Do not attempt to transfer a data set that is larger than the quota. Also, please monitor your transfers (particularly those using the 5TB data set), and be prepared to cancel and immediately clean up any transferred files if you bump up against space limits.
Thanks for being good data mobility citizens!
Data Sets
| Name | Total Size | Description |
|---|---|---|
| ds01 | 100MB | 10,000 x 10KB files in single directory |
| ds04 | 10GB | 10,000 x 1MB files in 100 non-nested directories, 100 files/directory |
| ds06 | 100GB | 100,000 x 1MB files in single directory |
| ds08 | 1TB | 50 x 10GB; 350 x 1GB; 1,000 x 100MB; 5,500 x 10MB; 23,176 x 1MB files in single directory |
| ds10 | 1TB | 100 x 10GB files in single directory |
| ds12 | 100GB | 1 x 100GB file in single directory |
| ds14 | 5TB | 50 x 100GB files in single directory |
| ds16 | 1TB | 4 x 250GB files in single directory |
Endpoints
The following endpoints contain a copy of each data set; they are read-only, so you may use them as the source endpoint for your transfers. Endpoint IDs are in parentheses, in case you are scripting or otherwise automating your transfers.
Important Notes
- You must be a member of the "DME Endpoint Access" Globus group to access the endpoints.
click here to request membership. - Only endpoints with a number in the leftmost column are currently active and available for use in the DME.
| No. | Display Name | Description |
|---|---|---|
| 1 | cac_dtn_test (606579ae-5b03-11e9-bf32-0edbf3a4e7ee) |
Cornell Center for Advanced Computing in Ithaca, NY |
| 2 | NCAR Data Mobility Exhibition (651903ec-e892-460e-bfcf-f824d66509fb) |
NCAR GLADE (IBM Spectrum Scale) filesystem in Boulder, CO |
| 3 | Data Mobility Exhibition - Argonne (e6cc344e-91e5-4cc3-afda-3603d02b8a00) |
Lustre filesystem on the Eagle system at the Argonne Leadership Computing Facility |
| — | Holystore01-testing (d73f12ec-4094-11eb-b55b-02d9497ca481) |
Lustre filesystem located in Harvard's datacenter |
The following endpoints are write-enabled so you may use them as the destination endpoint for your transfers. Click on the Display Name link to open the endpoint in the Globus web app file manager.
| No. | Display Name | Description | Quota |
|---|---|---|---|
| 1 | cac_dtn_test (606579ae-5b03-11e9-bf32-0edbf3a4e7ee) |
Cornell Center for Advanced Computing in Ithaca, NY | Unknown |
| 2 | NCAR Data Mobility Exhibition (651903ec-e892-460e-bfcf-f824d66509fb) |
NCAR GLADE (IBM Spectrum Scale) filesystem in Boulder, CO This endpoint has multiple 40Gbps links to the Internet2 100Gbps backbone so you should see great performance (assuming the destination endpoint is well-provisioned).
|
Unknown |
| 3 | Data Mobility Exhibition - Argonne (e6cc344e-91e5-4cc3-afda-3603d02b8a00) |
Lustre filesystem on the Eagle system at the Argonne Leadership Computing Facility | 10TB |
| 4 | DME PerfTest - Michigan (0ddccd16-c04a-11e9-9cec-0edb67dd7a14) |
Turbo system on the Flux cluster at the University of Michigan | 100GB |
| — | DME PerfTest - Boston (38b0ee04-c0b4-11e9-939a-02ff96a5aa76) |
Lustre filesystem located in Harvard's Boston datacenter | 10TB |
| — | DME PerfTest - MGHPCC (8c799234-be52-11e9-9ceb-0edb67dd7a14) |
Lustre filesystem located at the MGHPCC in Holyoke, MA | Unknown |
| — | DME PerfTest - Google Drive (fc185009-345f-4a62-9ae3-21e8e79960c6) |
Google Drive cloud storage accessed via an EC2 DTN in the us-east-1 region Note: Google Drive imposes rate limits on their API so expect throttling when transferring datasets with many files.
|
None |
| — | DME PerfTest - Box (401f8d2d-8797-4611-8521-196870dbf9fd) |
Box cloud accessed via an EC2 DTN in the us-east-1 region Note: In addition to API rate limits, Box imposes a filesize limit of 15GB; do not attempt to transfer ds12, ds14, or ds16.
|
None |
Useful Links
- The Globus Connect Server installation guide contains instructions for setting up your Globus endpoint.
Note: Unless you are a paying Globus subscriber at the High Assurance level or above, you must follow the instructions for Globus Connect Server v4; Globus Connect Server v5 is not the latest, general purpose release. - The Globus Command Line Interface (CLI) is great for scripting a set of transfers that you can run automatically from multiple endpoints.
- Additional (and more complete) examples of scripts are available in the Globus GitHub repositories; in particular, this repo has bash and Python equivalents for common automation tasks.
- Here are some (many!) of the frequently asked questions about Globus.
- The Globus support team is always available to help; please contact support@globus.org.
Click here for more documentation and other helpful Globus resources.
