File transfers in and out of the SRE

The Research Extract provided to the researchers on the SRE must only be used as permitted, as outlined in the Data Access Request, the approval letter / research agreement, and in the Population Data BC Researcher Services Agreement.

That said, researchers still need to be able to get information out of the SRE (e.g. aggregate results of analysis, scripts, etc.) as well as into the SRE (e.g. scripts, documentation).

In order to balance ease of use, protection and accountability, Population Data BC has developed what are called the ‘Yellow Folders’. Information that is to be transferred out of the SRE are dropped in this special “Yellow Folder.” The user triggers a transfer program, which logs the data, and transfers it to a location that is accessible from the outside.

This process allows Population Data BC to keep an audit trail for transfers and the explicit action required by the researcher will minimize accidental releases of data. The log information is reviewed by Population Data BC regularly and upon request by the applicable Data Steward(s.)

It is important that all researchers adhere to these restrictions in order to comply with the agreements made with the Data Stewards, the Pledge of Confidentiality that has been signed, as well as the project’s agreement with Population Data BC. Failure to comply constitutes a breach, and will result in resaechers' access being shut off, and may result in termination of the project and an inability to access data in the future. If breaches are suspected, SRE access is terminated immediately until the matter is investigated.

 

Downloading and uploading data to the SRE

Uploading data to the SRE

Researchers wishing to upload data to the SRE must request approval to do so from the Data Access Unit. If it is approved, the data will be added to the researcher's folder on the SRE by Population Data BC.

  • Researchers are NOT allowed to upload data (including their personal dataset) directly to the SRE
  • Researchers are NOT allowed to upload data not specified in the Data Access Request to the SRE

Guide to allowable downloads from the SRE

Unauthorized downloads
stop Research Extracts

Research Extracts or their subsets (i.e.the data received on the SRE for your project) as this considered raw data.

Note: Study IDs or program code that contains individual Study IDs are not allowed to be downloaded.

Stop Researcher-collected data uploaded to the SRE Any researcher-collected data that is housed on the SRE as part of a research project, as it is considered part of the Research Extract (see above).
Stop Archived/compressed or encrypted files Any files compressed (ZIP, RAR, 7Z, GZ...) or encrypted (e.g. password protected M.S.Office documents) as they are unsuited to PopData's automated analysis.
Stop Files that do not meet cell size restrictions

Unless otherwise specifically noted in the project’s Research Agreement(s), each table cell size must NOT be less than five (5) as per standard guidelines for aggregation of data..

Example:

Output file containing a column with a small cell size count of less than 5:

Age OBS Mean Count
38 100 1234.567
65 8 765.432
83 2 300.123

 

 

Authorized downloads
tick Research Output

Products from the results of research analysis that are aggregated (i.e., related to broad classes, groups, or categories) and therefore cannot be linked or identified are permitted to be downloaded from the SRE.

Examples of allowed downloads (noting cell size restrictions):

  • maps
  • tables
  • graphs
  • log files
tick Supporting documentation

Documentation used to support research that does not contain personal information.

Examples of allowed downloads:

  • scripts/code
  • syntax
  • research notes (not including data)

Using the Yellow Folders for uploading and downloading other files

While connected through VPN, you can transfer files between the SRE and your computer using a temporary file sharing connection. Direction names are relative to SRE.

Basically there are folders with the same name (IMPORT_TO_SRE , EXPORT_FROM_SRE) on the inside and outside, and clickable programs (IMPORT_IT , EXPORT_IT) which trigger the transfer between the inside and outside.

For Windows 10

Either use "File Explorer" menu in lower left corner as in the screenshot, or type 'explorer' in the search box "I'm Cortana. Ask me anything", you will be prompted to open the program

For Windows 8

For windows 8, start by moving the mouse to the Upper Right corner to bring up the "charm" menu, and in Search enter "File Explorer", then follow the steps as for Windows 7 below.

For Windows 7  

  • Right-click on Computer and select "Map network drive...". 
  • Select a drive letter (ex: Z:) and in box "Folder" enter  \\fraser.popdata.bc.ca\PROJECT_SPECIFIC_USERNAME
  • De-select "Reconnect at logon"; select "Connect using different credentials"; click "Finish". 
  • A box will ask for your project-specific username (popdata\PROJECT_SPECIFIC_USERNAME) and password. 

WARNING: this is complicated but necessary. Microsoft makes it difficult to use a different username and/or password.

Alternately you can download one or more ".bat" scripts for Windows, customized with your project login(s). Go to https://my.popdata.bc.ca/account/login and enter your short username and password, then click on "Account" at the top right, then "Account Info" and scroll down to "SRE Project Accounts" where you will find the script appropriate for your project(s). Follow instruction in the notes below.

Accessing shared files from a MacIntosh

Select Finder; Menu Go => Connect to Server (CMD-K)
In box Server Address enter cifs://PROJECT_SPECIFIC_USERNAME@fraser.popdata.bc.ca/PROJECT_SPECIFIC_USERNAME (substitute your project-specific username for PROJECT_SPECIFIC_USERNAME twice), press + to add to list Favorite Servers.

Transfer process

Getting files into the SRE

Your computer

On your own computer connect to the network share that you setup above. Change to folder IMPORT_TO_SRE and copy into it the files you want to send to SRE.

Meanwhile on the Remote SRE Desktop

Go into the SRE and navigate to R:\PROJECT_SPECIFIC_USERNAME\TRANSFER

Double click on the IMPORT-IT icon.

Getting files out of the SRE

Go into the SRE and navigate to R:\PROJECT_SPECIFIC_USERNAME\TRANSFER\EXPORT_FROM_SRE

Copy here the files you want to send out of the SRE.

Go up one folder, and double-click on the EXPORT-IT icon.

On your own computer connect to the network share for outgoing that you setup earlier.

How to avoid blocked transfers

In order to reassure Data Stewards that their data is safe in the SRE, while minimizing delays for researchers who need to export results and code or import code, Population Data BC uses a combination of education/training, retrospective inspection of files recently transferred, and automatic blocking of certain types of files. Automatic blocking for an account stays in effect until manual inspection is completed

If you try to import or export an archive or encrypted file (e.g. ZIP, 7Z, RAR etc.) the transfer will be automatically blocked for inspection by PopData staff. Instead of creating an archive before transfer, instead please copy or move your files into a folder and copy that folder to your EXPORT_FROM_SRE (or IMPORT_TO_SRE) folder, then trigger the transfer.

Other triggers for blocking include size or content-matching files under the project's DATA folder (i.e. files in your EXPORT folder will be compared to files in the project’s DATA folder) unless in a folder named 'docs', or otherwise registered as non-data.

If you are unsure whether your files will meet the export criteria after placing the files into the transfer folder email sre@popdata.bc.ca requesting a manual inspection and transfer.

After one of your transfers gets automatically blocked, all subsequent transfers will be blocked until we inspect the files and remove the restriction. So it is important to double-check that the contents of your IMPORT or EXPORT folder are acceptable and you meet PopData's import and export rules.

For example if you are not sure whether your files include sequences of digits that look like Study IDs, download scan-studyids-how.pdf and use the scanning tool described there.