Thursday, September 13, 2018

AWS - S3


Amazon S3 Storage

For institutions and organizations looking to use Amazon's S3 cloud storage for their installation, this can be set up manually through creation of a credentials file or automatically via the aws console commands.
You'll need an AWS account with an associated S3 bucket for your installation to use. From the S3 management console (e.g. https://console.aws.amazon.com/), you can poke around and get familiar with your bucket. We recommend using IAM (Identity and Access Management) to create a user with full S3 access and nothing more, for security reasons. See http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html for more info on this process.
Make note of the bucket's name and the region its data is hosted in. Dataverse and the aws SDK rely on the placement of a key file located in ~/.aws/credentials, which can be generated via either of these two methods.

Setup aws manually

To create credentials manually, you will need to generate a key/secret key. The first step is to log onto your aws web console (e.g. https://console.aws.amazon.com/). If you have created a user in AWS IAM, you can click on that user and generate the keys needed for dataverse.
Once you have acquired the keys, they need to be added to``credentials``. The format for credentials is as follows:
[default]
aws_access_key_id = <insert key, no brackets>
aws_secret_access_key = <insert secret key, no brackets>

Place this file ina a folder named .aws under the home directory for the user running your dataverse installation.

Setup aws via command line tools

Begin by installing the CLI tool pip to install the AWS command line interface if you don't have it.
First, we'll get our access keys set up. If you already have your access keys configured, skip this step. From the command line, run:
pip install awscli
aws configure
You'll be prompted to enter your Access Key ID and secret key, which should be issued to your AWS account. The subsequent config steps after the access keys are up to you. For reference, these keys are stored in ~/.aws/credentials.

Configure dataverse to use aws/S3

With your access to your bucket in place, we'll want to navigate to /usr/local/glassfish4/glassfish/bin/ and execute the following asadmin commands to set up the proper JVM options. Recall that out of the box, Dataverse is configured to use local file storage. You'll need to delete the existing storage driver before setting the new one.
./asadmin $ASADMIN_OPTS delete-jvm-options "\-Ddataverse.files.storage-driver-id=file"
./asadmin $ASADMIN_OPTS create-jvm-options "\-Ddataverse.files.storage-driver-id=s3"
Then, we'll need to identify which S3 bucket we're using. Replace your_bucket_name with, of course, your bucket:
./asadmin create-jvm-options "-Ddataverse.files.s3-bucket-name=your_bucket_name"
Lastly, go ahead and restart your glassfish server. With Dataverse deployed and the site online, you should be able to upload datasets and data files and see the corresponding files in your S3 bucket. Within a bucket, the folder structure emulates that found in local file storage.

Branding Your Installation

The name of your root dataverse is the brand of your installation of Dataverse and appears in various places such as notifications. Notifications and support links also contain the name of your support team, which will either be "[YourBrand] Support" or a name of your choosing when you configure a name for the :SystemEmail database setting described below.
Dataverse provides configurable options for easy-to-add (and maintain) custom branding for your Dataverse installation. Downloadable sample HTML and CSS files are provided below which you can edit as you see fit. It's up to you to create a directory in which to store these files, such as /var/www/dataverse in the examples below.
You can add a custom welcome/homepage as well as other custom content, to further brand your installation and make it your own. Here are the custom branding and content options you can add:
  • Custom welcome/homepage
  • Logo image to navbar
  • Header
  • Footer
  • CSS stylesheet

Custom Homepage

Dataverse allows you to use a custom homepage or welcome page in place of the default root dataverse page. This allows for complete control over the look and feel of your installation's homepage.
Download this sample: :download:`custom-homepage.html </_static/installation/files/var/www/dataverse/branding/custom-homepage.html>` and place it at /var/www/dataverse/branding/custom-homepage.html.
Once you have the location of your custom homepage HTML file, run this curl command to add it to your settings:
curl -X PUT -d '/var/www/dataverse/branding/custom-homepage.html' http://localhost:8080/api/admin/settings/:HomePageCustomizationFile
Note that the custom-homepage.html file provided has a "Browse Data" button that assumes that your root dataverse still has an alias of "root". While you were branding your root dataverse, you may have changed the alias to "harvard" or "librascholar" or whatever and you should adjust the custom-homepage.html file as needed.
For more background on what this curl command above is doing, see the "Database Settings" section below. If you decide you'd like to remove this setting, use the following curl command:
curl -X DELETE http://localhost:8080/api/admin/settings/:HomePageCustomizationFile

Custom Navbar Logo

Dataverse allows you to replace the default Dataverse icon and name branding in the navbar with your own custom logo. Note that this logo is separate from the root dataverse theme logo.
The custom logo image file is expected to be small enough to fit comfortably in the navbar, no more than 50 pixels in height and 160 pixels in width. Create a navbar directory in your Glassfish logos directory and place your custom logo there. By Glassfish default, your logo image file will be located at /usr/local/glassfish4/glassfish/domains/domain1/docroot/logos/navbar/logo.png.
Once you have the location of your custom logo image file, run this curl command to add it to your settings:
curl -X PUT -d '/logos/navbar/logo.png' http://localhost:8080/api/admin/settings/:LogoCustomizationFile

Custom Header

Download this sample: :download:`custom-header.html </_static/installation/files/var/www/dataverse/branding/custom-header.html>` and place it at /var/www/dataverse/branding/custom-header.html.
Once you have the location of your custom header HTML file, run this curl command to add it to your settings:
curl -X PUT -d '/var/www/dataverse/branding/custom-header.html' http://localhost:8080/api/admin/settings/:HeaderCustomizationFile

Custom Footer

Download this sample: :download:`custom-footer.html </_static/installation/files/var/www/dataverse/branding/custom-footer.html>` and place it at /var/www/dataverse/branding/custom-footer.html.
Once you have the location of your custom footer HTML file, run this curl command to add it to your settings:
curl -X PUT -d '/var/www/dataverse/branding/custom-footer.html' http://localhost:8080/api/admin/settings/:FooterCustomizationFile

Custom Stylesheet

You can style your custom homepage, footer and header content with a custom CSS file. With advanced CSS know-how, you can achieve custom branding and page layouts by utilizing position, padding or margin properties.
Download this sample: :download:`custom-stylesheet.css </_static/installation/files/var/www/dataverse/branding/custom-stylesheet.css>` and place it at /var/www/dataverse/branding/custom-stylesheet.css.
Once you have the location of your custom CSS file, run this curl command to add it to your settings:
curl -X PUT -d '/var/www/dataverse/branding/custom-stylesheet.css' http://localhost:8080/api/admin/settings/:StyleCustomizationFile

Going Live: Launching Your Production Deployment

This guide has attempted to take you from kicking the tires on Dataverse to finalizing your installation before letting real users in. In theory, all this work could be done on a single server but better would be to have separate staging and production environments so that you can deploy upgrades to staging before deploying to production. This "Going Live" section is about launching your production environment.
Before going live with your installation of Dataverse, you must take the steps above under "Securing Your Installation" and you should at least review the various configuration options listed below. An attempt has been made to put the more commonly-configured options earlier in the list.
Out of the box, Dataverse attempts to block search engines from crawling your installation of Dataverse so that test datasets do not appear in search results until you're ready.

Letting Search Engines Crawl Your Installation

For a public production Dataverse installation, it is probably desired that search agents be able to index published pages (aka - pages that are visible to an unauthenticated user). Polite crawlers usually respect the Robots Exclusion Standard; we have provided an example of a production robots.txt :download:`here </_static/util/robots.txt>`).
You have a couple of options for putting an updated robots.txt file into production. If you are fronting Glassfish with Apache as recommended above, you can place robots.txt in the root of the directory specified in your VirtualHost and to your Apache config a ProxyPassMatch line like the one below to prevent Glassfish from serving the version of robots.txt that embedded in the Dataverse war file:
# don't let Glassfish serve its version of robots.txt
ProxyPassMatch ^/robots.txt$ !
For more of an explanation of ProxyPassMatch see the :doc:`shibboleth` section.
If you are not fronting Glassfish with Apache you'll need to prevent Glassfish from serving the robots.txt file embedded in the war file by overwriting robots.txt after the war file has been deployed. The downside of this technique is that you will have to remember to overwrite robots.txt in the "exploded" war file each time you deploy the war file, which probably means each time you upgrade to a new version of Dataverse. Furthermore, since the version of Dataverse is always incrementing and the version can be part of the file path, you will need to be conscious of where on disk you need to replace the file. For example, for Dataverse 4.6.1 the path to robots.txt may be /usr/local/glassfish4/glassfish/domains/domain1/applications/dataverse-4.6.1/robots.txt with the version number 4.6.1 as part of the path.

Putting Your Dataverse Installation on the Map at dataverse.org

Congratulations! You've gone live! It's time to announce your new data respository to the world! You are also welcome to contact support@dataverse.org to have the Dataverse team add your installation to the map at http://dataverse.org . Thank you for installing Dataverse!

Administration of Your Dataverse Installation

Now that you're live you'll want to review the :doc:`/admin/index` for more information about the ongoing administration of a Dataverse installation.

JVM Options

JVM stands Java Virtual Machine and as a Java application, Glassfish can read JVM options when it is started. A number of JVM options are configured by the installer below is a complete list of the Dataverse-specific JVM options. You can inspect the configured options by running:
asadmin list-jvm-options | egrep 'dataverse|doi'
When changing values these values with asadmin, you'll need to delete the old value before adding a new one, like this:
asadmin delete-jvm-options "-Ddataverse.fqdn=old.example.com"
asadmin create-jvm-options "-Ddataverse.fqdn=dataverse.example.com"
It's also possible to change these values by stopping Glassfish, editing glassfish4/glassfish/domains/domain1/config/domain.xml, and restarting Glassfish.

dataverse.fqdn

If the Dataverse server has multiple DNS names, this option specifies the one to be used as the "official" host name. For example, you may want to have dataverse.foobar.edu, and not the less appealling server-123.socsci.foobar.edu to appear exclusively in all the registered global identifiers, Data Deposit API records, etc.
The password reset feature requires dataverse.fqdn to be configured.
Do note that whenever the system needs to form a service URL, by default, it will be formed with https:// and port 443. I.e.,
https://{dataverse.fqdn}/
If that does not suit your setup, you can define an additional option, dataverse.siteUrl, explained below.

dataverse.siteUrl

and specify the protocol and port number you would prefer to be used to advertise the URL for your Dataverse.
For example, configured in domain.xml:
<jvm-options>-Ddataverse.fqdn=dataverse.foobar.edu</jvm-options>
<jvm-options>-Ddataverse.siteUrl=http://${dataverse.fqdn}:8080</jvm-options>

dataverse.files.directory

This is how you configure the path to which files uploaded by users are stored.

dataverse.auth.password-reset-timeout-in-minutes


Users have 60 minutes to change their passwords by default. You can adjust this value here.

No comments:

Post a Comment

Hyderabad Trip - Best Places to visit

 Best Places to Visit  in Hyderabad 1.        1. Golconda Fort Maps Link :   https://www.google.com/maps/dir/Aparna+Serene+Park,+Masj...