02. Openedx Analytics - Hadoop/Hive/Sqoob

Configure hadoop, hive and sqoop

sudo su - hadoop 

cd /edx/app/hadoop




Please check Hadoop 

hadoop jar hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 2 100

Please check Hive is working

/edx/app/hadoop/hive/bin/hive







Test Dashboard is up and running




http://localhost:18110 

make suer hadoop dashboards are working

http://localhost:8088/cluster

hadoop job history

http://localhost:19888

hadoop node description

http://localhost:8042

Troubleshooting
01. Hadoop dashboard not working

hadoop/sbin/stop-all.sh

hadoop/sbin/start-all.sh

Determine Port is listening

telnet localhost 9000

lsof -i :9000

nc -l -p 9000

Test HDFS System

hadoop fs -ls /

Create Directory 

hadoop fs -mkdir /data

Put file

hadoop fs -put /home/file.txt /data

01. Openedx Analytics - Installation

Openedx Analytics Installation Steps




sudo apt-get update

sudo apt-get install git

sudo apt-get install python-pip

sudo apt-get install python-dev

  

sudo pip install virtualenv

  

# create an "ansible" virtualenv and activate it

virtualenv ansible

. ansible/bin/activate

git clone https://github.com/edx/configuration.git

cd configuration/

  

apt-get install libmysqlclient-dev

pip install -r requirements.txt

apt-get install mysql-server-5.7

apt-get install python-mysqldb

cd playbooks/edx-east/

  

apt-get install nginx

add folder

edx/app/nignx/sites-available

ansible-playbook -i localhost, -c local analytics_single.yml --extra-vars "INSIGHTS_LMS_BASE=$LMS_HOSTNAME INSIGHTS_BASE_URL=$INSIGHTS_HOSTNAME"

03-openstack-image-service(glance)

Image service (glance)

users to discover, register, and retrieve virtual machine images.

REST API that enables you to query virtual machine image metadata and retrieve an actual image

By default, this directory is

/var/lib/glance/images/

step 1 create database

CREATE DATABASE glance;

grant all privileges on glance.* to glance@10.0.3.11 identified by 'glance';

step 2 openstack

admin credentials to gain access

source admin-openrc

create glance user

openstack user create --domain default --password-prompt glance

unset all envrionment varialble (admin)

export all enironment variable again (admin)

password glance1

mysql database

add the admin role to the glance

$ openstack role add --project service --user glance admin

glance service entity

openstack service create --name glance --description "OpenStack Image" image

Create the Image service API endpoints:

openstack endpoint create --region RegionOne image public http://controller:9292

openstack endpoint create --region RegionOne image internal http://controller:9292

openstack endpoint create --region RegionOne image admin http://controller:9292

installing application

apt-get install glance

rabbitmq user=openstack/passwd=RABBIT_PASS

username must glance@10.0.3.11

/etc/glance/glance-api.conf

[database]

connection = mysql+pymysql://glance:glance@10.0.3.16:3306/glance

[keystone_authtoken]

# ...

auth_uri = http://controller:5000

auth_url = http://controller:35357

memcached_servers = controller:11211

auth_type = password

project_domain_name = default

user_domain_name = default

project_name = service

username = glance

password = GLANCE_PASS

[paste_deploy]

# ...

flavor = keystone

configure the local file system store and location of image files:

[glance_store]

# ...

stores = file,http

default_store = file

filesystem_store_datadir = /var/lib/glance/images/

/etc/glance/glance-registry.conf

[database]

# ...

connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance

[keystone_authtoken]

# ...

auth_uri = http://controller:5000

auth_url = http://controller:35357

memcached_servers = controller:11211

auth_type = password

project_domain_name = default

user_domain_name = default

project_name = service

username = glance

password = glance

[paste_deploy]

# ...

flavor = keystone

glance-manage db_sync

service glance-registry restart

service glance-api restart

verify operation

source admin-openrc.sh

wget http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img

openstack image create "cirros" --file cirros-0.3.5-x86_64-disk.img --disk-format qcow2 --container-format bare --public

Confirm upload of the image and validate attributes:

Dashboard

02-openstack-new-tenant

User can't create without having a tenant assign to them.

Create tenant called cookbook

keystone client to operate Keystone (python-keystoneclient)

Ensure that we have our environment set correctly to access our OpenStack environment

for administrative purposes:

export OS_TENANT_NAME=cookbook

export OS_USERNAME=admin

export OS_PASSWORD=openstack

export OS_AUTH_URL=https://10.0.3.11:5000/v2.0/

export OS_NO_CACHE=1

export OS_KEY=/vagrant/cakey.pem

export OS_CACERT=/vagrant/ca.pem

admin account

export OS_USERNAME=admin

export OS_PASSWORD=keystone

export OS_PROJECT_NAME=admin

export OS_USER_DOMAIN_NAME=Default

export OS_PROJECT_DOMAIN_NAME=Default

export OS_AUTH_URL=http://controller:35357/v3

export OS_IDENTITY_API_VERSION=3

Regular (non-admin) tasks

create demo user

password 123

create user role

Add the user role to the demo project and user:

$ openstack role add --project demo --user demo user

verify operation

As the admin user, request an authentication token:

openstack --os-auth-url http://controller:35357/v3 \

--os-project-domain-name default --os-user-domain-name default \

--os-project-name admin --os-username admin token issue

As the demo user, request an authentication token:

openstack --os-auth-url http://controller:5000/v3 \

--os-project-domain-name default --os-user-domain-name default \

--os-project-name demo --os-username demo token issue

create script file called admin-openrc

export OS_PROJECT_DOMAIN_NAME=Default

export OS_USER_DOMAIN_NAME=Default

export OS_PROJECT_NAME=admin

export OS_USERNAME=admin

export OS_PASSWORD=ADMIN_PASS

export OS_AUTH_URL=http://controller:35357/v3

export OS_IDENTITY_API_VERSION=3

export OS_IMAGE_API_VERSION=2

source admin-openrc

openstack --os-auth-url http://controller:35357/v3 --os-project-domain-name default --os-user-domain-name default --os-project-name admin --os-username admin token issue

demo-openrc

export OS_PROJECT_DOMAIN_NAME=Default

export OS_USER_DOMAIN_NAME=Default

export OS_PROJECT_NAME=demo

export OS_USERNAME=demo

export OS_PASSWORD=keystone

export OS_AUTH_URL=http://controller:5000/v3

export OS_IDENTITY_API_VERSION=3

export OS_IMAGE_API_VERSION=2

$ source admin-openrc

$ openstack token issue

openstack --os-auth-url http://controller:5000/v3 \

--os-project-domain-name default --os-user-domain-name default \

--os-project-name demo --os-username demo token issue

problem

The request you have made requires authentication. (HTTP 401) (Request-ID: req-21ace86f-4bc7-4364-bed8-d60090c55594)

solution

/etc/keystone/keystone.conf

export OS_TOKEN=737c639d9acbb9ba6501d6c0d46c3398

problem

__init__() got an unexpected keyword argument 'token'

solution

export OS_URL=http://controller:35357/v2.0

problem

solution is

password mismatch

Dashboard settings