02. Openedx Analytics - Hadoop/Hive/Sqoob

Configure hadoop, hive and sqoop

sudo su - hadoop 

cd /edx/app/hadoop




Please check Hadoop 

hadoop jar hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar pi 2 100

Please check Hive is working

/edx/app/hadoop/hive/bin/hive







Test Dashboard is up and running




http://localhost:18110 

make suer hadoop dashboards are working

http://localhost:8088/cluster

hadoop job history

http://localhost:19888

hadoop node description

http://localhost:8042

Troubleshooting
01. Hadoop dashboard not working

hadoop/sbin/stop-all.sh

hadoop/sbin/start-all.sh

Determine Port is listening

telnet localhost 9000

lsof -i :9000

nc -l -p 9000

Test HDFS System

hadoop fs -ls /

Create Directory 

hadoop fs -mkdir /data

Put file

hadoop fs -put /home/file.txt /data

01. Openedx Analytics - Installation

Openedx Analytics Installation Steps




sudo apt-get update

sudo apt-get install git

sudo apt-get install python-pip

sudo apt-get install python-dev

  

sudo pip install virtualenv

  

# create an "ansible" virtualenv and activate it

virtualenv ansible

. ansible/bin/activate

git clone https://github.com/edx/configuration.git

cd configuration/

  

apt-get install libmysqlclient-dev

pip install -r requirements.txt

apt-get install mysql-server-5.7

apt-get install python-mysqldb

cd playbooks/edx-east/

  

apt-get install nginx

add folder

edx/app/nignx/sites-available

ansible-playbook -i localhost, -c local analytics_single.yml --extra-vars "INSIGHTS_LMS_BASE=$LMS_HOSTNAME INSIGHTS_BASE_URL=$INSIGHTS_HOSTNAME"