Start the Jobs
This section provides the information on starting the jobs from active master node that were , stopped in section: Stop the Running Jobs.
To identify the active master node perform the following steps:
-
Log in to the master node using SSH.
-
Run the following command:
# hdfs haadmin -getServiceState `hostname -f`
The sample output may resemble as follows:
active
To start the jobs follow these steps in specified order on the active master node:
Clear the checkpoint
Run the following commands to clear the checkpoint value from the active name node:
hdfs dfs -rm -r -skipTrash /data/ddj/checkpoint hdfs dfs -rm -r -skipTrash /data/ddj2/checkpoint
Run the Master Job
-
Run master jobs from the active name node:
nohup sh /root/jobs/streaming_jobs/master_http_wrapper.sh > /var/log/mural_logs/master-http.log & nohup sh /root/jobs/streaming_jobs/master_nonhttp_wrapper.sh > /var/log/mural_logs/master-nonhttp.log &
-
Check the logs in the files, /var/log/mural_logs/master-http.log and /var/log/mural_logs/master-nonhttp.log and wait for zero counter to be displayed.
-
Start the input data flow, that was stopped in section: Stop the Running Jobs.
Run the Talend Job
nohup sh /root/jobs/ingestion_jobs/run-talend-nonhttp-job.sh > /var/log/mural_logs/talend-nonhttp.log & nohup sh /root/jobs/ingestion_jobs/run-talend-http-job.sh > /var/log/mural_logs/talend-http.log &
Run the Aggregation Job
Remove ts file
hdfs dfs -rm -r -skipTrash /data/streaming/*-ts
Run the 5-minute aggregation Job
nohup sh /root/jobs/aggregation_jobs/run-5min-agg-mgr_sh.sh > /var/log/mural_logs/5min-agg-mgr.log &
Run the Hourly Aggregation Job
nohup sh /root/jobs/aggregation_jobs/run-hourly-agg-mgr_sh.sh > /var/log/mural_logs/hourly-agg-mgr.log &
Run the Daily Aggregation Job
nohup sh /root/jobs/aggregation_jobs/run-daily-agg-weekReport_sh.sh > /var/log/mural_logs/run-daily-agg-weekReport.log &
Run the Monthly Aggregation Job
nohup sh /root/jobs/aggregation_jobs/run-monthlyagg-monthlyReport.sh > /var/log/mural_logs/monthlyagg-monthlyReport.log &
Run the Cleanup Job
nohup sh /root/jobs/misc_jobs/run_cleanup_job.sh > /var/log/mural_logs/cleanup.log &