Start the Jobs

This section provides the information on starting the jobs from active master node that were , stopped in section: Stop the Running Jobs.

To identify the active master node perform the following steps:

  1. Log in to the master node using SSH.

  2. Run the following command:

# hdfs haadmin -getServiceState `hostname -f`

The sample output may resemble as follows:

active

To start the jobs follow these steps in specified order on the active master node:

Clear the checkpoint

Run the following commands to clear the checkpoint value from the active name node:

hdfs dfs -rm -r -skipTrash /data/ddj/checkpoint
hdfs dfs -rm -r -skipTrash /data/ddj2/checkpoint

Run the Master Job

  1. Run master jobs from the active name node:

    nohup sh /root/jobs/streaming_jobs/master_http_wrapper.sh > /var/log/mural_logs/master-http.log &
    nohup sh /root/jobs/streaming_jobs/master_nonhttp_wrapper.sh > /var/log/mural_logs/master-nonhttp.log &
    
  2. Check the logs in the files, /var/log/mural_logs/master-http.log and /var/log/mural_logs/master-nonhttp.log and wait for zero counter to be displayed.

  3. Start the input data flow, that was stopped in section: Stop the Running Jobs.

Run the Talend Job

nohup sh /root/jobs/ingestion_jobs/run-talend-nonhttp-job.sh > /var/log/mural_logs/talend-nonhttp.log &
nohup sh /root/jobs/ingestion_jobs/run-talend-http-job.sh > /var/log/mural_logs/talend-http.log &

Run the Aggregation Job

Remove ts file

hdfs dfs -rm -r -skipTrash /data/streaming/*-ts

Run the 5-minute aggregation Job

nohup sh /root/jobs/aggregation_jobs/run-5min-agg-mgr_sh.sh > /var/log/mural_logs/5min-agg-mgr.log &

Run the Hourly Aggregation Job

nohup sh /root/jobs/aggregation_jobs/run-hourly-agg-mgr_sh.sh > /var/log/mural_logs/hourly-agg-mgr.log &

Run the Daily Aggregation Job

nohup sh /root/jobs/aggregation_jobs/run-daily-agg-weekReport_sh.sh > /var/log/mural_logs/run-daily-agg-weekReport.log &

Run the Monthly Aggregation Job

nohup sh /root/jobs/aggregation_jobs/run-monthlyagg-monthlyReport.sh > /var/log/mural_logs/monthlyagg-monthlyReport.log &

Run the Cleanup Job

nohup sh /root/jobs/misc_jobs/run_cleanup_job.sh > /var/log/mural_logs/cleanup.log &